This thread is for discussing website features (including forum features) and tech stack for the New Community Website Project.
This thread is for discussing website features (including forum features) and tech stack for the New Community Website Project.
Want to discuss this? Join my forum.
(Due to multi-year, sustained harassment from David Deutsch and his fans, commenting here requires an account. Accounts are not publicly available. Discussion info.)
Messages (30 of 219) (Show All Comments)
#191
> the concept of "it's super flexible" can be alluring.
the phrase "chasing universality" comes to mind.
#190 You could also have a nodes table and one table for each different type of relationship between nodes. E.g., you could have these tables for relationships: is_parent_of, links_to, references. Each of those tables would have two columns, source and target, each of which is a node ID.
db/postgres: some surprising and good results
I don't quite know how to explain these results, but I've made an interesting development.
I refactored sql that I used in views to use Arel. In doing that I found a better way to do a particular join (and what I was joining with).
In SQLite the query takes about the same time: 2000ms (actually a bit worse than before).
nodes_admin_can_read_with_parent_root 2068.22 ms | result: 6
get_nodes_readable_by 2218.54 ms | result: 25019
But in postgres -- which used to be an order of magnitude worse at like 30000ms -- is now an order of magnitude better at around 300-500ms. That's including rails's overhead; the query itself is 200-300ms.
nodes_admin_can_read_with_parent_root 274.80 ms | result: 6
get_nodes_readable_by 452.95 ms | result: 25019
relevant code is here - the benchmark-db-fixes branch on fi-tools/cf-forum
You can run the benchmarks with
rails runner tools/benchmark.rb
and
rails runner -e devpg tools/benchmark.rb
(provided you have postgres set up)
some screenshots of the benchmarks with other results included:
(Note that just returning all children with no filtering is ~150ms for both)
I like the Ruby Weekly newsletter and read some stuff from today's issue:
https://rubyweekly.com/issues/536
https://github.com/hotwired/hotwire-rails
> Hotwire is an alternative approach to building modern web applications without using much JavaScript by sending HTML instead of JSON over the wire. This makes for fast first-load pages, keeps template rendering on the server, and allows for a simpler, more productive development experience in any programming language, without sacrificing any of the speed or responsiveness associated with a traditional single-page application.
> The heart of Hotwire is Turbo. A set of complimentary techniques for speeding up page changes and form submissions, dividing complex pages into components, and stream partial page updates over WebSocket. All without writing any JavaScript at all. And designed from the start to integrate perfectly with native hybrid applications for iOS and Android.
> While Turbo usually takes care of at least 80% of the interactivity that traditionally would have required JavaScript, there are still cases where a dash of custom code is required. Stimulus makes this easy with a HTML-centric approach to state and wiring.
https://www.learnhotwire.com
> Hotwire is a new approach to building applications you'd typically lean on React or Vue for - without all the Javascript. It's brought to you by the great minds behind Ruby on Rails, Basecamp, and Hey.com - so you know it puts developer productivity and happiness first.
https://github.com/hotwired/turbo
Note: Turbo was formerly called Turbolinks. There are search results under the old name, e.g.:
https://thoughtbot.com/upcase/videos/turbolinks
And Rails has Action Cable for WebSockets:
https://guides.rubyonrails.org/action_cable_overview.html
From the first email from the learnhotwire.com people:
> Hotwire helps us fulfill the dream most of us have as developers: building reactive, modern applications, quickly and easily. Frameworks like React and Vue are great, but it's not the code we love - it's the results. The truth is, we want to build our applications fast and not go crazy doing it. Unfortunately over the past few years, it feels like we have no choice but to use complex build chains, learn advance state management techniques, drown in an ocean of JSON responses, and write our code in multiple languages or dialects.
>
> We do all of this work, just to eventually render HTML in the browser.
>
> Hotwire frees us of all of this, allowing us to build the full experience in Ruby on Rails, avoiding the complex machinery, and delivery HTML directly to the browser, over the wire. We can still leverage the power of Javascript when we need it, but it is now a power tool, not something you use to build everything.
>
> Of course, most of these ideas aren't new, but rather, it leverages patterns that are battle tested in frameworks like Elixir's Phoenix (LiveView) and .NET (Blazor). There's even a pre-existing option for Ruby on Rails (Stimulus Reflex). However, Hotwire is created by the same experts, and with the same philosophies, as Ruby on Rails, Basecamp, and Hey.com.
#200 sounds pretty good. Will check it out tomorrow.
AGE / AgensGraph
> #182 maybe they renamed to Apache AGE? not clear on what's going on. this repo has recent updates https://github.com/bitnine-oss/AgensGraph-Extension
yup.
https://www.postgresql.org/about/news/announcing-age-a-multi-model-graph-database-extension-for-postgresql-2050/
> AGE, a multi-model graph database extension for PostgreSQL has been announced. **AGE is the successor to AgensGraph.** AGE will offer the same integration of SQL and Cypher without users having to discard their existing solutions, allow for a cleaner integration of AGE with PostgreSQL’s robust collection of other extensions, and expand scalability without sacrificing performance.
that page has a download link which goes to
https://github.com/bitnine-oss/AgensGraph-Extension
which then redirects you again to
https://github.com/apache/incubator-age
> The project is in alpha stage now and it is currently being developed in the form of an extension for PostgreSQL 11.
> The next Apache AGE release (0.3.0-incubating) will be available around Jan 15, 2021.
recent history:
> acdc110 11 hours ago
https://age.apache.org/
> AGE is currently being developed for the PostgreSQL 11 release and will support PostgreSQL 12 and 13 in 2021 and all the future releases of PostgreSQL.
under *Installing AGE* (no direct link)
> Docker images are available on Docker Hub and are based on the official PostgreSQL 11 Debian and Alpine images.
https://hub.docker.com/r/sorrell/agensgraph-extension
manual installation looks easy too. IDK if we lose anything by moving to postgres 11, but it's probs worth thinking about; and the right time to stdize build env if we're going to go with the current architecture. (on that note: we should probably figure out how to evaluate whether the architecture is good enough or not.)
rich text editor
Rails added a rich text editor. It's optional. At a glance it looks nice.
https://fullstackrubyonrails.com/blog/how-to-use-actiontext-in-rails-6
Demo and more info:
https://trix-editor.org
https://guides.rubyonrails.org/action_text_overview.html
It has nested quoting (click quote once on some text, then click the right indent button to add nesting levels).
But so far I haven't figured out what format it saves its data as, or what it can export to. So I don't know if it's markdown compatible or not (my initial thought is we want to save posts in markdown format in the db).
#203 I know it can output html, which we could potentially convert to markdown. That might require limiting the feature set. I don't know what the alternatives are and how good they are.
#168 I should amend what I wrote to say "transactions that write" instead of "writes". You can do a *bunch* of writes in a single transaction in SQLite and it'll be very fast. But if you need to update the database with a bunch of writes and have the results of each write be instantly available to other readers after each write completes, then that requires a lot of transactions and would be slower in SQLite than it would be in other databases.
#204 I think html -> markdown will be harder than markdown -> html. I had a look around Trix docs but couldn't see anything advertising markdown support. There are other options tho, like some pure JS libraries for rendering markdown.
mb a good way to handle things: on the edit/create page we show raw markdown with a preview; the backend only sees markdown. then we can render markdown as HTML (user configurable?) for posts/comments etc. Adding buttons to do bold, etc, should be pretty straight forward even without a library (i.e. implementing it manually ourselves)
#202 it'd be nice and more user friendly if we had a rich text input option (preferably save as markdown) in addition to markdown input.
#207 think about tech illiterate people who repeatedly screw up quoting on this site or in email. markdown input with some buttons will confuse them and make the site a lot harder for them to use than FB or twitter. a WYSIWYG option will be easier for lots of people and is expected.
#206 I found this blogpost which goes through a few attempts to find a good markdown integration. https://www.codefellows.org/blog/how-to-create-a-markdown-friendly-blog-in-a-rails-app/
The author ends up going with the ruby library https://github.com/vmg/redcarpet (which looks active). apparently it was developed at github. looks like it has a decent feature-set and is configurable (so we could add extensions to do stuff like link comments like curi.us does). the rendering is done serverside, tho, so my guess is that we'd want to cache that. IDK tho; like maybe it'd make html-injection attacks easier. mb serverside rendering would be low overhead.
#206 i don't see much in the way of docs for trix but it looks nice so i skimmed some code:
https://github.com/basecamp/trix/blob/main/src/trix/models/document.coffee
some document export methods there:
toSerializableDocument
toString
toJSON
toConsole
and imports:
fromJSON
fromHTML
fromString
https://github.com/basecamp/trix/blob/main/src/trix/models/text.coffee
the text class has the same exports and the fromJSON import
and there's an html parser class
https://github.com/basecamp/trix/blob/main/src/trix/models/html_parser.coffee
#207
> it'd be nice and more user friendly if we had a rich text input option (preferably save as markdown) in addition to markdown input.
I think we can do that entirely client-side (in the UI) if we want. It can be the default.
Also WRT quoting I think there are win-win options like a feature that shows a 'quote' button when you select someone else's text that auto-inserts it into the reply field. it's win-win in the sense that it's useful for ppl who already know how to do good quoting.
> markdown input with some buttons will confuse them and make the site a lot harder for them to use than FB or twitter.
IDK, like reddit has pretty basic formatting stuff from memory. or at least WRT their old UI (can't remember wrt new UI).
#209 curi.us (and FI) uses redcarpet 2.3.0 for posts (but not comments). I chose redcarpet because I found a version that was compatible with old ruby. my research on this is some years old, but i recall there are a few major alternatives.
https://www.sitepoint.com/markdown-processing-ruby/
> We’re going to focus on 4 Ruby implementations of Markdown: kramdown, maruku, rdiscount, and redcarpet.
IIRC kramdown is what's used by my jekyll setup for https://www.elliottemple.com
if you wanted browser markdown rendering (good for e.g. live preview while writing a post, though i'd prefer a nice whole js editor someone already made) then you'd google for javascript markdown libraries not ruby ones.
> IDK, like reddit has pretty basic formatting stuff from memory. or at least WRT their old UI (can't remember wrt new UI).
most people who use reddit do not know how to use their text input stuff correctly, and a ton of people do not post or post less because they don't know how. reddit is incompetent and is massively harming their business by having an awful website (in dozens of ways). a tiny, self-selected group of reddit users do know how to use it and write quite a few of the comments, but a lot of people who write comments use few to no formatting features.
#209 that blogger is confused:
> Jekyll uses Liquid as a templating language, and I was using Rails’ out-of-the-box ERB. As far as I can tell, there is no way to make those two play nicely together.
you don't make templating languages play nicely with each other. but Rails supports using multiple templating languages in one project – just name files with different extensions. i don't know if there is any issue using Liquid files with Rails, but the issue is not about getting Liquid to play nice with ERB. also templating language and markdown to html conversion are separate issues.
btw there are some command line markdown to html tools, some of which may be pretty good, and they may also be relevant for exporting forum content as pdf, epub, mobi, etc. (cuz they'll do markdown -> html -> various. maybe some is done without the html in the middle, idk.) one of the tools that looks particularly promising is https://pandoc.org/index.html which has a ruby wrapper https://github.com/xwmx/pandoc-ruby
#214 re pandoc: yeah that's the 'anything -> anything' converter. we might lose stuff going from html -> markdown, tho.
pandoc is definitely useful for exporting.
personally I strongly prefer doing posts in markdown and just wrapping that on the client side. more flexible imo and can still have a rich WYSIWYG style editor. otherwise we entirely lose plaintext editing. plus, with a rich editor, users couldn't paste in markdown. IDK how they'd past stuff in without doing markdown->html on paste, anyway. Paste does support rich text but it'd need to be parsed. mb the rich editors support that already tho.
#213 yup, i didn't take into account selection bias there (regarding posts and posters). also my own bias of which subreddits i've browsed in the past.
#215 i don't understand what you're arguing with. i wasn't talking about not having markdown input support. i want markdown as an input. i also want a user friendly input option.
rails recursive includes
This isn't general, but i did find a way to do 'recursive' includes. One reason I did all the arel stuff (see models/node.rb) was b/c doing this sort of thing wasn't obvious. Note: it's not as good as the recursive method IMO, and doesn't consider permissions:
Node.includes(direct_children: [:direct_children, { direct_children: [:direct_children] }])
The debut output for SQL queries looks like:
Node Load (1.6ms) SELECT "nodes".* FROM "nodes" WHERE "nodes"."id" = $1 LIMIT $2 [["id", 0], ["LIMIT", 1]]
Node Load (1.0ms) SELECT "nodes".* FROM "nodes" WHERE "nodes"."parent_id" = $1 [["parent_id", 0]]
Node Load (1.2ms) SELECT "nodes".* FROM "nodes" WHERE "nodes"."parent_id" IN ($1, $2, $3, $4, $5, $6) [[nil, 2], [nil, 3], [nil, 4], [nil, 1], [nil, 14], [nil, 18]]
Node Load (1.1ms) SELECT "nodes".* FROM "nodes" WHERE "nodes"."parent_id" IN ($1, $2, $3, $4, $5, $6) [[nil, 5], [nil, 6], [nil, 15], [nil, 20], [nil, 21], [nil, 19]]
cf forum rails db progress
done a bunch of work on using db views and caching some stuff we can easily calculate incrementally.
using Node.insert_all! the seeds file can generate 88k nodes (forum posts) in under 3 minutes - fast enough to be useful.
using materialized views: all the permissions stuff is cached and refreshes are fast - 400ms with concurrently=false (blocks reads) and 800ms with concurrently=true.
rendering is fast: although there's some n+1 ish stuff going on, viewing a topic with 30k descendants takes less than 200ms and the SQL queries take like 4ms each. Note: this is capped at a depth of 3, and the tree has branching factor 3, so 40 nodes or so.
When I set max_depth = 4 (so ~120 nodes), this is the split between active record and views:
Completed 200 OK in 558ms (Views: 411.1ms | ActiveRecord: 139.1ms | Allocations: 248131)
with max_depth=3, some stats:
db progress: all pages rendering in <100ms with permissions.
some pictures / profile stats: https://github.com/fi-tools/cf-forum/pull/23#issuecomment-768514477
one thing I forgot to mention with #218 is that it's a very rails-y solution, too:
Ruby on Rails Creator Takes on JavaScript Frameworks with Hotwire
todo lists (that u can check off) and calendar stuff (e.g. notify at certain time, or recurring events like being asked a question on a schedule) are possible features.
Trying more to find something that someone else already made that could work. Anyone want to research that or make recommendations? That'd be helpful, thanks.