Syncing updates when multiple users are changing the same thing
So imagine you have a note taking app like notion. If you have a get query and an update mutation on the content of that note - what's a good way to keep it in sync but not overreactive on each user's client?
If I'm typing something and the update mutation fires, then the get query sets the old value of what i typed in the note and it appears to be very buggy. I feel like I'm using convex wrong here and not sure if i should be doing this at all?
The same goes for if another user types/deletes something while im typing in that note - it goes crazy on both our clients since itll sync a value a bit delayed from the current one u typed ( since the api call itself takes lets say 200ms )
33 Replies
I'm trying this in Nextjs with something called tiptap btw - A solution that would work is if i use a separate websocket server for each note that uses somehting like hocuspocus to do the merging of updates from various clients and I only use convex to load data initially in that server ( when a client connects ) - rest of the updates is synced by the separate websocket server
I wonder how well it would work to use y.js direct in convex functions to merge updates against a stored note
@star ✨ as it so happens, @CodeWithAntonio is about to release a notion clone built on convex very soon
👀
i was considering this too
so that should help provide some guidance. but until then, here's another example of collaborative editing: https://discord.com/channels/1019350475847499849/1153069931487830117/1153069931487830117
at the end of the day, the strategy most editors using for multiple simultaneous authors when faced with network delay or network disconnectivity is Operational Transformation (https://en.wikipedia.org/wiki/Operational_transformation )
Operational transformation
Operational transformation (OT) is a technology for supporting a range of collaboration functionalities in advanced collaborative software systems. OT was originally invented for consistency maintenance and concurrency control in collaborative editing of plain text documents. Its capabilities have been extended and its applications expanded to i...
Scroll shows off how to do that with convex
yjs uses crdt right?
is the notion clone also gonna be using OT?
taking a look at this 👀
It's using https://www.typecell.org/ according to https://twitter.com/YTCodeAntonio/status/1708979418666860832
TypeCell
TypeCell
where did you read that? i cant find it in the tweet
but that clone is very cool, would be massively useful as an example with convex
tho this one answers my question for now tbh, gonna try doing the OT thing in the convex function
oh nevermind im blind
dont understand how to use typecell in your own projects tho :catThink: theres no 'npm install' or anythin
me neither. part of why I'm looking forward to antonio's video
i am so excited and pumped for that project ☺
same!
Interestingly, Typecell's main page says it's powered by yjs crdt
(the collaborative functionality, that is)
Here's RJ's ProseMirror with Convex example , it also uses TipTap it's lower-level than TipTap https://github.com/rjdellecese/scroll/
GitHub
GitHub - rjdellecese/scroll: An infinite-scrolling note-taking app ...
An infinite-scrolling note-taking app with collaborative editing, built on Convex - GitHub - rjdellecese/scroll: An infinite-scrolling note-taking app with collaborative editing, built on Convex
I'd love to see Yjs in Convex! It should be a great platform for it.
Let me try all of these and see if i can come up with some simple example that would help future ppl coming here
@star ✨, I actually did use Tiptap in that project, but I used a ProseMirror plugin (https://github.com/rjdellecese/scroll/blob/b2f0caf4bf588141db8eafc7b54f63db59c7745d/src/elm-ts/note.tsx#L562-L564) rather than the Tiptap collaboration extension (https://tiptap.dev/api/extensions/collaboration) because doing the latter would have required building a Yjs provider for Convex, and at least at the time, there was no documentation on how to build a custom Yjs provider. From some cursory internet searching, it looks like that's still the case (e.g. https://docs.yjs.dev/tutorials/creating-a-custom-provider).
But generally speaking, I do think that CRDTs and Yjs in particular are where all the momentum is at right now.
Although OT is still possibly simpler to integrate and definitely works well for the vast majority of regular collaborative editing use cases!
Thank you so much for the insight!!
I WAS struggling with yjs cus yea there isn’t much docs 😭
I’m currently tryna figure out ur open source notes app :3
Yes, I struggled with that too! I knows it's a lot to take in, let me know if I can answer any questions about how it works 🙂
omg thank you id love that! let me try to figure it out on my own a bit first 👀
Yeah, I think in general the biggest difference is if you have a central server, OT is usually much easier to manage. If you want truly universally reconcilable divergences (e.g. peer to peer communication), they you'll need to use CRDTs b/c you can't rely on an entity providing one serialized order of the steps. so when you're using convex [which acts as a centralized mediator], OT is probably simpler
thanks for the correction RJ, updated my comment
I have a question - you're storing the steps in the db. So as more n more users have notes will the steps table keep having forever more objects in it?
Correct @star ✨. There are strategies you could employ to delete old steps—for example, you could keep track of the each client that's currently connected to a document and the latest version of the document that they have, and then delete (or compress) steps older than that version.
I wrote about this in the README as well: https://github.com/rjdellecese/scroll#technical-notes-and-limitations
o oops didnt read it fully
maybe i cud just simply delete old steps if everyone is disconnected?
ah but what if they were offline i cant do that hmph
okay ur right after all, need some version number
im curious what was your motivation to make scoll.ink? i think its pretty cool
My gut says just let the table grow forever and see if you run into issues, and by the point at which that happens (if ever), you'll have a better sense of the needs/habits of your app/users and therefore a better sense of how to fix it
Thanks 😀 I spent ~2 yrs (👴) trying to build a note-taking/knowledge management app to sell to businesses/small teams. After I gave up on that, I took some time off and discovered Convex, and used Convex (plus my experience building the note-taking/knowledge management app) to prototype some ideas about what I'd want in a personal note-taking app.
Specifically the idea behind Scroll was to treat note-taking as a continuous process without explicit boundaries between text. Like a never-ending scroll, conceptually—so the next thing you write is always after the last thing you wrote, unlike most note-taking apps, which asks you to create a new note at the top every time.
ohh i see
im building one specifically for unorganized notes because despite there being a lot of note taking apps my inability to organize/find things doesnt allow me to use any of them effectively
Similar energy 😄 I'm curious to see what you come up with, share it with us when you're done (if you're willing)!
i definitely will! im almost done with like a barebones version of the concept but got stuck on syncing the notes xD
but thanks to ur repo i kinda get how to make it work with "OTs" ( that prosemirror plugin mentions its not exactly how the paper implemented it but whatever it works xd)
Collaborative Text Editor | Liveblocks Example
This example shows how to build a collaborative text editor with Liveblocks, Yjs, Tiptap, and Next.js.
i found this recently '
Which one of these works best with convex?