RJ•3y ago

ProseMirror collab editing continued

tl;dr of what’s to follow: - I attempted to write a “plug-and-play” ProseMirror collaborative editing/React/Convex npm package. I encountered one pain point, but generally saw that it would be possible, and then gave up on it because I didn’t want to use React’s default state management solution and thus wouldn’t want to maintain a package built with it. - I re-implemented ProseMirror collaborative editing with Convex and a different state management strategy, and I don’t know why but unlike my first attempt, performance was really good!

10 Replies

RJOP•3y ago

A while ago, I wrote a proof-of-concept of using Convex to achieve collaborative editing with ProseMirror’s first-party OT implementation (https://github.com/ProseMirror/prosemirror-collab). It worked, which was really neat! But performance was not very good, for reasons I wasn’t ever able to ascertain (to the point that it would have probably been painful to use real-time with another person). After that I thought it would be fun to try to create a ProseMirror collaborative editing package specifically built for Convex. The idea was that, since schema definition and queries/mutations are all described in TypeScript, one could distribute everything necessary to accomplish collaborative editing (backend and frontend!) in a single npm package. Which is an impressive proposition, I think! I ended up abandoning that because I realized that I didn’t want to use React’s state management strategy (useState/useEffect) for my own project, which meant that I wouldn’t want to actually support a library built in that way. But I think I got close enough to a working solution to see that it would have panned out just fine, with one caveat—figuring out a convenient and type-correct way for a user to provide the generated API functions to the React component was a bit tricky! Particularly getting the types correct—the library code needed to use the types from the Convex function definitions, rather than the generated API types, to derive the type signatures of the functions that would be generated, so that it could validate that the user is providing to the component the correct Convex functions, which are those that were generated by the library’s provided Convex function definitions. I realize all of this might be a bit hard to follow, especially without code examples. I’m happy to provide more details if anyone is interested! Here is a link the repository containing this experiment—note that it’s a bit messy and doesn’t quite work (I had some nice infinite loop bug in the React code that I never resolved), but the overall architecture is perhaps still discernible: https://github.com/rjdellecese/prosemirror-collab-convex After all that I ended up creating a new project after that which used a UI state management solution I was happier with, and got the ProseMirror collaborative editing working again. I don’t know why, but this time around, performance has been great! It would definitely be sufficient, I think, for multiple people editing the same portion of the same document at the same time. I can’t quite share the demo of it yet, but I think I’ll be able to soon.

GitHub

GitHub - rjdellecese/prosemirror-collab-convex: ProseMirror collabo...

ProseMirror collaborative editing with Convex. Contribute to rjdellecese/prosemirror-collab-convex development by creating an account on GitHub.

ballingt•3y ago

This is great! ~~My discord app might be buggy but I see this post twice.~~ Really exciting to hear about making a third party library, I'll dig into the code first but want to hear more about getting the types right

RJOP•3y ago

Oops I saw the duplicate post too, I had a weak connection when I posted so that might have been the cause (deleted now)

ballingt•3y ago

I think I can see where submitting user functions vs library functions to convex would be a little funny, wondering about what the right apis for this are. I can certainly see wanting to ship the generated code for a library with it so not wanting all generated code to have to be generated together.

RJOP•3y ago

Some questions that come to mind for me: What is the boundary between the Convex documents defined by a library and those defined by the user? In the library I started to write, for example, the user should probably be able to read a ProseMirror document’s state, but they probably shouldn’t be able to access any of the steps used for conflict resolution. They also shouldn’t be able to modify a PM document’s state directly—all changes should be applied through steps. How do you express/enforce these constraints in library code? Is it possible right now? Escape hatches might be fine, but we need to be able to hide collaborative editing internals somehow. Exposing Convex queries and mutations (not the sort that generate API endpoints, but the sort that you can use in other queries/mutations—those with type signatures like

export const getDoc = async (
  db: DatabaseReader,
  docId: Id<"docs">
): Promise<string>

export const getDoc = async (
  db: DatabaseReader,
  docId: Id<"docs">
): Promise<string>

) might be one important ingredient in a solution! As mentioned above, how does the UI aspect of library code (e.g. a React component) get access to the correct Convex queries/mutations that it needs to function? The solution I came up with was to ask the user of the library to pass in the generated Convex functions the React component needed explicitly, but besides the TypeScript typing issue I ran into (which definitely seems like it could be solved in a straightforward way, if Convex exposed some more of its type transformation types to the library author), that’s annoying because it doesn’t seem like the user should need to concern themselves with wiring up these specific functions correctly. Generating the code beforehand could solve these problems, but I actually think that the wiring issue (needing to explicitly pass in some means of referencing the correct endpoints to React components) is more a consequence of the fact that a Convex function’s identifier is determined, at least in some part, by its location on the file system. Imagine an alternative means of defining Convex function identifiers that looked like this:

// Convex library type; the CLI expects one value of this type to be exported by the user in their `convex` directory

type ConvexFunctions =
  { queries: Record<string, ConvexQuery>,
    mutations: Record<string, ConvexMutation>
  }

// Convex library type; the CLI expects one value of this type to be exported by the user in their `convex` directory

type ConvexFunctions =
  { queries: Record<string, ConvexQuery>,
    mutations: Record<string, ConvexMutation>
  }

// User code (their API!)
// Maybe this is in `convex/functions.ts`, mirroring `convex/schema.ts`

export const functions: {
  queries: {
    userQuery,
    ...libraryQueries
  },
  mutations: {
    userMutation,
    ...libraryMutations
  }
};

// User code (their API!)
// Maybe this is in `convex/functions.ts`, mirroring `convex/schema.ts`

export const functions: {
  queries: {
    userQuery,
    ...libraryQueries
  },
  mutations: {
    userMutation,
    ...libraryMutations
  }
};

”userQuery” would the name/identifier for that query, and ”userMutation” the name/identifier for that mutation. A library could then just define the names of its own queries/mutations, and thus a library React component would only need access to the Convex client to be able to know how to access and use its Convex functions! I’ll note also that this just mirrors how schema definition currently works. Imagine if schema definition worked how Convex function definition did, where you define the schemas for separate Convex documents in separate files, and their names were derived from their file names! I think that would suck 😬 Why treat the Convex functions which comprise your API any differently?

ballingt•3y ago

I agree there are limitations with file-based routing, but working within them for a sec: how about instructions to the developer to import and re-export collab functions from a specific file? @colinclerk does something like this in their recommended approach to integrating Clerk with Next.js which has some analogous file-based routing. Absolutely agree that a library should be able export components hard-coded to use convex functions from that same library. re private data tables, lots of possibilities. Could be solved at the type level, all tables are runtime accessible but different subsets of the schema are used by different code. This is great stuff, thanks for raising it! We've talked about this and it's so good to hear the external perspective.

RJOP•3y ago

Yeah, that seems like the best solution under the current regime, but still seems worse to me! But I won’t harp on it :p I know that file system-based routing is popular in a lot of these mainstream frameworks, even going way back to Ruby on Rails. Yeah, I think that would be a great solution!

ballingt•3y ago

Totally, and it's not out of the question! We're committed to building a system that allows developers to make it do what they need it to do, and that could include opt-outs from file-based routing. All about balancing simplicity and prioritization as I'm sure you know. personally, Rails gems are an inspiration for a potential library system, along with Firebase and Heroku as general inspirations

RJOP•3y ago

Yeah for sure 🙂 Agreed that gems are a great source of inspiration, I mentioned this a while ago but I always think of https://github.com/paper-trail-gem/paper_trail in this context (which manages its own table/model(s?) in your app)

GitHub

GitHub - paper-trail-gem/paper_trail: Track changes to your rails m...

Track changes to your rails models. Contribute to paper-trail-gem/paper_trail development by creating an account on GitHub.

ballingt•3y ago

i forgot, i might have stolen this Rails inspiration thing from you after our conversation!

ProseMirror collab editing continued

Did you find this page helpful?