offline first?
Convex still not comparable to either Firebase AWS Amplify/Appsync or MongoDB Realm. No offline sync between client libraries and DB.
14 Replies
Hey there Sean. Isn't this a duplicate of ( https://discord.com/channels/1019350475847499849/1237500621783830638/1237500621783830638 ), or is there a new wrinkle to this thread?
Hey Jamie, I guess there's been no progress on this aspect then?
Do we stick with Firebase, Amplify, Urql, MongoDB Realm Sync, or will Convex implement this feature and become a true industry hero that finally gives offline capaable apps a proper altnerative to firebase....
No progress on this yet and no timeline that we can promise. But it's certainly pretty close to the top of the list of things that we want to tackle next.
Can you say more about your use case? Can offline sync be something you add later in the development cycle of building this product?
+1 to getting more information about specifics in what you'd want @Sean Knowles . I see you listed some in the linked thread. Also there was some discussion of folk's experience with OT / CRDT w.r.t. syncing local state: https://discord.com/channels/1019350475847499849/1230303876641587240/1231096038471761990
A ton is possible today without any built-in support:
- use queries but fall back to a local store
- update the local store from query results that you know the shape of
- keep the client up to date with a set of queries on all the data they want synced. if the query is shared amongst clients, it's cached automatically. "syncing" with convex is just subscribing to a consistent view via queries, which already works. It just doesn't do the delta - you'd have to manage that with pagination over some sequence ID.
- the convex client will hold onto mutations for at least a week to send up the next time it has service, if you want to buffer them. It may not work across page refreshes, however.
- You can also write out mutations you want to apply to local storage and use idempotency keys to avoid dual-writing. Or do an either-or: you try a mutation, and if it fails you write it to your local buffer, which you then re-apply when you reconnect.
Knowing more about what you have in mind would help us build something to fill in the gaps. Ultimately the same queries & mutations will not (and should not) execute locally, unless your app has no authentication or information hiding, so you'll always end up with something application-specific about which documents are fine to "sync" and what subset or combination of documents should be synced.
I'm in the process of experimenting with switching from AWS DataStore to Convex for "offline mode". The "offline first" approach that AWS DataStore uses causes a lot of duplicate logic and dependencies that make things overly complex, in my view.
I'm seeking "offline mode" as opposed to "offline first" which I believe can be solved using @ian suggestions. There are essentially two parts, the reading and writing of data.
Reading: This seems clear, I just need to update my local store with the convex query results.
Writing: This is the part I'm not so sure about. @ian you mention the convex client will hold onto the mutations. Can this be disabled, as I would need to maintain these pending mutations between app restarts, so I would need to write them out to local storage. What do you suggest saving to local storage exactly? The mutation JSON payload?
As a suggestion, some kind of convex-helper or middleware to handle this would be great. If this helper/wrapper could allow a local storage adapter for convex client. This could behave just as it does now, but it cloud persist all query results and pending mutations, so that it works across app restarts. If the convex client fails to reach the convex query endpoints (ie has no internet), then it could (optionally) read from local storage as fallback? Could Zustand Persist be used for inspiration? https://github.com/pmndrs/zustand?tab=readme-ov-file#persist-middleware
Only thing i'm not sure about is conflict handling. I think convex has some ability to handle conflicts, but i'm not sure if it is comparable to AWS conflict handling (if 2+ offline devices change the same db record).
To answer your question around conflict handling: Convex uses MVCC / OCC w/ automatic retries & backoff automatically, so for a given set of mutations being submitted, you generally won't see failures from conflicts.
However, there's a logical layer you need to care about.
If both competing mutations do something like "move money from account A to B" and the reads/writes are within a transaction, you don't really need to worry - both of those will be serialized - one will happen before the other (via ACID w/ serializable isolation guarantee).
If A and B both update the name of the team and they are serialized one of them may overwrite the others' work. This is also mostly expected.
Some traditional gotchas exist- like commenting on a document someone else deleted. You'll need to account for these sorts of races anyways even with online due to client latencies.
Generally when you're making assumptions about global state and queueing mutations while offline, you need to make sure that they still make sense (or can bail out safely) later even if the state of the world has changed.
Your description is very well put. I think that there's a workable API layer here. There would be some pieces to figure out around idempotentcy and optimistic updates, but you have my gears turning on a possible API wrapper.
To be honest I haven't played around much with the offline / online transition behavior vs. page refresh to speak to the specifics, but what I'm thinking so far is:
1. Wrap
useQuery and useMutation (maybe useAction too).
2. useQuery caches the values locally for a given fn/args, with some expiry. When you get the value from useQuery, it has the normal, consistent, trustworthy data, along with a stale value that has the most recent version. This version will be available offline, provided you've queried those parameters before.
3. useMutation writes all the function/arg combos locally, the attempts them via the client. If they succeed, it deletes them locally. When the page loads, it checks the local store for pending mutations to submit.
a. It'd need to lock against other browser tabs competing.
b. There's a race where a mutation was submitted but the old client never received the "ack" so the client re-submits a mutation. This is where idempotency comes in. [^1]
c. There will be some client-side optimistic update logic that will likely skew from the server logic. There are two challenges here: how to represent the mutation locally, and how to have that representation live across browser refreshes.[^2]
4. useAction is always less reliable than useMutation since it can't be
[^1]: There's a few ways to make your mutations safe against dual-submission (I avoid the term retry here to not conflate with the automatic OCC retries). OT / CRDT are strategies here, but not the only ones, if you're willing to think about how they could be interleaved.
a. A robust but more expensive one is to pass up an idempotency key (e.g. uuid) attached to every mutation. When processing it, it checks in a table if the key already exists, and if so, aborts. Otherwise it adds the key to the table. You'd vacuum the table on occasion.
b. Another strategy is to pass up the state of the world that are your assumptions for the mutation. e.g. that the user still exists, or that the line I'm deleting looks like X. A simple pessimistic version just aborts if an updatedAt field is greater than what the client saw when it made its changes. This gets into the world of Operational Transform (OT).
[^2]: When a mutation creates a document, we don't yet have a way for you to pass up the ID of the document you want to create later, so any later mutations referencing the document can't use its _id. As for reflecting changes across restarts, I think you'd need something like a reducer that applies all of the changes, that doesn't depend on capturing values in a closure like today's optimistic update API allows you to do. I would want to play with that API, as it would likely be the make-or-break piece for ergonomics for more complex apps.
I have some other helpers & layers on top of mind right now, so I won't be developing this immediately, but I'm feeling more optimistic about the possibility of something that is in user-space, makes some assumptions, is a bit opinionated, and ultimately enables easier local-mode development. Then from there a local-first offering will be easier to evaluate re: feasibilityHey there, sorry been heads down on getting the right stack for product. @Michal Srb Unfortunately we need offline first capabilities form the begening as our users will have no internet connection for extended periods of time. When they get back to internet connectivity any writes the local device or browser should sync to the backend with conflict resolutioon. Likewise we want to be able to control what data/tables are stored locally. MongoDB Atlas does a great job with this with Realm and their client sdks. We will probably move ahead with MongoDB as that gives us the most flexiblitity across clouds in the future and provides all the functionality we need right now. Convex would have been perfect but we are starting development now unfortunately.
Interesting alternatives in the space are
- https://electric-sql.com/
- https://ditto.live/ <------ this is wild stuff (edge syncing between devices, hops across edge devices)
- https://www.powersync.com/
The other solutions cover much of the functionality required but require extra devops effort and maintaince so we are most likely going with MongoDB Atlas at this point.
Thank you for sharing @Sean Knowles !
That being said, if convex release something this year, we could be persuaded to shift potentially...
@Sean Knowles I know that some people combined Convex with Replicache
You cant use these "alternatives" together with Convex right?
You mentioned "If A and B both update the name of the team and they are serialized one of them may overwrite the others' work. This is also mostly expected." If both A and B update a document containing an array, would the array be merged? For example, I have
order.payments which is an array of payments.
To give a bit more context, my application is react native, which I would assume would be the most common use case for offline support?
- Native apps: Just need data to work offline.
- Web apps: Need html/css/js (eg PWA) + data to work offline.
In my use case, out of 15 or so tables, I would only need to cache mutations for 1 of these tables. There are another 3 tables that can be mutated by the app, but I would just disallow mutations on these during offline mode. The remaining tables would be query only tables.
Re 3c: useMutation: Given _id is generated server side, I would need to have a client side generated id and add this as a table index (eg uuid). For client side optimistic updates, here is what I am thinking:
- When online + offline (as to not have multiple code paths), always use the cached results of useQuery, and then apply any local pending mutations on top of this. useQuery would only receive new data when online, which may contain a newly mutated record (that still exists in local mutations cache). The wrapper should only return unique records, with preference given to useQuery results.
It would be good to have the optimistic update/offline cache ability on a per table basis to avoid adding complexity where it's not needed. In summary, I think this one wrapper could address two needs: optimistic updates and offline mode. I like your thinking on the approach you are considering with this.At the risk of just chiming in without being useful. This topic is dear to my heart too. I am thinking about building a podcast player that uses convex.
Obviously podcast players also need to work offline (on planes or whatnot) so although it is possible that I do this with some data offline and some online it might be nice if there was a Convex solution.
I was thinking about using Loro but that hit a bit of a wall unfortunately: https://discord.com/channels/1019350475847499849/1197172506084265984/1197172506084265984
@Michal Srb Any updates on this? It's nearly end of the year and offline first is gaining traction... Many applications need it. It was said back in May that offline sync features are top of the list and what you are hoping to tackle next however see no updates or any progress in that direction.
Looking for the client SDKs to store local writes in a queue when offline and when back online should sync the writes conflict free. Likewise reads should work locally. Everything should work locally and then efficiently sync when connectivity resumes.
What would be interesting is to batch the local writes/transactions when syncing so that we don't incure multiple request loads when syncing local to remote db.
Theres some movement on using Legend-State on this thread, that might be relevant to this thread https://discord.com/channels/1019350475847499849/1289187277465059338/1291152193591836692