Migrations suck
I might be off here, but migrations in Convex feel kind of rough.
My workflow: I’m on a dev branch tweaking the schema, I write migrations, and I migrate my local data to test. Then in a later update I need to drop some old fields and make more code changes.
In “classic” SQL, I can bundle schema changes and data migrations together and let the pipeline roll forward automatically. In Convex, it seems like I have to roll the code back to an intermediate state, run the migrations on prod, deploy that intermediate state, and only then make the rest of the code changes again. It works, but it feels pretty clunky.
Am I missing a better pattern for handling schema + data changes without the intermediate deploy step? How are people doing this smoothly in Convex?
7 Replies
Thanks for posting in <#1088161997662724167>.
Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets.
- Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.)
- Use search.convex.dev to search Docs, Stack, and Discord all at once.
- Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI.
- Avoid tagging staff unless specifically instructed.
Thank you!
We had a discussion on this in the general channel. The short answer is you have to roll your own system if you want to handle running automatically in multiple envs. I've been working on my own system for a while that I think is actually getting close to what I want. I am testing it out over a few more migrations and once i feel good about it i'll make a github repo with a writeup on it. In our discussion with the convex team we came to the conclusion you just have to always do backward-compat migrations (favoring things like dual writes) which is fine, but i think there's just tons to the conversation being left out. For example, if you are doing specifically data migrations and want to run in multiple environments how do you do it? This is what you are getting to its clunky and i still think the ocnvex team should come up with an official solution
For my solution i basically modelled dbmate ( https://github.com/amacneil/dbmate ) where I have a migration tracking table and a release table and I create typical time incremeneted migration file (for example a recent one i had was
20251008_153000_backfill_chat_room_user_events.ts ) and i wrote my own custom script to run them in CI in the different environments i release to. My migration talbe is
GitHub
GitHub - amacneil/dbmate: 🚀 A lightweight, framework-agnostic da...
🚀 A lightweight, framework-agnostic database migration tool. - amacneil/dbmate
And my release tables are
In another thread we were just discussing a potential definition of a
migratingSchema that would keep the online-migration mindset with full typescript support and removes complex infra.
Essentially only 1 change is needed to define a strict migration-end-schema and your inserts & patches would work as if the new end schema is already fully enforced.
After the migration is ran you only need to clean up the 'migrating' schema and everything will be fully typesafe.
Discussion happening here if anyone interested: https://discord.com/channels/1019350475847499849/1421114757213327480/1421492336784965703thanks
great feedback and ideas here - thanks all.
A few things to surface:
- Most of the time you can get away with adding optional fields to add fields or marking fields as optional when they're no longer used. the
deprecated helper can help self-annotate dead fields, which makes changing branches / contexts not require running anything.
- You can always do a snapshot export on a branch with special data changes, then snapshot import back to it, so you don't have to figure out how to do up/down migrations without data loss (which in the general case is impossible to not lose data along the way)
- You can have a list of migrations that all attempt to get run, and part of changing branches can be to run these. Note: this doesn't handle the "down" migration - but more on that below.
- There's a pretendRequired and generic pretend validator helper for when you're adding a field and want TS to assume the value is there / is something specific even though it might not be (yet). Doing this along with adding a migration to the list allows you to make forward progress and later on (after it's been merged to main e.g.) clean up.
- One idea that this has brought up is to have some function you could call to do "down" migrations, where you pass in the known migrations on the current branch to a CLI-driven function call (before deploying code), and the currently deployed code runs all the down migrations that aren't known, then you deploy the current code and run the "up" migrations added by the current branch. Bundling this as a CLI tool (e.g. npx @convex-dev/migrations) would be pretty handy.
Ultimately there are many paths for migrating data, and things may radically change workflows, like having many one-off local dev instances or a per-branch cloud deployment.
Improving the "standard" tooling (today: my opinionated component) seems high leverage. Also figuring out what building blocks or advice folks need to build better / different tools is really exciting. Ideally there's no limitation on what the community can provide - e.g. I'm not using any secret APIs in @convex-dev/migrations - but it's still not as easy as it could be