Workflow for migrations?
The workflow for migrations in the documentation seems to indicate a very involved process for doing migrations.
1. Define the table with new optional fields.
2. Run the migration
3. Tighten up the table with the final version of the fields.
This seems extremely annoying. Surely there is a better way to do this? I want to the ability to do a deploy, have the migrations all run, and I'm not having to worry about manually doing every release. Are there any workflows for this that I'm not seeing?
It feels like in order to get my schema updated, we'll have to get our clients compiling with each step in the schema, instead of just the final product, and either manually go through this process of checking and releasing code for each step instead of just being able to rely on my data being in the final state. This seems like a massive headache.
Is there a better way to do this that I'm not seeing?
9 Replies
Thanks for posting in <#1088161997662724167>.
Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets.
- Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.)
- Use search.convex.dev to search Docs, Stack, and Discord all at once.
- Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI.
- Avoid tagging staff unless specifically instructed.
Thank you!
It seems like my schema is defined in code, and my migrations don't effect schema, just migrate data. This seems particularly annoying.
If I have a list of all my migrations that need to run, it also seems like I need to remove all run migrations from that list after every release (meaning I have to keep track of what's been run and not run in production), and then after it runs, I can update my schema to it's final, tightened value.
This dance between run migrations, and the current schema, because migrations don't change schema, seems particularly cumbersome. Are there any solutions?
I'm assuming you've seen https://stack.convex.dev/lightweight-zero-downtime-migrations ?
And the admittedly minimal docs on making it part of your deploy workflow: https://www.convex.dev/components/migrations#running-migrations-as-part-of-a-production-deploy
Lightweight Migrations
Patch all of your data in your database table with the bulk edit feature on the Convex dashboard, without writing migration code.
Ultimately the challenge is that when you want to change your schema, your data won't all magically change instantly. So you have to decide whether you want to handle the intermediate state where your app is "broken" . Other platforms do this synchronously with a deploy - essentially failing requests during the deploy & migration. I've worked at multiple companies where this pattern led to hour+ outages when a migration failed midway continually timing out, preventing rolling back or fixing forward until manual DB surgery was performed. I'll admit this was with millions of users, not something most apps run into.
If you only want to handle the types of the final result (and accept that sometimes they'll be wrong at runtime), you can use this helper: https://github.com/get-convex/convex-helpers/blob/main/packages/convex-helpers/validators.ts#L295-L334
which allows you to pretend the type is required.
GitHub
convex-helpers/packages/convex-helpers/validators.ts at main · get...
A collection of useful code to complement the official packages. - get-convex/convex-helpers
As for how to manage old migrations, I recommend commenting out the old migration and also pasting in the old schema, so you have a running log without checking git history, and you can patch in old things to migrate multiple previous steps if necessary. But the whole "migrate up" workflow is admittedly annoying right now - you can't have a dev deployment jump multiple schema phases easily. I've thought about whether scripts make sense here. but often folks go with a "seed" script and just blow away their dev data liberally
Thanks for your write-up
Maybe migrations take a bit longer in convex, or maybe it's best practice to always do them with zero-downtime taken into account, but there is an overhead cost to that decision that I now have to take on
migrations typically only impact a small portion of the data that a user could access typically, and convex's ability to "continue" migrations or re-run migrations without migrating the same data twice seems particularly valuable here
When you only have a few thousand customers, migrations can still be done in seconds, even on convex I'd imagine
Until I'm at the scale where I need to worry that going down for 5 minutes at 3 AM is a problem, I'd rather not worry about it with my migrations.
I hear you, I’m still thinking about the right way to balance that. I suspect you’ll appreciate the “pretends” validators. Also remember you can just turn off schema validation for a while and turn it back on later. You’ll still get types. I wrote a Stack post on moving fast - search for “yolo”
I might leverage turning on and off schema validation
Does that incur costs when turning the schema on and off? I assume when it’s turned back on it’s going to check the schema again?