Eva•4w ago

Recommended migrations workflow for local environments

We're using the Convex migrations component and I've read through the Intro to Migrations guide—super helpful, thank you! I'm wondering what the recommended workflow is for ensuring that migrations are run on individual developers' local environments? E.g, right now, we have an established workflow for running migrations in production, and that works fine. But once a schema change is pushed to main and pulled down to a developer's local environment, running convex dev will fail without the necessary migration. In past roles, my teams have had a generic npm run migrate or similar function to handle running any new or un-run migrations. Can we setup Convex this way with the migrations component to prevent devs from encountering a broken schema? Bonus points if this is something that can be integrated into predev or seed.ts so they don't have to manually run a migration at all...

Convex

Migrations

Framework for long running data migrations of live data.

Intro to Migrations

There are as many ways to migrate data as there are databases, but here’s some basic information to set the stage.

10 Replies

Convex Bot•4w ago

Thanks for posting in <#1088161997662724167>. Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets. - Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.) - Use search.convex.dev to search Docs, Stack, and Discord all at once. - Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI. - Avoid tagging staff unless specifically instructed. Thank you!

EvaOP•4w ago

Hm, asked Kapa in #Recommended migrations workflow for local environments and got the recommendation to just delete and re-seed data as mentioned in the MVP guide

YOLO: Get to an MVP fast

Before you burden yourself with “best practices” for large-scale companies, focus on what will reduce your feedback cycles and help you ship early and...

jamwt•4w ago

yeah, there are exceptions where individual devs keeping things make sense. but in general, in dev, I would suggest the wipe and reseed approach. the bigger your system gets and the more devs you have, it's nice if everyone is working from the same basis

EvaOP•4w ago

For the recommended one-liner:

for tableName in `npx convex data`; do npx convex import --table $tableName --replace -y --format jsonLines /dev/null; done

for tableName in `npx convex data`; do npx convex import --table $tableName --replace -y --format jsonLines /dev/null; done

do you recommend putting that in a script and running it before seed? Or is there a more Convex-y way to do that as part of the seed.ts itself

jamwt•4w ago

@ian ^ still the best way?

ian•4w ago

There's a few options. The way that has the most "correctness" / maintains your data, would be to add migrations that need to run to a list like

// maybe in convex/migrate.ts
export default migrations.runner([
  internal.migrations.setDefaultValue,
  internal.migrations.validateRequiredField,
  internal.migrations.convertUnionField,
]);

// maybe in convex/migrate.ts
export default migrations.runner([
  internal.migrations.setDefaultValue,
  internal.migrations.validateRequiredField,
  internal.migrations.convertUnionField,
]);

Which enables you to run npx convex run migrate. Or you can put this in your seed.ts function:

await migrations.runSerially(ctx, [
  internal.migrations.setDefaultValue,
  internal.migrations.validateRequiredField,
  internal.migrations.convertUnionField,
]);

await migrations.runSerially(ctx, [
  internal.migrations.setDefaultValue,
  internal.migrations.validateRequiredField,
  internal.migrations.convertUnionField,
]);

Documented here: https://www.convex.dev/components/migrations#running-migrations-serially However, this requires that you make your schema changes in backwards-compatible ways, so new schema changes don't prevent pushing. There also isn't an "onComplete" handler, so it wouldn't be trivial to wait for it to happen before running seed, but they should be able to run side-by-side if your data matches the old & new schema (backwards compatible)

Convex

Migrations

Framework for long running data migrations of live data.

ian•4w ago

If you're developing truly locally (opted into local dev, not cloud dev), then you could delete the sqlite3 file in ~/.convex/convex-backend-state/... , then restart npx convex dev --run seed. The one-liner works for cloud & local dev.

ian•4w ago

Last thought is you could have a testing:wipeAllTables which paginates through all the tables and calls delete on it. You could pass through a function to call at the end. AI Town has something like this. But be warned - you sould set an env variable to make sure PROD never gets wiped! https://github.com/a16z-infra/ai-town/blob/main/convex/testing.ts#L23-L32

GitHub

ai-town/convex/testing.ts at main · a16z-infra/ai-town

A MIT-licensed, deployable starter kit for building and customizing your own version of AI town - a virtual town where AI characters live, chat and socialize. - a16z-infra/ai-town

EvaOP•4w ago

Thank you for the detailed answer! Super helpful to have all the options spelled out. I've decided to circumvent the local migrations entirely for now and just wipe/re-seed the db, which should hopefully encourage improving the seed setup since data won't persevere. Followup q: is there a straightforward way to seed a default user using Convex Auth? Right now I can seed users to my users table, but it doesn't include any of the auth credentials. Would love to have a way to seed dummy accounts in different states / with different data and permissions.

ian•4w ago

You can write documents to the auth tables as part of seed. Check out the documents that are there after logging in successfully, logging out, etc. I haven't done that but it sounds like a great idea It can also help get around the annoyance of needing to actually log in with an oauth provider to test - you can create the credentials with a known token, then put that in a JWT in your browser... but I guess that'd take a bit of reverse-engineering convex auth..

Recommended migrations workflow for local environments

Did you find this page helpful?