BluePenguin•12h ago

First-class offline migrations

I realise that Convex's focus on online migrations is good policy for apps that run at scale. As an pre-PMF company we just want to rapidly prototype and having to do multi-step actions for a new non-nullable field (1. add optional field, 2. migrate, 3. remove optionality) really kills our speed. This approach also kills typesafety. I provide an example below We'd love to be able to do offline-migrations that still keep type-safety & schema validation. Example: * Write migration * Deploy convex * Transaction start * [Convex]: disable schema validation * [Convex]: execute migration (this would be a non-typesafe step) * [Convex]: validate new schema * [Failed]: fail transaction & revert * [Success]: enable schema validation * Transaction end * [Convex]: if success: deploy everything else * [Convex] server online Why does the current approach kill type safety? Between step 2 (run migration) and step 3 (remove optionality) there is a non-trivial amount of time, because both require separate deployments and a code change. Therefore the optionality is still present AFTER the migration and BEFORE the new schema. This means any typescript verification will NOT catch inserts that leave the new document's value unset. Therefore there is a possibility of breaking the ability to execute step 3 during that time. This to me breaks one of the big values of Convex; end-to-end type safety. It leaves me leaving optionality everywhere and running run-time checks (opposed to deploy-time-checks) which is inherintly a non-TS approach. If you got this far thanks for reading.

5 Replies

Convex Bot•12h ago

Thanks for posting in <#1088161997662724167>. Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets. - Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.) - Use search.convex.dev to search Docs, Stack, and Discord all at once. - Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI. - Avoid tagging staff unless specifically instructed. Thank you!

erquhart•6h ago

Agree that migrations should be possible with less steps for sure. But type safety comes down to implementation. The recommendation would be to do dual writes while the new field is optional, run the migration while dual writes are active, then set the field to required, and then update reads. At no point in that process would you be reading in a required field that's typed as optional.

BluePenguinOP•2h ago

Hi! Thanks for the response. Let me illustrate the type issue with an example: Old schema

foobar: defineSchema({
  foo: v.string();
})

foobar: defineSchema({
  foo: v.string();
})

New wanted schema:

foobar: defineSchema({
  foo: v.string();
  bar: v.string(); // default: "something"
})

foobar: defineSchema({
  foo: v.string();
  bar: v.string(); // default: "something"
})

I'd have to take the following steps: Step 1 - transitioning schema

foobar: defineSchema({
  foo: v.string();
  bar: v.optional(v.string());
})

foobar: defineSchema({
  foo: v.string();
  bar: v.optional(v.string());
})

Step 2 - run migration

migrations.define({
  table: 'foobar',
  migrateOne: async (ctx, document) => {
    await ctx.db.patch(document._id, {
      bar: 'something'
    })
  },
})

migrations.define({
  table: 'foobar',
  migrateOne: async (ctx, document) => {
    await ctx.db.patch(document._id, {
      bar: 'something'
    })
  },
})

Step 3 - make bar required This is where we want to change the schema to:

foobar: defineSchema({
  foo: v.string();
  bar: v.string(); // default: "something"
})

foobar: defineSchema({
  foo: v.string();
  bar: v.string(); // default: "something"
})

This is what could break. Breakage Let's say after step 2 I have some code I forget to update with the new schema that does the following

ctx.db.insert('foobar', {
  foo: 'hello',
  // no bar - typescript says it's a valid insert
})

ctx.db.insert('foobar', {
  foo: 'hello',
  // no bar - typescript says it's a valid insert
})

In the time after my migration this could have been run. Now I have executed my migration, but I can't apply Step 3 anymore since we now have a new document that does not match the new migration. This is what I'm trying to address. Essentially with a migration we want to guarantee a type change, which is not possible in a live migration in the current setup, which prevents true typesafe end-to-end migrations. Let me know you thoughts.

erquhart•2h ago

That's fair. There's a whole layer of functionality that Convex doesn't provide out of the box, but that are pretty straightforward to implement with the primitives available. The problem you're describing here is similar to enforcing uniqueness, which has the same caveat of "what if I forget to update an insert call somewhere". The answer to this whole class of issues is not using ctx.db.insert/patch/delete directly in your endpoints. What you do instead can look a number of different ways, but it boils down to having a single point of insert/patch/delete where rules can be enforced over and beyond what the schema enforces. You can write helpers, you can use custom functions in place of the standard query/mutation/action, there's Ents, which is in maintenance mode, but some folks still swear by. I personally have helper functions for each table in my non-trivial apps. For your 'foobar' table, then, you'd have a function for inserting a foobar document, and use that everywhere. I handle enforcement by using an eslint rule to disable ctx.db access outside of the helper files. That's just one approach, but it illustrates the core idea. When your Convex app goes from prototyping to having real complexity, those low level ctx.db calls become sort of dangerous when used directly, because the schema can't enforce all of your requirements. So some sort of broad adaptation to this reality is necessary.

BluePenguinOP•1h ago

Allright, essentialy DRY up any atomic DB interaction. Thanks for linking to Ents. Looks like it's great for defaults, but I don't see native support for A->B schema migrations where B is not a superset of A. I would feel a new v. function would help. Imagine

foobar: defineSchema({
  foo: v.string();
  bar: v.transitioning({
    // atRest would always have to be a superset of `writes`
    atRest: v.optional(v.string());
    writes: v.string();
  });
})

foobar: defineSchema({
  foo: v.string();
  bar: v.transitioning({
    // atRest would always have to be a superset of `writes`
    atRest: v.optional(v.string());
    writes: v.string();
  });
})

which would lead to types:

ctx.db.get(foobarId) // { foo: string, bar: string | undefined }
ctx.db.insert({
  foo: "hello",
  // error: `bar` is missing
});
ctx.db.insert({
  foo: "hello",
  bar: "something" // accepted
});
ctx.db.patch(foobarId, {
  foo: "world", // accepted
});
ctx.db.patch(foobarId, {
  bar: undefined, // error: `bar` should be `string`
});

ctx.db.get(foobarId) // { foo: string, bar: string | undefined }
ctx.db.insert({
  foo: "hello",
  // error: `bar` is missing
});
ctx.db.insert({
  foo: "hello",
  bar: "something" // accepted
});
ctx.db.patch(foobarId, {
  foo: "world", // accepted
});
ctx.db.patch(foobarId, {
  bar: undefined, // error: `bar` should be `string`
});

This would give full end-to-end type guarantees since we prevent any invalid types to be written, while still allowing a transitionary type at rest. This way we can run a migration and not worry we're accidentally writing invalid types through completely native type support. Anyway, realise there's a lot on your plates. Love the direction Convex is going!

First-class offline migrations

Did you find this page helpful?