Help upgrading with unvalidated documents
I'm trying to upgrade, but have old data that doesn't conform to my schema.
It's pretty challenging to try to get it conforming one item at a time. Is there an option to just drop data that doesn't conform easily?
11 Replies
@ian has a slack post in progress about this.
Thanks!
In the dashboard if I try to filter to the ID that I want to remove, it crashes Event ID: d1161471b2b14c839070588b6b814c12
presumably filter doesn't work past a certain number of docs?
One thing that can help is filtering by type in the dashboard - if there's an extra field or the wrong type, you can find them all, then edit or delete them. If there's a ton of them, you can write a migration script to walk all your data and modify / delete it
Note you can also disable schema validation and deal with this issue separate from the upgrade.
Thanks Presley, how do I disable validation?
it's kind of buried in https://blog.convex.dev/announcing-convex-0-14-0/ but there's a flag
{schemaValidation: false}Convex News
Announcing Convex 0.14.0
Meet the brand new Convex Rust client! We’ve also open sourced the JS client, added schema validation, and more. 0.14.0 is the best version of Convex yet!
Schemas | Convex Developer Hub
Schema validation keeps your Convex data neat and tidy. It also gives you end-to-end TypeScript type safety!
@magicseth Going to chime in here, but also try not to derail the thread. Thanks for reporting the dashboard issue, looks like it's related to a known task we have to improve pagination when searching through many documents.
We'll follow up when this specific issue is fixed, but in the meantime, Ian's suggestion of creating a filter to search for documents that don't match your schema should work better.
We have some other enhancements to the dashboard filtering experience coming soon which should ease migrations as well, but I know that a feature next week doesn't help you today, so I empathize with the difficulty you're experiencing migrating right now
I wrote a post on doing migrations in Convex: https://stack.convex.dev/intro-to-migrations
And the code to help out: https://github.com/get-convex/convex-helpers/blob/main/convex/lib/migrations.ts
Intro to Migrations
There are as many ways to migrate data as there are databases, but here’s some basic information to set the stage.
GitHub
convex-helpers/migrations.ts at main · get-convex/convex-helpers
A collection of useful code to complement the official packages. - convex-helpers/migrations.ts at main · get-convex/convex-helpers
Actually the more helpful post is this one:
https://stack.convex.dev/migrating-data-with-mutations
Migrating Data With Mutations
Using mutations to migrate data in Convex.
In your case you could check if a document was mis-formatted and delete it