aleks_glushko
aleks_glushko•2mo ago

Dev to prod table export/import

Hi, My goal is to create deployment version and port all of the data from dev tables to prod. I have done the following to
- npx convex deploy to create deployment branch in convex - exported backup as ZIP from dev branch - run npx convex import --prod --replace /Users/aleks/Downloads/snapshot_expert-lark-122_1731932235731925884.zip But get an error, even with an empty activityLogs:
New table authAccounts has IDs that conflict with existing table activityLogs
Is something done in a wrong way and what could be the solution? Thanks.
15 Replies
Convex Bot
Convex Bot•2mo ago
Thanks for posting in <#1088161997662724167>. Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets. - Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.) - Use search.convex.dev to search Docs, Stack, and Discord all at once. - Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI. - Avoid tagging staff unless specifically instructed. Thank you!
aleks_glushko
aleks_glushkoOP•2mo ago
Restoring from a newly created backup also throws an error that one of the tables has unsupported IDs: Hit an error while importing: _id k1706d0778z3fx7gp8jnn317v574m313 cannot be imported into 'customerUsers' because its IDs have a different format Why, if all of the deployment tables are empty and schema is exactly the same
Clever Tagline
Clever Tagline•2mo ago
As the messages are indicating, the issue is with the IDs, and that's something out of your control. Here's my guess at what's happening. You've got references between tables in your dev deployment; e.g. some field in tableA is an ID of a record in tableB. The way that Convex builds IDs for each table, it knows internally that a given ID matches with a given table. However, when generating the ID format for each table, sometimes the format for a table in dev will be used for a different table in prod. Using the above example, perhaps the ID that pointed to a tableB document in dev now matches the format of tableD documents in prod. In the end, this messes up all of your relationships because the schema says that the field in tableA should be an ID from tableB, but the way the IDs were generated between dev and prod conflicts with that design. That's my gut feeling on why this is happening. As far as how to fix it, that's a conundrum. I wonder if the migration component could help with this, but at first glance it appears to only solve the problem of migrating data within a given environment. If that won't work, you may have to build a custom migration tool that tracks the relationships and generates the documents from scratch via code.
$2a$06$ZaPf8mpjvzeiGl1SGS9x1.ie
i have same problem now
Clever Tagline
Clever Tagline•2mo ago
I'm hoping that the devs will have a more elegant solution.
aleks_glushko
aleks_glushkoOP•2mo ago
@Clever Tagline , thank you for your reply! Could you elaborate on what is the migration component exactly? If it is the convex ui components for backup and restore, then, unfortunatelly, it doesn't solve the issue
FleetAdmiralJakob 🗕 🗗 🗙
Convex components is convex backend code that you can call inside your convex code
FleetAdmiralJakob 🗕 🗗 🗙
Components
Independent, modular, TypeScript building blocks for your backend.
FleetAdmiralJakob 🗕 🗗 🗙
Migrations
Framework for long running data migrations of live data.
nipunn
nipunn•2mo ago
Hi! We're taking a look into this. Hang tight. Hi. We spotted an issue/inconvenience that y'all have noticed, regarding restoring into a deployment that has pre-existing tables. I'm sorry for the inconvenience!! We got a few paths forward for you. 1) Wait a couple of days and we'll make sure this is handled automatically so you can just do your restoration/import. 2) Manually delete all the tables in your destination deployment (prod deployment in your case) before doing the import. Then it'll work. 3) Restore into a totally fresh deployment (make a new project) Any of these should work. Let me know if any of those work for you! Again - I'm sorry for the inconvenience and we'll make this experience smoother for in the future (give us a few days).
nipunn
nipunn•2mo ago
Delete table button is here on the dashboard in case you want to go down route (2)
No description
aleks_glushko
aleks_glushkoOP•2mo ago
Thank you @nipunn , I will try one of the offered solution, while you are fixing the backend. Quick question: is there a way to get an amount of unallocated concurrent actions that still could potentially run?
Sano
Sano•4w ago
Hey @nipunn, just checking in on the fix for the data import ID conflicts. Is it ready yet, or should I still go with one of the temporary solutions? Thanks!
nipunn
nipunn•4w ago
@Sano fix should be in and working
aleks_glushko
aleks_glushkoOP•4w ago
@nipunn I have tried again the following: - npx convex deploy - backup dev branch data - import into prod version the dev branch data It doesn't work, because of the IDs conflicts, e.g. New table authAccounts has IDs that conflict with existing table activityLogs The following solution didn't work:
2) Manually delete all the tables in your destination deployment (prod deployment in your case) before doing the import. Then it'll work.
Error: Hit an error while importing: _id k1706d0778z3fx7gp8jnn317v574m313 cannot be imported into 'customerUsers' because it came from a different deployment and conflict with preexisting tables in this deployment. Try deleting preexisting tables or importing into an empty deployment. The following solution worked: 1) Create a backup on the dev branch and download 2) Extract the zip file 3) Manually clean the documents.jsonl for the tables that can be avoided to import, in the generated_schema.jsonl put "never" 4) Compress into zip again 5) Use the following CMD command: npx convex import --prod <path to the file>/<compressed file name>.zip

Did you find this page helpful?