Dev to prod table export/import
Hi,
My goal is to create deployment version and port all of the data from dev tables to prod.
I have done the following to
-
-
npx convex deploy
to create deployment branch in convex
- exported backup as ZIP from dev branch
- run npx convex import --prod --replace /Users/aleks/Downloads/snapshot_expert-lark-122_1731932235731925884.zip
But get an error, even with an empty activityLogs:
New table authAccounts has IDs that conflict with existing table activityLogsIs something done in a wrong way and what could be the solution? Thanks.
15 Replies
Thanks for posting in <#1088161997662724167>.
Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets.
- Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.)
- Use search.convex.dev to search Docs, Stack, and Discord all at once.
- Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI.
- Avoid tagging staff unless specifically instructed.
Thank you!
Restoring from a newly created backup also throws an error that one of the tables has unsupported IDs:
Hit an error while importing: _id k1706d0778z3fx7gp8jnn317v574m313 cannot be imported into 'customerUsers' because its IDs have a different format
Why, if all of the deployment tables are empty and schema is exactly the same
As the messages are indicating, the issue is with the IDs, and that's something out of your control.
Here's my guess at what's happening. You've got references between tables in your
dev
deployment; e.g. some field in tableA
is an ID of a record in tableB
. The way that Convex builds IDs for each table, it knows internally that a given ID matches with a given table. However, when generating the ID format for each table, sometimes the format for a table in dev
will be used for a different table in prod
. Using the above example, perhaps the ID that pointed to a tableB
document in dev
now matches the format of tableD
documents in prod
.
In the end, this messes up all of your relationships because the schema says that the field in tableA
should be an ID from tableB
, but the way the IDs were generated between dev
and prod
conflicts with that design.
That's my gut feeling on why this is happening. As far as how to fix it, that's a conundrum. I wonder if the migration component could help with this, but at first glance it appears to only solve the problem of migrating data within a given environment. If that won't work, you may have to build a custom migration tool that tracks the relationships and generates the documents from scratch via code.i have same problem now
I'm hoping that the devs will have a more elegant solution.
@Clever Tagline , thank you for your reply! Could you elaborate on what is the migration component exactly? If it is the convex ui components for backup and restore, then, unfortunatelly, it doesn't solve the issue
Convex components is convex backend code that you can call inside your convex code
Components
Independent, modular, TypeScript building blocks for your backend.
Migrations
Framework for long running data migrations of live data.
Hi! We're taking a look into this. Hang tight.
Hi. We spotted an issue/inconvenience that y'all have noticed, regarding restoring into a deployment that has pre-existing tables.
I'm sorry for the inconvenience!!
We got a few paths forward for you.
1) Wait a couple of days and we'll make sure this is handled automatically so you can just do your restoration/import.
2) Manually delete all the tables in your destination deployment (prod deployment in your case) before doing the import. Then it'll work.
3) Restore into a totally fresh deployment (make a new project)
Any of these should work. Let me know if any of those work for you!
Again - I'm sorry for the inconvenience and we'll make this experience smoother for in the future (give us a few days).
Delete table button is here on the dashboard in case you want to go down route (2)
Thank you @nipunn , I will try one of the offered solution, while you are fixing the backend.
Quick question: is there a way to get an amount of unallocated concurrent actions that still could potentially run?
Hey @nipunn, just checking in on the fix for the data import ID conflicts. Is it ready yet, or should I still go with one of the temporary solutions? Thanks!
@Sano fix should be in and working
@nipunn I have tried again the following:
- npx convex deploy
- backup dev branch data
- import into prod version the dev branch data
It doesn't work, because of the IDs conflicts, e.g. New table authAccounts has IDs that conflict with existing table activityLogs
The following solution didn't work:
2) Manually delete all the tables in your destination deployment (prod deployment in your case) before doing the import. Then it'll work.Error: Hit an error while importing: _id k1706d0778z3fx7gp8jnn317v574m313 cannot be imported into 'customerUsers' because it came from a different deployment and conflict with preexisting tables in this deployment. Try deleting preexisting tables or importing into an empty deployment. The following solution worked: 1) Create a backup on the dev branch and download 2) Extract the zip file 3) Manually clean the documents.jsonl for the tables that can be avoided to import, in the generated_schema.jsonl put "never" 4) Compress into zip again 5) Use the following CMD command: npx convex import --prod <path to the file>/<compressed file name>.zip