33 Replies
Changing the filename doesn't break anything does it?
hi yes i just saw this in sentry. sorry about that, fixing imminently
Thank you!
should be fixed now. was an issue on our side, unrelated to filename
thanks for reporting!
Thanks!
Now seeing this:
Now it's 502 bad gateway
the bad gateway is because i restarted your deployment to fix it. the concurrency control failure is unexpected, but might be transient 🤔
okay, trying again
503 again, last pasted message
It tries for a little longer now before the message though
the import is going to conflict with any ongoing mutations, which might be causing the OCC if you have a lot of mutations (but it should be around the same as an
npx convex deploy). i'll look into itOkay. I'm importing to my dev server, which has no queries or mutations running right now
😦 looks like you've hit a limitation of
npx convex import (one i need to document better and fix the error message). it works reliably in two cases (1) importing a deployment's own backup into the same deployment, and (2) importing one deployment's export into another deployment if both deployments were created with versions >1.7. unfortunately your import doesn't fit either case.
the problem is your table with id 48 (you can see each table's id in the snapshot's _tables/documents.jsonl file) from the prod deployment has the same id as a system table in the dev deployment. so if we performed the import, the _id fields would conflict.Is there anything I can do to make the export work?
is it possible to somehow redeploy a given deployment in place to bring it up to date w/ 1.7 conventions? I could do that with both deployments if so
even a one-off fix on your end would be welcome, downtime is no problem as we're not live yet. I just need the data and relations preserved. We can make a new project, whatever is needed.
thinking it over on our end. one of the possible solutions is to rewrite all
_id rows for the conflicting table. ideally that would be done by npx convex import but you could also do it manually.that's the biggest table 😭
there may be other options, i'll ask around
btw by "manually" i meant write a python script, not actually manually. still, this is unfortunate if there are foreign references
yeah I can script it if need be
do you have relations pointing to the
_id field of this table? or do you care about the _id field staying stable for another reason (it's persisted in stripe or something)? curious for helping decidehmm actually no, those ids don't matter for that particular table
if I deploy to prod does that make prod a 1.7 deployment? Like I could do that and then both dev and prod would be 1.7?
Okay, I'll redeploy prod and try an export from that
sorry, no. there's a problem if either deployment was created before 1.7. redeploying won't change the table ids, so there will still be a conflict.
i think we have a reasonable solution, but we need some changes on our side to make it work. the solution is to remove
_id fields from the conflicting table's (with id 48) documents.jsonl in the zip file -- i did this in a text editor replacing the regex "_id":"[^"]+", with an empty string. and also remove the table from _tables/documents.jsonl. then the import will pick a new table id and new _id fields for all documents.
it requires a code change on our side though, which i'm working on nowsounds great, thank you! I'll stay tuned
@lee just checking to see if there's any update here
hi! still working on the code change to make this work. i expect to ship it later today
thank you!
sorry for the delays. should be ready now to try. you will need to remove the
_id fields from that table and remove the entry from _tables/documents.jsonl
Oh and if you get an error about some other table having a conflict, or you see the OCC error again, let me know. We can probably work out a solution. Sorry about the difficulties.Still need to try this, will update when I'm up and running, hopefully today or tomorrow
@lee (for whenever you're on working hours) I've done this with table 48, and am now getting a (helpful!) error message that table 57 also has conflicting ids with a system table. The ids in table 57 have multiple relations per row :/
Sorry for the delayed feedback on this, my app was down for a backend refactor and I just got back to shipping
darn i missed that conflict. how feasible would it be to start a new project and import the data into that new project? the old projects have these sorts of conflicts. if starting a new project is cumbersome, we've been thinking about solutions that can reset a dev deployment without touching prod, but it's not implemented yet.
Wouldn't importing old data into a new project still have the same issues potentially?
I can try it tomorrow
I think i made new projects have system tables that don't conflict with old projects. Let's see if it works 🤞
Ran for a while before timing out.
Subsequent attempts result in:
well i'm glad we got past the previous errors. thanks for stress testing the system! i'm not sure where this timeout is coming from (i don't think we set a timeout explicitly), but i can repro by trying to import a large enough file. looking into it
Thank you!
Mine was 3.3mb
Well, the zip is 3.3mb
My two largest tables are 16mb and 10mb
unzipped
documents.jsonl files that issorry forgot to update here. i found the timeout and i'm working on a fix. i think it requires CLI changes though so it would be released with convex 1.9
basically currently
npx convex import does a single HTTP request, which uploads the file and imports the data. by splitting up the APIs into more phases we should become more resilient to timeouts and also transient errorsGot it 👍
npx convex import should support much bigger files now. Thanks for your patience https://news.convex.dev/announcing-convex-1-9/
Convex News
Announcing Convex 1.9
Big Import and Export Improvements
When we released ZIP file imports and exports in Convex 1.7, it wasn't long before industrious developers started hitting the limitations. We’ve got some big improvements for you.
* File storage can now be included in Snapshot Export
* Snapshot import via npx convex import
Thank you!!