Migrate MongoDb documents to convex
Hey, I have two table in my mongodb which has lots of documents and now I want to migrate them to convex. I can export the documents from mongo but to import it in convex I have to remove " " from keys and its not possible to do that for large no. of documents
14 Replies
If you're working with a large number of documents then you may want to explore https://docs.convex.dev/database/import-export/streaming
Other folks have also written custom Python scripts for their particular needs
Streaming Data in and out of Convex | Convex Developer Hub
Streaming Data in and out of Convex
To check my understanding, are you getting an error because of spaces in document keys? Spaces are generally allowed in document keys, so it would be great to see the error message. How are you attempting the import (i'm guessing
npx convex import)?not spaces, "key" its string quotes are not allowed in add document. I am directly adding json data to add document
Is this the correct way to import the data?
What does the data look like? Convex has limits on values so they can be retrieved efficiently. If you have objects with >1024 fields, i would restructure the data, maybe splitting it into multiple tables. https://docs.convex.dev/production/state/limits#documents
Limits | Convex Developer Hub
We’d love for you to have unlimited joy building on Convex but engineering
Quotes are also allowed in document keys, although you probably don't want them. You could potentially call JSON.parse, but it would be great to see more of the code to know for sure
This is my object format
That doesn't look like it can be converted to csv (so i'm not sure what csv you imported). I recommend importing with jsonl format
I am exporting it from mongo
"visuals_by_visual_id": {}. Now its saying Row 32 wasn't a valid Convex value: Field name cannot be empty
What file are you importing to get that error?
Some tips:
- you can make a jsonl file with https://stack.convex.dev/json-array-to-json-lines-jsonl
- you will need to rename the "_id" field because it's a reserved field in convex, just like in mongo, and their formats aren't compatible (you can rename fields with jq)
- you will need to rename fields starting with "$" because those are also reserved
Convert your .json array to a .jsonl (JSON Lines)
JSON Lines stores one JSON object per line in a file, which is more scalable. Here's the one-liner to convert your .json file to .jsonl
All these steps I am already doing but I can't store empty value fields
what is on row 32?
I am importing this json
Ah i see. Empty field names are also not allowed in convex, although i don't recall why. Can you replace occurrences of
"": with something else?
I'm now realizing this is what you mentioned in the original message (sorry for being slow). I'll look into why we don't support empty strings as object keys
update here: we do intend to allow empty strings as object keys, as part of more generally relaxing constraints. i'll add a task internally