Jules•3w ago

Exporting component-specific data from production to development

Hi, right now I am using ConvexHttpClient to query specific documents from production and subsequently use npx convex import to import them into my development deployment. I would also like to copy over specific documents like this for data stored inside non-app components. Is it possible to do this right now? I do see that npx convex import has an option to specify a --component path, but I don't know if it's possible to programmatically query specific documents from a non-app component. I know it's possible to export all data using npx convex export, but I'd like to only export specific documents since exporting the full production database can take a long time.

8 Replies

Convex Bot•3w ago

Thanks for posting in <#1088161997662724167>. Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets. - Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.) - Use search.convex.dev to search Docs, Stack, and Discord all at once. - Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI. - Avoid tagging staff unless specifically instructed. Thank you!

ian•3w ago

I would suggest: - Make queries for a subset of documents in the component- as an internalQuery if you want extra isolation / ensure no one else uses them - Run npx convex run --component <path> myFile:getRepresentativeTestData from the shell, saving the output as json - Import the data either with a mutation you pass the data to, or via import, provided the output json is in the right format

JulesOP•3w ago

Is it possible to add custom queries to components that you've added through a npm package? For instance, we have subscription data stored in the polar component (which is imported from @convex-dev/polar). I suppose we could fork the package and add it to our own repository, but that feels like overkill just to add support for exporting component data.

ian•3w ago

ah gotcha- have you considered manually curating your dev data? Might be a good idea to avoid putting prod payment metadata in dev. Also have you experimented with npx convex data? I believe you can fetch component data directly Also not unreasonable for components to expose more data.

JulesOP•3w ago

Might be a good idea to avoid putting prod payment metadata in dev.

I agree. The main reason we want to have this ability is to debug user-specific errors, for which it does help to have the most representative copy of their data for debugging. Ideally we add a feature for seeding (part of) the data in the copy script. I just looked at npx convex data, thanks for mentioning it. Unfortunately this command doesn't have support for providing a --component name. Also, the outputted data is in a pretty-printed table, which doesn't easily allow for converting the data to a JSONL file.

ian•3w ago

npx convex data --component agent messages works for me. However yes that format is not very friendly to parsing. I just put up a PR to do json, but I imagine it's still not there since you'd want some custom filter.. The best option right now feels like doing it in a customQuery in the dashboard, then copying those results into json and pasting them in the dashboard in dev. Or if it's a reasonable API for general purpose we can add it to the component

JulesOP•3w ago

npx convex data --component agent messages works for me.

Ah interesting. Then a heads up that the flag isn't present in the help output (version 1.25.4):

$ npx convex data --help
Usage: convex data [options] [table]

Inspect your Convex deployment's database.

  List tables: `npx convex data`
  List documents in a table: `npx convex data tableName`

By default, this inspects your dev deployment.

Arguments:
  table                               If specified, list documents in this table.

Options:
  --limit <n>                         List only the `n` the most recently created documents. (default: 100)
  --order <choice>                    Order the documents by their `_creationTime`. (choices: "asc", "desc", default: "desc")
  --env-file <envFile>                Path to a custom file of environment variables, for choosing the deployment, e.g. CONVEX_DEPLOYMENT or CONVEX_SELF_HOSTED_URL.
                                      Same format as .env.local or .env files, and overrides them.
  --prod                              Inspect the database in this project's production deployment.
  --preview-name <previewName>        Inspect the database in the preview deployment with the given name.
  --deployment-name <deploymentName>  Inspect the database in the specified deployment.
  -h, --help                          display help for command

$ npx convex data --help
Usage: convex data [options] [table]

Inspect your Convex deployment's database.

  List tables: `npx convex data`
  List documents in a table: `npx convex data tableName`

By default, this inspects your dev deployment.

Arguments:
  table                               If specified, list documents in this table.

Options:
  --limit <n>                         List only the `n` the most recently created documents. (default: 100)
  --order <choice>                    Order the documents by their `_creationTime`. (choices: "asc", "desc", default: "desc")
  --env-file <envFile>                Path to a custom file of environment variables, for choosing the deployment, e.g. CONVEX_DEPLOYMENT or CONVEX_SELF_HOSTED_URL.
                                      Same format as .env.local or .env files, and overrides them.
  --prod                              Inspect the database in this project's production deployment.
  --preview-name <previewName>        Inspect the database in the preview deployment with the given name.
  --deployment-name <deploymentName>  Inspect the database in the specified deployment.
  -h, --help                          display help for command

(and on that note, the --local flag also isn't present in the help output of the npx convex dev command)

I just put up a PR to do json, but I imagine it's still not there since you'd want some custom filter..

That's great, thank you. Yes, ideally we want to filter the data by a specific user id.

Or if it's a reasonable API for general purpose we can add it to the component

Right, we’ll think about what API would best meet our requirements.

ian•2w ago

good catch - I'm updating the data command and will un-hide the arg

Exporting component-specific data from production to development

Did you find this page helpful?