Does anyone know of any way to track the
Does anyone know of any way to track the progress of either an export or table update? I added an index to a table with ~7million documents and it has been running for over 24 hours now. And a full export has been running for over 2 days. I know they are actually doing something as the database container has a consistent ~10% cpu utilisation. At this point it would be quicker to just wipe the table and reimport everything with the index already set. This is really concerning me as trying to pull my data out of convex in the future can't take multiple days. Everything is running on a fast local instance with NVME storage so iops is not the problem. Any insights would be helpful.
4 Replies
Check the backend logs from the
convex-backend container to see what's going on in there.No errors are shown other than some on boot related to search but I have ignored them since the initial install as I assume they are just a consequence of not configuring any search fields in the schema. I believe I ended up solving the index backtrace problem by doubling the
maintenance_work_mem in postgres and increasing the available memory for the database from 8GB to 16GB. It seems excessive for the size of the tables but if it's required then so be it. I have not spotted anything in your docs about changing the log level so if there are ways to enable a debug mode of sorts please let me know.
I do think it would be very helpful to have an additional section of the dashboard that shows all running backend tasks such as backtraces, and exports/imports.Yeah - I hear you, especially for exports/imports. It's definitely a shortcoming of the self-hosted dashboard currently. There is code for it in the cloud dashboard - which with some eng effort we could adapt for self hosting.
I just filed https://github.com/get-convex/convex-backend/issues/105 to track that - we're probably not going to work on it right away, but wanted to get it noted as a good idea.
GitHub
Export / Import tab on dashboard · Issue #105 · get-convex/convex...
Would be great to have an export/import tab on the dashboard deployment page. We have a Backups section in the cloud dashboard (npm-packages/dashboard) which covers a superset of this behavior. Wou...
You mentioned two separate issues in the original post, one around adding an index and a separate one around exports. Lets cover them one by one.
While an export is going, there should be some logs like these ones
https://github.com/get-convex/convex-backend/blob/514603e3c6498fd0deee57da27707b584bbe01a1/crates/application/src/exports/mod.rs#L315
Are you seeing those in the backend logs?
For adding an index - you should be able to see loglines like this one https://github.com/get-convex/convex-backend/blob/514603e3c6498fd0deee57da27707b584bbe01a1/crates/database/src/index_worker.rs#L326
It does sound like it would be really useful to get progress indicators on large backfills - seems like a great feature idea.
GitHub
convex-backend/crates/database/src/index_worker.rs at 514603e3c6498...
The open-source reactive database for app developers - get-convex/convex-backend
GitHub
convex-backend/crates/application/src/exports/mod.rs at 514603e3c64...
The open-source reactive database for app developers - get-convex/convex-backend