Production deployment locked
Production deployment oceanic-kingfisher-7 has been locked for 30+ minutes by a hanging index backfill on by_user_workspace_folder index in the prompts table. Unable to deploy schema changes to cancel it. Request manual intervention to cancel backfill or rollback deployment.
7 Replies
Thanks for posting in <#1088161997662724167>.
Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets.
- Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.)
- Use search.convex.dev to search Docs, Stack, and Discord all at once.
- Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI.
- Avoid tagging staff unless specifically instructed.
Thank you!
Production index backfill hung for 20+ minutes on tiny dataset - is this expected?
Context:
- Deployed schema change adding new index to production (deployment: oceanic-kingfisher-7)
- Dataset: Only 67 records in prompts table
- Index: .index("by_user_workspace_folder", ["userId", "workspaceId", "folder"]) where folder is v.optional(v.string())
What happened:
- Backfill hung at "66/67 ready" for 20+ minutes with no progress
- All Convex queries returned 503 Service Unavailable
- Frontend stuck on loading screen
- Could not deploy schema fixes (deployment endpoint also returned 503)
- Convex dashboard slow/unresponsive
What I tried:
- Waited 20+ minutes for backfill to complete (no progress)
- Attempted to deploy fix removing the index (got 503 errors)
- Eventually created new Convex project to recover
Questions:
1. Is 20+ minute backfill expected for 67 records? (https://docs.convex.dev/database/reading-data/indexes/ suggest this should take seconds)
2. Is there an issue indexing optional fields with many null values, or was this a transient infrastructure issue?
3. Is there a way to recover from locked deployments without creating new project?
Indexes | Convex Developer Hub
Speed up queries with database indexes
other users are reporting outages too (see this channel), so my assumption is that you're affected by the outage
Convex Status
Welcome to Convex's home for real-time and historical data on system performance.
thanks. it got stuck on indexing so i thought it was something to do with my data?
It might be, but it's more likely that you're affected by the outage