Workpool Issues
🪲Bug Report❔Advice
Hey, we hit what looks like a producer-side Workpool OCC issue in our self-hosted staging environment. We use the standard enqueueAction() API from a Convex mutation to schedule background processing after
a write.
In a burst of 100 independent jobs from 32 concurrent requests, 7 requests eventually returned 500 after retries. The relevant error was:
Looking at the source, it seems each enqueue goes through:
while dispatch/completion also update that state. So action maxParallelism limits execution, but doesn’t limit concurrent producers calling enqueueAction.
This seems related to https://github.com/get-convex/workpool/issues/166. We also noticed the unreleased BatchWorkpool work: https://github.com/get-convex/workpool/pull/167. Is that intended to address this
runStatus contention pattern? Is there a supported mitigation for high-fan-in enqueue before it lands?
We have redacted logs and can try to extract a minimal Workpool-only repro if that would help.
a write.
In a burst of 100 independent jobs from 32 concurrent requests, 7 requests eventually returned 500 after retries. The relevant error was:
Looking at the source, it seems each enqueue goes through:
while dispatch/completion also update that state. So action maxParallelism limits execution, but doesn’t limit concurrent producers calling enqueueAction.
This seems related to https://github.com/get-convex/workpool/issues/166. We also noticed the unreleased BatchWorkpool work: https://github.com/get-convex/workpool/pull/167. Is that intended to address this
runStatus contention pattern? Is there a supported mitigation for high-fan-in enqueue before it lands?
We have redacted logs and can try to extract a minimal Workpool-only repro if that would help.
GitHub
Summary In a self-hosted setup, I can reproduce repeated OCC conflicts in internal workpool/workflow tables (runStatus, pendingCompletion) even with a single request path. Repro repository Main bra...
GitHub
Summary
Adds BatchWorkpool, a new execution mode where a small number of long-lived executor actions each run many task handlers concurrently. This eliminates the 1-action-per-task overhead of stan...
Adds BatchWorkpool, a new execution mode where a small number of long-lived executor actions each run many task handlers concurrently. This eliminates the 1-action-per-task overhead of stan...
