Parallel actions
Hello, I have a small questions - I need to run in parallel (ideally) 90 actions for modifying json files (from Convex storage). I know that workflow component can run parallel steps. Would it be the ideal solution or there are better options? If workflow ideal - does it make sense to split parallel actions to multiple workflows, so it doesnt run all 90 actions on a single parallel? My use case: user updating json for primary language (localization) and then i need to sync all new values/keys with other languages (max 90). Thank you!
8 Replies
Thanks for posting in <#1088161997662724167>.
Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets.
- Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.)
- Use search.convex.dev to search Docs, Stack, and Discord all at once.
- Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI.
- Avoid tagging staff unless specifically instructed.
Thank you!
Do these actions need to be coordinated in some way? For example, do we need some code to run at the end once all 90 are done?
If not, you might consider using the Workpool component. You can enqueue all of the actions and it can handle
If so, you might consider the Workflow component. It has an onComplete function that can run after all steps are completed. Or you can do more steps after the 90 actions are completed in the workflow definition itself.
For both Workflow and Workpool, you define the maximum amount of parallelism: https://www.convex.dev/components/workflow#specifying-step-parallelism
Convex
Workflow
Simplify programming long running code flows. Workflows execute durably with configurable retries and delays.
Hey, thank you for the answer! One quick question - it says that recommended parallelism is 20 (and 100 for pro), but I can use
enqueueActionBatch . If i batch all 90 actions with enqueueActionBatch - does it count as 1 parallel for workpool? (I decided to go with workpools, because we don't need any code at the end)It's not going to run all 90 at a time regardless of whether you enqueue them individually or as a batch
But it would "solve" the parallelism limit because it's gonna launch only 1 workpool call, if I understood correctly?
Im just trying to understand the best way to do 90 actions as fast as possible and as stable as possible with all limits.
90 actions probably a wrong term because technically I could send few IDs into the same action and then just loop through. It would make actions longer and heavier but create less action instances overall
I'm not sure I 100% understand what you're saying. The workpool has a limit of how many actions it will run in parallel. If you have a maxParallelism of 20, for example, it will run 20 at a time and the rest will wait in a queue until the workpool has capacity. Does that clear it up or am I misunderstanding?
Yes, this part is clear. Does it mean that
enqueueActionBatch is effectivly just a fast way to add a lot of actions to the queue which would call component only once but respect parallel limit (for example 20)? So with batch im just gonna schedule all 90 actions at once but then workpool gonna decide when to call them in parallel?
It seems like batch path is the correct way because workpool component called only once. So (from my understanding) if 3 users decided to sync languages, it's gonna create maximum 270 actions for each language, but the workpool component itself gonna be called only 3 times because we called 3 batches.Yes, this is correct
Yes, it is best to batch if you can