Server side event workers
I've been through the ai-town, llama farm, and the work stealing stack and still unclear on all available options.
Since we building more of a turn-based style slower game with lots of roles, permissions, acl it fits well in-between any two examples.
In llama farm the /worker/client.ts is being run by the client via the console.
Our goal is to generate events into a events table most of these process fairly quick, analytics and such but will get more compute heavy as time goes on, some events we want to run immediately . Its not something where we need a remote worker. Our worker needs to process the events and do often many mutations etc. The base question is how would you best run this worker on the hosting server?
Examples like AI-town seem to push us towards just doing as much as possible within convex. But will we regret putting too much compute on convex from a cost standpoint eventually. Our db interactions are just going to be unavoidably heavy. I don't exactly the idea of a 1 second tick type thing as with ai-town.
Is it possible to have server side(webhost) event listeners that respond immediately to pushed update from convex? Are there examples?
Seems like we might end up with a bit of everything.
24 Replies
Yeah. You can use the node library and have some container subscribe to events and then act on them.
Most teams just end up using convex workflow though. It’s way more convenient!
Yeah, so In the covex workflow is it possible to have a setup where there isn't a looping check, when work comes in its processed immediately? Or I'm thinking the event could also fire the same checkforWork/events in its own loop for something that needs to get done now.
Yes, when you make a change you can use the scheduler to kick off something in the background. This is what 90+% of people do for workflow with Convex. The AI Town / llama chat / work stealing optimizations are geared around more complex setups. Sorry they led you astray.
One article that goes one layer above the regular scheduler capabilities is https://stack.convex.dev/background-job-management
Background Job Management
Implement asynchronous job patterns using a table to track progress. Fire-and-forget, cancelation, timeouts, and more.
Generally it's like:
1. A request comes in.
2. You write some stuff to the DB and if there's anything longer, you schedule a follow-up action (it's scheduled transactionally so will only happen if your mutation succeeds).
3. That action can kick off mutations or other actions - immediately or scheduled in the future.
You can have logic like "whenever I update the user email, schedule an action to send them a confirmation email to both addresses" by writing some code that wraps updating the user.
Yeah the background-job-management page was what i was looking for. I just didn't show up i n my searches, i know i had seen it before. I also think it could probably use a couple updates, maybe a reference to the convex-helpers action retries function etc. I also tend to bias towards newer content as the best content.
Overall its not a loss as we will have parts almost exactly like ai-town and llama-farm but its nice to have the core events system sorted out now. Hopefully i it doesn't take any major rework later.
A question I have that this article didn't seem to have unless I missed it. How do I check the status of on going long running background jobs? Say for instance I have a 3rd party API that transcodes videos. In convex, I upload the video to a storage and using the storage ID I create a mutation to a table to save this file upload id. From this mutation I then want to kick off an action, that will talk to the transcoding API and start a transcoding job. How do I monitor the status of this job so I can update some table once the job is complete? One Idea I had was to use a "one off" CRON job that polls the Transcode API every X mins that I would kill once the job is complete but that doesn't seem possible unless I'm missing something from the docs.
After you schedule the action from the mutation, you can track its status using the system table
_scheduled_functions: https://docs.convex.dev/scheduling/scheduled-functions#retrieving-scheduled-function-statusScheduled Functions | Convex Developer Hub
Convex allows you to schedule functions to run in the future. This allows you to
If you have a table tracking the job - you can call it "jobs", then within the transcode action, you can be updating it with progress.
then write queries against the relevant job ID
but how would I poll the status of the transcode job?
you mean from the action how do you talk to the 3rd party doing the transcode?
that's up to the transcode API docs
how do I say every X mins check the status of this 3rd party api?
if the action is running the whole time, you can do it with a setInterval. If the action isn't running, you can schedule an action to check, which then schedules another action to check
scheduler.runAfter(5*60*1000, internal.videos.checkProgress, { someKey: identifyTheVideo })hmm so just always scheduling actions one after the other to check status?
ideally I wanted something that would like a cron job that I could "run x action every y mins" and within x action, I could kill the cron job if it was no longer needed
Maybe this is just better handled using something like trigger.dev?
or AWS step functions
I guess this isn't as bad of an option though
There are cron jobs in Convex, but honestly a recursively scheduled action is better - then it isn't running when you don't want it to.
You can also implement Crons on top of convex. An example is Cronvex: https://www.cronvex.com/
I definitely wouldn't use trigger.dev or AWS step functions - everything those can do can be done with vanilla Convex. That's just what Cronvex implements
It's open source and will have a stack post out soon: https://github.com/JamesCowling/cronvex
GitHub
GitHub - JamesCowling/cronvex: Send periodic http requests on a cro...
Send periodic http requests on a cron schedule. Contribute to JamesCowling/cronvex development by creating an account on GitHub.
How do a recursively call an action? Do I have ref to the action from within itself?
GitHub
cronvex/convex/cronlib.ts at main · JamesCowling/cronvex
Send periodic http requests on a cron schedule. Contribute to JamesCowling/cronvex development by creating an account on GitHub.
so any internal/action you declare will be avaliable on the
internal objectHas to be via scheduling, but recursive is same as initial
even within itself
yep
nice
exactly - all internal functions are in
internal, whereas public functions are on api https://docs.convex.dev/functions/internal-functionsInternal Functions | Convex Developer Hub
Internal functions can only be called by other functions
nice thanks for the help