Are there any existing self hosting
Are there any existing self hosting conversations about scale? Like rules of thumb for how much throughput certain common sized servers can handle. And maybe basic things to know about any kind of horizontal scaling if possible?
4 Replies
One thing we just did recently was open source our (previously) internal tool called load-generator
https://github.com/get-convex/convex-backend/tree/main/crates/load_generator
It can slam a backend to see what it's capable of.
@Emma has been working on a benchmark mode for load generator and has a lot of good ideas on how we could use something like that to test for these things in practice.
We still owe some documentation and a blog post about load-generator so that's why it's been quiet so far. We'll get around to it eventually, but you're welcome to try it out.
For horizontal scaling, the Convex cloud product has good horizontal scaling involved. However, I think it's unlikely that an individual instance would require horizontal scaling in this way.
Convex cloud uses horizontal scaling to broadly get cost savings for smaller customers, while making larger customers able to get more resources.
There's a function_runner https://github.com/get-convex/convex-backend/tree/main/crates/function_runner here which we wrap up into a separate service to horizontally scale the function execution separately from the committer. There's a good blog post on this stuff here https://stack.convex.dev/how-convex-works
Regarding horizontal scaling, there are a few ways in which horizontal scaling can manifest.
1) Concurrent Websockets (live subscription clients)
2) Concurrent longrunning actions
3) Concurrent mutations
They tend to have different required solutions. A full horizontally scaled stack requires solutions to all of them - which is part of why we recommend cloud often.
Our cloud product solves with:
1 - frontend service that specializes in keeping a large volume websockets alive and routing the queries/mutations/actions appropriately.
2 - needs horizontally scaled function runner. For regular actions/http actions, we have the aformentioned function runner as a separate service.
3 - Mutations can be run in parallel with optimistic concurrency control, but they are often bottlenecked on the commit to the database.
There's also separate scaling concerns if you use search or vector search.
But the pieces are out there for you to put something together yourself.
A lot of the cloud scaling is only necessary because we serve 10s of 1000s of instances. If you're only serving one or two, it may just be fine on a single self-hosted machine.
like probably 1000 concurrent websockets, 100s of concurrent actions, and mutation throughput - well gotta test that one with load generator. But I'd estimate 10s / second.
might be more or less depending on your specifics.... and like... tested numbers better than my napkin maths 😄
it also obviously depends on what the mutations/longrunning actions are, and what the websockets are doing actively - but napkin maths can handwave that all away
that makes sense. and yeah, this is exactly the kind of stuff I'm just making sure I'm ready to think through. So like right now, CEO is saying that this is an institutional product. So the current plan is that it would need to serve maybe a few thousand institutions with a few users each. But if he suddenly says.. this would make a great retail banking product. I need to be ready with the plan to handle a lot more signups.
I just ran load generator locally and it was pretty easy.
Your mileage is going to vary a ton based on your setup, so you have to do your own benchmarking to answer all the questions relevant to your setup, but you have the tools now. Good luck
I added instructions here for using load generator https://github.com/get-convex/convex-backend/blob/main/crates/load_generator/README.md
GitHub
convex-backend/crates/load_generator/README.md at main · get-conve...
The open-source reactive database for app developers - get-convex/convex-backend