2024 Q1 plans
Onwards and upwards, time for our plans going into Q1 (which starts Feb at Convex). Our primary focus is on scaling up and improving the experience for growing teams that are already heavily invested in Convex.
⚖️ Scale up and out
There are a lot of projects lined up to improve internal Convex scalability but the tasks that most directly affect users are:
* Speed up code sync for large codebases. * Finish deploying Fun Run to increase the level of parallelism we can support. 💳 Plans and Billing
This is mostly an internal project to decouple our plans and billing from our backend codebase so we can iterate more quickly on providing ideal pricing to users. User-visible tasks include:
* Don't charge for cached database reads. * Make cached function calls cheaper. 🐛 Debugging and Profiling
We pride ourselves on Convex just working out of the box but there are times when some additional insight is required into why a function isn’t performing well. We can do a lot better to help folks help themselves:
* Allow logging of execution time within a function. * Stream logs out of long-running or failed functions. * Log system errors more clearly. * Add support for fine-grained tracing of function execution. * Preemptively notify customers who are getting close to execution limits. * Better local testing and unit testing support. * Publish best practices for monitoring and alerting for customer projects. 📖 Open Source
Finish the swing on open sourcing the core of the Convex backend. The main remaining tasks are extending the CLI to provide better support for managing a local backend. This will be a single-binary version of the Convex backend that is intended for testing and running locally, not the highly durable scale-out distributed system we run customer backends on.
We’re looking forward to getting these changes out! Our community is growing quickly and we think this set of projects will get us well ahead of both the influx of customers and larger projects onboarding to the platform.
There are a lot of projects lined up to improve internal Convex scalability but the tasks that most directly affect users are:
* Speed up code sync for large codebases. * Finish deploying Fun Run to increase the level of parallelism we can support. 💳 Plans and Billing
This is mostly an internal project to decouple our plans and billing from our backend codebase so we can iterate more quickly on providing ideal pricing to users. User-visible tasks include:
* Don't charge for cached database reads. * Make cached function calls cheaper. 🐛 Debugging and Profiling
We pride ourselves on Convex just working out of the box but there are times when some additional insight is required into why a function isn’t performing well. We can do a lot better to help folks help themselves:
* Allow logging of execution time within a function. * Stream logs out of long-running or failed functions. * Log system errors more clearly. * Add support for fine-grained tracing of function execution. * Preemptively notify customers who are getting close to execution limits. * Better local testing and unit testing support. * Publish best practices for monitoring and alerting for customer projects. 📖 Open Source
Finish the swing on open sourcing the core of the Convex backend. The main remaining tasks are extending the CLI to provide better support for managing a local backend. This will be a single-binary version of the Convex backend that is intended for testing and running locally, not the highly durable scale-out distributed system we run customer backends on.
We’re looking forward to getting these changes out! Our community is growing quickly and we think this set of projects will get us well ahead of both the influx of customers and larger projects onboarding to the platform.
26 Replies
Comments and feedback in here 🙂
Good stuff
@james What will the open source backend version include?
It’s a fully functional Convex instance but running on a single node, so none of the distributed systems scale-out stuff, and will just store data locally. The initial release won’t include the dashboard and instead all management will be via the CLI, but we’ll follow up later with a release that includes dashboard too.
We have a release basically ready to go and are just making some final tweaks so expect to see that soon.
@james really nice, when are you planning to release the dashboard? Some weeks after the initial release?
We want to get the initial release out first and then follow up with dashboard support as a separate project. I think it’s likely we can get something usable in a month or so but as it won’t be quite as high priority as some other projects it won’t be in weeks.
It’s not as great a user experience but the CLI includes the requisite functionality to do things like set environment variables, so the gap really is just in terms of usability
@james Oh I see, dashboard is quite important and makes it easier for the developers also it reflects the data nicely
Yep absolutely
In the meantime it is of course included with the hosted product 🙂
Really nice, what will be the used stack?
Most of Convex is custom built in Rust but the persistence layer at the bottom is sqlite. On the actual hosted product it’s a distributed database cluster underneath
We also have a blog post coming out soon (soon @sujayakar? 😛) that will describe how Convex works internally
Really nice 🔥
Thought that everything is based on Typescript but I’ll study Rust then 😂
Yep frontend (client) code is typescript but the backend is all in Rust. Rust has an unfortunately steep learning curve but is a great systems language after that
@james May I ask why did you choose Rust? How is it different from Typescript for example?
Rust is quite a different language and more suited to low-level performance critical applications. There's a much bigger focus on type safety, memory safety, and it compiles directly to machine code rather than running within a runtime, with implications on performance and efficiency.
I wouldn't use Rust for frontend or application code since it's a somewhat difficult language to use and less ergonomic for UI work but it's a more modern alternative to use-cases when one would have otherwise used C or C++.
@james any chance we'll see add\update\delete triggers in the upcoming months?
And another one, Convex Ents vs Convex Components vs plain Convex schema and handcrafted functions — it there a vision about different approaches? will those converge to some stable\recommended\approved\blessed way to utilize the Convex platform?
re triggers i like the idea of something like
ctx.scheduler.runAfterChange(myHandlerFunction, myQuery)
i.e., the ability to schedule a one-time trigger that will fire whenever a query changes
i think this can be a really powerful primitive and something that's very likely for us to implement
re ents, components and vanilla convex, the way i think about it is:
- ents is a layer for folks who prefer that style of syntax and functionality. we'll have more layers over time like this and can move fast on layers that don't change core Convex functionality. it doesn't replace vanilla convex and the authoritative way of using convex will remain the vanilla API.
- components don't really replace vanilla convex. they allow us to build more userspace functionality and to easily drop in libraries that extend convex functionality. there'll be a whole bunch of components that will show up on stack.convex.dev and other places over time, some will be community-developed, but again they just make things simpler rather than change the core of what vanilla convex is. we'll also see layers like ents that leverage components internally
- vanilla convex is the officially supported way of using the platform, is the stuff we always stand behind and promise to support, and our docs will continue to be written in terms of vanilla convexJust two cent on triggers, it would be really cool if they could be code driven rather than activated by calling a function, eg., a
triggers.ts file or something. The presence of the trigger in code enables it, similar to cronjobs.We should chat more about this sometime. I think what you're asking for are latent triggers that get registered in the background and always execute whenever data changes. Something I'm slightly nervous about is situations where developers inadvertently create cycles in the triggers where they just keep triggering each other in an infinite loop, or a trigger on a rapidly updating dataset that gets called very frequently and blows through someone's quota. The nice thing about function-based triggers is that they just run once and then you can decide in that function whether to reschedule it or not.
Of course there are similarities in both models and you can still get in trouble pretty quickly by implementing recursive triggers in either case.
Anyway that's just the main design constraint right now. It would be pretty easy for us to implement any of these features but we want to make them as ergonomic as possible.
Yeah the concern makes sense for sure. Ergonomically, I suspect latent triggers are what folks requesting the feature are generally thinking of. But again, definitely understand the concern around cycles. Maybe a configurable “depth” limit with a low default could help.
I guess, if trigger declarations are tied to schema table(and\or field) declaration, so they could be statically analyzed and not created imperatively, there's a good chance to detect those potential cycles early, maybe even with some crazy typescript magic 🤔
or maybe a new kind of table and or field could be introduced, like generated columns in classical rdbms, but with a wider idea. like these special tables\fields are only allowed to be the sink of some triggers, they cannot have triggers defined on them
we could definitely detect cycles at runtime. runtime isn't ideal though because it's a surprising dev experience
static detection is tricky, especially given that triggers could potentially span code pushes
but we'll put on our thinking caps for this one. it's on our list for next quarter!
Hey @james , where can I find the post?
We're editing it as we speak 🙂 Stay tuned
Really nice, when will it be out? Will you be able to link it?
let's say before end of week. yep can link it
Huge thanks 🙏🏻