Convex Effect bindings & framework(?)
Hi everyone!
I've begun working on an
Effect (https://effect.website) bindings library for Convex, which I see evolving into something of a framework over time. I want to share some of my goals and anticipated features to get folks' feedback.17 Replies
Here are some of the things I'd like do in this project:
1. Wrap the Convex server APIs such that they maximally integrate with the
Effect ecosystem. This will take a few different shapes.
- db queries and mutations (and auth methods, storage methods, etc.) will be made to return the Effect<R, E, A> type rather than a simple Promise<A> (see https://effect.website/docs/effect-vs-promise if you're curious).
- Validated functions will accept a Schema (https://github.com/Effect-TS/schema) rather than a Convex Validator for validating its args (and will additionally accept a second Schema for validating its output). Convex function args will still be validated with Convex Validators, as I will compile a Schema<A> to its corresponding Validator<A>, raising a build-time error if it is not possible to perform the conversion for a given Schema. Args will also be parsed by the provided Schema, though, which will allow you to express a richer set of constraints than you would be able to otherwise (and couple those constraints with branded types, if you wish).
- Convex schema definition will happen via Schema (with compilation to Validators) as well, and the Schema for each field will be used automatically (as it will be integrated directly with the db wrapper) to validate data going into and coming out of the database. This means you will, for example, be able to define a Money schema which stores a serialized dinero.js value (DineroSnapshot<number>) in the database and deserializes it into a Dinero<number> whenever it is retrieved.
- Ctxs will be provided as Effect services (https://effect.website/docs/context-management/services), which I expect will make Convex functions easier to test.
2. Add support for row-level security (a la Lee's https://github.com/get-convex/convex-helpers/blob/main/convex/lib/rowLevelSecurity.ts). This will be relatively easy because I will have already wrapped the db APIs.
3. Add better support for middleware, likely as Effect services and layers (https://effect.website/docs/context-management/layers).
4. Add infrastructure for managing migrations. Since I will have wrapped the Convex schema definition, I could do cool stuff like create the migrations table (maybe if the user opts in to that feature?) and manage it for the user, all automatically!
5. Add support for foreign key constraints.
Since I will have wrapped all of the Convex APIs, it could even be possible to eventually begin supporting a little plugins ecosystem, where plugins can opaquely manage whole tables (or even just particular fields) in the database, and provide their preferred API to the data that they manage.
I have wanted to write something like this for a long time, but was too nervous to build something so close to the contours of the entire Convex API in the pre-1.0 days 😉 But it seems like now might be the right time!
Curious to hear folks' thoughts!
Oh and a small aside--I would find it useful for sake of the db wrapper to have the WithOptionalSystemFields type exposed.in short, this is super cool, and parts of this have been in vague plans internally as layers on top of the 1.0 api ourselves. so I'm pretty excited to see you diving in to creating some of this! I think it's great the more we see the layers on top of the core API being built as OSS by the developer wider ecosystem
from https://news.convex.dev/convex-1-0/:
Expose low-level primitives and higher-level composition. Magical high-level abstractions are great until they suddenly can’t express what you need. Then you’re stuck. So in Convex, the built-in primitives in our platform are quite low-level: they’re just TypeScript functions! Over time, we’re shipping higher-level capabilities like middleware, ORMs, and integrations with 3rd parties as open-source libraries built on these few primitives. That way developers on our platform are never stuck, and can always take apart, rebuild, or customize these higher-level abstractions to fit their specific needs.but I think having these layers developed by the community is even better because they're keeping the work tested by the demands of their actual projects 😄 re:
where plugins can opaquely manage whole tables (or even just particular fields) in the database, and provide their preferred API to the data that they manage.yeah, there's an idea we've tossed around about a "state machine marketplace" where you can have modules of code + workflow + data that drop into place. ideally, eventually you'd want some sort of platform support for this [which doesn't exist yet] so those modules, say, can only touch data in their own namespaced/managed tables and have some sort of transactional calling relationship with you, but they can't direct insert/update etc. but this is probably a little further down the road
Yeah, in general I think it will be incredible when Convex can support something like this.
What do you mean by "transactional calling relationship", exactly?
meaning, let's say there's some interface you have to implement. the nice part is, in a mutation, when it calls you to fulfill the "your table" part of it, you are still in the same transactional window as the module
so it will all commit together or not
but they do not have direct access to your tables
so they can't futz with your stuff
an example of this would be like a stripe module that deals with background jobs that reconcile failed payments, yada yada yada. but all you do is install the module and provide the implementation of the interface (secrets, handlers, etc). the module registers convex jobs, handlers, uses ephemeral tables that you can completely see in your dashboard. but it can only directly manipulate its namespace
you have full "payment system handling" in a box
code + data together
Ahh ok, I think I understand (better, anyways).
What are some examples of ephemeral tables that a Stripe integration might use?
well, let's say how many retry attempts, or last poll time of checking the status of an invoice, or something like that
things it needs to drive its workflow
Would the input to what goes in those tables not be sensitive data which you don't want the third-party integration to see? Or could it be?
it's still in your deployment, so the third party developer can't see it. it's just the plugin can only write to tables
com.stripe..... or whatever in your deployment to manage whatever workflow state machine thing it's operating that attaches to your app and performs some job
Hmm, I actually perhaps don't understand why you would need official platform support to prevent these kinds of "plugins" from only ever touching data that you'd like them to. I can imagine writing something on top of Convex as-is that would ensure that doesn't happen. Or is it more just the idea that official platform support would provide more peace of mind?
yeah, the idea is both for clarity (these tables / jobs / functions are mine [including understanding resources], these are module X, Y, and Zs) and security (I can pull this module forward and it's can't touch "my" records or accidentally collide with another module's tables/records and cause some unintended issues)
basically, the module is actually modular
and not by-convention modular
Hmm ok, cool! I'll be curious to see how close it might be possible to get to that without official platform support, as I get further along!
yeah, for now, I think best to zoom back in to what you're actually doing -- just wanted to make you aware of those idea for the distant future
I think there's a ton of value to be had long before that exists
so very excited about this
Oh yes absolutely, appreciate it though! It's a fun thing to look forward to!
definitely curious to hear what the rest of the team thinks about this though, they'll probably have more helpful feedback on some of your specific points
Hey @RJ , touching on couple of the points here:
- Integrating with Effect is exciting! Effect is getting a lot of traction and definitely has its pros!
- As Jamie mentioned we're also thinking about layers above the DB, enabling RLS, defaults, custom validators and triggers. So we'd love to see how you approach this!
- Even without an additional layer we're also interested in better support for migrations. We know that we can do much better than today.
- WithOptionalSystemFields type - yeah, we should definitely expose this (will try in 1.4)
- I know Effect isn't everyone's cup of tea, but I hope it will still benefit some folks 🙂 I use Effect comprehensively (
fp-ts before that), so it's what I'll benefit the most from building.
- I forgot about triggers! That's one I meant to include, I'd like to explore that as well, particularly for the use-case of caching values. In fact, I would probably explore just that use-case, specifically (at least at first).
- I always envisioned likely migrating (pun not intended) my implementation over to a simple wrapper around whichever of these features you do end up deciding you'd like to build first-class support for, like migrations.
- Thanks! 🙂
I'll also say that I bet this would draw at least some Effect folks over if I advertised it over there in that community.
(though that's not my motivation for building it, I just love both projects!)Agreed, I'm sure a lot of Effect folks would be interested in and benefit from Convex!