Correlating function references with `Schema` args validators
I've made some good progress on this (https://discord.com/channels/1019350475847499849/1019350478817079338/1157131989825093772), but have recently run into an unfortunate roadblock and am looking for thoughts and suggestions!
4 Replies
Progress
So far I've written a bunch of (TypeScript) functions which wrap Convex function definition (TypeScript) functions, which look like the following:
These are cool and work--specifically, they accept an
@effect/schema
Schema
and use it to decode the Convex function's arguments (DatabaseValue
) into a (possibly richer) value of type TypeScriptValue
, producing (as you can see above) a RegisteredQuery
whose input is the DatabaseValue
. And they accept a handler which expects the decoded TypeScriptValue
and returns an Effect
.
But...
The trouble arises when you want to invoke the Convex functon. See, @effect/schema
is extra cool compared to a library like zod
because it supports both serialization and deserialization. A Schema<From, To>
can Schema.parse
an unknown
into a To
, but it can also decode
a From
to a To
, or encode
a To
to a From
. This means that I can use a single Schema
to decode
a DatabaseValue
into a TypeScriptValue
(useful in the Convex function handler), and also to encode
a TypeScriptValue
into a DatabaseValue
(useful at the Convex function's callsite). The advantage here is that, after defining the Schema<DatabaseValue, TypeScriptValue>
once, a user can always interact with the richer TypeScriptValue
both in the context of their calling code and in their Convex function handler. You might see how this same feature would be useful when using @effect/schema
to define fields in schema.ts
.
In order to take advantage of this, I need to be able to match each Convex function with its corresponding Schema
. However, to invoke a Convex function, you need a function reference, but these function references (whether a FunctionReference
or a string in the old format, like myDir/myMod:myFun
) are defined by the path of the function in the file system, which I don't have access to without executing some shell comands which reach out to the filesystem, like the Convex CLI does.
Imagine
Imagine a fantasy world in which function references were defined within the runtime, without regard for where the code is in the project's directory structure. It might look something like this:
In this world, it would be easy to write a wrapper for Convex function invocation which both
1. Knows how to reference the appropriate Convex function, and
2. Knows which Schema
corresponds with that function
It would look something like this:
I think there are ways to sort of hack around this limitation, but they're all pretty ugly.
Very curious for thoughts and/or suggestions! I would love to hear that I've missed something and that there's a good path forward as-is 🙂
And of course please let me know if the problem is still unclear, or you'd like me to elaborate further on anything.Excited about your exploration here, RJ. I might be missing sone of your points, would love to hear more regardless.
A more explicit interface like defineApi is something we've talked about, especially because it's more amenable to wrapping like this. There are some code-splitting issues (now you have to bundle everything together instead of per-endpoint) but fundamentally it's a great idea.
Slightly different codegen could help here too, right now only arguments and return types reach to frontend but if there were a place to put types on a Convex function that we passed through to the client you could use them there.
Let me think about ways you might be able to do this today for a bit. Really appreciate the ideas, we absolutely intend for packages like this to be possible in Convex so interesting to see how far away we are today.
Thanks for your thoughts Tom!
There are some code-splitting issues (now you have to bundle everything together instead of per-endpoint)Hmm yeah good point, I hadn't considered that.
Slightly different codegen could help here too, right now only arguments and return types reach to frontend but if there were a place to put types on a Convex function that we passed through to the client you could use them there.The tricky thing here is that just having access to the
TypeScriptValue
type in the generated API isn't sufficient for my needs, for two reasons:
1. I actually need a Schema
value in hand on the frontend in order to perform the encoding and call the function with the correct DatabaseValue
, and
2. I really need the specific Schema<DatabaseValue, TypeScriptValue>
value that is being used to decode the DatabaseValue
in that Convex function's handler. If in each Convex function reference I had access to the exact type of the required Schema
I could then ask the user to provide it manually. This would be inconvenient but provide some confidence that the TypeScriptValue
provided will be encoded properly. Unfortunately it would not provide certainty that the encoding is correct, because it's possible to define two different Schema
s whose types are equal but which function differently. A trivial example:
So really I would need a way to pass through values to the client through the codegen. I'd have to think about it a bit more, but if that were possible that could end up working out reasonably well.
The best idea I've had so far for how to do this today is to simulate a defineApi
-like scenario, by creating a "wrapper", perhaps something like:
Which is then used (server-side) like so:
This way, the only thing I would need to know in my implementation of toEffectReact
, in order to be able to properly reference the functions using Convex's useQuery
etc. and generated function references, is the name of the file in which the export toConvexApi(effectConvexApi)
takes place. This could always be api
--required by convention--or defineConvexApi
could be parameterized by the file's name/path within the convex
directory.
(as an aside, if it's ever interesting enough to hop on a call to discuss/explain further, I'd be more than happy to)Let's chat this week! One way to do this could be more significant codegen, which is something we'll need for Python, Rust, etc. clients: once we have output validators we'll have type information we'll want to share with clients that can't use our current very light TypeScript approach. Some JavaScript clients can use this too. First we'd probably do types for our first-party Python client, but this be an interface you could hook into.
At that point API metadata should be available (input! output! docstring!) to code genererators and arbitrary metadata could be tacked on too. This is all without changing how function registration works.
But once we're doing this in-depth type generation it's reasonable for the (currently server-only) analyzer to be in the loop, at which point dynamic registration is back on the table.