RJ
RJ•17mo ago

Correlating function references with `Schema` args validators

I've made some good progress on this (https://discord.com/channels/1019350475847499849/1019350478817079338/1157131989825093772), but have recently run into an unfortunate roadblock and am looking for thoughts and suggestions!
4 Replies
RJ
RJOP•17mo ago
Progress So far I've written a bunch of (TypeScript) functions which wrap Convex function definition (TypeScript) functions, which look like the following:
const query = <
DatabaseValue extends DefaultFunctionArgs,
TypeScriptValue,
Output,
>({
args,
handler,
}: {
args: Schema.Schema<DatabaseValue, TypeScriptValue>;
handler: (
ctx: EffectQueryCtx<DataModel>,
a: TypeScriptValue
) => Effect.Effect<never, never, Output>;
}): RegisteredQuery<"public", DatabaseValue, Promise<Output>>;
const query = <
DatabaseValue extends DefaultFunctionArgs,
TypeScriptValue,
Output,
>({
args,
handler,
}: {
args: Schema.Schema<DatabaseValue, TypeScriptValue>;
handler: (
ctx: EffectQueryCtx<DataModel>,
a: TypeScriptValue
) => Effect.Effect<never, never, Output>;
}): RegisteredQuery<"public", DatabaseValue, Promise<Output>>;
These are cool and work--specifically, they accept an @effect/schema Schema and use it to decode the Convex function's arguments (DatabaseValue) into a (possibly richer) value of type TypeScriptValue, producing (as you can see above) a RegisteredQuery whose input is the DatabaseValue. And they accept a handler which expects the decoded TypeScriptValue and returns an Effect. But... The trouble arises when you want to invoke the Convex functon. See, @effect/schema is extra cool compared to a library like zod because it supports both serialization and deserialization. A Schema<From, To> can Schema.parse an unknown into a To, but it can also decode a From to a To, or encode a To to a From. This means that I can use a single Schema to decode a DatabaseValue into a TypeScriptValue (useful in the Convex function handler), and also to encode a TypeScriptValue into a DatabaseValue (useful at the Convex function's callsite). The advantage here is that, after defining the Schema<DatabaseValue, TypeScriptValue> once, a user can always interact with the richer TypeScriptValue both in the context of their calling code and in their Convex function handler. You might see how this same feature would be useful when using @effect/schema to define fields in schema.ts. In order to take advantage of this, I need to be able to match each Convex function with its corresponding Schema. However, to invoke a Convex function, you need a function reference, but these function references (whether a FunctionReference or a string in the old format, like myDir/myMod:myFun) are defined by the path of the function in the file system, which I don't have access to without executing some shell comands which reach out to the filesystem, like the Convex CLI does. Imagine Imagine a fantasy world in which function references were defined within the runtime, without regard for where the code is in the project's directory structure. It might look something like this:
// convex/api.ts

import { defineApi } from "convex/server";

import myNamespace1 from "./convex/myNamespace1";
import myNamespace2 from "./convex/myNamespace2";

// const defineApi: (convexApi: ConvexApi) => ConvexApiDefinition;
export default defineApi({
myNamespace1: {
myFunction: myNamespace1.myFunction,
},
myNamespace2: {
myFunction: myNamespace2.myFunction,
},
});
// convex/api.ts

import { defineApi } from "convex/server";

import myNamespace1 from "./convex/myNamespace1";
import myNamespace2 from "./convex/myNamespace2";

// const defineApi: (convexApi: ConvexApi) => ConvexApiDefinition;
export default defineApi({
myNamespace1: {
myFunction: myNamespace1.myFunction,
},
myNamespace2: {
myFunction: myNamespace2.myFunction,
},
});
In this world, it would be easy to write a wrapper for Convex function invocation which both 1. Knows how to reference the appropriate Convex function, and 2. Knows which Schema corresponds with that function It would look something like this:
// This would actually describe not just queries, but any function
type EffectConvexFunction = {
args: Schema<DatabaseValue, TypeScriptValue>,
handler: (
ctx: EffectQueryCtx<DataModel>,
a: TypeScriptValue
) => Effect.Effect<never, never, Output>;
}

// I'd have to think more about how this type would look, but it should
// basically be the same as `ConvexApi`, except its leaves would be of type
// `EffectConvexFunction` rather than `RegisteredQuery | RegisteredMutation |
// RegisteredAction`.
type EffectConvexApi = // ...

const defineEffectApi: (effectConvexApi: EffectConvexApi) => EffectConvexApiDefinition;

const toConvexApiDefinition: (effectConvexApiDefinition: EffectConvexApiDefinition) => ConvexApiDefinition;

const toEffectActionCtx: (effectConvexApiDefinition: EffectConvexApiDefinition) => EffectActionCtx<DataModel>;

const toEffectReact = (effectConvexApiDefinition: EffectConvexApiDefinition) =>
{
// A wrapped `useQuery` which, for any given Convex functon (which we can
// reference properly because that information is contained in
// `effectConvexApiDefinition`), expects `TypeScriptValue` and uses `Schema`
// to encode `TypeScriptValue` into the correct `DatabaseValue`.
useQuery: // ...
// ...
}
// This would actually describe not just queries, but any function
type EffectConvexFunction = {
args: Schema<DatabaseValue, TypeScriptValue>,
handler: (
ctx: EffectQueryCtx<DataModel>,
a: TypeScriptValue
) => Effect.Effect<never, never, Output>;
}

// I'd have to think more about how this type would look, but it should
// basically be the same as `ConvexApi`, except its leaves would be of type
// `EffectConvexFunction` rather than `RegisteredQuery | RegisteredMutation |
// RegisteredAction`.
type EffectConvexApi = // ...

const defineEffectApi: (effectConvexApi: EffectConvexApi) => EffectConvexApiDefinition;

const toConvexApiDefinition: (effectConvexApiDefinition: EffectConvexApiDefinition) => ConvexApiDefinition;

const toEffectActionCtx: (effectConvexApiDefinition: EffectConvexApiDefinition) => EffectActionCtx<DataModel>;

const toEffectReact = (effectConvexApiDefinition: EffectConvexApiDefinition) =>
{
// A wrapped `useQuery` which, for any given Convex functon (which we can
// reference properly because that information is contained in
// `effectConvexApiDefinition`), expects `TypeScriptValue` and uses `Schema`
// to encode `TypeScriptValue` into the correct `DatabaseValue`.
useQuery: // ...
// ...
}
I think there are ways to sort of hack around this limitation, but they're all pretty ugly. Very curious for thoughts and/or suggestions! I would love to hear that I've missed something and that there's a good path forward as-is 🙂 And of course please let me know if the problem is still unclear, or you'd like me to elaborate further on anything.
ballingt
ballingt•17mo ago
Excited about your exploration here, RJ. I might be missing sone of your points, would love to hear more regardless. A more explicit interface like defineApi is something we've talked about, especially because it's more amenable to wrapping like this. There are some code-splitting issues (now you have to bundle everything together instead of per-endpoint) but fundamentally it's a great idea. Slightly different codegen could help here too, right now only arguments and return types reach to frontend but if there were a place to put types on a Convex function that we passed through to the client you could use them there. Let me think about ways you might be able to do this today for a bit. Really appreciate the ideas, we absolutely intend for packages like this to be possible in Convex so interesting to see how far away we are today.
RJ
RJOP•17mo ago
Thanks for your thoughts Tom!
There are some code-splitting issues (now you have to bundle everything together instead of per-endpoint)
Hmm yeah good point, I hadn't considered that.
Slightly different codegen could help here too, right now only arguments and return types reach to frontend but if there were a place to put types on a Convex function that we passed through to the client you could use them there.
The tricky thing here is that just having access to the TypeScriptValue type in the generated API isn't sufficient for my needs, for two reasons: 1. I actually need a Schema value in hand on the frontend in order to perform the encoding and call the function with the correct DatabaseValue, and 2. I really need the specific Schema<DatabaseValue, TypeScriptValue> value that is being used to decode the DatabaseValue in that Convex function's handler. If in each Convex function reference I had access to the exact type of the required Schema I could then ask the user to provide it manually. This would be inconvenient but provide some confidence that the TypeScriptValue provided will be encoded properly. Unfortunately it would not provide certainty that the encoding is correct, because it's possible to define two different Schemas whose types are equal but which function differently. A trivial example:
// Same types, different implementations. No string exists which is decodable by both `Schema`s.
const a: Schema.Schema<string, string> = Schema.string.pipe(Schema.maxLength(5))
const b: Schema.Schema<string, string> = Schema.string.pipe(Schema.minLength(6))

// Same types, different implementations. No string exists which is decodable by both `Schema`s.
const a: Schema.Schema<string, string> = Schema.string.pipe(Schema.maxLength(5))
const b: Schema.Schema<string, string> = Schema.string.pipe(Schema.minLength(6))

So really I would need a way to pass through values to the client through the codegen. I'd have to think about it a bit more, but if that were possible that could end up working out reasonably well. The best idea I've had so far for how to do this today is to simulate a defineApi-like scenario, by creating a "wrapper", perhaps something like:
const defineConvexApi: (api: Record<string, EffectConvexFunction>) => EffectConvexApi;

const toConvexApi: (effectConvexApi: EffectConvexApi) => Record<string, RegisteredQuery | RegisteredMutation | RegisteredAction>;

const toEffectReact = (effectConvexApi: EffectConvexApi) =>
{
useQuery: // ...
useMutation: // ...
useAction: // ...
};
const defineConvexApi: (api: Record<string, EffectConvexFunction>) => EffectConvexApi;

const toConvexApi: (effectConvexApi: EffectConvexApi) => Record<string, RegisteredQuery | RegisteredMutation | RegisteredAction>;

const toEffectReact = (effectConvexApi: EffectConvexApi) =>
{
useQuery: // ...
useMutation: // ...
useAction: // ...
};
Which is then used (server-side) like so:
// convex/api.ts

const effectConvexApi = defineConvexApi({
myNamespace1_myFunction: // ...
myNamespace2_myFunction: // ...
});

export toConvexApi(effectConvexApi);
// convex/api.ts

const effectConvexApi = defineConvexApi({
myNamespace1_myFunction: // ...
myNamespace2_myFunction: // ...
});

export toConvexApi(effectConvexApi);
This way, the only thing I would need to know in my implementation of toEffectReact, in order to be able to properly reference the functions using Convex's useQuery etc. and generated function references, is the name of the file in which the export toConvexApi(effectConvexApi) takes place. This could always be api--required by convention--or defineConvexApi could be parameterized by the file's name/path within the convex directory. (as an aside, if it's ever interesting enough to hop on a call to discuss/explain further, I'd be more than happy to)
ballingt
ballingt•17mo ago
Let's chat this week! One way to do this could be more significant codegen, which is something we'll need for Python, Rust, etc. clients: once we have output validators we'll have type information we'll want to share with clients that can't use our current very light TypeScript approach. Some JavaScript clients can use this too. First we'd probably do types for our first-party Python client, but this be an interface you could hook into. At that point API metadata should be available (input! output! docstring!) to code genererators and arbitrary metadata could be tacked on too. This is all without changing how function registration works. But once we're doing this in-depth type generation it's reasonable for the (currently server-only) analyzer to be in the loop, at which point dynamic registration is back on the table.

Did you find this page helpful?