RJ•2y ago

Upgrading "middleware" functions to Convex 0.13

I have some "middleware" functions (à la https://stack.convex.dev/sessions-wrappers-as-middleware) which I'm trying to upgrade to be compatible with convex 0.13, and am having trouble figuring out how to properly type (and, consequently, modify) them accordingly. Are there any examples of functions like this that are up-to-date with convex 0.13?

26 Replies

jamwt•2y ago

@ian actually just updated all those middleware projects, so this is fresh on his mind!

ian•2y ago

It's an active project actually. Still figuring out the types I have a 0.13 branch on convex-helpers that I'm iterating on Sorry for lagging the release

RJOP•2y ago

I suppose one issue I'm running into is that, with the new validated arguments API, instead of having a function like this which I expect as a parameter to my Convex function:

func: (ctx: ActionCtx, ...args: Args)

func: (ctx: ActionCtx, ...args: Args)

I have something like this:

validatedFunction: ValidatedFunction<ActionCtx, any, Output>

validatedFunction: ValidatedFunction<ActionCtx, any, Output>

(I'm still not sure what should go in the second type parameter for ValidatedFunction) So whereas before the body of the middleware function would look something like this:

// do some stuff
func(ctx, ...args)

// do some stuff
func(ctx, ...args)

Now I have to figure out how to invoke the function with the argument validators. Hmm ok. Any tips on the above?

ian•2y ago

One thing that simplifies things is if you can put your wrapper inside the query/mutation. e.g.:

export default mutation({
  args: {...},
  handler: withUser(async ({user}) => {
    ...
  })
});

export default mutation({
  args: {...},
  handler: withUser(async ({user}) => {
    ...
  })
});

RJOP•2y ago

Ah of course, makes sense

ian•2y ago

For something like sessions, passing an extra argument means either putting that argument in args (annoying), or doing a fancier thing to wrap mutation / query

RJOP•2y ago

Hmm right Ok, not so easy as I was hoping I think I'll likely put this off until you come up with a solution in those examples then, @ian. Would love it if you wouldn't mind pinging me when you do! Or if you open a PR for it, I could also just subscribe to that.

ian•2y ago

Sounds good, I'll update you. Sorry for the delay

RJOP•2y ago

No worries, thank you!

ian•2y ago

I've gotten them working and pushed to the 0.13.0 branch. There are a few types for withSession that we didn't export that I find useful, so I inlined them at the bottom. withUser is a simple example of doing wrappers like mutation({ handler: withUser(fn) }) . withSession is a more complicated example that patches your args and injects into the function. I wrote it to support all three styles: mutation(withSession(fn)), mutation(withSession({handler: fn})) mutation(withSession({args, handler: fn})). I also updated the queryWithSession and mutationWithSession types - which is an example of how to type the response from query / mutation. https://github.com/get-convex/convex-helpers/tree/0.13.0/convex

GitHub

convex-helpers/convex at 0.13.0 · get-convex/convex-helpers

A collection of useful code to complement the official packages. - convex-helpers/convex at 0.13.0 · get-convex/convex-helpers

ian•2y ago

I'll push them to main once the types are exposed (if we can get a patch release out)

RJOP•2y ago

@ian why are there so many anys in these withSession examples? I'm using that code as reference and am still failing to get the code to typecheck, unless I liberally add anys like you did there, but that doesn't give me much confidence that any of it will work, and doesn't bode well for maintainability in general.

ian•2y ago

Yeah it's a good tradeoff to call out. I went for the "do all the fancy types in the overrides, and "just trust me" in the body, so you can get all the types and autocomplete when using it as a library, without having to bend over backwards to get typescript to agree that the arguments no longer has a sessionId, and ctx now has a session. I'm considering changing more of the helpers to a .js and .d.ts file structure, so you can use them in either js or ts projects without modifying them. The downsides, as you mention, are extensibility and confidence. When you split out the types from the implementation, someone modifying the function isn't as confident it'll still work. Unfortunately, with the change to named args which may not exist if there are none, and also it possibly being an object which may or may not have args, explaining that to TypeScript felt like it would be so complex, it would be hard for someone to extend anyways. One thing that would help is if we exposed the right types and hooks so you could write middleware with simple types, and have the crazy convoluted typescript types in one place. It's definitely in this gray area between a user-owned, user-maintained thing, and a library which hides the mess and doesn't let you modify it. Open to ideas too. @alexcole and I paired on it for a while and agree it's non-ideal. Thankfully ones like withUser are easier since they don't rely on arguments and can just wrap the inner function.

alexcole•2y ago

Yeah, with the addition of argument validators, the types for middleware have definitely gotten more complicated. The anys should all be constrained to the implementation and not the external interface so you're not losing much type safety. And this pattern of separating the interface with fancy types from the implementation with simple ones is IMO a good one. If you're doing complex enough stuff it's too hard to do otherwise. Lmk if you have questions or want to pair on your middleware! And yeah I think splitting this into .d.ts and .ts files would make this all more clear

RJOP•2y ago

I dug into the types in src/values/validator.ts and src/values/value.ts and noticed some things: - Validator looks like it could be more constrained, right now it looks like

export declare class Validator<
  TypeScriptType,
  IsOptional extends boolean = false,
  FieldPaths extends string = never
> { ... }

export declare class Validator<
  TypeScriptType,
  IsOptional extends boolean = false,
  FieldPaths extends string = never
> { ... }

but couldn't it instead be

export declare class Validator<
- TypeScriptType,
+ TypeScriptType extends Value,
  IsOptional extends boolean = false,
  FieldPaths extends string = never
> { ... }

export declare class Validator<
- TypeScriptType,
+ TypeScriptType extends Value,
  IsOptional extends boolean = false,
  FieldPaths extends string = never
> { ... }

? - There are a lot of anys in validator.ts also—such as:

export type PropertyValidators = Record<string, Validator<any, any, any>>;

export type PropertyValidators = Record<string, Validator<any, any, any>>;

Could this instead be:

export type PropertyValidators = Record<string, Validator<Value, boolean, string>>;

export type PropertyValidators = Record<string, Validator<Value, boolean, string>>;

or maybe even:

export type PropertyValidators = Record<string, Validator<unknown>>;

export type PropertyValidators = Record<string, Validator<unknown>>;

? In general, is it ever necessary to use any as an argument to the Validator type? --- The reason I'm asking is because my gut (which certainly could be wrong) says that these types are not so complex that TypeScript should be totally failing here, and it does appear to be failing somewhere having to do with the PropertyValidators. It would be a real bummer IMO if what began as a happily simple and straightforward concept (Convex functions are just JS functions, so Convex "middleware" is just higher-order functions!) became an untypable mystery box (Convex middleware is higher-order functions sort of but you can't really write them yourself because they can't be written in a type-safe way). The fact that zod and similar libraries have the ability to create a schema which is the intersection of two schemas (validators in Convex lingo) (https://github.com/colinhacks/zod#extend) is also what makes me think that this should be possible to manage somehow with improved or altered typings (since that's basically what we're trying to do in these middleware functions) I think that in the worst-case, where this can't be typed properly for some reason, this would be pretty necessary to reasonably claim that "Convex supports middleware for functions with validated arguments"

alexcole•2y ago

Thanks for the thoughts! I'm not sure that any of this changes are too high priority given that they don't change the interface of our npm package. In general, any can be a cancer that spreads throughout a TS codebase. That being said, all of these examples are using any in internal types that aren't exposed in the Convex interface. Counterintuitively, when writing complex TS types, I'm a big fan of separating interfaces and implementation and only using the fancy types on the interfaces. Our main goal is to give developers great type safety in their apps. We're really serious about this and do things like add type-level tests to make sure they work (for example https://github.com/get-convex/convex-js/blob/main/src/api/api.test.ts). It's nice when the types can also catch bugs that Convex devs make, but thats a secondary goal. In a lot of these cases, convincing TS that all of our implementation matches these complex types is difficult and would slow down our work. Sometimes it's impossible (as @ian and I found with some middleware). To both move fast on the impls and have correct types we often split them up entirely. Ex https://github.com/get-convex/convex-js/blob/main/src/server/query.ts and https://github.com/get-convex/convex-js/blob/main/src/server/impl/query_impl.ts We do this in codegen too with the .d.ts vs .js files. TypeScript won't totally catch if they don't match but thats okay. We can test for this in other ways. No doubt! And it is possible! But if you read the Zod source, they are doing the exact same thing. I see over 300 anys in https://github.com/colinhacks/zod/blob/master/src/types.ts But back to your middleware - It's definitely possible to write it given the new validator types, but I wouldn't stress over removing every any from your implementation if it isn't exposed in the interface. Happy to hop on a call and help you figure out how to do this for your project.

RJOP•2y ago

I want to make sure I'm being clear about my intent here, as I think there are a few different issues at play and I'm not sure that what I'm trying to convey is quite coming across. First of all, the reason I dug into those internal types is not because I'm trying to critique your internal development philosophy, or ask you to refactor internal details of your code that don't affect any end users, but because I was wondering if the typings of e.g. PropertyValidators might be the cause of the typing issues that I (an end user) am experiencing . I'm not sure exactly what you mean by "all of these examples are using any in internal types that aren't exposed in the Convex interface", because PropertyValidators uses a number of anys in its definition and is exported (and indeed currently is required to type these middleware functions properly). Maybe there is a meaningful line to be drawn (e.g. "we consider use of any in our packages public only if we expose functions which take or return values labeled as any, and all other uses of any are internal"), I haven't thought about it enough to have a good sense myself. But that's why I was exploring PropertyValidators and the type it parameterizes with any (Validator)—the typing issues I was experiencing in my code as a user seemed to originate there. Second but more importantly, the real reason I'm digging into this at all is that I'm trying to upgrade the Convex client and I'm finding that upgrading the few "middleware"-style functions that I have to be incredibly difficult and much more time-consuming than I'd like. I can't just copy and paste the withSession function that you all have kindly rewritten to be correct, as my middleware is doing different things than withSession is, and so I have to engage in the process of trying to refactor this without being able to really employ the type system, using these types (like PropertyValidators) that I'm not familiar with and didn't write. As I'm sure you can infer, I think the type theory stuff is pretty interesting, but in this instance I really just want my code here to be working, bug-free, and then to move on to more important things. I'm also just trying to be helpful by suggesting things (which may be wrong or not aligned with how y'all want to design your software, or which may be misaligned with other requirements y'all have which I'm not aware of—and that's fine!).

alexcole•2y ago

Wanna hop on a call and get your middleware fixed up? Seeing how yours works might inform what helpers I should put in the npm package to make this easier for other devs. But I think that the anys you found are unrelated to the difficulty updating the types. The any we needed in withSession was because TypeScript wasn't simplifying a type automatically anymore

RJOP•2y ago

Sure, I'll be available in ~10 mins if that works for you

alexcole•2y ago

Oops. I'm a meeting. Around this afternoon?

RJOP•2y ago

Ah sorry, I misunderstood. Yeah I'm free all afternoon, but I do have to end my day right around 5pm Eastern time

The Aleks•16mo ago

Where did you end on this @RJ and @alexcole? I also find myself missing an "intersection" helper

ballingt•16mo ago

@The Aleks can you say more, do you want an intersection helper for validators and what's the use case leading you to want it?

The Aleks•16mo ago

I'd like to re-use some of my types defined in my schema in some of my create and update mutations - having intersection types would allow me to break up the types so I can re-use without overexposing fields that shouldn't be changeable through the api.

v.intersection(
   v.object({created by: v.id("accounts")}),
  // Changeable fields
  v.object({...rest})
)

v.intersection(
   v.object({created by: v.id("accounts")}),
  // Changeable fields
  v.object({...rest})
)

ballingt•16mo ago

Thanks, that makes sense. Ideally we can make v.object({created by: v.id("accounts"), ...rest}) work, this works today if rest is

const rest = {
  a: v.string(),
  b: v.string(),
};

const rest = {
  a: v.string(),
  b: v.string(),
};

but would be cool if you could do it with a v.object({ ... })

The Aleks•16mo ago

Oh that would be amazing!

Upgrading "middleware" functions to Convex 0.13

Did you find this page helpful?