David Alonso•2mo ago

Patching convex-helpers package

I'd love to add a new type of Zod Id (similar to Zid but our own custom Fid) to my schema and have it work with zodToConvex. I'm wondering what the easiest approach would be and whether there are any docs on how to add a patch to the helpers package

39 Replies

Convex Bot•2mo ago

Thanks for posting in <#1088161997662724167>. Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets. - Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.) - Use search.convex.dev to search Docs, Stack, and Discord all at once. - Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI. - Avoid tagging staff unless specifically instructed. Thank you!

David AlonsoOP•2mo ago

trying the pnpm patch approach for now after changing the ts files, is there an easy way to recompile the helpers package?

lee•2mo ago

if you have cloned the convex-helpers repo, you can

cd <convex-helpers-repo>/packages/convex-helpers
npm run build
npm pack
cd <your project>
npm install <convex-helpers-repo>/packages/convex-helpers/convex-helpers-0.1.60.tgz

cd <convex-helpers-repo>/packages/convex-helpers
npm run build
npm pack
cd <your project>
npm install <convex-helpers-repo>/packages/convex-helpers/convex-helpers-0.1.60.tgz

i don't know how this interacts with pnpm

David AlonsoOP•2mo ago

didn't manage to get this to work, will try again. We have a monorepo with pnpm which complicates things. are there other resources i could check out on how to best work the convex-helpers / convex package e.g. to add a custom validator or something?

David AlonsoOP•2mo ago

I realize that what i need is actually described here: https://stack.convex.dev/using-branded-types-in-validators except i use zod for schemas and zod's branded string doesn't get preserved in zodToConvex. Is this something that could be quickly added to the helpers package? I guess it'd take Ian a couple of minutes

Using branded types in validators

If you have a more specific type than what you can express with Convex validators, you can still document that at the type level in Convex by casting ...

David AlonsoOP•2mo ago

hmm very confusing, I see ZodBranded is already there, but it doesn't seem to work... if I do:

const test = zodToConvex(z.string().brand("test"));

const test = zodToConvex(z.string().brand("test"));

test is inferred as: VString<string, "required">

David AlonsoOP•2mo ago

This fixes things for me: https://github.com/get-convex/convex-helpers/pull/326

GitHub

Fix zodBranded conversion to Convex validator by davidoort · Pull R...

See https://discord.com/channels/1019350475847499849/1298976854316814336 By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the t...

David AlonsoOP•2mo ago

Still not super clean though, if I do:

export const zFid = <T extends string>(brand: T) =>
  z.string().length(FID_LENGTH).brand<T>();
export const vFid = <T extends string>(brand: T) => zodToConvex(zFid(brand));
export type Fid<T extends string> = Infer<ReturnType<typeof vFid<T>>>;

export const zFid = <T extends string>(brand: T) =>
  z.string().length(FID_LENGTH).brand<T>();
export const vFid = <T extends string>(brand: T) => zodToConvex(zFid(brand));
export type Fid<T extends string> = Infer<ReturnType<typeof vFid<T>>>;

And then something like:

    const parsedId = zFid("collectionViews").parse(args.viewFid); // Will check for length of id

    const parsedId = zFid("collectionViews").parse(args.viewFid); // Will check for length of id

this is of type string & BRAND<"collectionViews"> which is different from string & { _: "collectionViews" } (which comes from the convex validator's branded string)

RJ•2mo ago

I left a comment on your PR @David Alonso—I think you need to use the brand from Zod to get the behavior you're looking for here

David AlonsoOP•2mo ago

i wasn't able to get this to work - is this the only way to successfully build a patch of the package? would be easier to get this merge if it's easy to test these changes

ian•2mo ago

I submitted https://github.com/davidoort/convex-helpers/pull/1 for your review, and published convex-helpers@0.1.64-alpha.0 for you to test with my changes (on top of your changes)

GitHub

use zod brand by ianmacartney · Pull Request #1 · davidoort/convex-...

I'm going to keep the brandedString separate from the zod branding for now, so folks not using zod don't have to install it as a peer dependency. how does this look / work for you?

David AlonsoOP•2mo ago

This works for me! Thanks for pushing a new patch 🙂 The only weird thing I'm still observing is with zCustomFunctions, check this out

David AlonsoOP•2mo ago

No description

David AlonsoOP•2mo ago

while the zodToConvex function actually seems like it works as expected so I'm confused

No description

David AlonsoOP•2mo ago

would really love this to work so that I don't have to manually add a parsing call for every zFid i pass into functions (which happens a lot)

ian•2mo ago

odd. is the type that comes in via args branded? or the value that shows up by the client? I'll try to come back to this soon

David AlonsoOP•2mo ago

not sure what you mean, but this is the definition of zFid:

export const zFid = <TableName extends string>(tableName: TableName) =>
  z
    .string()
    .length(FID_LENGTH)
    .brand<TableName>()
    .refine(
      (value) => {
        if (!TABLE_NAME_HASHES[tableName]) {
          throw new Error(`No hash found for tableName ${tableName}.`);
        }
        // Extract the tableName hash from the FID (first 3 characters)
        const tableNameHash = value.slice(0, tableName_HASH_LENGTH);
        return tableNameHash === TABLE_NAME_HASHES[tableName];
      },
      {
        message: `Invalid tableName for FID. Expected ${tableName}.`,
      }
    );

export const zFid = <TableName extends string>(tableName: TableName) =>
  z
    .string()
    .length(FID_LENGTH)
    .brand<TableName>()
    .refine(
      (value) => {
        if (!TABLE_NAME_HASHES[tableName]) {
          throw new Error(`No hash found for tableName ${tableName}.`);
        }
        // Extract the tableName hash from the FID (first 3 characters)
        const tableNameHash = value.slice(0, tableName_HASH_LENGTH);
        return tableNameHash === TABLE_NAME_HASHES[tableName];
      },
      {
        message: `Invalid tableName for FID. Expected ${tableName}.`,
      }
    );

ian•2mo ago

From my testing this works the same as zod right now: the input is typed as string, but the output is typed as a branded string. So for

export const brandTest = zQuery({
  args: { branded: z.string().brand("foo") },
  handler: async (ctx, args) => {
    const a = args.branded;
    return args.branded;
  }
});

export const brandTest = zQuery({
  args: { branded: z.string().brand("foo") },
  handler: async (ctx, args) => {
    const a = args.branded;
    return args.branded;
  }
});

calling the function asks that branded is string and the type of a and the return type is branded, same as zod. What issues are you seeing?

No description

No description

David AlonsoOP•2mo ago

Ah okay, well what’s not really nice for us is that when we call the query on the client we’d want type warnings there if a raw string is being passed, does that make sense? I thought this would make sense as expected behavior

RJ•2mo ago

Makes sense, that would be really nice IMO, but this isn't something that's supported right now. This would require some kind of additional codegen/transformation step to accomplish. I'm going to look into doing this soon for my library (using Effect Schemas), but for now I think the best you can do (and the best I've been able to do) is: 1. Extract your args and returns Zod validators into their own values/files 2. Manually connect these to the Convex function handlers they correspond to 3. Manually connect the same validators to the correct Convex functions where they're used on the frontend I wrote helpers for accomplishing #3 for Confect, here. You should be able to do basically the same thing with Zod. Actually sorry I may be wrong, maybe you can't do this with Zod—Zod schemas would need to be bidirectional for this to work. If they are, this strategy should work fine. In other words, if you have a Zod schema which converts a string to an FId, you would also need to be able to convert an FId back into a string using the same schema.

ian•2mo ago

I think the current code might also give you what you want when you're using zodToConvex[Fields] as you saw - where technically that's the output type, not input. One thing that I haven't added (but there is an issue for) is providing a function / type to go from zod output types to convex - e.g. if you were to parse using zod, then pass it to Convex / store it in a table. Then the validator for the output would have the branded string type, as well as things like having fields with .default be required / non-optional.

David AlonsoOP•2mo ago

an example would be really useful here! if you think there's a quick way to solve this I'd love to hear! Right now I'm just doing Convex raw queries with argName: zodToConvex(zFid) for proper type checking on the client and then doing zFid(parse) manually inside the query... Can i ask a side question: is there both convex runtime AND zod validation for the args in a zFunction? Asking as it might help explain <#1300865291366174762>

RJ•2mo ago

Is this directed at Ian or me?

David AlonsoOP•2mo ago

Whoever knows the answer hehe

RJ•2mo ago

I don't use Zod so I'm not positive, but I think something like this would work, as long as the runtime representations of the Zod input and output types are the same:

import {
  useAction as useConvexAction,
  useMutation as useConvexMutation,
  useQuery as useConvexQuery,
} from "convex/react";
import type { FunctionReference } from "convex/server";
import z from "zod";

export const useZodQuery =
  <Query extends FunctionReference<"query">, Args, Returns>({
    query,
    args,
    returns,
  }: {
    query: Query;
    args: z.ZodType<Args, any, Query["_args"]>,
    returns: z.ZodType<Returns, any, Query["_returnType"]>;
  }) =>
  (actualArgs: Args): Returns | undefined => {
    const parsedArgs = args.parse(actualArgs);

    const actualReturnsOrUndefined = useConvexQuery(query, parsedArgs as Query["_args"]);

    if (actualReturnsOrUndefined === undefined) {
      return undefined;
    } else {
      const parsedReturns = returns.parse(
        actualReturnsOrUndefined,
      );

      return parsedReturns;
    }
  };

import {
  useAction as useConvexAction,
  useMutation as useConvexMutation,
  useQuery as useConvexQuery,
} from "convex/react";
import type { FunctionReference } from "convex/server";
import z from "zod";

export const useZodQuery =
  <Query extends FunctionReference<"query">, Args, Returns>({
    query,
    args,
    returns,
  }: {
    query: Query;
    args: z.ZodType<Args, any, Query["_args"]>,
    returns: z.ZodType<Returns, any, Query["_returnType"]>;
  }) =>
  (actualArgs: Args): Returns | undefined => {
    const parsedArgs = args.parse(actualArgs);

    const actualReturnsOrUndefined = useConvexQuery(query, parsedArgs as Query["_args"]);

    if (actualReturnsOrUndefined === undefined) {
      return undefined;
    } else {
      const parsedReturns = returns.parse(
        actualReturnsOrUndefined,
      );

      return parsedReturns;
    }
  };

That implementation is probably a bit more verbose than it needs to be And you may need to do more type assertion inside the body But I think the API and functionality is what you're looking for So like I was saying above, you just need to extract your args and returns Zod schemas and pass them in here for the query you're trying to invoke, each time I guess you could also wrap each query in their own hook, too. Anyways, a more full example:

// convex/documents/zodSchemas.ts

export const getDocumentArgs = z.object({ fid: zFid() })
export const getDocumentReturns = z.object({ stuff: z.string() }).brand<"Document">()

// convex/documents/zodSchemas.ts

export const getDocumentArgs = z.object({ fid: zFid() })
export const getDocumentReturns = z.object({ stuff: z.string() }).brand<"Document">()

// convex/documents.ts

export const getDocument =
  // define your zod query, using `getDocumentArgs` and `getDocumentReturns`

// convex/documents.ts

export const getDocument =
  // define your zod query, using `getDocumentArgs` and `getDocumentReturns`

// whatever/component.tsx

// ...
  const fid: Fid = // ...

  const document = useZodQuery(
    api.getDocument,
    getDocumentArgs,
    getDocumentReturns
  )({ fid })
  //   ^ should typecheck arg (and return value) correctly
// ...

// whatever/component.tsx

// ...
  const fid: Fid = // ...

  const document = useZodQuery(
    api.getDocument,
    getDocumentArgs,
    getDocumentReturns
  )({ fid })
  //   ^ should typecheck arg (and return value) correctly
// ...

Hopefully you get the gist of it from this Something like this would probably work

ian•2mo ago

With the patch to your existing brand PR (which is on an alpha release but not merged in currently), you get the branded type out of the argument validator, just not type errors on passing a string to the backend. we could make another zod wrapper utility that results in the input also being a branded string I bet, in a similar way to zId. Also - currently zod custom functions do both convex validation and zod validation. Turning off convex validation would have this impact: 1. Any customFunctions that provide input args (extra arguments that can be consumed by the custom function) would not have the argument validation applied. Function args currently needs to be entirely validated or entirely un-validated (unless you nest other args under something like { etc: v.any() }. 2. Validating Convex v.id would not happen. Currently it relies on Convex to validate that the ID is for the right table. Without making a change to more deeploy validate zId as arguments, an attacker could pass an ID to a different table and if you're doing a blind get or patch they could trick your code into doing the wrong thing. Does that make sense? How do those limitations sound @David Alonso?

RJ•2mo ago

How would this be different from my suggestion above?

ian•2mo ago

This would be a lighter-weight option that would allow using the zCustomQuery / et. al. .. - without a special client wrapper - without splitting out arg/return validator definitions from the function - without specifying the validators on the client It wouldn't provide the features yours would, like eager validation (before it's on the wire), though. It'd just be what we have today + an "input" type that is branded too

RJ•2mo ago

That sounds great, my solution is tedious. I just don't understand how that would be implemented in user-space. Actually looking at https://github.com/get-convex/convex-helpers/compare/main...ian/zod-brand I would think that if it were possible, this would be the way? That is, the proposal as it already stands

ian•2mo ago

yeah it'd probably look like another custom class that's special cased to have a different type, like Zid: - special class that gives zod different input / output type: https://github.com/get-convex/convex-helpers/blob/main/packages/convex-helpers/server/zod.ts#L790-L793 - special casing the type for convex translation: https://github.com/get-convex/convex-helpers/blob/main/packages/convex-helpers/server/zod.ts#L467-L468 It also may be as easy as casting like z.string() as ZodType<string & BRAND<B>, ZodBrandedDef<string>, string & BRAND<B>> . By default the brand looks like z.string() as ZodType<string & BRAND<B>, ZodBrandedDef<string>, string> (note the missing brand on the last param - the input type)

David AlonsoOP•2mo ago

Nice! What version is this? I have to say, for us having the errors is kinda crucial to avoid weird runtime errors (it's very easy to pass around the wrong kind of id to these functions), so I'd be really interested in a zId like wrapper for branded zod strings... that makes sense, v.id validation is what i suspected as an issue, but i was wondering if the code for this is open source and could potentially be exposed so we run it in our zod parse methods or something. We're also doing table-level validation in this way for zFid

ian•2mo ago

you can use await db.normalizeId to validate ids, but it's hard to put in a zod validator, since they don't have access to db when defining them for args, and validators typically aren't async. But it'd be possible to make something like await validateIds(ctx.db, zodToConvex(zodSchema), data); that walks through the convex validator tree and validates any IDs The same one as before: convex-helpers@0.1.64-alpha.0 https://discord.com/channels/1019350475847499849/1298976854316814336/1300636690917888032

David AlonsoOP•2mo ago

i wonder how expensive this operation is for a schema like ours...

ian•2mo ago

It'd be a function of how many ID validators there are - and specifically how many different tables they reference. I suspect it'd be pretty fast, but you could benchmark. At that point I guess the zod-flavored custom functions could just opt into doing their own argument & return validation. It would lose out on things like the OpenAPI generation / other language bindings (which read validators to produce type signatures) but could be an interesting option if you're all-in on zod

RJ•2mo ago

Ah I see, and this is where it happens: https://github.com/get-convex/convex-helpers/blob/dc26186ced81a0c7e39020f6dc69e804536f81b9/packages/convex-helpers/server/zod.ts#L429 Yeah that would be an amazing feature to support!

ian•2mo ago

which one? 1. branded string that has a branded input type 2. validateIds(ctx.db, validator) 3. zCustomQuery(query, ..., { skipConvexValidators: true })

RJ•2mo ago

I meant 1, though I imagine all could be valuable

ian•2mo ago

Ok I added zBrand to convex-helpers@0.1.66-alpha.0 (which also includes the other zod branding improvements). You can use it like zBrand(z.string(), "mybrand") and it brands the input & output. @David Alonso if it works for you I'll ship it in v 0.1.66

David AlonsoOP•2mo ago

You're the man!! This now allows me to use zFunctions pretty much everywhere and to the custom id parsing implicitly which is great I'm not directly using zBrand but ZodBrandedInputAndOutput like so:

export const zFid = <TableName extends string>(tableName: TableName) =>
  z
    .string()
    .length(FID_LENGTH)
    .brand<TableName>()
    .refine(
      (value) => {
        if (!TABLE_NAME_HASHES[tableName]) {
          throw new Error(`No hash found for tableName ${tableName}.`);
        }
        // Extract the tableName hash from the FID (first 3 characters)
        const tableNameHash = value.slice(0, tableName_HASH_LENGTH);
        return tableNameHash === TABLE_NAME_HASHES[tableName];
      },
      {
        message: `Invalid tableName for FID. Expected ${tableName}.`,
      }
    ) as unknown as ZodBrandedInputAndOutput<z.ZodString, TableName>;

export const zFid = <TableName extends string>(tableName: TableName) =>
  z
    .string()
    .length(FID_LENGTH)
    .brand<TableName>()
    .refine(
      (value) => {
        if (!TABLE_NAME_HASHES[tableName]) {
          throw new Error(`No hash found for tableName ${tableName}.`);
        }
        // Extract the tableName hash from the FID (first 3 characters)
        const tableNameHash = value.slice(0, tableName_HASH_LENGTH);
        return tableNameHash === TABLE_NAME_HASHES[tableName];
      },
      {
        message: `Invalid tableName for FID. Expected ${tableName}.`,
      }
    ) as unknown as ZodBrandedInputAndOutput<z.ZodString, TableName>;

Let me know if there's a better way!

Convex Community Join

Join the Convex Discord! Explore Convex, the fullstack TypeScript platform for developers and startup founders.

6.2KMembers

View on Discord