oscklm•6mo ago

Working with return validator and a query that enriches data

Hello, In our project, we use relational tables and in a query like the one below, where we wanna enrich video with its related data. We wanna use this new return validation, but find it a bit hard to get the returned object to actually match the validator we define. I'd love to ask for some advice on how to best approach this, as we are looking to do it quite a lot with other pieces of data in our app. Return validator:

export const videoEnrichedValidator = v.object({
  _id: v.id('video'),
  _creationTime: v.number(),
  ...videoValidator.fields,
  enrichments: v.object({
    activity: v.union(v.null(), activityValidator),
    category: v.union(v.null(), categoryValidator),
    thumbnails: v.union(
      v.null(),
      v.object({
        small: v.union(v.string(), v.null()),
        medium: v.union(v.string(), v.null()),
        large: v.union(v.string(), v.null()),
      })
    ),
    muxAsset: muxAssetValidator,
  }),
});

export const videoEnrichedValidator = v.object({
  _id: v.id('video'),
  _creationTime: v.number(),
  ...videoValidator.fields,
  enrichments: v.object({
    activity: v.union(v.null(), activityValidator),
    category: v.union(v.null(), categoryValidator),
    thumbnails: v.union(
      v.null(),
      v.object({
        small: v.union(v.string(), v.null()),
        medium: v.union(v.string(), v.null()),
        large: v.union(v.string(), v.null()),
      })
    ),
    muxAsset: muxAssetValidator,
  }),
});

9 Replies

oscklmOP•6mo ago

getOneEnriched query function:

export const getOneEnriched = query({
  args: {
    id: v.string(),
  },
  returns: videoEnrichedValidator,
  handler: async (ctx, { id }) => {
    const videoId = ctx.db.normalizeId('video', id);

    if (!videoId) {
      throw new Error(`Video could not be found with id ${id}`);
    }

    // Get the video
    const video = await ctx.db.get(videoId);

    // Get related image & thumbnails
    const imageRelationId = await ctx.db
      .query('videoImageRelation')
      .withIndex('videoId', (q) => q.eq('videoId', videoId))
      .unique();
    const thumbnails =
      imageRelationId && (await getThumbnailsFromImageId(ctx, imageRelationId.imageId));

    // Get related muxAsset
    const muxAssetRelation = await ctx.db
      .query('videoMuxAssetRelation')
      .withIndex('videoId', (q) => q.eq('videoId', videoId))
      .unique();

    const muxAsset = muxAssetRelation?.muxAssetId
      ? await ctx.db.get(muxAssetRelation.muxAssetId)
      : null;

    // Get related category
    const categoryRelation = await ctx.db
      .query('videoCategoryRelation')
      .withIndex('videoId', (q) => q.eq('videoId', videoId))
      .unique();
    const category = categoryRelation && (await ctx.db.get(categoryRelation.categoryId));

    // Get related activity
    const activity = await ctx.db
      .query('activity')
      .withIndex('videoId', (q) => q.eq('videoId', videoId))
      .unique();

    return {
      ...video,
      enrichments: {
        category,
        activity,
        thumbnails,
        muxAsset,
      },
    };
  },
});

export const getOneEnriched = query({
  args: {
    id: v.string(),
  },
  returns: videoEnrichedValidator,
  handler: async (ctx, { id }) => {
    const videoId = ctx.db.normalizeId('video', id);

    if (!videoId) {
      throw new Error(`Video could not be found with id ${id}`);
    }

    // Get the video
    const video = await ctx.db.get(videoId);

    // Get related image & thumbnails
    const imageRelationId = await ctx.db
      .query('videoImageRelation')
      .withIndex('videoId', (q) => q.eq('videoId', videoId))
      .unique();
    const thumbnails =
      imageRelationId && (await getThumbnailsFromImageId(ctx, imageRelationId.imageId));

    // Get related muxAsset
    const muxAssetRelation = await ctx.db
      .query('videoMuxAssetRelation')
      .withIndex('videoId', (q) => q.eq('videoId', videoId))
      .unique();

    const muxAsset = muxAssetRelation?.muxAssetId
      ? await ctx.db.get(muxAssetRelation.muxAssetId)
      : null;

    // Get related category
    const categoryRelation = await ctx.db
      .query('videoCategoryRelation')
      .withIndex('videoId', (q) => q.eq('videoId', videoId))
      .unique();
    const category = categoryRelation && (await ctx.db.get(categoryRelation.categoryId));

    // Get related activity
    const activity = await ctx.db
      .query('activity')
      .withIndex('videoId', (q) => q.eq('videoId', videoId))
      .unique();

    return {
      ...video,
      enrichments: {
        category,
        activity,
        thumbnails,
        muxAsset,
      },
    };
  },
});

sshader•6mo ago

This is pretty similar to what I've tried (example of something similar https://github.com/sshader/proset/pull/9/files#diff-0a5f54785ba844435b2cd590cfeb9375ae75ec8392af0a5b5d957cae5219ca0eR79) I found that having a mergeObjects helper that works similar to the spread operator helped. As well as the doc helper to add in the _id and _creationTime fields for me. I want to add some of these to convex-helpers (alongside https://github.com/get-convex/convex-helpers/blob/main/packages/convex-helpers/README.md#validator-utilities which already has some pretty good stuff like nullable)

GitHub

Return value validators with doc helpers by sshader · Pull Request ...

Main change is that I cherry-picked the "exposing validator innards" PR onto Convex, and then used it to make betterV which has a doc("Messages") type on it. It ...

GitHub

convex-helpers/packages/convex-helpers/README.md at main · get-conv...

A collection of useful code to complement the official packages. - get-convex/convex-helpers

sshader•6mo ago

When doing this myself I also found it pretty frustrating that TypeScript is perfectly fine with your function returning an object with extra properties while the return validators would not be. I don't have a ton of advice there other than to manually test your functions in dev

oscklmOP•6mo ago

Yeah we've been through the same frustrations as well, i would love to know the why behind ts being fine with extra properties, as long as it gets the one it needs. Thanks for sharing all of this though, i'm sure it will help - i will report back if i struck some nice way of going about it. Actually my biggest pain is trying to make any sense of what typescript errors, when whats expected by the handler dont match the validator (I am talking about the rather cryptic long typescript error, when hovering the handler. I feel even with the pretty typescript vscode extension, its not an easy task to make sense of it 😆 it might just be me

sshader•6mo ago

Ah yeah. Believe it or not, the type errors are a little more readable now than they were before. One tip for isolating types / type errors is temporarily saving the types you care about to variables (e.g. type TEMP = ReturnType<typeof myFunctionToValidate> , type TEMP2 = TEMP["users"]). The you can compare it to the types from your validators (Infer<typeof myValidator>) and hopefully have a slightly easier time narrowing down which part of your validator might be mismatching with the function. Also the general advice of starting with a lot of things as v.any() and slowly adding in the more precise types applies

oscklmOP•6mo ago

That's great advice. Thanks!

erquhart•6mo ago

The history of typescript not supporting exact object types is storied. If you have a few hours to kill you can check out the 8 year old canonical issue on the repo: https://github.com/microsoft/TypeScript/issues/12936

GitHub

Exact Types · Issue #12936 · microsoft/TypeScript

This is a proposal to enable a syntax for exact types. A similar feature can be seen in Flow (https://flowtype.org/docs/objects.html#exact-object-types), but I would like to propose it as a feature...

erquhart•6mo ago

Just look at those upvotes lol

oscklmOP•6mo ago

Awesome, def wanna take a look at that! Thanks mate

Working with return validator and a query that enriches data

Did you find this page helpful?