mvols Comments - Convex Community

mvols

•Created by mvols on 1/17/2025 in #general

Batch insertions

dude get outta here, your kidding.. this has got to be one of the best (and coolest) services ive used to date! LOVE how this is all in TS and how unbelievably cool the dashboard is when writing/testing this stuff!

51 replies

CCConvex Community

•Created by mvols on 1/17/2025 in #general

Batch insertions

so within that insert function, im doing this to check if its already been added:

export async function ensureUniqueCardPriceDoc(
  ctx: QueryCtx,
  args: CardPriceDoc
): Promise<Doc<"cardPricing"> | null> {
  const { cardId, isoSnapshotDate, variant } = args;

  return ctx.db
    .query("cardPricing")
    .withIndex("unique_card_id_and_snapshot_date_and_variant", (q) =>
      q
        .eq("cardId", cardId)
        .eq("isoSnapshotDate", isoSnapshotDate)
        .eq("variant", variant)
    )
    .unique();
}

export async function ensureUniqueCardPriceDoc(
  ctx: QueryCtx,
  args: CardPriceDoc
): Promise<Doc<"cardPricing"> | null> {
  const { cardId, isoSnapshotDate, variant } = args;

  return ctx.db
    .query("cardPricing")
    .withIndex("unique_card_id_and_snapshot_date_and_variant", (q) =>
      q
        .eq("cardId", cardId)
        .eq("isoSnapshotDate", isoSnapshotDate)
        .eq("variant", variant)
    )
    .unique();
}

so this being within the internalMutation would still count as only 1 fn call?

51 replies

CCConvex Community

•Created by mvols on 1/17/2025 in #general

Batch insertions

thats much more palletable you think right to run each day. so the ctx.db.insert or ctx.db.query doesn't count towards function calls monthly it sounds like? just the actual internalActions or query from the client side?

51 replies

CCConvex Community

•Created by mvols on 1/17/2025 in #general

Batch insertions

damn you a mad genius

51 replies

CCConvex Community

•Created by mvols on 1/17/2025 in #general

Batch insertions

so here im only inserting 1 each time, but if i bump this to say take in an array of 200 cards, then the call to this internalMutation would only count as one per 200 cards? even tho we are calling ctx.db.insert for each single card (lets just say at the end of this function daily, i would be inserting 100k docs to keep numbers easy

51 replies

CCConvex Community

•Created by mvols on 1/17/2025 in #general

Batch insertions

hm okay i think im grasping better now - so your saying in the internalMutation handle inserting an array of docs

/**
 * Internal mutation to insert a card price document
 * - runs validation checks on the input arguments
 * - checks for uniqueness
 */
const internalInsertCardPriceDoc = internalMutation({
  args: zodToConvex(cardPriceSchema),
  handler: async (ctx, args) => {
    const result = validateCardPriceArgs(args);
    if (result.error) {
      const errors = result.error.issues
        .map((issue) => issue.message)
        .join(", ");
      throw new Error(
        `[internalInsertCardPriceDoc] ${errors}, args:${JSON.stringify(args)}`
      );
    }

    const existingCard = await ensureUniqueCardPriceDoc(ctx, args);
    if (existingCard) {
      throw new Error(
        `[internalInsertCardPriceDoc] card pricing already exists, doc:${existingCard._id}, args:${JSON.stringify(
          args
        )}`
      );
    }

    return ctx.db.insert("cardPricing", args);
  },
});

export default internalInsertCardPriceDoc;

/**
 * Internal mutation to insert a card price document
 * - runs validation checks on the input arguments
 * - checks for uniqueness
 */
const internalInsertCardPriceDoc = internalMutation({
  args: zodToConvex(cardPriceSchema),
  handler: async (ctx, args) => {
    const result = validateCardPriceArgs(args);
    if (result.error) {
      const errors = result.error.issues
        .map((issue) => issue.message)
        .join(", ");
      throw new Error(
        `[internalInsertCardPriceDoc] ${errors}, args:${JSON.stringify(args)}`
      );
    }

    const existingCard = await ensureUniqueCardPriceDoc(ctx, args);
    if (existingCard) {
      throw new Error(
        `[internalInsertCardPriceDoc] card pricing already exists, doc:${existingCard._id}, args:${JSON.stringify(
          args
        )}`
      );
    }

    return ctx.db.insert("cardPricing", args);
  },
});

export default internalInsertCardPriceDoc;

51 replies

CCConvex Community

•Created by mvols on 1/17/2025 in #general

Batch insertions

im working on my own historical pricing chart LOL so this is storing pricing on each card!

51 replies

CCConvex Community

•Created by mvols on 1/17/2025 in #general

Batch insertions

omg get outta here this is for pokemon.. thats literally what im doing for my side project 😆

51 replies

CCConvex Community

•Created by mvols on 1/17/2025 in #general

Batch insertions

ah okay okay - so in terms of the pricing model - anytime a function is called thats how the pricing is computed right? (don't need exactly just a rough estimate) So it would call the internalAction once (1 fn call) then call the internalMutation (take the higher end to be safe of 750 * 75(number of pages im going through the api) which is round up to 60k function calls. Am i right in this in terms of how these function calls work?

51 replies

CCConvex Community

•Created by mvols on 1/17/2025 in #general

Batch insertions

Yup yup for sure, will look to do that - im curious tho - how come no built in way for bulk insertions?

51 replies

CCConvex Community

•Created by mvols on 1/17/2025 in #general

Batch insertions

which would happen within the asyncMap thats calling the internalMutation to insert one doc at a time

51 replies

CCConvex Community

•Created by mvols on 1/17/2025 in #general

Batch insertions

oh you know what, i miss spoke early too, so currently right now, with how this internalAction is setup, theres more like 500-750 documents to be inserted into the db per page from the query

51 replies

CCConvex Community

•Created by mvols on 1/17/2025 in #general

Batch insertions

hm, okay do you have any docs I can read through because to be honest im not sure i understand in one mutation if theres no like bulk insert

51 replies

CCConvex Community

•Created by mvols on 1/17/2025 in #general

Batch insertions

is doing something like this daily expensive through Convex? in terms of the function calls - trying to grasp the pricing model & do's & don'ts for convex ya know

51 replies

CCConvex Community

•Created by mvols on 1/17/2025 in #general

Batch insertions

so this is what my current internalAction looks like

51 replies

CCConvex Community

•Created by mvols on 1/17/2025 in #general

Batch insertions

const trackCardPricing = internalAction({
  handler: async (ctx) => {
    try {
      let page = 1;
      let hasMorePages = true;
      let totalCardVariants = 0;
      let totalProcessed = 0;

      console.log("starting to track card pricing...");

      while (hasMorePages) {
        const response = await fetchPokemontcgPrices(page);
        const { data, page: currPage, count, totalCount } = response;

        hasMorePages = currPage < totalCount / count;
        page++;

        const cardVariants: CardPriceDoc[] = [];

        data.forEach((card) => {
          const baseFields: Omit<CardPriceDoc, "variant"> = {
            cardId: card.id,
            cardName: card.name,
            cardSetId: card.set.id,
            isoSnapshotDate: transformApiDateToIsoDate(
              card.tcgplayer.updatedAt
            ),
            tcgplayerUrl: card.tcgplayer.url,
          };

          Object.entries(card.tcgplayer.prices).forEach(([variant, prices]) => {
            cardVariants.push({
              ...baseFields,
              variant,
              ...filterNullishValues(prices),
            });
          });
        });

        totalCardVariants += cardVariants.length;

        await asyncMap(cardVariants, async (card) => {
          await ctx.runMutation(
            internal.cardPricing.mutations.internalInsertCardPriceDoc.default,
            card
          );
          totalProcessed++;
        });

        console.log(
          `Processed ${cardVariants.length} variants from page ${currPage} (Total processed: ${totalProcessed}/${totalCardVariants})`
        );

        await new Promise((resolve) => setTimeout(resolve, 100));
      }

      console.log(
        `Finished tracking card pricing! Processed ${totalProcessed} total card variants`
      );
    } catch (error) {
      console.error(`[trackCardPricing] ${error}`);
    }
  },
});

const trackCardPricing = internalAction({
  handler: async (ctx) => {
    try {
      let page = 1;
      let hasMorePages = true;
      let totalCardVariants = 0;
      let totalProcessed = 0;

      console.log("starting to track card pricing...");

      while (hasMorePages) {
        const response = await fetchPokemontcgPrices(page);
        const { data, page: currPage, count, totalCount } = response;

        hasMorePages = currPage < totalCount / count;
        page++;

        const cardVariants: CardPriceDoc[] = [];

        data.forEach((card) => {
          const baseFields: Omit<CardPriceDoc, "variant"> = {
            cardId: card.id,
            cardName: card.name,
            cardSetId: card.set.id,
            isoSnapshotDate: transformApiDateToIsoDate(
              card.tcgplayer.updatedAt
            ),
            tcgplayerUrl: card.tcgplayer.url,
          };

          Object.entries(card.tcgplayer.prices).forEach(([variant, prices]) => {
            cardVariants.push({
              ...baseFields,
              variant,
              ...filterNullishValues(prices),
            });
          });
        });

        totalCardVariants += cardVariants.length;

        await asyncMap(cardVariants, async (card) => {
          await ctx.runMutation(
            internal.cardPricing.mutations.internalInsertCardPriceDoc.default,
            card
          );
          totalProcessed++;
        });

        console.log(
          `Processed ${cardVariants.length} variants from page ${currPage} (Total processed: ${totalProcessed}/${totalCardVariants})`
        );

        await new Promise((resolve) => setTimeout(resolve, 100));
      }

      console.log(
        `Finished tracking card pricing! Processed ${totalProcessed} total card variants`
      );
    } catch (error) {
      console.error(`[trackCardPricing] ${error}`);
    }
  },
});

51 replies

CCConvex Community

•Created by mvols on 1/17/2025 in #general

Batch insertions

doc size is very small, only about 7-10kb in size

51 replies

CCConvex Community

•Created by mvols on 1/17/2025 in #general

Batch insertions

Wow Jamie! Ive been watching all your videos on youtube - new to convex just starting up a side project now to use convex! Im drinking all the koolaid from all your videos and so far have loved working with (albeit my first week lol) but no not within one action. So currently i have an internalMutaion setup to insert one doc that ive been testing through the dashboard & running the function. now I have another internalAction that im using to fetch through the api & then call that intermalMutation The internalAction is set to loop over & pagination the query per 250 items from the api.

51 replies