Sronds•2mo ago

help me setup a cronjob that loops through my entire user table and updates a number

currently the following code errors out because i have more than 4096 user rows

export const replenishPointsForAllUsers = internalMutation({
  args: {},
  handler: async (ctx) => {
    const userQuery = ctx.db.query('user')

    for await (const user of userQuery) {
      await ctx.db.patch(user._id, {
        points: 100,
      })
    }
  },
})

export const replenishPointsForAllUsers = internalMutation({
  args: {},
  handler: async (ctx) => {
    const userQuery = ctx.db.query('user')

    for await (const user of userQuery) {
      await ctx.db.patch(user._id, {
        points: 100,
      })
    }
  },
})

doesn't seem like indexes would help so not sure how to solve this. i know i should batch the queries but not sure how to do it here is the specific error i'm getting

Uncaught Error: Too many reads in a single function execution (limit: 4096). Consider using smaller limits in your queries, paginating your queries, or using indexed queries with a selective index range expressions.

Uncaught Error: Too many reads in a single function execution (limit: 4096). Consider using smaller limits in your queries, paginating your queries, or using indexed queries with a selective index range expressions.

2 Replies

Convex Bot•2mo ago

Thanks for posting in <#1088161997662724167>. Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets. - Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.) - Use search.convex.dev to search Docs, Stack, and Discord all at once. - Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI. - Avoid tagging staff unless specifically instructed. Thank you!

SrondsOP•2mo ago

managed to fix it by reading through this article https://stack.convex.dev/migrating-data-with-mutations specifically this piece of code

export const myMigrationBatch = internalMutation({
  args: { cursor: v.union(v.string(), v.null()), numItems: v.number() },
  handler: async (ctx, args) => {
    const data = await ctx.db.query("mytable").paginate(args);
    const { page, isDone, continueCursor } = data;
    for (const doc of page) {
      // modify doc
    }
    if (!isDone) await ctx.scheduler.runAfter(0, internal.example.myMigrationBatch, {
      cursor: continueCursor,
      numItems: args.numItems,
    });
  }
);

export const myMigrationBatch = internalMutation({
  args: { cursor: v.union(v.string(), v.null()), numItems: v.number() },
  handler: async (ctx, args) => {
    const data = await ctx.db.query("mytable").paginate(args);
    const { page, isDone, continueCursor } = data;
    for (const doc of page) {
      // modify doc
    }
    if (!isDone) await ctx.scheduler.runAfter(0, internal.example.myMigrationBatch, {
      cursor: continueCursor,
      numItems: args.numItems,
    });
  }
);

Stateful Online Migrations using Mutations

Online migrations in Convex using mutations, including a Convex Component to manage them.