Gi•7d ago

seeding a project

What would be the best way to seed a project? Currently running against read and write limits. I want to 1) purge an entire table, then 2) insert a lot rows. Is there anything better than 1) paginate to get all _ids and remove and for 2) create internal mutations for batches?

6 Replies

jamwt•7d ago

article here: https://stack.convex.dev/seeding-data-for-preview-deployments

Seeding Data for Preview Deployments

Now that we've launched Preview Deployments on Convex, you can test out backend changes easier than ever. But you may want to seed your project with d...

GiOP•7d ago

Nice, missed that one. Thanks!

jamwt•7d ago

re: (1), there's an import trick you can do to wipe your whole database from the command line:

for tableName in npx convex data; do npx convex import --table $tableName --replace -y --format jsonLines /dev/null; done

this immediately drops all your records quickly without having to enumerate ids by importing empty tables

GiOP•7d ago

Hmm. I'd probably want to steer away from CLI "tricks" where possible. Will probably just create a helper mutation to clear the table like AITown's vacuum one

export const vacuumTable = internalMutation({
  args: {
    tableName: v.string(),
    before: v.number(),
    cursor: v.union(v.string(), v.null()),
    soFar: v.number(),
  },
  handler: async (ctx, { tableName, before, cursor, soFar }) => {
    const results = await ctx.db
      .query(tableName as TableNames)
      .withIndex('by_creation_time', (q) => q.lt('_creationTime', before))
      .paginate({ cursor, numItems: DELETE_BATCH_SIZE });
    for (const row of results.page) {
      await ctx.db.delete(row._id);
    }
    if (!results.isDone) {
      await ctx.scheduler.runAfter(0, internal.crons.vacuumTable, {
        tableName,
        before,
        soFar: results.page.length + soFar,
        cursor: results.continueCursor,
      });
    } else {
      console.log(`Vacuumed ${soFar + results.page.length} entries from ${tableName}`);
    }
  },
}

export const vacuumTable = internalMutation({
  args: {
    tableName: v.string(),
    before: v.number(),
    cursor: v.union(v.string(), v.null()),
    soFar: v.number(),
  },
  handler: async (ctx, { tableName, before, cursor, soFar }) => {
    const results = await ctx.db
      .query(tableName as TableNames)
      .withIndex('by_creation_time', (q) => q.lt('_creationTime', before))
      .paginate({ cursor, numItems: DELETE_BATCH_SIZE });
    for (const row of results.page) {
      await ctx.db.delete(row._id);
    }
    if (!results.isDone) {
      await ctx.scheduler.runAfter(0, internal.crons.vacuumTable, {
        tableName,
        before,
        soFar: results.page.length + soFar,
        cursor: results.continueCursor,
      });
    } else {
      console.log(`Vacuumed ${soFar + results.page.length} entries from ${tableName}`);
    }
  },
}

Would be a nice to have this on mutation/action ctx by default like ctx.db.clear(table) similar to the Clear Table button in dashboard

jamwt•7d ago

yeah, we could probably ship this as an MCP tool as well

GiOP•7d ago

Should I make a ticket for this somewhere? Would be useful in general to have a simple way to prune data older than x time

seeding a project

Did you find this page helpful?