erquhart•2y ago

Batch deletions limits

Is it safe to run multiple deletes on a table at once in a mutation? Eg., batches of 10 instead doing all in sequence. In the immediate case, total size could technically be hundreds but most often maybe 100-200 records tops. Curious if there's a recommended approach here.

11 Replies

Michal Srb•2y ago

You can delete as many documents as you like in one mutation, as long as you fit in the limits described here: https://docs.convex.dev/functions/error-handling#readwrite-limit-errors (so 8192 is a theoretical maximum). All deletions in one mutation will either succeed together or not happen. If you have concurrent writes to the same documents you might run into OCC errors, in which case deleting in smaller batches might be a good idea (you can schedule them from a mutation or run them from an action).

Error Handling | Convex Developer Hub

There are three reasons why your Convex

sshader•2y ago

To add on, if you're deleting a potentially unbounded / large amount of data, and don't necessarily need it to be atomic, some of the patterns for migrations (https://stack.convex.dev/migrating-data-with-mutations) can be useful. As an example, one of my apps has a cron job that deletes stale data in batches

Migrating Data With Mutations

Using mutations to migrate data in Convex.

erquhartOP•2y ago

So I do understand general migrations, I'm more specifically asking about running db.delete() against multiple records in the same table concurrently. Eg.,:

const ids = [...]
for { const idChunk of chunk(ids, 10) } {
  await Promise.all(idChunk.map(id => db.delete(id))
}

const ids = [...]
for { const idChunk of chunk(ids, 10) } {
  await Promise.all(idChunk.map(id => db.delete(id))
}

There I'm keeping concurrency limited to 10. Is that alright? Should I go lower or can I go higher?

Michal Srb•2y ago

That's actually sequential. Concurrency against the DB only happens if you have multiple mutations running at the same time. (I know the Promise.all makes this confusing, but that's for us to potentially batch the deletes - you don't have to worry about concurrency inside a single mutation).

erquhartOP•2y ago

Hmm so if I had a couple hundred ids there, you're saying I don't need to chunk, Convex just handles it?

Michal Srb•2y ago

If you don't have concurrent (other mutations) writes to them, I think it should be fine.

erquhartOP•2y ago

So what happens if one user triggers these deletions and another separately triggers writes to any of the same records at the same time?

jamwt•2y ago

@erquhart so, 99% of the time you'll never notice. convex handles it under the covers by retrying the transaction that conflicted if you have a highly, highly contended record set, it's possible for it to retry "too many times" and convex will throw a temporary failure but I wouldn't worry about this; it's rare

jamwt•2y ago

https://docs.convex.dev/database/advanced/occ

OCC and Atomicity | Convex Developer Hub

In Queries, we mentioned that determinism

jamwt•2y ago

making the batch sizes you're deleting in a single transaction smaller will also lower the chances of conflict as well

erquhartOP•2y ago

Got it, thanks for clarifying 👍

Batch deletions limits

Did you find this page helpful?