abdullahislam_•6mo ago

De-duplicate table with existing values

I have a table that already has data. I want to do a find a remove rows where there's duplicates. What's the best way to approach this?

5 Replies

ballingt•6mo ago

Duplicates values for a specific field? add an index on whatever field(s) it is that are duplicated and walk over them. If it's less than a few thousand you may be able to do this in once mutation, if it's more then you can paginate this.

abdullahislam_OP•6mo ago

Ah got it. So something like this should work:

interface CityObject {
  city: string;
  [key: string]: any;
}

function deduplicateCities<T extends CityObject>(arr: T[]): T[] {
  return Array.from(
    new Map(arr.map(item => [item.city, item])).values()
  );
}

interface CityObject {
  city: string;
  [key: string]: any;
}

function deduplicateCities<T extends CityObject>(arr: T[]): T[] {
  return Array.from(
    new Map(arr.map(item => [item.city, item])).values()
  );
}

ballingt•6mo ago

This looks like a good way to return a deduplicated list, not a way to delete the data from the database. If this works for you great, I thought you wanted to delete the duplicates.

abdullahislam_OP•6mo ago

So instead of returning, I would just put it in a const and do a promise all on await ctx.db.delete(DOC_ID_FROM_MAP); ?

ballingt•6mo ago

Well just delete the duplicates, so different logic to find those, but yeah.

De-duplicate table with existing values

Did you find this page helpful?