How to do an array contains query over a large table?
Hi,
I have a table with over 100k docs and I have a field (
fieldA) on the table, values of which are arrays of strings.
What is the best way to find the documents in this table, such that the corresponding array at fieldA contains a certain string?
It looks like convex doesn't support an arrayContains filter, and I can see the suggestion to read the full table into memory and then filter this using js...?
But I think with a large table like this then this won't work because I will either hit the (~16k) document scan limit, or run out of memory?2 Replies
You're correct on all counts. We recommend paginating the query so you don't read too much at once. Check out https://stack.convex.dev/complex-filters-in-convex
Using TypeScript to Write Complex Query Filters
There’s a new Convex helper to perform generic TypeScript filters, with the same performance as built-in Convex filters, and unlimited potential.
There are also a lot of resources on implementing "tags" with a "join" table instead of with a list of strings, which allows for much faster lookups (looking up each string with an indexed lookup). https://search.convex.dev/?q=tags
Convex Developer Search
Search Docs, Stack, Discord all at once