How do I rebuild a table index after adding documents?

Reading through the documentation, I've read the following;

Similarly, even after an index is defined, Convex will have to do a bit of extra work to keep this index up to date as the data changes. Every time a document is inserted, updated, or deleted in an indexed table, Convex will also update its index entry. This is analogous to a librarian creating new index cards for new books as they add them to the library.

Similarly, even after an index is defined, Convex will have to do a bit of extra work to keep this index up to date as the data changes. Every time a document is inserted, updated, or deleted in an indexed table, Convex will also update its index entry. This is analogous to a librarian creating new index cards for new books as they add them to the library.

So what I've done, is I've created a large amount of documents that are beyond the non-indexed return limit, and have then generated an index for that table. I've then added new data, and am testing out the returns, to find the new data hasn't been indexed yet. New documents returns fine via .take(3), but as far as the index return is concerned - those new documents don't exist. I'd like to understand if it's possible to trigger the manual rebuild of an index, I guess my interval would when approaching an additional ~8000 records to trigger the rebuild - which would allow for either a table look up of new non-indexed records, or an indexed lookup of the previously indexed records. Generally speaking, if index rebuild is automated - how long is the wait before new documents appear in the index?

3 Replies

lee•17mo ago

When a new document is inserted, it's added to indexes immediately and atomically, so by the time await ctx.db.insert returns, the document has been added to all indexes on the table. If you're observing that new documents don't appear in the index, that is not expected and we should dig into it. Can you describe more about what you are observing, with the index definition and the queries on the index?

lee•17mo ago

One exception is if it's a vector index, only the first 100k documents are indexed (we're actively working on this) https://docs.convex.dev/vector-search#limits

Vector Search | Convex Developer Hub

Run vector search queries on embeddings

KarlstensOP•17mo ago

Thanks @lee - that's very helpful. I had a flag in my cloud function that I totally forgot which was filtering the otherwise instantly/correctly indexed data. Woo!

How do I rebuild a table index after adding documents?

Did you find this page helpful?