yarrichar•15mo ago

Multiple vector indexes on the same table

I have a bunch of embeddings stored in a table. I want to be able to filter those on different columns depending on the scenario. I assume I'm meant to create two different vectorIndex's. But I keep getting the error when I try:

400 Bad Request: IndexFieldsNotUnique: Hit an error while evaluating your schema:
Index fields must be unique within an index.

400 Bad Request: IndexFieldsNotUnique: Hit an error while evaluating your schema:
Index fields must be unique within an index.

The indexes I have on that table are:

        .vectorIndex("by_embedding", {
            vectorField: "embedding",
            dimensions: 1536,
            filterFields: ["clerkUserId"],
        })
        .vectorIndex("by_embedding_doc", {
            vectorField: "embedding",
            dimensions: 1536,
            filterFields: ["documentId"],
        }),

        .vectorIndex("by_embedding", {
            vectorField: "embedding",
            dimensions: 1536,
            filterFields: ["clerkUserId"],
        })
        .vectorIndex("by_embedding_doc", {
            vectorField: "embedding",
            dimensions: 1536,
            filterFields: ["documentId"],
        }),

If I remove the second vectorIndex then it works. Any ideas what I'm doing wrong?

7 Replies

lee•15mo ago

Interesting, this looks to me like it should work, so we'll look into it. But it also looks like you could use a single index .vectorIndex("by_embedding", {vectorField: "embedding", dimensions: 1536, filterFields: ["clerkUserId", "documentId"]}). For each vector search query you can filter on any subset of the filter fields.

ian•15mo ago

To expand on Lee's point a bit, a vectorIndex is different from a normal index in the way that filterFields work. You don't have to specify them in prefix order. You can filter on just documentId or just clerkUserId. As a result, you don't benefit from having two indexes on the same field. Of course let us know if you have a use case that would benefit from multiple indexes that I haven't thought of

yarricharOP•15mo ago

So specifying both fields in the one index ["clerkUserId", "documentId"] seems to work, thanks! This mostly meets my needs - the only thing I'm missing is the ability to do ANDs in vector searches (intellisense and the docs both seem to indicate it's not available). When I'm filtering by documentId I would have liked to be able to also filter by clerkUserId to make sure I haven't stuffed up and am only looking at documents for the current user 🙂

ian•15mo ago

Gotcha. For that case, you could check the results when they come back, which would require fetching the document for the id returned from the search

lee•15mo ago

@sshader and @sujayakar looked into supporting ANDs for filters on vector search, and iirc we discovered that it's an open research problem (as long as you want any kind of guarantees about the accuracy or runtime) If the research problem is solved, we'll work on supporting it 😛. Isn't it fun working on cutting edge ai stuff

yarricharOP•15mo ago

Yeah, and checking the results after the fact is fine for this use case - hopefully it has no effect 😉 FWIW it looks like Pinecone has support for ANDs. See: https://docs.pinecone.io/docs/metadata-filtering#querying-an-index-with-metadata-filters

lee•15mo ago

yep i believe it can be implemented but not with a reasonable time complexity 😅

Multiple vector indexes on the same table

Did you find this page helpful?