Tomer Moran•2mo ago

RAG component dimensionality mismatch

Hey guys, found a small issue/footgun with the RAG component. If the specified embedding dimension doesn't match the actual dimensionality of the model, the search doesn't work. Example:

// Problem! OpenAI's text-embedding-3-large
// produces embeddings of dimensionality 3072 by default.
// It seems that Convex doesn't specify desired dimensionality when calling the AI SDK.
const rag = new RAG(components.rag, {
  textEmbeddingModel: AISDKOpenAI.embedding("text-embedding-3-large"),
  embeddingDimension: 1356 
})

// When adding entries: the embeddings will be stored in vector_3072
await rag.add(ctx, { ... })

// When searching entries, nothing is returned, probably because vector_1356 is empty
await rag.search(ctx, { ... })

// Problem! OpenAI's text-embedding-3-large
// produces embeddings of dimensionality 3072 by default.
// It seems that Convex doesn't specify desired dimensionality when calling the AI SDK.
const rag = new RAG(components.rag, {
  textEmbeddingModel: AISDKOpenAI.embedding("text-embedding-3-large"),
  embeddingDimension: 1356 
})

// When adding entries: the embeddings will be stored in vector_3072
await rag.add(ctx, { ... })

// When searching entries, nothing is returned, probably because vector_1356 is empty
await rag.search(ctx, { ... })

Note that, per OpenAI's docs [1], it may be desirable to use a large model with a smaller dimensionality. [1] https://platform.openai.com/docs/guides/embeddings

[...] a text-embedding-3-large embedding can be shortened to a size of 256 while still outperforming an unshortened text-embedding-ada-002 embedding with a size of 1536.

1 Reply

Convex Bot•2mo ago

Thanks for posting in <#1088161997662724167>. Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets. - Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.) - Use search.convex.dev to search Docs, Stack, and Discord all at once. - Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI. - Avoid tagging staff unless specifically instructed. Thank you!

RAG component dimensionality mismatch

Did you find this page helpful?