1 reply

RAG component dimensionality mismatch

Hey guys, found a small issue/footgun with the RAG component. If the specified embedding dimension doesn't match the actual dimensionality of the model, the search doesn't work. Example:

// Problem! OpenAI's text-embedding-3-large
// produces embeddings of dimensionality 3072 by default.
// It seems that Convex doesn't specify desired dimensionality when calling the AI SDK.
const rag = new RAG(components.rag, {
  textEmbeddingModel: AISDKOpenAI.embedding("text-embedding-3-large"),
  embeddingDimension: 1356 
})

// When adding entries: the embeddings will be stored in vector_3072
await rag.add(ctx, { ... })

// When searching entries, nothing is returned, probably because vector_1356 is empty
await rag.search(ctx, { ... })

// Problem! OpenAI's text-embedding-3-large
// produces embeddings of dimensionality 3072 by default.
// It seems that Convex doesn't specify desired dimensionality when calling the AI SDK.
const rag = new RAG(components.rag, {
  textEmbeddingModel: AISDKOpenAI.embedding("text-embedding-3-large"),
  embeddingDimension: 1356 
})

// When adding entries: the embeddings will be stored in vector_3072
await rag.add(ctx, { ... })

// When searching entries, nothing is returned, probably because vector_1356 is empty
await rag.search(ctx, { ... })

Note that, per OpenAI's docs [1], it may be desirable to use a large model with a smaller dimensionality.

[1] https://platform.openai.com/docs/guides/embeddings

[...] a text-embedding-3-large embedding can be shortened to a size of 256 while still outperforming an unshortened text-embedding-ada-002 embedding with a size of 1536.

RAG component dimensionality mismatch

Hey guys, found a small issue/footgun with the RAG component. If the specified embedding dimension doesn't match the actual dimensionality of the model, the search doesn't work. Example:

// Problem! OpenAI's text-embedding-3-large
// produces embeddings of dimensionality 3072 by default.
// It seems that Convex doesn't specify desired dimensionality when calling the AI SDK.
const rag = new RAG(components.rag, {
  textEmbeddingModel: AISDKOpenAI.embedding("text-embedding-3-large"),
  embeddingDimension: 1356 
})

// When adding entries: the embeddings will be stored in vector_3072
await rag.add(ctx, { ... })

// When searching entries, nothing is returned, probably because vector_1356 is empty
await rag.search(ctx, { ... })

// Problem! OpenAI's text-embedding-3-large
// produces embeddings of dimensionality 3072 by default.
// It seems that Convex doesn't specify desired dimensionality when calling the AI SDK.
const rag = new RAG(components.rag, {
  textEmbeddingModel: AISDKOpenAI.embedding("text-embedding-3-large"),
  embeddingDimension: 1356 
})

// When adding entries: the embeddings will be stored in vector_3072
await rag.add(ctx, { ... })

// When searching entries, nothing is returned, probably because vector_1356 is empty
await rag.search(ctx, { ... })

Note that, per OpenAI's docs [1], it may be desirable to use a large model with a smaller dimensionality.

[1] https://platform.openai.com/docs/guides/embeddings

[...] a text-embedding-3-large embedding can be shortened to a size of 256 while still outperforming an unshortened text-embedding-ada-002 embedding with a size of 1536.

RAG component dimensionality mismatch

RAG component dimensionality mismatch

Continue the conversation

Convex Community

Continue the conversation

Convex Community

Similar Threads