@convex-dev/agent Vector Embedding Size Limitations
If I use google's multimodalEmbedding001, which outputs 1408 dimensions, won't it break with existing code above?
7 Replies
Thanks for posting in <#1088161997662724167>.
Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets.
- Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.)
- Use search.convex.dev to search Docs, Stack, and Discord all at once.
- Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI.
- Avoid tagging staff unless specifically instructed.
Thank you!
Does this mean we can only use openai?
I tried to include all the dimensions for all the providers I could find, but I guess I missed that one. I'll add it to the list!
I added it in version
0.1.1
but just FYI the embeddings are currently only on the text part of the message, which this model seems to not be well optimized for. Adding a future image/video embedder may make sense, but right now I'd suggest using a different embedding model.Yeah 32 tokens is hilarious
Thank you
I also ask because the agent component has some methods in the source code like
Which currently don't seem to be utilizied?
And I was wondering if there was some multimodal work going on there to make file embeddings easier
Oh yeah that's for adding images/ files to messages, but the context RAG isn't currently doing vector search on anything but text fields.
For files outside of the chat history I was thinking a RAG component would be better suited. But for things in the chat itself doing embeddings on more things could be beneficial - if the surrounding conversation doesn't pull it in. would be interesting to do some evals
Do you send the text message and then add a file to it after the fact? So it can be displayed inline?
You can add a file, put it in a message (as a URL to convex storage), and send it to the LLM in the
messages
argument. I put an example in the repo that doesn't (yet) use the Agent file tracking: https://github.com/get-convex/agent/blob/ian/text-streaming/example/convex/example.ts#L344-L370GitHub
agent/example/convex/example.ts at ian/text-streaming · get-convex...
Build AI agents on Convex with persistent chat history - get-convex/agent