Optimizing embedding
is there any smart way of optimizing updates to the embedding of text content? For example - in a notes app, idk how expensive updating the embeddings for document might be if we update the text every time the note is changed
5 Replies
Hey @star ✨, embedding text does have a cost, you can check the price with the provider you’re using (like OpenAI).
To lower the cost, you can debounce the embedding, using scheduled functions or crons.
You could also split the input text (user note) into parts (say paragraphs) and embed each separately (to avoid having to recompute the embedding of the whole text).
Would splitting a larger text block into paragraphs result in any loss of context or anything like that?
Currently I’ve debounced input changes with a setTimeout of 1sec - I think the only main problem here is if there’s a large text block and there’s even a single word change, I’m recomputing the whole thing
Because a single word could potentially change the meaning of a whole paragraph/sentence
The way some text splitters handle the context loss is often they have an overlap, for instance keeping a sentence or paragraph before / after each chunk.
ohh i see
There are also strategies to provide a summary of larger sections to use as a short context within chunks. I'm no expert but I'd look at what the splitter is producing and apply heuristics that make it more understandable to you. Ultimately these models are trained on regular language, so my assumption is that they'll be most useful when that's their input