Helmet Posts - Convex Community

Helmet

•Created by Helmet on 11/28/2023 in #support-community

How do vectorstores work on Convex with Langchain?

I am making a chatbot of sorts with retrieval QA on langchain using Convex's vectorstore. I'm storing documents embedded in Convex, but wondering how I can access multiple vectorstores by name? Currently in schema.ts, I have a basic schema example for a simple vector db.

export default defineSchema({
  documents: defineTable({
    embedding: v.array(v.number()),
    text: v.string(),
    metadata: v.any(),
  }).vectorIndex("byEmbedding", {
    vectorField: "embedding",
    dimensions: 1536,
  })
},
{
    strictTableNameTypes: false,
});

export default defineSchema({
  documents: defineTable({
    embedding: v.array(v.number()),
    text: v.string(),
    metadata: v.any(),
  }).vectorIndex("byEmbedding", {
    vectorField: "embedding",
    dimensions: 1536,
  })
},
{
    strictTableNameTypes: false,
});

I'm wondering how I can make more vector dbs than just one named "documents", ie, for multiple users where each user might have their own vectdb with their name. Currently accessing the vector db in langchain through new ConvexVectorStore(new OpenAIEmbeddings({ openAIApiKey: secretKey }), { ctx });, and I don't see anywhere to pass in string names of the vector db to use or if I wanted to create a new vector db with a specific name. Curious how I might be able to achieve this?

9 replies

CCConvex Community

•Created by Helmet on 11/11/2023 in #support-community

Could not locate the bindings file

I'm using HNSWLib-node for a vectorstore, but when I run npx convex dev, I get

400 Bad Request: InvalidModules: Hit an error while pushing:
Loading the pushed modules encountered the following
    error:
Uncaught Failed to analyze _deps/node/EWAOYEO5.js: Could not locate the bindings file. Tried:
 → /tmp/source/e9f38876-a6ef-41e1-b3c9-c5d51185bdcc/build/addon.node
 → /tmp/source/e9f38876-a6ef-41e1-b3c9-c5d51185bdcc/build/Debug/addon.node
 → /tmp/source/e9f38876-a6ef-41e1-b3c9-c5d51185bdcc/build/Release/addon.node
 → /tmp/source/e9f38876-a6ef-41e1-b3c9-c5d51185bdcc/out/Debug/addon.node
 → /tmp/source/e9f38876-a6ef-41e1-b3c9-c5d51185bdcc/Debug/addon.node
 → /tmp/source/e9f38876-a6ef-41e1-b3c9-c5d51185bdcc/out/Release/addon.node

    at bindings (../../../node_modules/bindings/bindings.js:126:4)
    at node_modules/hnswlib-node/lib/index.js (../../../node_modules/hnswlib-node/lib/index.js:19:17)
    at __require2 (convex:/user/_deps/node/R44RVPLZ.js:16:50)
    at <anonymous> (../../../node_modules/hnswlib-node/lib/index.js:19:17)

400 Bad Request: InvalidModules: Hit an error while pushing:
Loading the pushed modules encountered the following
    error:
Uncaught Failed to analyze _deps/node/EWAOYEO5.js: Could not locate the bindings file. Tried:
 → /tmp/source/e9f38876-a6ef-41e1-b3c9-c5d51185bdcc/build/addon.node
 → /tmp/source/e9f38876-a6ef-41e1-b3c9-c5d51185bdcc/build/Debug/addon.node
 → /tmp/source/e9f38876-a6ef-41e1-b3c9-c5d51185bdcc/build/Release/addon.node
 → /tmp/source/e9f38876-a6ef-41e1-b3c9-c5d51185bdcc/out/Debug/addon.node
 → /tmp/source/e9f38876-a6ef-41e1-b3c9-c5d51185bdcc/Debug/addon.node
 → /tmp/source/e9f38876-a6ef-41e1-b3c9-c5d51185bdcc/out/Release/addon.node

    at bindings (../../../node_modules/bindings/bindings.js:126:4)
    at node_modules/hnswlib-node/lib/index.js (../../../node_modules/hnswlib-node/lib/index.js:19:17)
    at __require2 (convex:/user/_deps/node/R44RVPLZ.js:16:50)
    at <anonymous> (../../../node_modules/hnswlib-node/lib/index.js:19:17)

I am not sure the exact cause of this error as I have run the exact same file locally with no issues. I am following Langchain's usage instructions, and as it runs locally, I can only assume this issue is related to Convex not having some sort of support for bindings and node addons. I'd appreciate some help resolving this issue.

16 replies

CCConvex Community

•Created by Helmet on 11/10/2023 in #support-community

Error related to Node.js (package not found on filesystem)

Hey guys, I'm currently attempting to prototype a langchain-based retrieval QA system with user chat messages. Currently handling a user's query within action, in which I send the query string to a function from a separate file in the same convex directory to do the retrieval augmentation work (dir looks somewhat like convex/mainFunctions.ts and convex/retrievalQA.ts). However, running the convex code is giving the following error:

[ERROR] Could not resolve "node:path"

    node_modules/langchain/dist/vectorstores/hnswlib.js:165:32:
      165 │       const path = await import("node:path");
          ╵                                 ~~~~~~~~~~~

  The package "node:path" wasn't found on the file system but is built into node. Are you trying to
  bundle for node? You can use "platform: 'node'" to do that, which will remove this error.

[ERROR] Could not resolve "node:path"

    node_modules/langchain/dist/vectorstores/hnswlib.js:165:32:
      165 │       const path = await import("node:path");
          ╵                                 ~~~~~~~~~~~

  The package "node:path" wasn't found on the file system but is built into node. Are you trying to
  bundle for node? You can use "platform: 'node'" to do that, which will remove this error.

It's giving errors for other imports such as node:fs/promises as well. HNSWLib, a vectorstore that langchain provides a wrapper for, imports these node utilities, and from what my aggressive googling has informed me the issue has something to do with the environment in which Convex is running. Most of the googling shows Cloudflare or Sveltekit-related issues (ie https://stackoverflow.com/questions/73178856/sveltekit-packages-not-being-able-to-access-node-functions, https://github.com/evanw/esbuild/issues/1996, etc). I'm not really sure if Convex's run environment doesn't support node or isn't up-to-date past v18 (which introduced prefixed imports), or if even is related to Convex at all. What I want: for those who have used external functions from other files, how do you recommend accomplishing something similar to what I am trying to do? Is it ok that I use functions from other files within the convex directory? Does Convex run node, and if so, what would be causing this issue? I've read that it might be related to polyfills, but not sure. Any and all help appreciated!!

11 replies