Helmet•14mo ago

Error related to Node.js (package not found on filesystem)

Hey guys, I'm currently attempting to prototype a langchain-based retrieval QA system with user chat messages. Currently handling a user's query within action, in which I send the query string to a function from a separate file in the same convex directory to do the retrieval augmentation work (dir looks somewhat like convex/mainFunctions.ts and convex/retrievalQA.ts). However, running the convex code is giving the following error:

[ERROR] Could not resolve "node:path"

    node_modules/langchain/dist/vectorstores/hnswlib.js:165:32:
      165 │       const path = await import("node:path");
          ╵                                 ~~~~~~~~~~~

  The package "node:path" wasn't found on the file system but is built into node. Are you trying to
  bundle for node? You can use "platform: 'node'" to do that, which will remove this error.

[ERROR] Could not resolve "node:path"

    node_modules/langchain/dist/vectorstores/hnswlib.js:165:32:
      165 │       const path = await import("node:path");
          ╵                                 ~~~~~~~~~~~

  The package "node:path" wasn't found on the file system but is built into node. Are you trying to
  bundle for node? You can use "platform: 'node'" to do that, which will remove this error.

It's giving errors for other imports such as node:fs/promises as well. HNSWLib, a vectorstore that langchain provides a wrapper for, imports these node utilities, and from what my aggressive googling has informed me the issue has something to do with the environment in which Convex is running. Most of the googling shows Cloudflare or Sveltekit-related issues (ie https://stackoverflow.com/questions/73178856/sveltekit-packages-not-being-able-to-access-node-functions, https://github.com/evanw/esbuild/issues/1996, etc). I'm not really sure if Convex's run environment doesn't support node or isn't up-to-date past v18 (which introduced prefixed imports), or if even is related to Convex at all. What I want: for those who have used external functions from other files, how do you recommend accomplishing something similar to what I am trying to do? Is it ok that I use functions from other files within the convex directory? Does Convex run node, and if so, what would be causing this issue? I've read that it might be related to polyfills, but not sure. Any and all help appreciated!!

Stack Overflow

SvelteKit: Packages not being able to access node functions

I am building an app with SvelteKit and publishing it to Cloudflare Pages, but it is some packages I am using are not able to access node native functions. Here's the build log: 2022-07-30T19:05:14.

GitHub

Issues · evanw/esbuild

An extremely fast bundler for the web. Contribute to evanw/esbuild development by creating an account on GitHub.

6 Replies

HelmetOP•14mo ago

I used "use node"; directive at the top of the functions for convex file, but I get a different error clearMessages defined in functions.js is a Mutation function. Only actions can be defined in Node.js. What I ended up doing is moving the main action to a separate file with "use node"; directive instead, seems to work If anyone has a better idea of how to manage this, feel free to let me know

Michal Srb•14mo ago

Hey @Helmet,

What I ended up doing is moving the main action to a separate file with "use node"; directive

That is the correct use of "use node", the file can only contain actions. But latest Convex + langchain should not require Node.js at all btw.

HelmetOP•14mo ago

The js langchain vectorstore wrapper for HNSWLib imports node:path, so unless I specify "use node" it throws an error, originally described in the post above

Michal Srb•14mo ago

Ah, I see! Have you tried using ConvexVectorStore instead? That one doesn't need to use Node, and will use data stored in one of your tables.

HelmetOP•14mo ago

I haven't, looked into it and I plan to try it out. Just wondering about performance, as it stores the vectors in tables. So when using a similarity search on a table database of perhaps 1000 entries, I wonder if it will be kinda overkill Having to make all the db reads and whatnot

Michal Srb•14mo ago

Check out the docs for our vector search: https://docs.convex.dev/vector-search It's optimized for exactly this workload, and we're working on supporting millions of rows

Vector Search | Convex Developer Hub

Run vector search queries on embeddings

Error related to Node.js (package not found on filesystem)

Did you find this page helpful?