Could no resolve "util"

How does the convex backend actually run ? Does it use node / can I use packages intended for node and how?

45 Replies

The "use node" thing is on the right track, every file in the convex directory that imports (or imports a file that imports, etc.) "util" or any other Node.js builtin needs to have "use node" at the top. To narrow down which file it is you can try removing files to see when the problem disappears. It looks like it's something that imports @tensorflow External packages the @v links to are mostly a performance optimization, you'll still need to avoid importing from "use node" files from non-"use node" files.

LightwavesOP•7mo ago

"util" was just an example, I I'm only trying to import tensorflow but it depends on many packages, and so far every single tf dependency is giving me that error

v•7mo ago

I thought using an external package would fix dynamic imports, my bad on that(because I thought that might be the issue)

LightwavesOP•7mo ago

it's too big to be one apparently

ballingt•7mo ago

Yeah sounds like they all need to be imported only from "use node" convex files, and you can't import anything from those "use node" files from non-"use node" files All the "use node" files get bundled together and all the non-"use node" files get bundled together (into separate bundles, but it's the same bundle process" and if you import a "use node" file from one of the non-"use node" files you'll see these errors

LightwavesOP•7mo ago

I'm only importing it from filers with "use node"

ballingt•7mo ago

You might be importing one of those files from a non-"use node" file We may have some linters before long for this sort of thing, alerting if you every import a "use node" from a non-"use node"

LightwavesOP•7mo ago

At the moment the only file tensorflow is imported it has "use node" and all it does is import tf. That file is not imported anywhere else At least not that I can see

ballingt•7mo ago

What's the name of that file, try searching your project files ((ctrl/cmd)-shift-f in VSCode) for that filename

v•7mo ago

Off topic, that profile photo looks dope @Lightwaves

LightwavesOP•7mo ago

lol thanks, i've had it for years, I can't remember where it's from Okay so, it's just named tf_model.ts Here it is

  "use node"

const tf = require("@tensorflow/tfjs-node")


// export const build_model = () =>
// {
//     const model = tf.sequential();

//     model.add (tf.layers.conv2d({
//         inputShape: [ 28 , 28, 1],
//         kernelSize: 3,
//         filters: 8,
//         strides:1,
//         activation:"relu",
//         kernelInitializer: "varianceScaling"
//     }));
//     model.add(tf.layers.maxPooling2d({poolSize:[2,2]}));
//      model.add (tf.layers.conv2d({
//         kernelSize: 3,
//         filters: 16,
//         strides:1,
//         activation:"relu",
//         kernelInitializer: "varianceScaling"
//     }));
//     model.add(tf.layers.maxPooling2d({poolSize:[2,2], strides:[2,2]}));

//     model.add(tf.layers.flatten());
//     model.add(tf.layers.dense({
//         units: 128,
//         kernelnitializer: 'varianceScaling',
//         activation: 'relu'
//     }));
//     model.add(tf.layers.dense({
//         units: 10,
//         kernelnitializer: 'varianceScaling',
//         activation: 'softmax'
//     }));
//     const optimizer = tf.train.adam();
//     model.compile({
//         optimizer: optimizer,
//         loss: 'categoricalCrossentropy',
//         metrics: ['accuracy'],
//     });

//   return model;
// }

  "use node"

const tf = require("@tensorflow/tfjs-node")


// export const build_model = () =>
// {
//     const model = tf.sequential();

//     model.add (tf.layers.conv2d({
//         inputShape: [ 28 , 28, 1],
//         kernelSize: 3,
//         filters: 8,
//         strides:1,
//         activation:"relu",
//         kernelInitializer: "varianceScaling"
//     }));
//     model.add(tf.layers.maxPooling2d({poolSize:[2,2]}));
//      model.add (tf.layers.conv2d({
//         kernelSize: 3,
//         filters: 16,
//         strides:1,
//         activation:"relu",
//         kernelInitializer: "varianceScaling"
//     }));
//     model.add(tf.layers.maxPooling2d({poolSize:[2,2], strides:[2,2]}));

//     model.add(tf.layers.flatten());
//     model.add(tf.layers.dense({
//         units: 128,
//         kernelnitializer: 'varianceScaling',
//         activation: 'relu'
//     }));
//     model.add(tf.layers.dense({
//         units: 10,
//         kernelnitializer: 'varianceScaling',
//         activation: 'softmax'
//     }));
//     const optimizer = tf.train.adam();
//     model.compile({
//         optimizer: optimizer,
//         loss: 'categoricalCrossentropy',
//         metrics: ['accuracy'],
//     });

//   return model;
// }

This actually works but if I uncoment the "build model" function then it doesn't work

LightwavesOP•7mo ago

I get

LightwavesOP•7mo ago

Are you only installing packages that are being used cause if so that's pretty cool

ballingt•7mo ago

Yeah, that's an esbuild thing, it does tree-shaking And nothing else imports from tf_model.ts? Oh wait, I thought you had a different error

LightwavesOP•7mo ago

neat, anyways, removing that convex.json file, when I uncomment the function I get these I don't kno wwhy it's only these

ballingt•7mo ago

This error is about not being able to mark this as external

LightwavesOP•7mo ago

yes I was just saying it was weird that it only happened when i uncomment the function lol but yeah Ive removed that file

ballingt•7mo ago

Hm I wonder what library requires this @mapbox/node-pre-gyp thing, whatever's using that needs to be marked as external

LightwavesOP•7mo ago

tensorflow does

v•7mo ago

Rip

ballingt•7mo ago

ah hm, anything involving binaries like that shoudl be an external dependency, because you can't bundle binaries

LightwavesOP•7mo ago

but if I mark it as external it's apparently too big

ballingt•7mo ago

You may be out of luck for this library for now, hm I wonder if anyone else has used it

v•7mo ago

I thought that's what external was for Is this limit configurable with convex-backend

LightwavesOP•7mo ago

🥲 RIp to my hackathon project

v•7mo ago

@Tom

LightwavesOP•7mo ago

I'll try with the non-node package

v•7mo ago

Or would it have to be rebuilt. Not that is an issue but it would be easier.

ballingt•7mo ago

Oh there's a non-node package! Yeah use that Likely yes? @v check the code This is an issue with what we can deploy to AWS Lambda

v•7mo ago

Hmm okay

LightwavesOP•7mo ago

😔 well the non-node package works, obviously, unfortunately this make everything much slower lol. I know tf-node is a pain of a package though, because it require this gyp library and python. I just hoped it would work because the regular package is the opposite of performant

v•7mo ago

LightwavesOP•7mo ago

nice to be bale to use any tensorflow though

v•7mo ago

I wonder if the CLI also checks the package size too

ballingt•7mo ago

Shoot, yeah sounds like you're stuck deploying a container to deploy that code @Lightwaves I don't think the the external packages check doesn't happen locally in the CLI, I think we have to actually install the packages on AWS Lambda?

v•7mo ago

That would be better I just wonder how different the behavior is for the local node executor I'ma check

v•7mo ago

Sike

v•7mo ago

-_- Does GitHub oauth still work I wonder Yes

LightwavesOP•7mo ago

heh, github works fine for me :p

v•7mo ago

Lucky ahhh Haha nah ittle be back soon Yup back

v•7mo ago

https://github.com/get-convex/convex-backend/blob/d1bfe3ebd8707921baae6ac469e1bba61b0b8fde/crates/model/src/source_packages/types.rs#L69

GitHub

convex-backend/crates/model/src/source_packages/types.rs at d1bfe3e...

Open source single-machine version of the Convex backend - get-convex/convex-backend

ballingt•7mo ago

Ah ok, yeah is designed specificaly for AWS Lambda

v•7mo ago

i know but i wanted to see what would happen is that limit still here on selfhosting ? i can see later on linux im just messing around

v•7mo ago

my pc too dookie to search for scalars i guess. it takes ages

v•7mo ago

yeah i know nothing about lambda

Could no resolve "util"

Did you find this page helpful?