loading protos

Hi team. im trying to load some protos into the convex backend but couldnt figure out how

// client.js
"use node"

import grpc from "@grpc/grpc-js"
import { loadSync } from "@grpc/proto-loader"

export class Client {
  constructor(address) {
    var PROTO_PATH = "./../../src/protos/client.proto"
    const packageDefinition = loadSync(PROTO_PATH, {
      keepCase: true,
      longs: String,
      enums: String,
      defaults: true,
      oneofs: true,
    })

// client.js
"use node"

import grpc from "@grpc/grpc-js"
import { loadSync } from "@grpc/proto-loader"

export class Client {
  constructor(address) {
    var PROTO_PATH = "./../../src/protos/client.proto"
    const packageDefinition = loadSync(PROTO_PATH, {
      keepCase: true,
      longs: String,
      enums: String,
      defaults: true,
      oneofs: true,
    })

my directory looks sth like

.
├── convex
│   ├── client
│   │   ├── client.js
├── src
│   ├── protos
│   │   └── client.proto

.
├── convex
│   ├── client
│   │   ├── client.js
├── src
│   ├── protos
│   │   └── client.proto

tried a few variations but keep running into

Uncaught Error: ENOENT: no such file or directory, open '../../src/protos/client.proto'

Uncaught Error: ENOENT: no such file or directory, open '../../src/protos/client.proto'

i also tried moving the protos/client.proto into the convex dir/ but got an error like

✘ [ERROR] No loader is configured for ".proto" files: convex/client/protos/client.proto

✘ [ERROR] No loader is configured for ".proto" files: convex/client/protos/client.proto

i'd apprecate any tips on how to debug the error. thanks!

8 Replies

winsoroaksOP•10mo ago

Error
    at openSync (node:fs:596:3)
    at readFileSync (node:fs:464:35)
    at fetch (bundledFunctions.js:7105:30)
    at load (bundledFunctions.js:7135:11)
    at loadSync (bundledFunctions.js:7145:19)
    at loadProtosWithOptionsSync (bundledFunctions.js:10612:31)
    at loadSync (bundledFunctions.js:11387:63)
    at getProof (bundledFunctions.js:19609:62)
    at handler (bundledFunctions.js:23494:24)
    at invokeFunction (bundledFunctions.js:1393:36) {
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: '/src/protos/client.proto'
}

Error
    at openSync (node:fs:596:3)
    at readFileSync (node:fs:464:35)
    at fetch (bundledFunctions.js:7105:30)
    at load (bundledFunctions.js:7135:11)
    at loadSync (bundledFunctions.js:7145:19)
    at loadProtosWithOptionsSync (bundledFunctions.js:10612:31)
    at loadSync (bundledFunctions.js:11387:63)
    at getProof (bundledFunctions.js:19609:62)
    at handler (bundledFunctions.js:23494:24)
    at invokeFunction (bundledFunctions.js:1393:36) {
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: '/src/protos/client.proto'
}

not sure if convex is doing some magic to the path 🤔 when i do

const loadedProto = loadSync('client.proto', {
  keepCase: true,
  longs: String,
  enums: String,
  defaults: true,
  oneofs: true,
  includeDirs: [`${__dirname}/../../src/protos],
});

const loadedProto = loadSync('client.proto', {
  keepCase: true,
  longs: String,
  enums: String,
  defaults: true,
  oneofs: true,
  includeDirs: [`${__dirname}/../../src/protos],
});

i get

path: '/var/task./../../src/protos/client.proto'

path: '/var/task./../../src/protos/client.proto'

erquhart•10mo ago

Convex functions don’t have file system access, they’re serverless functions And as you saw, non-js files can’t be imported. Maybe the team has a more creative solution, but my guess is any grpc use will need to happen in a separate service. I’ll also go ahead and ask: is it possible to use Convex without grpc to accomplish whatever you’re doing here?

ballingt•10mo ago

@winsoroaks Indeed the way Convex bundles code won't include a file just mentioned as a string in a JS/TS file. Is there a codegen-oriented approach here? So you generate JS/TS code which can be bundled?

winsoroaksOP•10mo ago

thanks, tom. i am not sure if i fully understand the question. it's not a codegen approach, as in it's always the same protos that i'd like to load to the server. is there an alternative i can try?

ballingt•10mo ago

I'm wondering if there's another way to consume these .proto files where instead of loading them at runtime you generate JavaScript from the .proto file. That sort if approach would definitely work. You do have some other options, it would be great to figure out the recommended way to work with proto specs in convex. "always the same protos" -> great, then codegen spunds reasonable?

ballingt•10mo ago

Just googling I see https://github.com/stephenh/ts-proto

GitHub

GitHub - stephenh/ts-proto: An idiomatic protobuf generator for Typ...

An idiomatic protobuf generator for TypeScript. Contribute to stephenh/ts-proto development by creating an account on GitHub.

ballingt•10mo ago

Convex currently bundles only .js, .jsx, .ts, .tsx, .wasm, and ither related files by following import statements in code. It does not follow filename strings, so it won't find this proto file. Let me know what you think, I've only used these sorts of codegen approaches, never the dynamically load style so I don't know if there's something I'm missing about what you're trying to do.

winsoroaksOP•10mo ago

got it. will check it out, thanks 🙂

loading protos

Did you find this page helpful?