spider•2mo ago

trouble pushing

I think convex backend is down, anybody seeing network errors?

convex dev
✖ Error: Unable to start push to https://....convex.cloud
✖ Error fetching POST  https://j....convex.cloud/api/deploy2/start_push 500 Internal Server Error: InternalServerError: Your request couldn't be completed. Try again later.
Failed due to network error, retrying in 1.1s...

convex dev
✖ Error: Unable to start push to https://....convex.cloud
✖ Error fetching POST  https://j....convex.cloud/api/deploy2/start_push 500 Internal Server Error: InternalServerError: Your request couldn't be completed. Try again later.
Failed due to network error, retrying in 1.1s...

23 Replies

lee•2mo ago

i see an error on our side that indicates it might be having trouble parsing your source code. still not sure why

spiderOP•2mo ago

Thanks for the quick response.

lee•2mo ago

so far i have determined that esbuild is generating a sourcemap, probably for your http.ts file, that causes the sourcemap parsing library to throw this error https://github.com/getsentry/rust-sourcemap/blob/5243de7ce80a4fa146e50a3bba5c5514d3a770b7/src/errors.rs#L31 is there anything interesting about your http.ts file that could help repro? (i can't look at it without permission)

spiderOP•2mo ago

Not sure, lately i have only updated libraries. There's a hono setup, you can take a look.

nipunn•2mo ago

Can we have (temporary) permission to look at your source code @spider ? We wanna try to repro.

spiderOP•2mo ago

sure thing! I think the hono endpoints that use c cause issues

nipunn•2mo ago

do you have a github or something like that we can look at? Or alternately, DM me a zip or tar of the convex/ directory.

nipunn•2mo ago

we were chatting in DMs - but to loop back, we found that the ai library https://www.npmjs.com/package/ai - seems to hit some kind of sourcemap parsing issue starting at 4.0.9. 4.0.8 works. We'll keep looking into fixing, but that should help unblock in the meantime.

npm

AI SDK by Vercel - The AI Toolkit for TypeScript and JavaScript. Latest version: 4.1.47, last published: a day ago. Start using ai in your project by running npm i ai. There are 537 other projects in the npm registry using ai.

nipunn•2mo ago

we just pushed out a fix today to tolerate sourcemap parsing issues. We're still not sure why the sourcemaps aren't parsing, but at least your pushes won't 500 now (though they may lack sourcemaps). Somewhere between the esbuild bundling and the sourcemap rust crate for parsing.

stefano•5w ago

I am getting OP error too. I am on latest Convex version and even tried to downgrade the ai library to 4.0.9 but the error still persists. Is it possible that any other package is causing the error? @nipunn Note: I am trying to run my project on Convex self host. Originally it is hosted on cloud and it works. Can share the zip if you want

nipunn•5w ago

Hi. Note that the workaround requires downgrading to 4.0.8 (not 4.0.9) as mentioned above. The self hosted last released 8 days ago https://github.com/get-convex/convex-backend/pkgs/container/convex-backend The fix to tolerate sourcemap parsing issues landed 4 days ago - and it's out on the cloud product. https://github.com/get-convex/convex-backend/commit/5ba8eedb14791be5576f5567eedbf7479151381f Hang tight and we'll push out a new self hosted docker image in the next day or so that will contain the fix.

stefano•5w ago

Thanks! Will update if the problem persists 🙂

nipunn•5w ago

@stefano - just launched a new docker image today - about an hour ago. Try it out. It should allow you to use newer versions of ai library.

stefano•5w ago

Hi! Saw the new docker image and I tried it. However it seems like the problem does not originate from the ai package. As you can see from the video I've been able to narrow down the problem to something strange: 1. if I use "use node" in a file defining an action and I use that specific table export as argument for the action I get an error when executing npx convex dev 2. If I remove "use node" but still use that table export for the arguments, the npx convex dev will work. I have not changed anything inside the docker-compose.yml and I am using SQLITE I will DM you my convex/ folder Thanks!

nipunn•4w ago

taking a look now - will try to repro - but I'm curious if you see anything in the logs of the backend (in the terminal where you did the docker compose up ) I was unable to repro on either self hosting or the cloud product. There's clearly something more tricky going on than just pushing the code. If you see any errors in your local-backend's logs (in the terminal with docker-compose-up) that would be great to see. It'll have more details on what the InternalServerError was and then we can go from there. Alternately, if you can repro on the cloud product, then I'll have more info on our side to debug it. So that's an option too. To repro I tried

docker compose up

docker compose up

and

# unzip stefano's zip into convex/
npm init
npm install convex
# install a bunch of other dependencies
# export CONVEX_SELF_HOSTED_URL and CONVEX_SELF_HOSTED_ADMIN_KEY
npx convex dev
# set the CLERK_JWT_ISSUER_DOMAIN to a placeholder value

# unzip stefano's zip into convex/
npm init
npm install convex
# install a bunch of other dependencies
# export CONVEX_SELF_HOSTED_URL and CONVEX_SELF_HOSTED_ADMIN_KEY
npx convex dev
# set the CLERK_JWT_ISSUER_DOMAIN to a placeholder value

it's a generic 500 - it may be totally unrelated to the OP's - the details will be in the logs

stefano•4w ago

Well, the strange thing is that I can't repro in the cloud product... Anyway, here are the local-backend's logs

backend-1    | 2025-03-08T09:04:43.495180Z DEBUG metrics::timer: 2.404542707s for timer "node_executor_total_seconds" {MetricLabel { key: "method", value: "analyze" }, MetricLabel { key: "status", value: "error" }}
backend-1    | 2025-03-08T09:04:43.498280Z ERROR common::errors: Caught error (RUST_BACKTRACE=1 RUST_LOG=info,common::errors=debug for full trace): Hit an error while pushing:\nEOF while parsing a value at line 1 column 65536: EOF while parsing a value at line 1 column 65536
backend-1    | 2025-03-08T09:04:43.498310Z DEBUG common::errors: Hit an error while pushing:
backend-1    | EOF while parsing a value at line 1 column 65536
backend-1    |
backend-1    | Caused by:
backend-1    |     EOF while parsing a value at line 1 column 65536
backend-1    |
backend-1    | Stack backtrace:
backend-1    |    0: <unknown>
backend-1    |    1: <unknown>
backend-1    |    2: <unknown>
backend-1    |    3: <unknown>
backend-1    |    4: <unknown>
backend-1    |    5: <unknown>
backend-1    |    6: <unknown>
backend-1    |    7: <unknown>
backend-1    |    8: <unknown>
backend-1    |    9: <unknown>
backend-1    |   10: <unknown>
backend-1    |   11: <unknown>
backend-1    |   12: <unknown>

backend-1    | 2025-03-08T09:04:43.495180Z DEBUG metrics::timer: 2.404542707s for timer "node_executor_total_seconds" {MetricLabel { key: "method", value: "analyze" }, MetricLabel { key: "status", value: "error" }}
backend-1    | 2025-03-08T09:04:43.498280Z ERROR common::errors: Caught error (RUST_BACKTRACE=1 RUST_LOG=info,common::errors=debug for full trace): Hit an error while pushing:\nEOF while parsing a value at line 1 column 65536: EOF while parsing a value at line 1 column 65536
backend-1    | 2025-03-08T09:04:43.498310Z DEBUG common::errors: Hit an error while pushing:
backend-1    | EOF while parsing a value at line 1 column 65536
backend-1    |
backend-1    | Caused by:
backend-1    |     EOF while parsing a value at line 1 column 65536
backend-1    |
backend-1    | Stack backtrace:
backend-1    |    0: <unknown>
backend-1    |    1: <unknown>
backend-1    |    2: <unknown>
backend-1    |    3: <unknown>
backend-1    |    4: <unknown>
backend-1    |    5: <unknown>
backend-1    |    6: <unknown>
backend-1    |    7: <unknown>
backend-1    |    8: <unknown>
backend-1    |    9: <unknown>
backend-1    |   10: <unknown>
backend-1    |   11: <unknown>
backend-1    |   12: <unknown>

lee•4w ago

Oh cool. Despite the lack of stacktrace, it's pretty clear what's going on: the "use node" modules are too big to send via command line to the node process. I believe @Emma is working on a general fix to elevate this limit

stefano•4w ago

Cool, so there isn't anything I can do on my end, right?

nipunn•4w ago

workaround might be to have less "use node" modules - but if you hang tight for a bit - we have a fix.

stefano•4w ago

got it!

Emma•4w ago

Should be fixed in the latest release! https://discord.com/channels/1019350475847499849/1216859935288201296/1348770561463160894

stefano•4w ago

Can confirm the fix. You are all amazing!

nipunn•4w ago

@spider - esbuild pushed out a fix https://github.com/evanw/esbuild/issues/4107 - which we rolled out with convex 1.20.0 or newer https://www.npmjs.com/package/convex/v/1.20.0 With that - sourcemaps should work with ai package too.

GitHub

Sourcemap of script containing the ai library produces invalid so...

Repro import { streamText } from "ai"; console.log(streamText); ai at version 4.1.50 esbuild at version 0.25.0 npx esbuild repro.js --bundle --sourcemap --outfile=dist/repro.js The genera...

npm

convex

Client for the Convex Cloud. Latest version: 1.20.0, last published: 3 days ago. Start using convex in your project by running npm i convex. There are 16 other projects in the npm registry using convex.

trouble pushing

Did you find this page helpful?