RJ•2mo ago

Increased incidences of generic failure + warning

I've been seeing greater incidences from my Convex actions and HTTP actions in the last few days of failures with the message Your request couldn't be completed. Try again later. and the warning Client disconnected. The most curious part (to me, anyways) is that in all of these instances the actions/HTTP actions seem to actually be functioning just fine. Even if the action in question is reported as a failure, it nonetheless appears to have behaved as though it succeeded. What do those messages mean? Should I be concerned? It's not clear to me what the pattern is here, it seems fairly unpredictable (although I can reproduce them, sometimes).

35 Replies

Convex Bot•2mo ago

Thanks for posting in <#1088161997662724167>. Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets. - Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.) - Use search.convex.dev to search Docs, Stack, and Discord all at once. - Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI. - Avoid tagging staff unless specifically instructed. Thank you!

lee•2mo ago

hi! this is a feature i've been working on. for a couple days it was broken and that would have caused the error Your request couldn't be completed. Try again later. . More recently it has been fixed, and the correct new behavior is the Client disconnected. warning. the thing we're newly detecting is clients calling HTTP actions and then disconnecting. you should be able to repro it reliably if you have a long-running http action by closing the browser while it's loading (or ^C during a curl). The feature this helps support is Request.signal, with usage example here https://discord.com/channels/1019350475847499849/1322109618910007327/1322109618910007327

RJOP•2mo ago

Ok I see, interesting. I am currently monkey-patching Request.signal (here) in HTTP actions, so perhaps that has something to do with what I've been seeing? But it's still inconsistent. I can hit the same HTTP action/endpoint from the browser, get back a 200 response, and still see the Client disconnected warning. I can also hit that same endpoint, get back a 200 response, and not see the warning. Maybe I just don't under HTTP very well, but it would seem to me that in neither case was the client disconnected?

lee•2mo ago

interesting; it shouldn't happen if the http client is staying connected. in this http action, is the response streamed? is the request streamed? (monkey-patching Request.signal should not affect anything)

RJOP•2mo ago

In this case it's an HTTP action, and I don't believe there's any streaming going on (I'm certainly not trying to stream anything). I'm testing this through a Scalar API playground in the browser. Not sure what other details are relevant, but happy to share whatever else might be helpful

lee•2mo ago

streaming would happen if you return a Response with a body which is a ReadableStream. Or if you read the request.body incrementally (contrast with await request.json() which reads it all at once). I'm trying to repro now

RJOP•2mo ago

I am using an Effect library to build this endpoint (https://github.com/Effect-TS/effect/blob/main/packages/platform/README.md#http-api, it's like Hono), and I'm not familiar with exactly what it's doing under the hood (although I don't assume it's doing any streaming, by default). This is how you'd handle streaming requests/responses, per their documentation: https://github.com/Effect-TS/effect/blob/main/packages/platform/README.md#streaming-requests I'm not doing any of those things

lee•2mo ago

hmm i haven't been able to repro

RJOP•2mo ago

Just tested this with curl as well and consistently seeing Client disconnected in spite of 200 responses You could repro my usage of @effect/platform's HttpApi module usage by taking the example code from here and tweaking it slightly Egh I should do this myself, but I don't think I'll have time this week Anyways, if you want to try to repro it like above lmk and I can help you, or else I'll try to do so myself later But final thought: this is appearing as a Convex system warning in my Axiom logs, and I had a monitor set up to notify me should any of these appear As my understanding was that these would be things like "this Convex action is using a lot of memory" or "this Convex query is reading a lot of documents" If I'm understanding the situation correctly, this is different, in that it's something that may be triggered by a client under somewhat more arbitrary conditions Like the user closing the browser tab while an action is in flight Anyways, it doesn't feel to me like the same category of issue And so now my monitor needs to be further refined or re-configured to serve the same purpose for me Which was originally "notify me when I need to take a look at potentially problematic performance characteristics of my Convex functions" I guess this warning could be that also But it also could not be

lee•2mo ago

all good to know this isn't something i realized would happen. thanks for pointing out btw we only log the [WARN] line if you've already returned a Response and it's in the middle of streaming the response body maybe Effect is doing that internally? what would you think if i made this log line be [INFO] instead of [WARN] ? the idea of the log line is to let you know that no more log lines will be coming from the http action, because it has been cancelled. any tips for tweaking? i'm not familiar with Effect; trying to repro anyway

lee•2mo ago

i found this 🙂 https://rjdellecese.gitbook.io/confect/http-apis

HTTP APIs | Confect

RJOP•2mo ago

Replace the last two blocks

// Set up the server using NodeHttpServer on port 3000
const ServerLive = HttpApiBuilder.serve().pipe(
  Layer.provide(MyApiLive),
  Layer.provide(NodeHttpServer.layer(createServer, { port: 3000 }))
)

// Launch the server
Layer.launch(ServerLive).pipe(NodeRuntime.runMain)

// Set up the server using NodeHttpServer on port 3000
const ServerLive = HttpApiBuilder.serve().pipe(
  Layer.provide(MyApiLive),
  Layer.provide(NodeHttpServer.layer(createServer, { port: 3000 }))
)

// Launch the server
Layer.launch(ServerLive).pipe(NodeRuntime.runMain)

with something like:

    const EnvLive = Layer.mergeAll(
      MyApiLive,
      HttpServer.layerContext,
    );

    const { handler } = HttpApiBuilder.toWebHandler(EnvLive);

    return handler(request);

    const EnvLive = Layer.mergeAll(
      MyApiLive,
      HttpServer.layerContext,
    );

    const { handler } = HttpApiBuilder.toWebHandler(EnvLive);

    return handler(request);

The above ☝️ is basically what's under the hood here: https://github.com/rjdellecese/confect/blob/8804fc1f3dd3cc6ea399749ac049293b0f4a6c52/src/server/http.ts#L67-L75 The tricky part is mostly getting a Layer with the right dependencies And HttpApiBuilder.toWebHandler just gets you a function that looks like (request: Request) => Promise<Response> (So once you have that function you can just hook it up to an HTTP action like normal) Just make sure it's at the correct route (/hello-word in the example)

lee•2mo ago

cool i got a repro!

RJOP•2mo ago

I don't fully understand the context(s) in which this log message is useful, but for my purposes that would definitely be helpful as I could use that distinction to filter it out from higher-priority signals (in the context of the log feed). Another note. The log entry I'm getting looks like this:

{
  "data": {
    "function": {
      "cached": false,
      "path": "GET /xxx",
      "request_id": "xxx",
      "type": "http_action"
    },
    "is_truncated": false,
    "log_level": "WARN",
    "message": "Client disconnected",
    "system_code": "error:httpAction",
    "timestamp": 1742330863080,
    "topic": "console"
  }
}

{
  "data": {
    "function": {
      "cached": false,
      "path": "GET /xxx",
      "request_id": "xxx",
      "type": "http_action"
    },
    "is_truncated": false,
    "log_level": "WARN",
    "message": "Client disconnected",
    "system_code": "error:httpAction",
    "timestamp": 1742330863080,
    "topic": "console"
  }
}

Maybe system_code should be something like warning:httpAction (or info:httpAction) instead? I'm not sure what the possible values here are, though Hooray!!

lee•2mo ago

i can also modify that system code to be anything

RJOP•2mo ago

Thanks a lot for looking into this Lee, I gotta go now but lmk if there's anything else I can do to help

lee•2mo ago

frickin cors making it hard to test

RJOP•2mo ago

ah (I lied I'm still here) you can use some middleware to make that easier one sec

lee•2mo ago

i have a repro with curl so it's probably fine i have also determined it happens with both http/2 and http1.1

RJOP•2mo ago

const { handler } = HttpApiBuilder.toWebHandler(EnvLive, { middleware: HttpMiddleware.cors() });

const { handler } = HttpApiBuilder.toWebHandler(EnvLive, { middleware: HttpMiddleware.cors() });

and then also connect that to the OPTIONS endpoint Oops, HttpMiddleware.cors() over HttpMiddleware.middlewareCors() anyways, just in case that's helpful

lee•2mo ago

is there a way to return a string without quotes?

$ curl -i --http1.1 'https://giant-firefly-602.convex.site/' 
HTTP/1.1 200 OK
Content-Length: 15
Content-Type: application/json
Date: Wed, 19 Mar 2025 01:03:32 GMT
X-Convex-Usher: 1

"Hello, World!"%

$ curl -i --http1.1 'https://giant-firefly-602.convex.site/' 
HTTP/1.1 200 OK
Content-Length: 15
Content-Type: application/json
Date: Wed, 19 Mar 2025 01:03:32 GMT
X-Convex-Usher: 1

"Hello, World!"%

RJOP•2mo ago

I think you would change the response encoding: https://github.com/Effect-TS/effect/blob/main/packages/platform/README.md#changing-the-response-encoding I'm not positive though

lee•2mo ago

i'm trying to repro against our local setup, where all of the prod services are running locally on my laptop. and... it doesn't repro which makes it much harder to debug 😦 i've been looking through the effect codebase but have found nothing now i have a repro that doesn't use effect at all, so that narrows it down. although it still only happens in prod, not with locally running services

RJOP•2mo ago

Hmm, that's annoying! I'm curious what you isolated as the cause?

lee•2mo ago

still haven't isolated the cause, unfortunately. the "effect" library makes it more frequent, but even the simplest http action return new Response("hello world") repros it occasionally. 😕

RJOP•2mo ago

Oh no lol

lee•2mo ago

i found the issue. will fix imminently, but in the meantime you can try to work around it by using a streamed response. the issue only happens if there's a 'Content-Length' response header

RJOP•2mo ago

Awesome! I can live with the erroneous logs for a little longer as long as it's going to be fixed. Glad you found the root cause!

jamwt•2mo ago

nice work @Lee !!

lee•2mo ago

The fix has deployed, i believe

lee•2mo ago

https://github.com/get-convex/convex-backend/commit/d6606d44fcf3028369e16aec95f86d9907445880

GitHub

fix http action warning (#35663) · get-convex/convex-backend@d6606d4

See here: https://github.com/hyperium/hyper/blob/68bae2b53f8db4b0440e49b7e151e7a99229146c/src/proto/h1/decode.rs#L153 If an http action returns a Content-Length header, hyper/axum will stop pollin...

RJOP•2mo ago

I don't seem to be able to trigger this warning erroneously anymore Thanks again Lee!

lee•2mo ago

hmm there is another case where it could happen, if Content-Length is 0. putting up a fix for that too thank you for reporting! and for helping get a repro btw there's a reason i didn't feel comfortable just reverting my original change: the old behavior in this case was that the HTTP action wouldn't show up in the execution log at all (whether they succeeded or failed). My original change just made such issues louder.

RJOP•2mo ago

Makes sense, definitely more important to have it show up at all!

justin•2w ago

Hi, I'm seeing something similar... I have a route

http.route({
  path: "/testUpload",
  method: "POST",
  handler: httpAction(async (ctx, request) => {
    console.log("testUpload request", request.headers);
    return new Response(JSON.stringify({ hello: "world" }), { status: 200 });
  }),
});

http.route({
  path: "/testUpload",
  method: "POST",
  handler: httpAction(async (ctx, request) => {
    console.log("testUpload request", request.headers);
    return new Response(JSON.stringify({ hello: "world" }), { status: 200 });
  }),
});

and I'm exercising it with either 32kB or 64kB of zeros:

curl -i 'https://tremendous-dodo-311.convex.site/testUpload' -X 'POST' --data-binary @32kB.bin -H "Content-Type: application/octet-stream"

curl -i 'https://tremendous-dodo-311.convex.site/testUpload' -X 'POST' --data-binary @32kB.bin -H "Content-Type: application/octet-stream"

In both cases I get the expected response, but the 64kB also shows a 500 in the logs with "Your request couldn't be completed. Try again later." In both cases output from curl is

HTTP/2 200 
content-type: text/plain;charset=UTF-8
date: Thu, 17 Apr 2025 22:07:47 GMT
x-convex-usher: 1
content-length: 17

{"hello":"world"}%

HTTP/2 200 
content-type: text/plain;charset=UTF-8
date: Thu, 17 Apr 2025 22:07:47 GMT
x-convex-usher: 1
content-length: 17

{"hello":"world"}%

Convex logs

4/17/2025, 3:25:28 PM [CONVEX H(POST /testUpload)] [LOG] 'testUpload request' Headers { host: 'tremendous-dodo-311.convex.site', 'user-agent': 'curl/8.7.1', 'content-length': '32768', accept: '*/*', 'content-type': 'application/octet-stream', 'x-forwarded-for': '23.93.82.19', 'x-forwarded-host': 'tremendous-dodo-311.convex.site', 'x-forwarded-proto': 'https', 'accept-encoding': 'gzip', 'convex-request-id': '2a7295d5e92b3420' }
4/17/2025, 3:25:39 PM [CONVEX H(POST /testUpload)] [LOG] 'testUpload request' Headers { host: 'tremendous-dodo-311.convex.site', 'user-agent': 'curl/8.7.1', 'content-length': '65536', accept: '*/*', 'content-type': 'application/octet-stream', 'x-forwarded-for': '23.93.82.19', 'x-forwarded-host': 'tremendous-dodo-311.convex.site', 'x-forwarded-proto': 'https', 'accept-encoding': 'gzip', 'convex-request-id': '44e18bed7a191bd8' }
4/17/2025, 3:25:39 PM [CONVEX H(POST /testUpload)] Your request couldn't be completed. Try again later.

4/17/2025, 3:25:28 PM [CONVEX H(POST /testUpload)] [LOG] 'testUpload request' Headers { host: 'tremendous-dodo-311.convex.site', 'user-agent': 'curl/8.7.1', 'content-length': '32768', accept: '*/*', 'content-type': 'application/octet-stream', 'x-forwarded-for': '23.93.82.19', 'x-forwarded-host': 'tremendous-dodo-311.convex.site', 'x-forwarded-proto': 'https', 'accept-encoding': 'gzip', 'convex-request-id': '2a7295d5e92b3420' }
4/17/2025, 3:25:39 PM [CONVEX H(POST /testUpload)] [LOG] 'testUpload request' Headers { host: 'tremendous-dodo-311.convex.site', 'user-agent': 'curl/8.7.1', 'content-length': '65536', accept: '*/*', 'content-type': 'application/octet-stream', 'x-forwarded-for': '23.93.82.19', 'x-forwarded-host': 'tremendous-dodo-311.convex.site', 'x-forwarded-proto': 'https', 'accept-encoding': 'gzip', 'convex-request-id': '44e18bed7a191bd8' }
4/17/2025, 3:25:39 PM [CONVEX H(POST /testUpload)] Your request couldn't be completed. Try again later.

Increased incidences of generic failure + warning

Did you find this page helpful?