Runtime exited unexpectedly despite all Promises awaited

I keep getting this error

Runtime exited unexpectedly. This typically happens due to error raised from dangling promises. Did you forget to await your promises? RequestId: 6579ea81-b938-4cc6-8a87-349cee46ffed Error: Runtime exited with error: signal: killed

Runtime exited unexpectedly. This typically happens due to error raised from dangling promises. Did you forget to await your promises? RequestId: 6579ea81-b938-4cc6-8a87-349cee46ffed Error: Runtime exited with error: signal: killed

But I have checked the code and I did await all promises. Is there a way to debug to see where it failed at?

20 Replies

milk enjoyerOP•12mo ago

After some testing, it seems like it MIGHT be related to fetching a large file and uploading it to the store. What are the limits when fetching then uploading a file (using convex storage) from node?

james•12mo ago

Hi @milk enjoyer some storage limits are documented at https://docs.convex.dev/file-storage/upload-files#limits and https://docs.convex.dev/production/state/limits#file-storage. In particular request size limits within an action are limited at 20MB and for POST-style uploads we don't have a specific file limit but the request will time out after 2 minutes. If you're hitting this error only on larger files then that is likely the issue. Some of our limits aren't documented as clearly as they could be and I think that "2 minute timeout" is probably a bit too underspecified for developers to reason about. Would love to hear what size files you're looking to work with and what specific circumstances you're using them. Is this running a node action, fetching some content from elsewhere on the web via fetch, and then writing it directly to Convex storage?

Limits | Convex Developer Hub

We’d love for you to have unlimited joy building on Convex but engineering

Uploading and Storing Files | Convex Developer Hub

Files can be uploaded by your users and stored in Convex.

milk enjoyerOP•12mo ago

yes, that's what i'm doing. so if the limit is 20MB that definitely would not work because some of these files are long videos that might exceed it.

milk enjoyerOP•12mo ago

milk enjoyerOP•12mo ago

but if the request max size is 20mb, how would we upload a 50gb file in this limit here? only via frontend?

ian•12mo ago

If you're not passing the file from one action to another, the 20MB doesn't come into play. It's possible to stream a network request - like a fetch - or an upload to an http action, directly into storage. If you're downloading the whole file before storing it, then you are bloating the runtime memory. Some syntax that does the streaming:

const response = await fetch(imageUrl);
const image = await response.blob();
// Store the image in Convex
const storageId: Id<"_storage"> = await ctx.storage.store(image);

const response = await fetch(imageUrl);
const image = await response.blob();
// Store the image in Convex
const storageId: Id<"_storage"> = await ctx.storage.store(image);

That works in a non-node action anyways. I'm assuming a node action has a similar implementation of blob streaming

milk enjoyerOP•12mo ago

This works for me in node:

const buffer = await response.arrayBuffer();
    const contentType =
      response.headers.get("Content-Type") || "application/octet-stream";

    const blob = new Blob([buffer], { type: contentType });

    const storageId = await ctx.storage.store(blob);

const buffer = await response.arrayBuffer();
    const contentType =
      response.headers.get("Content-Type") || "application/octet-stream";

    const blob = new Blob([buffer], { type: contentType });

    const storageId = await ctx.storage.store(blob);

so this should work? in that case fetching and streaming large files is not the cause of the error, am i right to say that?

ian•11mo ago

Your code seems to be getting the full buffer before creating the blob (I believe awaiting the arrayBuffer will download the full file). Does const blob = await response.blob() not work? It should get the content type and stream the buffer automatically. And is there a reason you're using a node action here - is it a library you're relying on? @milk enjoyer did you eventually get this working?

milk enjoyerOP•11mo ago

actually it's still not quite working. but I am not sure why. Same error:

Runtime exited unexpectedly. This typically happens due to error raised from dangling promises. Did you forget to await your promises? RequestId: ded0c2eb-ff74-4649-8e46-bb85cd4134c3 Error: Runtime exited with error: signal: killed

Runtime exited unexpectedly. This typically happens due to error raised from dangling promises. Did you forget to await your promises? RequestId: ded0c2eb-ff74-4649-8e46-bb85cd4134c3 Error: Runtime exited with error: signal: killed

After extensive testing I can confirm that these lines are causing the error:

  const response = await fetchWithOfAccount(source, ofSessionString);
    const blob = await response.blob();
    const storageId = await ctx.storage.store(blob);

  const response = await fetchWithOfAccount(source, ofSessionString);
    const blob = await response.blob();
    const storageId = await ctx.storage.store(blob);

I tried removing the ctx.storage line and i am getting a new error:

2/9/2024, 6:42:56 PM
0ms
failure
Action
ofApi/vault/actions:getVaultMediaItemsAction
failure
Transient error while executing action

2/9/2024, 6:42:56 PM
0ms
failure
Action
ofApi/vault/actions:getVaultMediaItemsAction
failure
Transient error while executing action

const blob = await response.blob();

const blob = await response.blob();

It seems like this line is causing at least some of the problems, but only for particular images. it's quite strange

milk enjoyerOP•11mo ago

something very weird is why it takes such a long time to fail

ian•11mo ago

Dangling promises are tricky bc the error can be from a previous node action. Each one can be causing the issue for the next. I’d check the fetchWithOfAccount and other previous code first

milk enjoyerOP•11mo ago

that's guaranteed not the problem, because i have many other functions using fetchWithOfAccount without this issue. also when i remove these lines it disappears:

const blob = await response.blob();
    const storageId = await ctx.storage.store(blob);

const blob = await response.blob();
    const storageId = await ctx.storage.store(blob);

when i exclude these 2 lines, all the dangling promise errors disappear

ian•11mo ago

Gotcha - sorry for all the back & forth. We'll dig into this tomorrow. My guess is it's running out of memory if it's downloading & uploading without garbage collecting the file. Can you provide some file sizes for files that are working, and confirm that this works with a smaller file?

milk enjoyerOP•11mo ago

It seems to have nothing to do with filesize at all. I managed to download much bigger files with no issues. It might be the problem with the server I am fetching from, but I hope for two things: 1. better errors 2. it should not cause the entire action and all dependent actions to fail

ian•11mo ago

That makes sense, we'll see what we can do. Have you tried putting a try/catch around it and seeing what error (if any) is thrown? If it's some server I wonder if it's downloading at a different rate. To confirm, you've tried from that server with a small (<10MB) file? To rule out it being that server + big files, where another server + big files might work?

milk enjoyerOP•11mo ago

yes i tried try catch, it doesnt stop the problem the issue is that the fetching doesnt error but takes a super long time to terminate, affecting the rest of the actions within the action to prevent a "stray" .blob() from affecting the other tasks, i now have to isolate every single instance that fetches and uses .blob() to it's own action so even if it fails it doesnt affect other tasks i can confirm. i have managed to download all kinds of files successfully from kbs to 300+mb files no problem while some of the files in question that throw error are only like 1mb or less.

ian•11mo ago

That's frustrating, sorry you're hitting this. We're working on some changes to make it so you can catch an error in more places, e.g. other actions calling this action will always be able to try/catch if the action fails. We can also try to figure out what the underlying issue is and why the error messages aren't more helpful. Is there a URL you could DM me that I can repro with?

milk enjoyerOP•11mo ago

I can add you to my repo on GitHub if you want, and the dev instance is https://handsome-hummingbird-567.convex.cloud

ballingt•11mo ago

@milk enjoyer re this only being an issue for particular images, would you be able to send in one of the files this doesn't work for (if it's indeed reproducible with specific ones) @milk enjoyer I'd be interested in seeing this code, you could send a zipfile to tom@convex.dev or add GitHub user thomasballinger to the repo.

milk enjoyerOP•11mo ago

I can't get the images because it is somewhat IP restricted and only accessible via the server (it is almost impossible for humans to navigate). But I can share the code. Let me tidy up some things first and I will send it over.

Runtime exited unexpectedly despite all Promises awaited

Did you find this page helpful?