Handling large video files from s3
Hi all, just wondering if there's a solution to handling large video files -
I'm integrating a meeting bot system (like Recall.ai) that sends a s3 blob url that I need to save into convex before the url expires. Seems like I run into
Connection lost while action was in flight error when the video is above certain size - any suggestions?8 Replies
Thanks for posting in <#1088161997662724167>.
Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets.
- Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.)
- Use search.convex.dev to search Docs, Stack, and Discord all at once.
- Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI.
- Avoid tagging staff unless specifically instructed.
Thank you!
Are you streaming the file?
await .blob() loads the entire file into memory, which could hit a memory limit. more info: https://stack.convex.dev/streaming-http-using-fetch
it would be nice to have better errors than Connection lost while action was in flight, do you get a better error than this in the Convex dashboard?nope thats all im getting @Tom
I am not streaming the file. I am receiving an s3 blob url from a third party system. I need to download this file and save it into convex filesystem before the url expires
I suggest streaming it
got it let me try with this
is it possible to stream the file into convex file api?
yeah, how can we stream files?
See https://docs.convex.dev/api/interfaces/server.StorageWriter#generateuploadurl about upload URLs, you can stream to that URL directly from the client or buffered through an action
nice ! thanks
@Tom just an update -
So for our case we need to stream this file buffered through the action. But seems like we are hitting the strict TLS related error (hard to find info on this bug)
Sending readableStream to client for digestion seems to be working for the case in the doc you shared, but I might be just confused on how to use node (I'm not really a JS developer)... currently still failing to stream file through the server into Convex files.