BakemonoHouse
BakemonoHouse2mo ago

Handling large video files from s3

Hi all, just wondering if there's a solution to handling large video files - I'm integrating a meeting bot system (like Recall.ai) that sends a s3 blob url that I need to save into convex before the url expires. Seems like I run into Connection lost while action was in flight error when the video is above certain size - any suggestions?
8 Replies
Convex Bot
Convex Bot2mo ago
Thanks for posting in <#1088161997662724167>. Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets. - Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.) - Use search.convex.dev to search Docs, Stack, and Discord all at once. - Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI. - Avoid tagging staff unless specifically instructed. Thank you!
ballingt
ballingt2mo ago
Are you streaming the file? await .blob() loads the entire file into memory, which could hit a memory limit. more info: https://stack.convex.dev/streaming-http-using-fetch it would be nice to have better errors than Connection lost while action was in flight, do you get a better error than this in the Convex dashboard?
BakemonoHouse
BakemonoHouseOP2mo ago
nope thats all im getting @Tom I am not streaming the file. I am receiving an s3 blob url from a third party system. I need to download this file and save it into convex filesystem before the url expires
ballingt
ballingt2mo ago
I suggest streaming it
BakemonoHouse
BakemonoHouseOP2mo ago
got it let me try with this is it possible to stream the file into convex file api?
mckoda09
mckoda095w ago
yeah, how can we stream files?
ballingt
ballingt5w ago
See https://docs.convex.dev/api/interfaces/server.StorageWriter#generateuploadurl about upload URLs, you can stream to that URL directly from the client or buffered through an action
BakemonoHouse
BakemonoHouseOP4w ago
nice ! thanks @Tom just an update - So for our case we need to stream this file buffered through the action. But seems like we are hitting the strict TLS related error (hard to find info on this bug)
Error
at node:internal/deps/undici/undici:12618:11
at processTicksAndRejections (node:internal/process/task_queues:95:5)
Error
at node:internal/deps/undici/undici:12618:11
at processTicksAndRejections (node:internal/process/task_queues:95:5)
Sending readableStream to client for digestion seems to be working for the case in the doc you shared, but I might be just confused on how to use node (I'm not really a JS developer)... currently still failing to stream file through the server into Convex files.

Did you find this page helpful?