tstepro•2y ago

Dealing with files

Hey there, this is what I'm trying to do. 1. Upload file to convex. 2. Upload that file to somewhere else (from convex). 3. .... 4. Profit I'm running into problems with 2. I'm passing my stored blob to a node env where I have the sdk installed. I am then trying to convert that Blob into a File object. That's needed for the sdk. I keep running into the error:

Blob {"_parts":[{"_data":{"0":.... is not a supported Convex type To learn about Convex's supported types, see https://docs.convex.dev/using/types.

From the rest of the trace it seems like the Blob is a special type in convex? And it's converted into a json object? Not entirely sure. For context, this is the api (https://platform.openai.com/docs/api-reference/audio/create-transcription) I'm trying to upload a blob to. I can probably work around this by manually calling this. But would rather use their sdk. Any thoughts on how to make this happen?

OpenAI Platform

Explore developer resources, tutorials, API docs, and dynamic examples to get the most out of OpenAI's platform.

9 Replies

nipunn•2y ago

Hi! Sounds pretty neat The issue you're running into is that the full contents of a file (Blob) is not valid as an argument or result (input/output) to a Convex function. To do what you're doing, you have to do the entire operation within a Convex action. From the action context, you should be able to get the file (storage.get) and then take that blob and use it with the open AI api you're aiming to use - all within the same action.

nipunn•2y ago

This stack post (https://stack.convex.dev/using-dall-e-from-convex) has an example of calling open AI's dalle and then uploading the result to Convex Sounds like you want the sort of reverse - downloading the file from convex and uploading it to open AI, but it should be pretty similar in what you have to build.

Using Dall-E from Convex

Use Convex to fetch an image from OpenAI’s image generation service based on a user-provided prompt.

ian•2y ago

In this case you’d just need to pass along the storageId and RE-fetch from the action. You can also turn a Blob into an AreayBuffer (the v.bytes() convex type) to send to the function. I believe with blob.arrayBuffer() may work. Downloading it in the action is probably the move, as there’s a limit to total argument size & bytes length, similar to Documents

tsteproOP•2y ago

Do you guys have any examples of uploading a file to an external service from within an action (using a blob)? A lot of the examples I run across use fetch (and dom apis like "File" and "FormData").

ballingt•2y ago

Do you want to use the node.js sdk library or fetch? If the library, this looks relevant: https://github.com/openai/openai-node/issues/77#issuecomment-1455247809

GitHub

[Whisper] cannot call createTranscription function from Node.js d...

Describe the bug Cannot call createTranscription function like below: ... const audio = await fs.readFile('path/to/audio.mp4'); // Compile Error at the first argument const response = await...

ballingt•2y ago

Sounds like you can write the file to disk and use that or get the bytes in memory using the linked comment

tsteproOP•2y ago

Thanks for the link. I was just trying to see an example of a fetch. Their workaround didn't seem to be working in a typescript environment. I was able to figure it out, just the nuance of fetch and formdata tripped me up.

ballingt•2y ago

Nice. If you don't mind, pasting a trimmed down version of the fetch command you got to work here it might help folks in the future.

ian•2y ago

Here's some code from the Dall-E demo before we had built-in storage access in actions:

// Download the image
  const imageResponse = await fetch(dallEImageUrl);
  if (!imageResponse.ok) {
    throw new Error(`failed to download: ${imageResponse.statusText}`);
  }
  const image = Buffer.from(await imageResponse.arrayBuffer());
  // Create a Convex url to upload the image to.
  const postUrl = await mutation("sendMessage:generateUploadUrl");
  // Upload the image to Convex storage.
  const postImageResponse = await fetch(postUrl, {
    method: "POST",
    headers: { "Content-Type": imageResponse.headers.get("content-type") },
    body: image,
  });

// Download the image
  const imageResponse = await fetch(dallEImageUrl);
  if (!imageResponse.ok) {
    throw new Error(`failed to download: ${imageResponse.statusText}`);
  }
  const image = Buffer.from(await imageResponse.arrayBuffer());
  // Create a Convex url to upload the image to.
  const postUrl = await mutation("sendMessage:generateUploadUrl");
  // Upload the image to Convex storage.
  const postImageResponse = await fetch(postUrl, {
    method: "POST",
    headers: { "Content-Type": imageResponse.headers.get("content-type") },
    body: image,
  });

in case it's helpful to anyone else . the critical bit is const image = Buffer.from(await imageResponse.arrayBuffer());

Dealing with files

Did you find this page helpful?