Signed URLs in a query
I want to generate signed urls for some images when I fetch them in a query and I cannot call an
action in a convex query, anyone have any other ideas or a better way to do it? (Note: using AWS S3 + Cloudfront and not convex storage)17 Replies
I cant really generate them ahead of time as I need them to expire every 24 hours. so to give you more context what i am trying to do is I have a bunch of services being provided on the website and every service has multiple pictures. every service is displayed as a card on my listings page which is a carousel of images and i need signed urls for those images. what would be the best approach in this case ?
You could have a cron job that runs every 12 hours to regenerate all the time-expiring URLs you'll need. Do you have to do this lazily (ie you want to wait for the request from the browser to generate them) or can you generate them ahead of time?
If you need to do this lazily, not generating the signed URLs until a user visits the page, then you'll need to fire off an action when the user hits the page to generate these signed URLs. If you can do this ahead of time you won't have to wait for the AWS API url signing for the images to show up.
I did think of a cron job but then I thought when the application scales let’s say 1000s of entries - it might be difficult to generate signed urls for that many entries ? Maybe I could be wrong here.
And I believe the second approach might be more efficient too as they are only generated on demand. That’s what I am thinking and I am not sure if you have anything else in mind as you guys have a lot of experience with scaling and such issues.
Thanks in advance. I really appreciate your input and having this conversation
Yeah if your average listing isn't viewed over a 24 hour period then it sounds less efficient. You probably still want to stored these signed URLs though, not generate them each time right? It seems inefficient to get new signed URLs every time someone visits a page they might visit dozens of times in a day?
You've got a few options. How are you building this site? Is it a website or a mobile app?
Oh, reading a bit more: generating a signed url is not a network request!
No, it doesn't cost anything. Generating a pre-signed URL is a purely client-side operation. There are no AWS costs associated with this and there is no network activity. The SDK you're using takes your current credentials, the bucket and key for your object, your method of choice (e.g. GET), an expiration time, optional HTTP headers, and calculates and signs a URL, all done locally. -https://stackoverflow.com/a/55232199/398212So you can do this in a query!
Stack Overflow
Does generating a pre-signed URL to an S3 object using the AWS SDK ...
Using the AWS SDK in any language, you can generate a pre-signed URL to an PRIVATE S3 object and then anyone who has the URL can use it download the object. This is explained here:
https://docs.aws.
As long as you can get the AWS client to run in a query.
If you can't (if the AWS SDK you need to use only works in a Node.js action, for example) then doing this is a cron absolutely makes sense, you can generate thousands in a single Convex action.
Oh that’s a good find. I totally forgot it’s not a network request
And you probably want to be able to call this action more than just every 12 hours, since ever time you add a new property you'd want to update it too
I think I did try generating signed urls in a query and it did throw an error on the import of crypto library in aws sdk when I checked if convex runtime supports crypto and I think it does which was weird to me but I probably did something else wrong. I think for now atleast I might try once again to generate urls in the query itself and if that doesn’t work I am going to do what you suggest - cron job for generating URLs
Ah that's annoying, there may be a version of the aws sdk that uses web crypto instead of Node.js crypto but if not then you may be stuck with Node.js actions.
Convex supports crypto, just not
import crypto. They have similar APIs but code that expects needs to be changed to use the other generally.that makes a lot more sense
now that I am reading the error again, is it solvable? i dont understand what exactly its saying though lol
No, not without a lot of work forking the library.
This probably needs to be run in Node.js.
i see. also one more question, i made the action, and required internal mutations and queries - and then I had to add my private key to convex dashboard. after that i tried to run the action and got this error. some googling led me to that the newlines are removed from the private key when pasted somewhere else - like in this case convex dashboard which puts everything in one line. do you have idea about this? (I tried putting newline characters but no luck)
Edit: phew fixed it😮💨 and it was super fast...thank you very much for your help today!