I am new to Convex and would like to
I am new to Convex and would like to host it myself. I want to build an application that revolves around file uploads. I assume that the system will have many terabytes of data in the medium term.
With Supabase, you can use S3/MinIo as file storage and thus scale horizontally (with multiple S3/Minio instances).
So my questions are:
* Is there also a way to scale file storage horizontally with Convex (self-hosted)?
* And in general, I would be interested to know how file storage works in the background?
Thank you for your help.
4 Replies
Hi - you can just use the Convex file storage. With self-hosting - it defaults to storing files on the local disk, but you can configure it to work with S3 (or something S3 compatible like R2). https://github.com/get-convex/convex-backend/tree/main/self-hosted#using-s3-storage
I don't totally understand what you mean by "horizontally scaling S3". S3 (and R2 and Convex file storage and Supabase and Minlo) hide this concept behind the abstraction. You can just make calls to put and get with relatively high QPS and not think about how many machines.
I'd probably recommend just using the APIs in a simple way - and not worrying about these scaling problems until you have an actual issue in practice. S3 routinely deals with terabytes of data - it's not a big number for them.
Just give it a shot and get something working!
GitHub
convex-backend/self-hosted at main · get-convex/convex-backend
The open-source reactive database for app developers - get-convex/convex-backend
Hi @Nipunn, thank you for your quick answer and the link for S3 configuration.
I will test this.
Regarding your question:
I don't totally understand what you mean by "horizontally scaling S3"I am thinking about hosting several MiniIO servers myself and then connecting them together. Based on this concept: https://blog.min.io/minio-replication-best-practices/ MinIO supports horizontal scaling in this way to combine storage space from different servers into one large pool.
S3 just hides this whole thing behind the abstraction. IMO # of servers isn't a thing you should have to think about for file storage.
Great, thank you @Nipunn