For the S3 thing, yes it's effectively implementing an interface/trait. For the scalability thing, t
For the S3 thing, yes it's effectively implementing an interface/trait. For the scalability thing, the code changes are similarly straightforward, but the tricky part is spinning up parts of the codebase (the parts that run functions or connect websockets) on separate servers and making sure they can talk to each other.
18 Replies
@Jamie, I am assuming it's SQLite as I have not supplied any specific DB configuration.
@Benanna19 Self-hosted Convex now supports streaming import so it's easy to set up Convex with your existing database! https://github.com/get-convex/convex-backend/blob/main/self-hosted/CHANGELOG.md
@Austin We also fixed a bug in HTTP actions routing
GitHub
convex-backend/self-hosted/CHANGELOG.md at main · get-convex/convex...
The open-source reactive database for app developers - get-convex/convex-backend
Wow! That was incredibly fast!
yep, shoutout to @nipunn for getting it done!
Fantastic, are there plans to include streaming
export for self-hosting? And not using Fivetran/Airbyte?Is the open-source version of the Convex backend open to suggestions or contributions? Can it deviate from the Convex Cloud backend?
For example, I noticed that you open-sourced the
aws_s3 and aws_utils crates, and I was able to use Cloudflare R2 as storage with minimal changes. Is it something you would be open to add to the repo?@James Cowling ^
It's just amazing, I've almost finished my implementation on the "privacy RAG" project I'm working on. Since I'm using convex most of the implementation is just plumbing things together. I'm 8h in and I'm almost done 🎉 , topics with file uploads, chats with LLM a nice user interface based on shadcn/ui and everything JUST WORKS 🎉 .
THANKS A LOT for this awesome project! If you guys ever need a helping hand, I got you 🙋♂️
We're going to keep the open source version and the convex cloud version the same, since apart from the extra stuff we need to run the hosted product it's the same code.
A few thoughts about contributions:
* if there are any bug fixes or tweaks etc we'd love to receive them and will upstream them into the hosted product
* if there are feature requests we always want to receive those too
* something that's a strategic shift like moving our storage to R2 that's likely not a PR we'd merge in, since it'll complicate us managing the hosted product
* will chat internally with folks about whether we might want to leverage R2 for the hosted product. would love to lean in to your suggestions/needs here
* of course you're always welcome to fork, although i'd love if we can keep people happy on the main repo as much as we can!
* we're hoping a lot of Convex development will happen on Convex Components rather than having to modify the codebase itself, since that way everyone can benefit from the changes, even on the hosted version. specifically for R2 we have a ready-made component that you might want to try! https://github.com/get-convex/r2
(this is for using r2 for file storage, which might be a different use-case than you're looking for)
GitHub
GitHub - get-convex/r2: Convex component for storing and serving fi...
Convex component for storing and serving files with Cloudflare R2 - get-convex/r2
Got it! R2 is supposed to be S3 compatible but it is not 100%. So I understand it is tricky for you to fix and merge things that are not broken in the Cloud version, especially if you don't plan to use it in prod.
Right now it was just a matter of adding a new env variable (s3 endpoint) and fixing unsupported api get_object_attributes but who knows how many other differences with AWS S3.
(I am talking about using R2 on the rust side to store convex metadata, files, modules,indexes)
@Emma do you think it might be relatively low-lift for our code to work when passing in either an S3 bucket or R2 endpoint?
Hi @stevanfreeborn great video on self-host Convex. Can you share the video in the #show-and-tell channel? Thanks
Stevan Freeborn
YouTube
How I Self-Host Convex
Blog Post: https://stack.convex.dev/self-hosted-develop-and-deploy
Convex announced last week their new and improved self-hosting option. I was super excited to hear about it. Up until now I've only used their cloud offering and I've been hesitant to do more with it because I didn't want to get stuck on someone else's cloud for a side project. ...
For sure!
Thanks!
Yep! Support for S3 just landed, will get out a new docker image today with these changes https://github.com/get-convex/convex-backend/commit/161e32648a971fb8ef591e61212f7b9fb7ff4f2c
GitHub
Support S3 storage for self-hosted backend (#34680) · get-convex/co...
Add S3 support for self-hosted-backend. Includes refactor to reduce code duplication for initializing storage, moving file, search, export, import storages to
ApplicationStorage struct that gets ...Released a new convex-backend docker image that supports S3! Notes here https://github.com/get-convex/convex-backend/blob/main/self-hosted/CHANGELOG.md and docs here https://github.com/get-convex/convex-backend/blob/main/self-hosted/README.md#using-s3-storage
GitHub
convex-backend/self-hosted/CHANGELOG.md at main · get-convex/convex...
The open-source reactive database for app developers - get-convex/convex-backend
GitHub
convex-backend/self-hosted/README.md at main · get-convex/convex-ba...
The open-source reactive database for app developers - get-convex/convex-backend
R2 support sounds like a great addition, we'd be happy to take a contribution! I took a look, and you're right, it looks like adding a new env variable for s3 endpoint and adding a get_object_attributes implementation. I think the head object API should be sufficient. Would you be interested in implementing this @Spioune ? Happy to support you along the way!
Sure. Should I make a PR?