llama farm
The intrepid @ian recently created a pretty cool project that allows a bunch of random computers (your laptop, fly.io machines, whatever) running ollama3 to collectively run all the LLM work for a chat application.
All this work is coordinated using Convex as a kind of job queue. This allows the LLM cluster to be scaled up, and nothing needs to expose ports to the Internet to participate.
Pretty neat idea!
Tweet: https://twitter.com/ianmacartney/status/1787663174394876000 (follow @ian ! )
Hosted: https://labs.convex.dev/llama-farm
Teh code: https://github.com/get-convex/llama-farm-chat
Ian Macartney (@ianmacartney) on X
Live demo of llama farm:
https://t.co/0aeOvbg6xj
- By default it's just you & 🦙
- Share the URL for others to join in.
- Streams to all users at once.
I'll add logins, rate limit, etc. if it gets traffic.
A couple 🦙's on @flydotio too: details in repo
Twitter
GitHub
GitHub - get-convex/llama-farm-chat: Use locally-hosted LLMs to pow...
Use locally-hosted LLMs to power your cloud-hosted webapp - get-convex/llama-farm-chat
1 Reply
@ian - after coding this project, do you personally believe we are living in a simulation? 🙂
Nice talk a couple weeks ago at essence VC conference!