Billing risk from bad actors
Firstly wanted to say I love what all of you have built so far!
This thread was added following discussion in general chat with Ian as per his request.
Currently, bad actors can connect via websockets and continue spamming functions. Rate limiters or any other criteria based functions at application layer can prevent access but functions are still called and result in billing. The bad actors will remain connected to the backend and we cant get rid of them. Only mitigation for billing risk at present is spending cap but this would result in downtime for the service.
Proposed fix:
1) Allow method to disconnect websocket of a client, server side.
2) After several attempts to reconnect following their forced reconnect, Pro users who can have own domains can utilise cloudflare or similar websocket proxies to mitigate attacks. https://developers.cloudflare.com/network/websockets/
The purpose of triggering the disconnect is to trigger cloudflare’s own rate limiters
PS: On a side note, I would love to try Pro as a startup if given a chance and would love to be considered. Still no VC money yet to qualify on the site :).
Thanks very much!
Cloudflare Docs
WebSockets
Cloudflare supports proxied WebSocket connections without additional configuration.
28 Replies
Thanks for posting in <#1088161997662724167>.
Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets.
- Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.)
- Use search.convex.dev to search Docs, Stack, and Discord all at once.
- Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI.
- Avoid tagging staff unless specifically instructed.
Thank you!
context from general:
WebSocket disconnection IMO is not a meaningful lever. Any web server can receive traffic from any source, without network-level blocking of IPs, etc.
Making HTTP or WebSocket or other requests to any web-hosted project will incur some cost, whether that pricing is passed through or not.
There is not a significant architecture difference from Convex hosting to any other hosting I'm aware of.
In terms of "vectors" - avenues by which bad actors operate:
DDoS: many people use this term to mean other things, but true DDoS attacks are rare. This is where you work with your hosting provider (e.g. Convex) to get network-level defense.
"fuzzing": where someone runs many random requests against (usually http) endpoints with the aim of discovering some vulnerability. This happens occassionaly and is annoying but often is coming from a small set of IPs or easily filterable without much actual cost.
A (usually small) set of malicious users making extra requests to rack up a bill. e.g. if a user sent 1,000,000 requests, it would cost you $2.
A few things that provide protection:
When your functions have validated arguments, those will be rejected before it runs any JS if the client's arguments don't match.
When your functions are authenticated, you can easily reject un-authenticated users. You can track untrusted users, or avoid authenticating them in the first place.
If you're having a real issue and you're a Pro customer, reach out to support and we'll help you handle it!
Worth mentioning: you don't pay per active websocket connection. We hold all of those open for free (they're very cheap when you handle it in Rust!)
I'll add more context in a second
As for the difference with Cloudflare:
Currently, bad actors can connect via websockets and continue spamming functions.FYI bad actors can spam your functions without connecting via websocket - via http actions / calling the http api for normal functions, etc.
Check out firestore’s implementation suggestion to mitigate
https://www.reddit.com/r/Firebase/s/7Zvq9ovBVh
Rate limiters or any other criteria based functions at application layer can prevent access but functions are still called and result in billing.Correct. Rate limiters that you define in your code (e.g. using the rate limiter component) will be able to reject traffic. You can't currently configure any rate limits in Convex directly that live outside of the application layer
They have app check and via running cloud functions it can sit behind cloudflare
Http Apis can also sit behind cloudflare. Problem right now is websockets cant sit behind cloudflare without the backend websocket disconnects
Gotcha, so the ask here is to support having your app behind a cloudflare proxy, and therefore needing to account for how cloudflare's rate limiting only happens at websocket connection time
Correct!
Check out the infamous hundred grand bill from firestore in the reddit post
Its definitely a vulnerability for us risking unnecessary billing. Just a simple method to force disconnect would fix this.
btw doing a code push will disconnect clients I believe - they'll auto-reconnect
Is this a theoretical concern of yours, or something you anticipate? I only ask b/c none of our startup customers have had this issue / been impacted.
Unlike other platforms where users are directly connected to the DB, functions are the entrypoint where you can shed load by returning early.
Yea but if we can have a method to run in functions would be awesome
In order to spend $100k in function calls, you'd need to send 50 billion requests
But I hear the concern
Not at the moment but i have had bad experiences in the past so would like to mitigate from the get go.
Out of curiosity, how would the function determine that the user should be disconnected?
If they hit the rate limiter on application layer
Gotcha
Generally we put the spending limits there so you can prevent crazy events / accidents. I believe we send you an email when you're approaching them, and send you invoices as you reach different bill sizes, so you can see if they're coming from real usage or a bug / attack. So you can bump them up if traffic is normal
If you're enforcing auth, then once their token expires you can avoid giving them another one. Whenever there's an auth error, the websocket reconnects. Which limits their window
Yup hopefully that should be worst case scenario as a backup as would disrupt entire service
yeah ideally you can fix the issue before the hard cap is hit.
Noted regarding the auth token expiry. Good to know
but if all they're doing is calling functions which throw a rate limiting error, they'd need to send millions and millions of requests for that to financially impact you
The risk is if you expose a function where they could do something expensive without a check in place, e.g. a public action that calls an LLM with no auth
True but as you mentioned auth token has to expire first before the disconnect.
And generally when users are being attacked / have huge spikes, we will work with them and don't send surprise huge bills
most of the time this is someone who has a runaway bug where they are scheduling functions recursively like a fork bomb & don't know how to cancel all the scheduled functions / or set up a third party service that is hammering them
Nevertheless, i think wouldnt be hard just to add another method to call the websocket disconnect. As inherently normal websockets have a method server side
I hear you. Currently the functions are agnostic to whether they're over http, websocket, or being called from another convex function, and this would be the first function that pierces that boundary, so we'd probably move slowly to find the thoughtful place to put the API (e.g. probably not on
ctx)
I'd give Pro a try - $25 is what I paid for a sandwich & drink the other day so hopefully not cost-prohibitive (SF prices 🙄)Would love to join the startup programme once we raise funds i guess. Currently bootstrapped. So workos integration with convex recently announced is a godsend vs clerk 😆 . Anyways thanks very much! I’ll monitor this thread as to if/when this gets implemented.
I'll put it in our issue tracking system
Hi any updates on this?
We've talked about a special exception you could throw to disconnect a user, but the work is not planned.
If you see antagonistic traffic (i.e. not your fault) we will definitely work with you and not surprise you with a big bill, etc.
So if it's a longer term concern, I'd not sweat it for now and eventually we'll have some more nuanced story, how does that land?
Just a random idea.. what is the feasibility of creating essentially something like a convex proxy possibly using the self hosting or testing code. Something that could act like a traffic analysis layer/filter for cases like this?
Not sure if you'd need the self hosting or testing code - a general purpose proxy should work, if you can terminate the SSL at your proxy and do another request to the backend forwarding the IP