Key Management
Is the Convex team working on any sort of key management service to separate storage of cryptographic keys/salts? Or any ideas on how to do that securely right now?
I'm storing API keys from external services and want to store the salted encrypted version of them for security
14 Replies
One quick thought (not sure if others have better ideas): what about storing the salt & key in environment variables, and store the encrypted contents in the DB? I'm assuming these values are generated statically and don't change often.
was thinking of going that route too
or connecting to AWS secrets manager
If you're connecting to an external service, the one hitch is that the interface exposed to queries & mutations is the database & environment variables. So you'd either need something reading from AWS and writing to tables (which is what I think you're trying to avoid to begin with?), or to do the encryption / decryption in an action, which can't be synchronously called from a query or mutation. So the most ergonomic choice I think is env variables. Sorry there isn't a deeper integration with secrets managers. We'll keep it in mind for the future.
@punn in a theoretical "secrets" or key management feature, would the difference between that and our current environment variables feature just be the level of encryption and access? Environment variables are currently stored in our database which is encrypted, but perhaps you're looking for us to store secrets in something more like Hashicorp Vault such that they're separately encrypted and not even readable from the dashboard once set, and are only available in your queries and mutations.
If that's something you (or anyone else) would find valuable, give this a ➕ and we will keep this in mind!
Exactly that! Thanks for considering it!
Gotcha sounds good I think we'll go with that for now since we won't need the advanced features of external services
This is currently my biggest concern with adopting Convex.. was really hoping there was a "Supabase Vault" equivalent within the product. Storing an encryption key within a convex environment variable and using that to encrypt field values does solve the plain text issue, but only delays an attacker from decrypting a convex column if they were to gain access to the dashboard.
But I'm not sure the concern is warranted - if the ideal scenario is that Convex holds the keys and doesn't display those within the dashboard, in this case if a hacker gains access to my dashboard then they can author an action to exfiltrate decrypted data (despite not knowing the keys).
So it seems whether Convex holds the encryption keys or not, if hacker gains access to dashboard, then bad thing happens. Am I thinking about that correctly? Like, even if I set up AWS Secrets Manager, again, if hacker gains access to dashboard, they can still exfiltrate unencrypted data via a bad action. So really, what encryption is doing for us is reducing the likelyhood we accidentally send API keys in plaintext to someone who isn't supposed to see it within our own application. Also, if Convex were hacked but somehow the bad actor did not get access to the Env Secrets Store, then our users are protected. Outside of that, hacker wins.
Am I thinking about that correctly? If so, then storing our encryption key as an env var is roughly as secure as if Convex did it for us? Beyond that, the only real thing we can do to reduce the likelyhood of a data breach is to use MFA and involve only the fewest, trusted users with the dashboard. Is that correct? would seem true of almost any managed db as a service
Yeah if an attacker gains access to your dashboard, AWS console, etc. then you're pwned and they can snapshot export / delete / anything, on Convex or AWS.
A big thing in hiding text on the dashboard is for compliance - to only look at the minimal user data possible and have an audit log for what you looked at.
Having data in plain text is easier to accidentally return the wrong data - e.g. if you did db.get with a string not validated by v.id and they passed in an ID to a sensitive table.
But for API keys I personally keep them in plain text.
Having a vault / secure store seems reasonable for a variety of scenarios - e.g. sharing secrets securely between environments, having secrets you can never see after entering, etc.
You keep API keys in plaintext? Interesting 🤔 one of our junior devs leaked one of our slack keys on accident today and, fortunately, Slack's automated system's caught and disabled it for us. Much worse scenario would be if we were responsible for getting our customer's API key's leaked/disabled (even when disabled is best case scenario). Hence why we use Supabase Vault to encrypt those along with PII.. just harder to mess up.
Yeah the profile looks different for different companies & usecases for sure.
One big difference is whether you're storing your own API keys (that you can invalidate yourself) vs. customer keys (where they need to be notified and take action, which for LLMs means they might be on the hook for big $$).
Doing an env var secret w/ symmetric encryption is a good idea, the more I think about it for this usecase.
Defense in depth is great for things like this. e.g. you can leak keys after pulling from a vault & decrypting - you end up with the raw value in codeone way or another - the question is what are the vectors you're prioritizing, what's the impact / what does recovery look like. This is a big risk on platforms like Supabase where you're querying from the client / exposing the DB by default & trusting you've covered all the access models via RLS
One thing convex should definitely have is non viewable secrets along side environment variables. so dashboard users cant view the secret value, but can update it.
and ensure printing secret into console gets obfuscated with ***
Additionally, require account to 2fa to confirm authority before being allowed to make write operations to any table data, schedules, secrets or env variables in production env for current session.
the above two should defend against dashboard breach?
having a non viewable secret key is kinda essential though, like if im going to use an external secret vault, i wouldnt want the api key to the vault to be visible/retrievable in an event of a breach. Is this something convex would consider prioritising @ian ?
I think part of the Enterprise plan in the works is having more granular controls over what team members can do / see / modify in the dashboard. I don't know exactly the feature set there. Also having env variables that you can't see after you've set them makes sense once that's the case, but currently you could write a custom query to read the env variable, and if you can push code, you can do anything you want. So ultimately the people who shouldn't have access to data or push etc. shouldn't be on the project at all IMO.
Having to re-authenticate to push to prod isn't something I've thought about before. That feels more like an IAM/ CLI credentials space and we'd likely match the aws model & lean on mfa/2fa
ultimately there's a trust chain - you put one secret somewhere, then you need a secret to get access to that secret, etc.
maturity & regulation controls determine where the line is. For now I think having a symmetric key in env variables suffices, or storing your vault key there.
You can always have a separate project just for prod that has more restricted access - so dev deployments are all on some project where "prod" is staging, and the prod project is locked down / isolated - maybe a separate team even. That's the workaround today for access controls when users ask
@ian on a similar note is having a separate project where "prod" is staging the canonical way to get a staging env? i'm new to convex and need permanent staging (so preview deployments wouldn't work) for beta testers/non-technical cofounder. just want to make sure that's not an anti-pattern
Yeah I haven't set up staging for a Convex project, but I have for GCP / other companies.
Here's how I recommend doing it:
Project A:
Dev deployment per team member
Default env variables for staging & sandboxed & other non-prod API services
"Prod" is a staging instance. Either deployed automatically from anything pushed to a git "staging" branch, or on every "main" commit that passes, or directly by users who want to (free for all)
Project B:
No dev deployments
"Prod" is real production. Deployed from a "production" git push, which is only pushed to from a PR / CI workflow that has been verified. This makes audits / SOC-2 happy. Emergency deploys are OK but trigger a review and documenting why.
Awesome really appreciate the response @ian, also your Convex blog posts have been super helpful (along with the convex-helpers package) 🙏