RJ•2y ago

Declarative schema validations and branding

I'm sure this is a big topic that you all have thought a lot about, but I've found myself more and more wishing that I could declare constraints for specific fields in my schema. It would be really neat (I think) if Convex supported something like this:

items: defineTable({
  quantity: s.number().validate((n) => n >= 0)
})

items: defineTable({
  quantity: s.number().validate((n) => n >= 0)
})

Such that whenever using db.insert, db.patch or db.replace, any validations would be run on new or modified fields, and an error would be raised if any fail (or, better yet, the validation failure is [even optionally] reflected in the type signature). I imagine there are much fancier alternatives to this, but this seems to have some nice advantages nonetheless: 1. It's effectively the same as what a user would have to do within each of their individual mutation functions if they're trying to enforce this invariant in a more ad-hoc way (the only option at the moment). By "effectively the same" I mean it is computationally identical (JS/TS code being executed at runtime), but a much nicer API. 2. It's what many ORMs do and have done for a long time (e.g. Ruby's Rails and Elixir's Phoenix, at least)—which implies a) that it's generally a useful pattern and also b) that it's familiar to people. 3. It doesn't require changing the database layer to implement (obviously database-layer guarantees are much nicer, but are, I imagine, also much harder—or at least more work). In a similar vein, I've often found myself wanting to be able to more easily declare that e.g. a s.string() is an ISO 8601 date. It would be cool to extend the above with type branding, such that you could have something like:

export default defineSchema({
  items: defineTable({
    quantity: positiveInteger
  })
});

const positiveInteger = s.number()
  .validate((n) => n >= 0)
  .brand("PositiveInteger");

export type PositiveInteger = Infer<typeof positiveInteger> // number & Brand<"PositiveInteger">

export default defineSchema({
  items: defineTable({
    quantity: positiveInteger
  })
});

const positiveInteger = s.number()
  .validate((n) => n >= 0)
  .brand("PositiveInteger");

export type PositiveInteger = Infer<typeof positiveInteger> // number & Brand<"PositiveInteger">

These would both be huge DX upgrades for me!

14 Replies

jamwt•2y ago

Good feedback! @alexcole is really passionate about this stuff too, and we have some changes about to go out in a release this week that start to address some of these needs

RJOP•2y ago

Awesome, I'm excited to see it!

alexcole•2y ago

Interesting stuff! We were just talking about something like this in the office last week. But I don't want to over-promise here. We're going to have some validation improvements in the new release this week and schema enforcement in the next one, but the type of custom validators that you're describing aren't currently on the roadmap. But I agree that it would be great if the framework could handle this kind of thing! Thanks for proposing it.

RJOP•2y ago

I was reflecting on this more this afternoon (as I spent a good portion of the day updating my schema.ts), and realized that Convex's SchemaDefinition could actually be used to do a lot of the same kinds of things that libraries like zod, io-ts (https://github.com/gcanti/io-ts), or effect/schema (https://github.com/Effect-TS/schema) can do. One example that might be particularly useful would be generating parsers (https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/) which might be used to... parse the parameters of Convex functions. This is in contrast to an approach like https://stack.convex.dev/wrappers-as-middleware-zod-validation, where you need to write both your Convex schema and your Zod schema, and you have to manually ensure you validate everywhere you're mutating particular fields. There may still be a place for custom schemas for parsing function parameters, of course, but that could make it even easier!

alexcole•2y ago

Yeah, we actually took some inspiration from Zod when building Convex schemas! The new npm release actually just went out and includes argument validation (https://docs.convex.dev/functions/args-validation) which does a bit of this. Now you can reuse the same validators between your schema and functions.

Argument Validation | Convex Developer Hub

Argument validators ensure that queries,

alexcole•2y ago

Now time to get the release notes up...

RJOP•2y ago

Wonderful!

ian•2y ago

And now we have schema validation too - so the convex validators will guarantee your data at rest matches the schema (or will fail your deploy telling you what doesn't match): https://blog.convex.dev/announcing-convex-0-14-0/ Interested to hear how close we're getting to what you originally asked for @RJ

Convex News

Announcing Convex 0.14.0

Meet the brand new Convex Rust client! We’ve also open sourced the JS client, added schema validation, and more. 0.14.0 is the best version of Convex yet!

RJOP•2y ago

This is a huge update, I love it! It's not really what I was asking for here, but it's still an extremely valuable change IMO. Now if y'all were to pair that with custom validators built into the database layer... then we'd really be in 🤯 territory. What I was asking for here works well if 1. TypeScript is your only interface to the Convex database (this is also a limitation of this approach in frameworks like Ruby on Rails/ActiveRecord and Phoenix/Ecto [Elixir lang]). If you ever insert data through e.g. direct SQL queries, these validations/constraints are simply absent, and you're on your own to enforce them ad-hoc (good luck). 2. You don't ever change such validations after you add them. If you do, you'd also have to find some (probably manual/ad-hoc) way to re-validate all existing data according to the new or altered validations. It would be really incredible if y'all were to build support for such arbitrary validations into the database layer and therefore also have them be enforced in other language client libraries. In other words, if you would let me write:

items: defineTable({
  quantity: s.number().validate((n) => n >= 0)
})

items: defineTable({
  quantity: s.number().validate((n) => n >= 0)
})

In a TypeScript Convex schema, and then ensure that in e.g. the Rust client whenever I try to insert a quantity, it is validated against that TypeScript validation function. That's much fancier stuff though (though a real dream in a lot of ways), which is why I suggested the simpler TypeScript-only approach. At a certain point down this line of thinking you end up asking for a DSL for executing db-layer validation logic (like SQL functions or whatever). And then you end up wanting bindings to this DSL in client libraries (TS, Rust, etc.).

ian•2y ago

Yeah, that makes sense, and I figured that's what you really want. It's a little tricky to do the js execution well - a function could capture variables, could change deploy to deploy, leaving us scanning all records all the time. One nice property of Convex today, though, is that the interface layer is already just javascript. Lee has some example code where he wraps the db object passed into query / mutations with access checks. You could add your own validation there, and enforce that your code always conforms, and do your own one-off scans when you want. Similar to the withSession or withUser wrapper, but a little more meta. His example is to implement RLS via a wrapper

RJOP•2y ago

Legit, I hear you. Still would be pretty cool if it could be done, though! And yeah, when I originally created this thread I thought about doing exactly that (what Lee experimented with). (What's RLS though?)

ian•2y ago

Row level security

RJOP•2y ago

Ah cool!

ian•2y ago

I mentioned it in #general , but this post by Lee has a pattern that could be extended here, where you check an object's validity on write and/or read: https://stack.convex.dev/row-level-security by wrapping the db object in each function's ctx

Row Level Security: Wrappers as "Middleware"

Implementing row-level security on Convex as a library. Wrap access to the database with access checks written in plain old JS / TS.

Declarative schema validations and branding

Did you find this page helpful?