Ignore extra fields
is there any way I can get convex mutations, queries and actions to ignore extra fields? it is very annoying (and unsafe) having to manually delete _id and _creationTime fields
22 Replies
Can you share a bit more of what your code does? Why are you passing in _id and _creationTime ?
Also note that you can include them in your validator type.
it is a basic CRUD internal mutation (update) generated by the convex-helpers package
but the same happens for any other way you could create this mutation
if you create a Create or Update mutation, and pass in any othey additional field (_id, _creationTime) which is added when you fetch an object from convex, you need to
delete obj._id; delete obj._creationTime or else it will reject the mutation call in runtime
typescript doesn't catch this in compile-time since typescript allows passing objects with additional fields, as long as the required fields are there
@Michal SrbYeah, TypeScript not having closed objects is super annoying (we're not the only ones running into that).
How are you using the update mutation?
ctx.runMutation(internal.chatbots.update, {...someObjectIFetchedFromConvex, ...updatedFields})
there isn't any bug or error, I'm just asking if convex can whitestrip additional fields automatically. if I pass only the required fields it works (but currently i have to do it manually)Why pass in
someObjectIFetchedFromConvex?
(update is a patch under the hood, so it doesn't need existing fields)
Also to answer the question: There is no way to do this currently.in this case, its a react form that merges a existing object with its updated values
seems like we could change convex-helpers here so it adds system fields to the validator https://github.com/get-convex/convex-helpers/blob/main/packages/convex-helpers/server/index.ts#L171
GitHub
convex-helpers/packages/convex-helpers/server/index.ts at main · ge...
A collection of useful code to complement the official packages. - get-convex/convex-helpers
but I've come across this many other times, even when just doing backend
ie it pulls an object from convex (a form), lets you update it, then sends the updated object back to convex
also if you're used to program in a functional style, its common to keep your mutations on the edge of the system, which means you'll usually be passing around the updated object as a whole instead of patching only what changed (since that would require a bunch of updates all around)
is convex's json validator written in rust? if it's using a library, that library probably has a whitelist option
Lee is correct that in this case we could update the helpers to accept the extra fields.
I am curious about cases where it seems like having an "open" object validator would be helpful.
If you can, can you show more of the code that's forcing this? I'm not sure I get how "functional style" impacts this.
Yeah, the validation is custom logic in Rust, we'd have to change our TypeScript bindings and the Rust implementation.
We've been hesitant to add this because in a lot of cases (especially for the arguments of public functions) having an open type can open a path to security issues and make it harder to evolve an API over time.
don't think you need open types, just strip the JSON for fields that aren't included in the validator
I'll give more context to my code
but replying to this, basically everywhere in javascript objects are passed around and fields that aren't used are ignored (that's what makes js quick for development as a untyped language, and what makes typescript fit javascript well)
not really related to functional programming at all, more of a dynamic language thing
but for my specific case
I have a "state" table which keeps all of the information which is relevant for my agent to do his decision-making and conversation flow
and on top of that state, I have many functions that use the current state to make decisions and contribute to what should happen next
its a pretty complex flow with many functions, so if I were to call
update functions inside each one one of them to update only what changed, understanding what's going on (and where) gets pretty unwiedly
all of these are pure functions which don't mutate the database, they only return new information which can then be considered to make new decisions. This lets me centralize the decision-making in a single function, instead of spreading it all around
^this part is functional programming related
so when I finally make the next decision at the end of the decision-taking function, and update the state with information that changed
I pass the whole object, instead of patching only what changed
i could probably diff() real changes and then send a PATCH with only what changed, but i didn't :p
my use case is probably more niche, but ^this is more common
especially if you are just retreiving an item from convex, mutating it and sending back the new version
i guess this error comes from the convex-helper crud being too strict
however AFAIR, I had this same problem validation when directly PUTing the full object back into convex without using helpers
since I was re-using the table definition as parameter validators, but then the mutation would complain _id was passed insince I was re-using the table definition as parameter validatorsYeah, we've seen others run into this. There can be an argument made that you don't want the API to be tied so directly to the database schema, but it can be convenient. @ian has been championing addressing these issues.
another example could be when we are receiving webhooks or saving data from external APIs, and they return extra data that is not in the validators. I would be fine with being able to just strip whatever is not in my validators
Thanks. IMO, part of makes convex so great to use is being able to call backend functions as if they were normal javascript/typescript functions. However, these are the small things that get in the way of simply being able to treat them as normal functions
and get the full ts developer experience
I think this is what you want?
I made
create and update take in optional system fields (_id and _creationTime) in convex-helpers@0.1.35Thanks. Ideally, what I'd want is a way to deactivate overly strict validation for convex functions in general. I will use this for now, thanks.
I hear you. I have a proposal internally about this but it’s still in discussion. Something like
v.object({…fields}, { extraFields: “omit” | “allow” | “throw” }
But what you’re proposing is more universal. I’d have to think through the implications there bc you could do an insert then a get and not have all the data. If you forgot to add a field you’d be dropping your data on the floor unknowinglyAlso open shapes play badly with optional fields (in any language).
For a type
{myGreatFieldName?: number},
this object doesn't show a typer error: {myGraetFieldName: 1242},
but it doesn't do what you expect (and it's very easy to miss, especially with many other fields present).