Unable to push Convex functions `Function execution timed out`
The full error I'm getting is:
Not really sure where to start looking... Typecheck passes, schema validation passes...
35 Replies
Thanks for posting in <#1088161997662724167>.
Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets.
- Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.)
- Use search.convex.dev to search Docs, Stack, and Discord all at once.
- Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI.
- Avoid tagging staff unless specifically instructed.
Thank you!
All I changed was some updates to the convex schema
Okay, we're also observing this when deploying to different environments. Deploying to our staging environment works and with the same code when we deploy to production we get the same error, this feels like an issue out of our control
@David Alonso can you guys file a support ticket to ensure we handle this ASAP?
Feel free to just link to this thread, or copy paste etc.
Sent!
Hi David! Sorry to hear you're unable to deploy. The error message here is a bit misleading - there is no function execution happening, the deploy is timing out on analyzing the code you're pushing. Could you try running
npx convex deploy --debug-bundle-path <debug-directory>? This should write the code you're pushing to the <debug-directory> so we can see what's going on.
If you run du -h <debug-directory> on both the current version that fails to deploy and the previous version of your code that worked, lmk if you see any significant changes in size. You're not hitting the module size limits, but it's possible the time out is size-relatedvery very strange
I've just sent the zip file of my bundle debug dir on the existing email support ticket thread
Would you mind trying to push again to
sensible-salmon-524? I've enabled tracing so I should be able to have more visibility into what's going onSorry, EU timezone 🙂 Just pushed again. A few questions:
- when you say push, do you mean npx convex dev or npx convex deploy?
- Are they doing anything differently?
- Are the seemingly strange results in the screenshot expected?
if you enable tracing for our prod env: uncommon-gull-185 then I can redeploy there as well (different code running into the same issue)
Hi @David Alonso , sorry for the delay.
* when I said push, I meant "push code to a deployment" and
npx convex dev pushes code to your dev deployment, and npx convex deploy pushes code to your prod deployment. They're not doing anything differently besides that.
* We're still investigating why your code is timing out when your deployment tries to analyze it. Do you know if you've added any more validators between the push that works and the push that fails? https://docs.convex.dev/functions/validation Validators look like
Argument and Return Value Validation | Convex Developer Hub
Argument and return value validators ensure that
@David Alonso I've increased the timeout for sensible-salmon-524 and uncommon-gull-185, so let me know if deploying code works!
this type of error is the bane of my existence right now. but you at least have a timeout on it.. so not a circular reference..
Id make sure your "use node" files are not importing ts functions or has functions being imported to as just referencing the "use node" file was causing it to get it bundled slowing things down for me slightly for the whole project.
Yeah definitely! We’ve made updates to our schema which probably affects certain function arg types
Good to know, but don’t think this is the case for us since we haven’t made any changes to those files
i only found my problem because i added a class to one of the files the 'use node' file imported a associated function within that file
So this error has to do with arg types for convex functions if i understand correctly? @Emma
What about generally with the size of our schema? Our schema (due to many unions) is above 15k lines when flattened
Do you know roughly how many of those are unique or just large unions? And do you see slow running mutations. Im extra curious as ive spent the last 2 weeks with convex support working on performance as my functions were taking a 400-800ms hit and my schema was at maybe 4,000 lines, but probably 2,000 v. validators, ive got most everything resolved (btw) and the new convex backend optimizations also played a role too
And, I had a unusually large bundle even getting rid of most of my packages. which was fixed by moving the convex folder as we finally decided it could live in the proper mono-repo spot. i don't know if re-initializing it somehow cleared stuff out. But it immediately reduced my bundle time in half and some mutations cut processing time in half, the best i could get getting rid of everything but clerk and stripe was 14s, and after it went to 7s. It did the same on our other dev environment. II think watching my bundling on traffic monitor it was 30megabytes, now 3.5
we have several very large unions for certain documents (probably up to 5k lines for some document types). What happened in my dev env is that we added an additional union to a doc type which is used as an arg in some functions probably leading to the issue. So it seems like we'll need to have longer timeouts @Emma . Could you increase the timeout also for:
- amiable-possum-452
- colorful-labrador-762
🙏
Are you sure you increased it? I'm still seeing:
@David Alonso did increasing the timeout work for sensible-salmon524 and uncommon-gull-185?
still running into the same issue on uncommon-gull-185 as well
okay nvm, our schema is 50k lines 😁 some types in there are being used as function args
Tbh 5k line validators will be extremely inefficient and is probably slowing down your functions quite a bit
And yeah @ampp this leads to mutation times of ~1.3s in some cases, so optimistic updates are a must for us
yeah i agree, we haven't had time to optimize this
when i reduced 1500 lines of my biggest validators i saved nearly 100ms on some functions 😅 but that was prior to the last update or two this last week.
but i had 5? different confounding factors playing into my issues (ive lost track)
I definitely had some of those or worse for a bit
the thing is: having these unions is very beneficial to us, so while we can maybe move some common fields up the tree, we'll keep using the union pattern in many tables
I understand their goal as getting it as optimized as possible. There isnt that many people that have gone to your level imho lol
I'm turning a few of mine back on to see what happens
I was asking for a way (if possible) to disable the validators without having to edit a line. I think the problems are not only validators and this would quickly show us how much time the backend is processing them.
I believe every time i reduced my bundle size i saw a improvement to some degree in mutation time
I need a script to process the axiom results so i have better data..
@David Alonso can you try deploying again? to any of the instances you mentioned
sensible-salmon524 worked! uncommon-gull-185 worked too! Thank you @Emma !!
Yup i added back all the validators, back to 5000 lines in the schema from 2500. took one query from a 85-90 to 100-105.. so not signifcant as the 90 i probably saved orginally.. still have most of my event mutations around 150ms. I just have a few oddball outliers still, 500 or 700ms queries for no good reason.
great @David Alonso ! I think you might see more performance issues with these large validators but I can lyk if we optimize this further.
this would be great, I honestly don't know how much freedom there is on our side to optimize things further if we want to keep strong runtime type safety. Would be happy to show you our schema in a call or sth
thank god for optimistic updates 🙌
for what its worth 2 weeks ago i was at 550-1300ms for our custom event mutation system as a baseline. now they are 125-150 without any code changes(to that system) or validator changes. so id just keep open the idea that there is something else going on possibly.
and we can avoid more optimistic updates right now 😅
I'm still confused about the screenshot I shared before with the two terminals side by side:
npx convex dev pushes code to your dev deployment, and npx convex deploy pushes code to your prod deployment. They're not doing anything differently besides that.If this is just pushing the same code to different environments, why would one succeed and the other fail? btw, would be useful to know how much of the mutation latency goes into validating the args vs validating what actually gets inserted into the db @Emma If arg validation is significant maybe we can skip that somehow or do zod validation (which might be faster when using discriminatedUnions, but just a hypothesis)
ah, so using the
--debug-bundle-path flag doesn't actually deploy functions, it just writes the bundle to that directory for debugging purposes. You can always check the dashboard history page (or functions page) to verify if your deployment went through. We should change that log line after that says "Deployed convex functions".
We've added more tracing to validators so I can get back to you about that next week, and we've already made some optimizations that went out last week (what @ampp is referencing).that's great, where can i read more about the optimizations to know if they are likely to affect us?
what kind of schema and function args do you have for this?
The optimization last week reduced the number of times we parse validators on our end. You are already benefitting from that one!