vitalii.s
vitalii.s8mo ago

Cannot run a schedule function

Hey guys! Maybe something stupid, but for some reason, each scheduled function is just stucking. No matter if it was scheduled with runAt or runAfter. I tried to set time in the future and the past - I just see the functions in the schedule - but they are not running. When I try to run a function manually from the Dashboard - everything is ok.
12 Replies
presley
presley8mo ago
Interesting. What is your instance name?
lee
lee8mo ago
(or can you verify in #verify so we can look it up)
vitalii.s
vitalii.sOP8mo ago
verified I guess, thanks for checking
lee
lee8mo ago
we see the issue on our side. Still trying to diagnose. Sorry about this
presley
presley8mo ago
Can you provide any context on what those actions do? Are they expected to be long running? Like run for few minutes? On our side it seems like they are running for a long time and for some reason we don't time them out properly.
vitalii.s
vitalii.sOP8mo ago
There is a "planning" function, which takes "leads" page by page and plans lead processing with intervals. Each lead processing function makes an API call (the particular call depends on what data is missed), it takes the result and checks if there are other missed data, and it schedules the same function again to enrich missed data. Usually, an API call takes a few seconds, but some of them can take 1-3 minutes. I tried to plan a dummy function, but the same result. Looks like I can't plan any function in this instance.
jamwt
jamwt8mo ago
@presley did you conclude anything here?
presley
presley8mo ago
Not yet. We are looking at it. The problem seems specific to this one project and happens over multiple instances with same code. We suspect there is something specific about those functions that make them timeout but for some reason the action 10 minute timeout is not kicking it. We are not able reproduce yet. The reason you can't run any scheduled functions on that instance is because the scheduler concurrency for free accounts is limited to 10. And you have 10 functions that are stuck executing and not timing out. We are investigating what exactly is happening here.
lee
lee8mo ago
hi @vitalii.s we're adding logging on our end, but it looks like all of your scheduled functions have been completed or canceled. Could you try scheduling more functions to cause the issue again? Would it be possible for you to share your source code with us (in a DM) or give us permission to look at your function's source (which we have, but won't look at without permission)? This would help us to attempt repro on our side. Thank you!
vitalii.s
vitalii.sOP8mo ago
Hey! Thanks for the response! I have planned more functions, and now it looks like it works on DEV. But on the Production environment, it processed 10 scheduled functions successfully, but after that next 10 scheduled functions are hanging Sure, you have my permission to check the code Let me know if you need more context
lee
lee8mo ago
hey thanks for the context. we discovered a place where we're not applying a timeout, so the node function can get stuck indefinitely. we're still trying to figure out why that happens on your functions, and in the meantime we're adding a timeout.
vitalii.s
vitalii.sOP8mo ago
Thanks! We don't feel any problems now. Hey! Just an update about this issue. It looks like it's related to the "use node" statement. I forgot to add it for some helper classes, no errors but functions were hanging. After fixing everything work well

Did you find this page helpful?