Issues with limits/timeouts doing cron jobs on large datasets; infrequent aggregates of data
Hi, does anyone have a good solution for doing bulk data operations within Convex cron jobs? I keep running into either data query limits or action timeouts. I have a table closing in on 1 million rows on which I want to run weekly-ish updates and store the aggregates results in another table (like 20 rows, that's the easy part). At the moment I was able to group by a field in the data, reducing the data set to maybe 50-100k rows and run the aggregates on that smaller dataset but I just keep running into limits or timeouts.
The aggregates can be slow to compute, that's no problem, the cron jobs just need to be stable. Is there maybe a way to offload it somewhere outside of a "regular" action?
The aggregates can be slow to compute, that's no problem, the cron jobs just need to be stable. Is there maybe a way to offload it somewhere outside of a "regular" action?
