Right now I just update daily buckets in
Right now I just update daily buckets in real-time.
Then the frontend will just pull all of the buckets and sum them, so it sums 7 documents for 7d etc.
11 Replies
It's getting a bit clustered, and i shouldn't have went with the convo without a thread for this long 😅
Sure no prob
I need to think about this, I remeber I've built something around the lines to grab data like that, but it was mostly estimates
You're NOT missing anything - this is a textbook case for pre-aggregation. The Convex Aggregate component would actually be overkill and slower for your needs.
Even Convex Chef is agreeing with me....but still AI bias hhaha
Are you vibe coding this by any chance?
No, not in the strong sense of the word. Some ideas have been AI driven, I've actually never used Chef until right now (to try and see if it steered me towards some magical solution)
I think you might be right here, a query with
withIndex(q=> q.lte("_creationTime", 7d)) would be the ideal use case here
Like call each function when the user requests them, and paginate the request itself if its larger than the limit
I wouldn't return data this big, for calculations I'd just use something like that, with the option to change the query args based on the user wants, my brain keeps on going to redis, maybe? Overkillconst twentyFourHoursAgo = Date.now() - (24 * 60 * 60 * 1000);
const recentAlerts = await ctx.db
.query("alerts")
.withIndex("by_creation_time", (q) =>
q.gte("_creationTime", twentyFourHoursAgo)
)
.collect();
Yeah so like the above, the issue is that if that data thats returned is above 8MB/16MB etc you run into a massive issue.
Pagination solves this for most use-cases, but not if you want to sum all of the results of it, such as 'how many alerts total'...at least I couldnt get a solutionIf you want to keep track of the calculations, how about a stat estimate count?
Imma head out for the day, too much internet for me
No prob, thanks for your help. TL;DR I still cant see past my existing bucket strategy, I think its the best solution but meh, who knows.
The bucket strategy is nice for it's simplicity, at scale it risks contention issues if writes are happening in parallel, because the written values are always based on the read value. Aggregate component and triggers feel like a lot to set up, but you're going to end up with some of this anyway as your project grows.
I'd say if it works for now, definitely use your buckets. When you start seeing issues of scale, aggregates and triggers will probably be more worth your time at that point.
Also note the sharded counter loses it's advantage if you're reading on every write, which would be the case here.