Need to jump pages not by _creationTime
I am working on an application that is going to have an extremely long activity ledger (think hundreds of thousands maybe millions of rows eventually). I want users to be able to "browse" the ledger, sort it, and yes page it... The problem comes when there are say 50 or 100 pages + of logs.. Scrolling isn't a great experience at that size and faking pages but then having to load all those records just to "hide" them to give a "jump to page" type experience is also super wasteful to both the client (memeory) and convex servers.
What is the work around when serverside collect then slice/skip&take is a bad fit due to volume, and the pagination feature is a bad fit due to User experience or well, also volume.
8 Replies
Thanks for posting in <#1088161997662724167>.
Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets.
- Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.)
- Use search.convex.dev to search Docs, Stack, and Discord all at once.
- Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI.
- Avoid tagging staff unless specifically instructed.
Thank you!
I would expect to reach for virtualization or a virtualized table (maybe TanStack table is good for this?) plus the aggregate component for jumping to a point in the result set: https://www.convex.dev/components/aggregate
that would give me a count for sure, but the skip/take would be a problem if I am following what the aggregate will do for me..
Also do we know if aggregate can handle more than 16k records (the rough limit of convex collect).
Aggregate is made to handle high volume, check out the offset based pagination example: https://www.convex.dev/components/aggregate#offset-based-pagination
oh yah I did see that in my initial analysis but it assumes only sorting on createdtime and if I have a list of say invoices and the user wants to sort on name or invoice_total then the example breaks down (and I need to create lots of aggregations to support all the verious sorting cases)..
I realize i am thinking in a very DB first way and that is probably getting in the way.... just trying not to paint myself into a corner developing LOB solutions on convex.
oh and thank you @erquhart for jumping on and chatting about this with me š
Iām interested in this topic. What solution did you end up with?
I wish I had a good answer but sadly I couldn't work one out that didn't require significant complexity trying to make the aggregator do things it wasn't meant to do..
Yeah in order to allow user to jump to page X for Y number of filters/sorting orders, the two clear options are:
1. Maintaining an aggregate for all Y possibilities
2. (Optionally create an index for each Y, and) Collecting/paginating all pages up to page X behind-the-scenes in order to show page X to the user
For option 1, you will have to face creating and maintaining an aggregate for each of your Y combinations, while option 2 will incur a higher bandwidth usage and probably much slower query times for large values of X. I don't really think this is specific to Convex however - this is a more basic data modelling problem that you will encounter with any solution.
My suggestion is also to go with option 1, that is one of the great use cases of aggregates, and not at all "trying to make the aggregator do things it wasn't meant to do".