Pagination, sorting and filtering
We're going through a migration to convex and one of our features has cards. We've filters for cards based on different values like owner, priority, dates, etc.
Users can sort by priority so that highest priority cards are available on top, then medium, low, etc. Users can also sort by creation date, due date, etc etc.
And lastly (the issue) we perform pagination, meaning that in the first page there may not be a high priority card but in the next page there are high priority cards available.
There are currently thousands of cards, meaning we can't perform a "collect all then sort then paginate" that won't work.
Is there any alternative to this?
7 Replies
Thanks for posting in <#1088161997662724167>.
Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets.
- Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.)
- Use search.convex.dev to search Docs, Stack, and Discord all at once.
- Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI.
- Avoid tagging staff unless specifically instructed.
Thank you!
Check out streams: https://stack.convex.dev/merging-streams-of-convex-data
Merging Streams of Convex data
New convex-helpers are available now for fetching streams of documents, merging them together, filtering them them out, and paginating the results. Wi...
recommendation is to use stream + indexes for the most common filters and then use withFilters on the stream. On my e-commerce like use case with hierarchical and refinement list filters + search + aggregates, I'm getting 3 to 6 seconds to get a result and setup is way too complex to be maintainable. resorted to use meilisearch. typesense or algolia are other alternative.
meilisearch is 1 to 5 milliseconds per query and easier/maintainable setup
@sbkl interesting! thanks for that, I'll be looking into it
@erquhart after some refactoring, that article helped us make the new sorting system. It consists mostly on creating indexes and working our way into fetching pages based on sorted fields.
Our system has:
search
filtering
sorting
pagination
and we're looking to introduce "pinned" cards, but we're good for now. Thanks! problem solved
@fede did you face any performance issues mentioned by @sbkl ?
Give me a second while I perform some tests.
@Amal Chandran Being honest, my performance could be greatly improved. In my use case, cards have relationships with other tables that aren’t considered by the “Merging Streams of Convex Data” article’s philosophy. My implementation here is poor and can definitely be improved.. we just don’t care about it right now.
Another important factor is that I’m running in a local environment, which affects query durations.
That being said, the query ran in 370 ms with no filtering and the default sort (creationTime) for only 10 cards per page. Great? No, again, we just don’t care too much about it right now. Fetching the next 10 cards took 25ms (not bad). Going back to page 1 and applying a sorting method took 50ms.
Hope it helps
This refactor took the whole day, and I didn’t have much time to put in enough effort. I would’ve looked into @sbkl approach, it sounds like a good solution, since I don’t expect Convex to handle this feature very well. I’d be happy to see any example of this stream+index method applied correctly. In the meantime, I’m enjoying what Convex is good at somewhere else.
@fede thanks for the detailed answer 🤟