ampp•6mo ago

Stopping infinite pagination

What is the best way to keep a paginated query from expanding in length forever as new results come in? I have a couple ideas but i couldn't find this talked about anywhere.

25 Replies

erquhart•6mo ago

This is generally limited by visibility. In react native, I'm typically using FlashList, which accepts a function to call when the user scrolls close to the bottom, and there are similar approaches for the web.

amppOP•6mo ago

Yeah, scrolling down forever is a different problem, I'm talking about you start with a 10 items and 7000 items come into the database, now you have 7010 (and 7000 re-renders) rows rendered on page. (im using nextjs)

erquhart•6mo ago

You have to pass a number into loadMore(), that number will limit the number of results per page. But I feel like I'm missing something about your implementation here.

amppOP•6mo ago

    const {
        results: events,
        status,
        loadMore,
        isLoading,
      } = useStablePaginatedQuery(
        api.engine.events.getLiveEventList,
        { 
        } ,
        { initialNumItems: itemsPerPage}
      );

    const {
        results: events,
        status,
        loadMore,
        isLoading,
      } = useStablePaginatedQuery(
        api.engine.events.getLiveEventList,
        { 
        } ,
        { initialNumItems: itemsPerPage}
      );

just controlling the max this expands to with new data, without the user taking any action. can be 100's or 1000s of rows very quickly

erquhart•6mo ago

You're calling loadMore() somewhere and passing a number in, correct Presumably you're passing in the value assigned to itemsPerPage

lee•6mo ago

even the first page can grow without calling loadMore

erquhart•6mo ago

But it can't grow past initialNumItems right

lee•6mo ago

it can grow arbitrarily large. initialNumItems only affects the initial page load

lee•6mo ago

(see https://stack.convex.dev/fully-reactive-pagination )

Stack

Fully Reactive Pagination

Paginating over large datasets is tricky when the data is changing. Naive approaches result in missing or duplicated data. The trick is to rewrite lim...

amppOP•6mo ago

Yeah, we managed to kill 4.5gb of bandwidth in minutes 😅 thanks to a poor optimization and a bug

lee•6mo ago

as for fixing it, there's a feature that if a page grows too large usePaginatedQuery will split the query into multiple pages. so at least it won't be doing huge queries if the list is append-only. but that doesn't completely help you since all of the pages stay subscribed. you can make a custom equivalent to usePaginatedQuery which unloads pages that are off-screen, or resets the query entirely when the page grows too large. i'm about to post an article about this kind of thing. it's solvable, but currently involves a lot of custom code to keep track of the pages.

amppOP•6mo ago

Yeah, i was thinking of something as simple as attempting to start it over if the page grows beyond 2x its initialNumItems. I was hoping someone had some code laying around that implemented a best practice.

lee•6mo ago

i'm not sure if this is a solution you want, but here's an example of a different way usePaginatedQuery can work https://discord.com/channels/1019350475847499849/1019350478817079338/1255262027731964016

amppOP•6mo ago

I'm just rendering events from my our custom event management system, so its not too different then logs but i have new stuff at the top. My main goal is to just avoid exploding the dataset and wasting db bandwidth.

lee•6mo ago

i don't think we have code for unloading or resetting pages lying around. if you come up with something, it would be useful to share. This kind of thing isn't built into usePaginatedQuery because it's hard to know whether you want to unload the top or bottom of the list; it's possible the user is interacting with either end, so it's hard to make a blanket judgement

amppOP•6mo ago

Yeah, this isn't exactly a high priority for us but perhaps its not really best to use pagination. I'm going to make a search page so perhaps the need for a load more goes to that page. However the need to keep table from expanding indefinitely would still exist. Generally viewing from the oldest first would be much less common. IMHO

ballingt•6mo ago

It sounds like you want non-reactive pagination: you don't want this list to grew when new items are added, or change when these items change, yeah?

amppOP•6mo ago

I would like to keep reactivity as recent additions are more likely to change once. But for this page i could skip the pagination and have a reactive query with the most recent 20 rows added to the db. But that would be a similar situation dealing with infinite growth from new rows i think.

lee•6mo ago

you could have a query that does

return ctx.db.query("tbl").order("desc").take(20);

return ctx.db.query("tbl").order("desc").take(20);

v•6mo ago

Rip

sshader•6mo ago

One potential UI pattern that could work here would be showing some indicator like "scroll to top to see new items" which shows up based on a query that queries for an item above the top of whatever your main query returned. And when the user clicks the button, you start a new query. If you're sorting by creation time, your main query would take in the current date, and load items created before that time, and then you could have another query that queries for the first item created after that time, which gates the "see new items" button. It means users have to click the button to see new things + you'll be loading some of the items multiple times across different queries

ian•6mo ago

FYI Lee just wrote this great article on a pagination helper that allows you to do a lot of custom behavior https://stack.convex.dev/pagination

Stack

Take Control of Pagination

Convex offers robust control over pagination with a powerful function, getPage, enabling complex edge cases. In this article, we go over how to use th...

ian•6mo ago

Two other related articles: https://stack.convex.dev/help-my-app-is-overreacting

Stack

Help, my app is overreacting!

Reactive backends like Convex make building live-updating apps a cinch, but default behavior might be too reactive for some use cases. Not to worry! L...

ian•6mo ago

and https://stack.convex.dev/coping-with-the-web-s-looming-global-reactivity-crisis

Stack

Managing Reactivity with useBufferedState

Reactivity has taken a dominant position today within web app development. Our components and app state are all reactive, and the world has adapted–mo...

v•6mo ago

😂