What is the best way to implement infinite scrolling with Convex

I have this function to call all the items in a table

export const getPortfolios = query({
  handler: async (ctx) => {
    return await ctx.db.query('portfolios').collect();
  },
});

export const getPortfolios = query({
  handler: async (ctx) => {
    return await ctx.db.query('portfolios').collect();
  },
});

I want to implement infinite scrolling, so rather than overloading the app only show more items when scrolling wanted to know if there is a way I can do this server side rather than client side.

41 Replies

erquhart•6mo ago

Does paginated query not work for your use case?

Hosna QasmeiOP•6mo ago

yes it does, just looked it up wasn't aware of it 😅 will try it, thank you!

erquhart•6mo ago

ah - no problem! realtime paginated query is one of the best parts of Convex, you'll love it

Hosna QasmeiOP•6mo ago

Wow paginated query is amazing! Wanted to know if you can help, with in the paginated query I want to sort and filter if possible. This is what I have right now, only the sorting. Not sure if it's right. Wanted to also do filtering,

 
export const getPortfolios = query({
  args: {
    paginationOpts: paginationOptsValidator,
    sortType: v.string(),
    filterType: v.string(),
  },
  handler: async (ctx, args) => {
    const { paginationOpts, sortType, filterType } = args;

    // TODO:Filtering
  
    // Sorting
    if (sortType === 'recentlyAdded') {
      return await ctx.db
        .query('portfolios')
        .filter((q) => q.eq(q.field('tags'), [filterType])) // Doesn't work
        .order('desc')
        .paginate(paginationOpts);
    } else if (sortType === 'mostPopular') {
      return await ctx.db
        .query('portfolios')
        .filter((q) => q.eq(q.field('tags'), [filterType])) // Doesn't work
        .withIndex('by_favoritesCount')
        .order('desc')
        .paginate(paginationOpts);
    } else if (sortType === 'alphabetical') {
      return await ctx.db
        .query('portfolios')
        .filter((q) => q.eq(q.field('tags'), [filterType])) // Doesn't work
        .withIndex('by_name')
        .order('asc')
        .paginate(paginationOpts);
    } else {
      return await ctx.db.query('portfolios').paginate(paginationOpts);
    }
  },
});

 
export const getPortfolios = query({
  args: {
    paginationOpts: paginationOptsValidator,
    sortType: v.string(),
    filterType: v.string(),
  },
  handler: async (ctx, args) => {
    const { paginationOpts, sortType, filterType } = args;

    // TODO:Filtering
  
    // Sorting
    if (sortType === 'recentlyAdded') {
      return await ctx.db
        .query('portfolios')
        .filter((q) => q.eq(q.field('tags'), [filterType])) // Doesn't work
        .order('desc')
        .paginate(paginationOpts);
    } else if (sortType === 'mostPopular') {
      return await ctx.db
        .query('portfolios')
        .filter((q) => q.eq(q.field('tags'), [filterType])) // Doesn't work
        .withIndex('by_favoritesCount')
        .order('desc')
        .paginate(paginationOpts);
    } else if (sortType === 'alphabetical') {
      return await ctx.db
        .query('portfolios')
        .filter((q) => q.eq(q.field('tags'), [filterType])) // Doesn't work
        .withIndex('by_name')
        .order('asc')
        .paginate(paginationOpts);
    } else {
      return await ctx.db.query('portfolios').paginate(paginationOpts);
    }
  },
});

here is my schema for portfolios

export default defineSchema({
  portfolios: defineTable({
    name: v.string(),
    link: v.string(),
    tags: v.optional(v.array(v.string())),
    image: v.id('_storage'),
    favoritesCount: v.optional(v.number()),
  })
    .index('by_favoritesCount', ['favoritesCount'])
    .index('by_name', ['name'])
    .index('by_tags', ['tags']),
...

export default defineSchema({
  portfolios: defineTable({
    name: v.string(),
    link: v.string(),
    tags: v.optional(v.array(v.string())),
    image: v.id('_storage'),
    favoritesCount: v.optional(v.number()),
  })
    .index('by_favoritesCount', ['favoritesCount'])
    .index('by_name', ['name'])
    .index('by_tags', ['tags']),
...

Michal Srb•6mo ago

Yes, you can filter, like you have it or in JS: https://docs.convex.dev/database/pagination#transforming-results

Paginated Queries | Convex Developer Hub

Load paginated queries.

Hosna QasmeiOP•6mo ago

Ah yes, I thouhght I saw that somewhere. Thanks @Michal Srb !

erquhart•6mo ago

I believe you can output anything in your pages, as long as it's deterministic based on the original pages received from the query.

Hosna QasmeiOP•6mo ago

okay so here is my dilemma

export const listWithTransformation = query({
  args: { paginationOpts: paginationOptsValidator },
  handler: async (ctx, args) => {
    const results = await ctx.db
      .query("messages")
      .order("desc")
      .paginate(args.paginationOpts);
    return {
      ...results,
      page: results.page.map((message) => ({
        author: message.author.slice(0, 1),
        body: message.body.toUpperCase(),
      })),
    };
  },
});

export const listWithTransformation = query({
  args: { paginationOpts: paginationOptsValidator },
  handler: async (ctx, args) => {
    const results = await ctx.db
      .query("messages")
      .order("desc")
      .paginate(args.paginationOpts);
    return {
      ...results,
      page: results.page.map((message) => ({
        author: message.author.slice(0, 1),
        body: message.body.toUpperCase(),
      })),
    };
  },
});

for this example, results only returns a subset of the data because of pagination. Lets say I want to filter and sort from All the data in the messages table . If I do it through this example, it will only do i from that subset from results right? how would i do it the way I need to

erquhart•6mo ago

pondering ah I see what you mean you can sort/filter the complete result in the frontend, but things will shift around as the user scrolls through, which I'm guessing you don't want filtering isn't the problem, it's sorting

Hosna QasmeiOP•6mo ago

no it's the opposite

erquhart•6mo ago

filtering shouldn't be impacted I don't think, let me know if that's not the case oh?

Hosna QasmeiOP•6mo ago

sorting is fine, it's the filtering that's giving the issue

Hosna QasmeiOP•6mo ago

https://www.portfolioshub.com/

PortfoliosHub

Find the best portfolios and showcase your work..

Hosna QasmeiOP•6mo ago

if you go here, and select one of the options that isn't All it gets glitchy

erquhart•6mo ago

I don't see a filtering problem in your code there okay looking Can you share how you're filtering? Filtering shouldn't cause an issue. I'd also expect the kind of filtering you're doing there to be index driven

Hosna QasmeiOP•6mo ago

yup, filtering on the front end

  const { results, status, loadMore } = usePaginatedQuery(
    api.portfolios.getPortfolios,
    {
      sortType: selectedSort || 'recentlyAdded',
      filterType: selectedFilter || 'All',
    },
    { initialNumItems: 6 },
  );

 const filteredData =
    selectedFilter === 'All' || selectedFilter === null || !results
      ? results
      : results.filter(
          (portfolio) =>
            portfolio.tags &&
            portfolio.tags.map((tag) => `${tag}s`).includes(selectedFilter),
        );

  const { results, status, loadMore } = usePaginatedQuery(
    api.portfolios.getPortfolios,
    {
      sortType: selectedSort || 'recentlyAdded',
      filterType: selectedFilter || 'All',
    },
    { initialNumItems: 6 },
  );

 const filteredData =
    selectedFilter === 'All' || selectedFilter === null || !results
      ? results
      : results.filter(
          (portfolio) =>
            portfolio.tags &&
            portfolio.tags.map((tag) => `${tag}s`).includes(selectedFilter),
        );

erquhart•6mo ago

It almost seems less like an actual glitch and more like the animations are appearing glitchy when loading partial pages

Hosna QasmeiOP•6mo ago

So I initially did the sort/filter the in the frontend approach it worked for me on desktop but mobile i was coming across this issue https://github.com/vercel/next.js/issues/34455 so to avoid it i wanted to do infinite scrolling

GitHub

iOS safari crashes when huge amount of next/image instances are ren...

Verify canary release I verified that the issue exists in Next.js canary release Provide environment information Operating System: Platform: linux Arch: x64 Version: Ubuntu 20.4.0 LTS Thu Feb 17 20...

erquhart•6mo ago

when you're not filtering you always load a full page so it seems smoother

Hosna QasmeiOP•6mo ago

okay gotcha, but you notice when you filter to another tab the load spinner shows up so im not sure if that's filtering based of the query and paginating to look for more results because all the data isnt there

erquhart•6mo ago

It's getting more pages more often because you're filtering on the frontend. This will be much smoother if you filter on the backend. And you'll use less data (user is loading way more than is being displayed currently, unless viewing the All tab)

Hosna QasmeiOP•6mo ago

right

erquhart•6mo ago

I would index on that filter field (assuming that filter is driven by a single field) and use that in the paginated query doesn't address the mobile issue though maybe smaller pages or lower weight images would help? Next is a monster lol, a lot going on in there You shouldn't have to change too much, you can keep filter state in the frontend as you currently are, and pass that into your paginated query

Web Dev Cody•6mo ago

if you're just trying to paginate, you could try checking this out https://github.com/webdevcody/thumbnail-critique/blob/main/convex/thumbnails.ts#L167

GitHub

thumbnail-critique/convex/thumbnails.ts at main · webdevcody/thumbn...

Contribute to webdevcody/thumbnail-critique development by creating an account on GitHub.

Hosna QasmeiOP•6mo ago

@Web Dev Cody i wish I was trying to just paginate lol, I want to filter and sort as well from the backend if possible

erquhart•6mo ago

Did my last comment make sense, anything still unclear on how to proceed

Hosna QasmeiOP•6mo ago

was the last comment just to stick with filter the frontend since I need access to all the data?

ampp•6mo ago

Maybe https://github.com/get-convex/fullstack-convex provides some examples of filtering? we used it to progress some of our stuff. Plus it has infinite scroll. Its just old now, id love it if convex would take a once through on some of these projects and see how they might be progressed with the latest improvements.

GitHub

GitHub - get-convex/fullstack-convex: Fullstack.app implementation ...

Fullstack.app implementation using Convex / Auth0. Contribute to get-convex/fullstack-convex development by creating an account on GitHub.

erquhart•6mo ago

No, filter on the backend using the selected filter as an index

Hosna QasmeiOP•6mo ago

so the selected filter is an array of strings dont think I can do q.contains, no sure what other way there is im trying to filter based on tags, so if the tab is in tabs of that item I want to include it

erquhart•6mo ago

Ahh it's tags based

Hosna QasmeiOP•6mo ago

Yeah 😅

erquhart•6mo ago

I know they added support for general js filtering in the query filter might require a library though checking

lee•6mo ago

https://stack.convex.dev/complex-filters-in-convex

Stack

Using TypeScript to Write Complex Query Filters

There’s a new Convex helper to perform generic TypeScript filters, with the same performance as built-in Convex filters, and unlimited potential.

erquhart•6mo ago

Thanks @lee that's it So if you filter in the query (as opposed to getting the page of data and then filtering in your query function), you can use doc.tags.includes(tag), for example, and get full pages that abide by the filter. As opposed to filtering after the fact and getting less than a full page, or maybe even an empty page. If you use the filter helper from the article, that is

Hosna QasmeiOP•6mo ago

Ah yes that article looks like it’s it, let me take a look and try it thanks @lee @erquhart

lee•6mo ago

the filter helper in the article is equivalent to filtering after the fact, btw. so the pages might be empty

erquhart•6mo ago

does that also apply to ctx.db.query().filter()? If so I misunderstood this part If there isn't a great way to get complete pages, I don't think backend filtering will improve your current situation You could either do some extra work to make indexing on these tags possible (search is one possibility, would probably involve adding a field with a string containing all tags and searching on that), or dig into the animations that are causing the glitchy presentation so that it's still smooth when loading small pages.

lee•6mo ago

ctx.db.query().filter().paginate() does apply the filter before getting the page, so you won't get empty pages. but it also has the potential to read way too many rows and hit a query limit.

ampp•6mo ago

It would be amazing if we could build a comprehensive master list of all things filtering and index, add it to this maybe https://labs.convex.dev/convex-vs-prisma . Complex queries is one of the few things I'm avoiding. It ends up being a lot of cross referencing and i still don't know if there is a more efficient way. Or i forgot the way i did it a week ago and do it a less efficient way, admittedly ents adds a layer to this. Now i need to try filter(..

Prisma vs Convex

Compare Prisma examples with Convex

erquhart•6mo ago

When you have to make it work, denormalization (sometimes extreme denormalization) is often the way I've had to exercise a lot more creativity in how I structure my data than I would with a traditional db, but it's unquestionably worth it for the benefits. In Hosna's case here, as long as there will only be a small number of tabs on that filter page, it'd be reasonable to make those filter tags correspond to individual boolean fields on the table that's being paginated. Then you can index directly against the fields. Conceptually that seems strenuous, but the reality is this is trivial to do with Convex. And when requirements change down the road, you can restructure if need be.

What is the best way to implement infinite scrolling with Convex

Did you find this page helpful?