oscklm
oscklm•13mo ago

Advanced paginated query use case... is an custom paginate() a possibility?

So i'm trying to build a better feed for our app. Currently wanting to use the .paginate() to paginate my query. So essentially what i want is to always receive X amount of results, but importantly i want a order of 2 horizontal video results, followed by 2 verticals results, since this is a feed consisting of vertical and horizontal videos that eventually gets build into a feed layout. This is what i got now 
export const feedList = query({
  args: {
    aspectRatio: v.union(v.literal('16:9'), v.literal('9:16')),
    paginationOpts: paginationOptsValidator,
  },
  handler: async (ctx, args) => {
    // Get all videos
    const results = await ctx.db
      .query('videos')
      .filter((q) => q.eq(q.field('muxData.aspectRatio'), args.aspectRatio))
      .order('desc')
      .paginate(args.paginationOpts)

    // Enrich videos with user and thumbnail
    const page = await Promise.all(results.page.map(async (video) => enrichVideo(ctx, video)))

    return {
      ...results,
      page,
    }
  },
})
export const feedList = query({
  args: {
    aspectRatio: v.union(v.literal('16:9'), v.literal('9:16')),
    paginationOpts: paginationOptsValidator,
  },
  handler: async (ctx, args) => {
    // Get all videos
    const results = await ctx.db
      .query('videos')
      .filter((q) => q.eq(q.field('muxData.aspectRatio'), args.aspectRatio))
      .order('desc')
      .paginate(args.paginationOpts)

    // Enrich videos with user and thumbnail
    const page = await Promise.all(results.page.map(async (video) => enrichVideo(ctx, video)))

    return {
      ...results,
      page,
    }
  },
})
And in the frontend i have 2 separate paginatedQueries, and a function that loads 2 results from each of those, order the results and builds the feed etc. 
// How i use them
const loadMoreVideos = () => {
    // these are from the usePaginatedQuery()
    loadHorizontalVideos(LOAD_AMOUNT) 
    loadVerticalVideos(LOAD_AMOUNT)

    if (horizontalStatus === 'Exhausted' || verticalStatus === 'Exhausted') return

    const newHorizontalVideos = getNewResults(horizontalVideoResults)
    const newVerticalVideos = getNewResults(verticalVideoResults)

    const newOrder = generateElements(newHorizontalVideos, newVerticalVideos, 2)

    setVideoBlocks([...videoBlocks, ...newOrder])
  }
// How i use them
const loadMoreVideos = () => {
    // these are from the usePaginatedQuery()
    loadHorizontalVideos(LOAD_AMOUNT) 
    loadVerticalVideos(LOAD_AMOUNT)

    if (horizontalStatus === 'Exhausted' || verticalStatus === 'Exhausted') return

    const newHorizontalVideos = getNewResults(horizontalVideoResults)
    const newVerticalVideos = getNewResults(verticalVideoResults)

    const newOrder = generateElements(newHorizontalVideos, newVerticalVideos, 2)

    setVideoBlocks([...videoBlocks, ...newOrder])
  }
And what i'm searching for is a better approach, it seems like not really the best way to go about it right now. But has worked fine for just testing it out 🙂 So would greatly appreciate, thoughts on this and any ideas!
6 Replies
Indy
Indy•13mo ago
Have you considered using q.or in the .filter to make this one paginated query? See second example here: https://docs.convex.dev/database/reading-data#combining-operators
Reading Data | Convex Developer Hub
Query and
oscklm
oscklmOP•13mo ago
I have played around with paginate for a few days now, but haven’t been able to really get around having to 2 individual paginated queries in the front end. Looking at the q.or in the docs and filter, I don’t really see how this could allow me to do the following; Database query the video table, paginating the results, and each time load 4 results where aspect ratio matches X followed by 4 results where it matches Y. I can’t seem to figure out if this is possible with convex. I could see it maybe being possible if making some sort of custom paginated query was a thing. If something like this was possible, I could see it work like; Example, yet .slice() obviously is imaginary here 
let lastSlicePosition;

const 4x16by9results = ctx.query(‘videos’).filter((q) => q.eq(q.field(‘aspect ratio’, ‘16:9’).slice(0,4)

const 4x9by6results = ctx.query(‘videos’).filter((q) => q.eq(q.field(‘aspect ratio’, ‘9:16’).slice(0,4)

return […4x16by9results,…4x9by6results]
let lastSlicePosition;

const 4x16by9results = ctx.query(‘videos’).filter((q) => q.eq(q.field(‘aspect ratio’, ‘16:9’).slice(0,4)

const 4x9by6results = ctx.query(‘videos’).filter((q) => q.eq(q.field(‘aspect ratio’, ‘9:16’).slice(0,4)

return […4x16by9results,…4x9by6results]
Just a thought, still tryna figure this out. Not sure if this kind of queries are something you guys have any advice for? Sorry if I’m missing something obvious here, kinda hit a wall with this one for now.
Michal Srb
Michal Srb•13mo ago
So i'm trying to build a better feed for our app.
When I hear "feed", I think of FB/Twitter, and those are precomputed/computed on demand. You might want to further complicate the ordering, and so it seems the most scalable is to move the logic from read time (trying to paginate multiple ways at the same time) to write time (for each user or global, compute the set of items to show them). Have you explored this direction?
oscklm
oscklmOP•13mo ago
That’s an interesting direction. Which I have not thought about Michael. So essentially moving the computation off the feed into when the video is added to the platform. In this case, it’s by us (staff) and right now while we scale up our user base, we still have a very limited amount of videos being uploaded weekly. This def gives some food for thought. I have no idea how I would go about doing that, but I will definitely look into it. I’m eager to find any resources on how YouTube and other video platforms go about generating a feed, especially one containing not only 1 aspect ratio type of content, that seems to me is where it becomes complicated, at least when wanting to still preserve some sort of system for paginating the queries.
Michal Srb
Michal Srb•13mo ago
If all you're after is replicating the experience of having landscape videos, then a portaits section, I image you can do that with simple paginated queries from the client. I'm thinking of YouTube, the main list is a paginated query for vertical, and then each portrait shorts section is a separate paginated query for portrait. You can reuse the same paginated query for the portrait sections (slice it up on the client).
oscklm
oscklmOP•13mo ago
Yeah, that’s pretty much the approach I’ve gone with for now, having 2 separate queries. It does work, and for now it’s a fine solution, while I search for any better way of going about it - if there is one worth implementing. still the only dev at our little startup for now 🚀 One thing is certain, even though I find it being rather complex. it’s very fun and interesting building a video recommendation feed and also planning out the algorithm and all that. What I wouldn’t give for 30 min of picking the brain of someone who has done it for a large platform before 😂 Appreciate your inputs Michael, as always. I will share my learnings here, and I warmly welcome any other inputs

Did you find this page helpful?