Starlord•2mo ago

starlord_gf's Thread

Hello, I have strange problem that when new item is added to the list while user is subscribed to this list via paginated query i get this error

Uncaught ConvexError: InvalidCursor: Tried to run a query starting from a cursor, but it looks like this cursor is from a different query.

Uncaught ConvexError: InvalidCursor: Tried to run a query starting from a cursor, but it looks like this cursor is from a different query.

but strange thing its happening only in convex production environement but on not in dev environment. Any idea what could be the reason?

51 Replies

convex-threads•2mo ago

Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets. If this is a support-related question, please create a post in #support-community. If not, carry on! Thanks!

lee•2mo ago

Yeah so the difference in dev vs prod is a bug that we'll fix in the next release. But you should figure out why it's happening. Can you share the code for the paginated query? In particular what is the .withIndex used before the .paginate()

StarlordOP•2mo ago

const productIdsWithStock = new Set(
      productsWithStock.map(variation => variation.productId)
    );

    // Use appropriate index based on sort field
    let query = ctx.db
      .query("products")
      .withIndex(sort.field, q => 
        q.eq("mainCategoryId", mainCategoryId)
      )
      .filter(q => 
        q.or(
          ...Array.from(productIdsWithStock).map(id => 
            q.eq(q.field("_id"), id)
          )
        )
      )
      .order(sort.direction);


    const results = await query.paginate(paginationOpts);

const productIdsWithStock = new Set(
      productsWithStock.map(variation => variation.productId)
    );

    // Use appropriate index based on sort field
    let query = ctx.db
      .query("products")
      .withIndex(sort.field, q => 
        q.eq("mainCategoryId", mainCategoryId)
      )
      .filter(q => 
        q.or(
          ...Array.from(productIdsWithStock).map(id => 
            q.eq(q.field("_id"), id)
          )
        )
      )
      .order(sort.direction);


    const results = await query.paginate(paginationOpts);

the bug happens only when filter is used and only in production environment productsWithStock is changing because of new added item

lee•2mo ago

Yep that would do it .paginate() only works consistently when the query isn't changing. If the filter is changing, the whole set of pages needs to reset

StarlordOP•2mo ago

and why is it working in dev environment?

lee•2mo ago

In dev environment it resets the pages, starting the pagination over. In prod environment it throws an error (this is the bad behavior that i'm fixing to make it reset the pages)

StarlordOP•2mo ago

ok do you know when the fix is live or do i need to change something in code?

lee•2mo ago

Something i don't understand about your query: if you have the full set of ids you want, why do you need pagination?

StarlordOP•2mo ago

because i dont want to send all the products to the client set of ids is there to filter valid items i have another table to track amount of available items this amount is used to return only products with higher amount than 0

lee•2mo ago

What if you fetch all the items in the query, and then filter them out before returning them

StarlordOP•2mo ago

but i need to return based on cursor. the user is scrolling down a list so need to send only new items that need to be fetched for the list there are 1k+ products

lee•2mo ago

Gotcha Are you okay with the client's set of pages resetting whenever there's a new product?

StarlordOP•2mo ago

 let variationsQuery = ctx.db
      .query("productVariations")
      .withIndex("by_amount", q => q.gt("amount", 0));

 const productsWithStock = await variationsQuery.collect();


    const productIdsWithStock = new Set(
      productsWithStock.map(variation => variation.productId)
    );

    // Use appropriate index based on sort field
    let query = ctx.db
      .query("products")
      .withIndex(sort.field, q => 
        q.eq("mainCategoryId", mainCategoryId)
      )
      .filter(q => 
        q.or(
          ...Array.from(productIdsWithStock).map(id => 
            q.eq(q.field("_id"), id)
          )
        )
      )
      .order(sort.direction);


    const results = await query.paginate(paginationOpts);

 let variationsQuery = ctx.db
      .query("productVariations")
      .withIndex("by_amount", q => q.gt("amount", 0));

 const productsWithStock = await variationsQuery.collect();


    const productIdsWithStock = new Set(
      productsWithStock.map(variation => variation.productId)
    );

    // Use appropriate index based on sort field
    let query = ctx.db
      .query("products")
      .withIndex(sort.field, q => 
        q.eq("mainCategoryId", mainCategoryId)
      )
      .filter(q => 
        q.or(
          ...Array.from(productIdsWithStock).map(id => 
            q.eq(q.field("_id"), id)
          )
        )
      )
      .order(sort.direction);


    const results = await query.paginate(paginationOpts);

also as far i can see its not possible to join queries in convex. so i have to fetch valid products first from one table to combine it with product data query to another table well yes but idk how the experience for the user would be. lets say he is scrolling a page down than new item is added. it would be better not to rerender all the items but to add this new item somehow just to the list its rare case because new products are added not that often but right now its breaking experience because of the error same error happens when item is deleted right now on dev environment it rerenders all the items when new product is added also regarding this i dont know what will the performance be and how much data would be consumed if there are thousands of productVariations. basically every paginate request need to refetch this table and i dont know better solution

lee•2mo ago

You could try upgrading to convex version 1.17.5-alpha.1 to see if it makes prod behave like dev

StarlordOP•2mo ago

can you send me npm for this please?

lee•2mo ago

npm install convex@1.17.5-alpha.1

StarlordOP•2mo ago

thanks

lee•2mo ago

Ok i have a new suggestion: can you do pagination on the productVariations query and do a join against the products table

lee•2mo ago

https://docs.convex.dev/database/pagination#transforming-results

Paginated Queries | Convex Developer Hub

Load paginated queries.

StarlordOP•2mo ago

let me check the things is each product has multiple variations. it can be one 1 - 10 variations or so. so pagination request is related to products table not to variations table

lee•2mo ago

Oh so you need to dedupe?

StarlordOP•2mo ago

i dont understand what you mean. each product has multiple variations with different amount. i need to fetch only products that have variations with amount > 0

lee•2mo ago

And sort i suppose

StarlordOP•2mo ago

the filter itself is quite complex i have sent only part of it. the user can filter products in a lot of ways but filter change is not a problem because it will restart query the problem is when items change like added or removed when admin adds products. varations change and products change

lee•2mo ago

How many productVariations can there be in total? In your current query you're fetching all of them, which makes the query large

StarlordOP•2mo ago

a lot there are 1k+ products and each of them can have multipe variations i can track amount inside product additionally and modify amount of products in both product and variation

lee•2mo ago

Are you aware that queries have a limit of 16k documents read https://docs.convex.dev/database/pagination#transforming-results

Paginated Queries | Convex Developer Hub

Load paginated queries.

StarlordOP•2mo ago

but i still need variations because variations have filter like color so thats why i need to filter products by variations variation = combination of amount, color, sizes etc its ecomerce app yes thats why i need to find better solution for pagination of products same error with this one

lee•2mo ago

Darn

StarlordOP•2mo ago

Uncaught ConvexError: InvalidCursor: Tried to run a query starting from a cursor, but it looks like this cursor is from a different query.

Uncaught ConvexError: InvalidCursor: Tried to run a query starting from a cursor, but it looks like this cursor is from a different query.

lee•2mo ago

I suppose my code change isn't in the alpha yet I still think this query needs some reworking

StarlordOP•2mo ago

yes it does

lee•2mo ago

So what goes wrong when you do pagination on the productVariations? I think we need to do that, and work through the issues one at a time 1. The join on "products" table -- use a .map on the page 2. More filters -- use a .filter on the page 3. Sort order -- denormalize fields onto productVariations table and put the indexes there

StarlordOP•2mo ago

hm lets say i paginate with request load more 10. how should it paginate variations if i need 10 products

lee•2mo ago

Call loadMore() until you have enough results

StarlordOP•2mo ago

but it will load variations that are not part of next products on the list i think i need to paginate products and load their variations for check after this and do manual pagination to load more products if some are missing to complete request

lee•2mo ago

That also works, good idea

StarlordOP•2mo ago

lets say loadmore = 10. i load 10 products check their variation = result is 9 products. i try to load 1 more product etc

lee•2mo ago

I would let the client call loadMore() until you have enough results. Doing it on the server is harder

StarlordOP•2mo ago

ok thanks. but this will not solve this exception or?

lee•2mo ago

If your paginated query doesn't change, the exception won't happen

StarlordOP•2mo ago

let me check if it works

lee•2mo ago

Also in case you haven't seen it, this may be useful https://stack.convex.dev/complex-filters-in-convex

Using TypeScript to Write Complex Query Filters

There’s a new Convex helper to perform generic TypeScript filters, with the same performance as built-in Convex filters, and unlimited potential.

StarlordOP•2mo ago

this is typescript filter that means it loads all the data from database first no? so normal database filter limits documents amount typescript will not work if there are more than 16k documents btw i dont see docs info about 16k limit

lee•2mo ago

Regular .filter will also not work if there are more than 16k documents

StarlordOP•2mo ago

and index? this is bad why is there no solution for large amount of data that needs to be filtered

lee•2mo ago

Whoops wrong link https://docs.convex.dev/production/state/limits#transactions

Limits | Convex Developer Hub

We’d love for you to have unlimited joy building on Convex but engineering

lee•2mo ago

Index does exclude documents from the scanned data, only filter doesn't

StarlordOP•2mo ago

ok so i can reduce with index first

lee•2mo ago

Yes. You can also do the pagination and filter afterwards, which can result in small/empty pages but then on the client you can call loadMore until you have enough

StarlordOP•2mo ago

this can result to a lot of loadmore requests if paginate doesnt find right items would be great if convex would support big data filtering out of the box on c# like backend i would cache results based on filters so i dont need to load them again from database Anyway. Thank you for the providing information. I will try to improve it on my end. But hope convex will get some improvements too for such logic ok new methods of paginating products and than fetch their variations works without exception. thanks

starlord_gf's Thread

Did you find this page helpful?