pagination and caching
So what's the point of this argument? Bite the dust and accept the 2x bills?
39 Replies
happy to talk through paging + caching strategies--but so you know, you can always go back to manual page management like traditional databases if you don't want to use convex's automatic pagination
using something like the aggregate module
if you want the full control, you can have it. you just don't get automatic reactive page management
what kind of aggregate module is it and where do i find it
caching on paginated queries got worse there is 0 caching right now
i am not only one who complains about data bandwith consumtion
so, same query, no mutations, no caching?
its query only and i wrote ticket 24 ago with no response
https://www.npmjs.com/package/convex-helpers#query-caching does this help?
npm
convex-helpers
A collection of useful code to complement the official convex package.. Latest version: 0.1.87, last published: a day ago. Start using convex-helpers in your project by running
npm i convex-helpers. There are 13 other projects in the npm registry using convex-helpers.jamie is basically the king of this topic
i know about client side caching has nothing to do with it here
well, I actually haven't worked on the caching guts of the databasde in awhile
but we have people that know it well
wow
we have sites at a lot of scale that go down without caching... so normally if there's a blip, we hear about it fast from our big customers.
but its still entirely possible we have a regression
so happy to take a look
no mutations, two queries in a row, no caching is what I'm hearing
on a paginated query
that does sound wrong
sounds bad
yep
how much less do cached queries cost? its only the bandwidth or more
i tried everything no caching at all on paginated queries and huge db consumtion
ill do some test rn
my bill jumped from 25 to 90 usd
with shrinking userbase
big baller
okay. yeah, that sounds crazy
will look
is it the same exact code
yes
small update to db structure only
i had peak active users when i had 25 usd bill
can you dm me your team slug name? that way we can dig into your specific account and try to find this issue
the only thing that changed is there are more documents in the database but that what paginated querie is there for
sure
do you think more than 5mb
i ask just because im going to run some test
purely for my own curiosity
yeah they are not caching for me either
this is with no auth, I take it? just regular ol paginated query on a table?
yes
got it. yeah that seems bad
will get some people looking into this
thanks
i even tried on convex 18
i guess it doesnt matter
maybe that doesn't work the way i thought
yeah, this is surprising. we'll look into it
this should be cached
w jamie as always and a very good report from starlord
so right now, is there no way to use this with pagination?
they are retrieving the data server side it seems so client side caching is of no use
sorta got the "hover cache" working with the query-caching helper
after multiple tests here is conclusion: query.paginate(paginationOpts) call invalidates cache of the query
i ended up writing own pagination solution that solved the problem with caching
also one more thing that invalidates all the cache of entire app = any change to the code of anything in convex
it should not happen and kind of stupid. it should invalidate cache only of affected functions
can you send these as codeblocks?
Pastebin
const { numItems, cursor: cursorString } = paginationOpts;const num...
Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.
but beware this is for my products table
also my client pagination code is slightly modified and default may not accept isdone parameter but hasMore
but now i see its not very good solution and is working because query did not hit document limit yet. its operates on filter to instead of index because index is not possible here