fakesilent
fakesilent•3mo ago

Ordered listings created within the last 6 days gives too many bytes read error

export const getTopRatedListings = internalQuery({
handler: async (ctx) => {
const sixDaysAgo = Date.now() - 6 * 24 * 60 * 60 * 1000;

return ctx.db
.query("listings")
.withIndex("by_cashflow")
.order("desc")
.filter((q) => q.gte(q.field("_creationTime"), sixDaysAgo))
.take(1);
},
});
export const getTopRatedListings = internalQuery({
handler: async (ctx) => {
const sixDaysAgo = Date.now() - 6 * 24 * 60 * 60 * 1000;

return ctx.db
.query("listings")
.withIndex("by_cashflow")
.order("desc")
.filter((q) => q.gte(q.field("_creationTime"), sixDaysAgo))
.take(1);
},
});
schema
...,
monthlyCashflow: v.optional(v.number()),
})
.index("by_cashflow", ["monthlyCashflow"])
...,
monthlyCashflow: v.optional(v.number()),
})
.index("by_cashflow", ["monthlyCashflow"])
This is giving me the following too many bytes read error:
Error: [CONVEX Q(fetchListings:getTopRatedListings)] [Request ID: 49905af9f0732044] Server Error
Uncaught Error: Too many bytes read in a single function execution (limit: 16777216 bytes). Consider using smaller limits in your queries, paginating your queries, or using indexed queries with a selective index range expressions.
Error: [CONVEX Q(fetchListings:getTopRatedListings)] [Request ID: 49905af9f0732044] Server Error
Uncaught Error: Too many bytes read in a single function execution (limit: 16777216 bytes). Consider using smaller limits in your queries, paginating your queries, or using indexed queries with a selective index range expressions.
I am genuinely puzzled on how to do this without running into this error. ~6k documents
12 Replies
Convex Bot
Convex Bot•3mo ago
Thanks for posting in <#1088161997662724167>. Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets. - Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.) - Use search.convex.dev to search Docs, Stack, and Discord all at once. - Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI. - Avoid tagging staff unless specifically instructed. Thank you!
Clever Tagline
Clever Tagline•3mo ago
Your withIndex syntax is incomplete. You're not using it to make any comparison using the cashflow field (guessing the field name from the index name), so it's effectively pushing the full table read through to the order step. Review the docs if needed: https://docs.convex.dev/database/reading-data/indexes/#querying-documents-using-indexes
Indexes | Convex Developer Hub
Speed up queries with database indexes
fakesilent
fakesilentOP•3mo ago
I wouldn't like to make a comparison on the cashflow field, I am merely using the index for sorting. I just need to get highest cashflow listings from the last 6 days.
Clever Tagline
Clever Tagline•3mo ago
Gotcha. I also overlooked that you'd included the table index in your original post, so I didn't need to guess the field name. 😂 While the index does help with sorting, by not applying any criteria it's going to return the entire table. Instead of doing the filter outside the index, do it in the index and remove the .filter() section. Per the docs, you'll need to first make at least one .eq() comparison, and then you can do the date comparison with the _creationTime field (that field is included in every index by default). I feel that this might yield better results than filtering outside the index, though I could be mistaken.
fakesilent
fakesilentOP•3mo ago
Well that is kind of the issue haha, i need to do one eq but it has to be on the monthlyCashflow field which is just not possible because that is what I am trying to sort. It seems so weird to not be able to do this normally. Granted of course I have wide documents but I don't mind if it takes longer, I just want to be able to do it lol. The docs mention if you run into this limit, to use a index, but that is precisely what I am using... very frustrating.
Clever Tagline
Clever Tagline•3mo ago
Yeah, I hear you. I'm still not familiar enough with the nuances of complex queries to wrap my head around how else to accomplish this. And I just dug deeper into the index part of the docs and saw some examples following the pattern you used (.withIndex but no range expression), which I somehow thought was less efficient, but there are some use cases mentioned there (though they aren't clicking as readily as I'd hoped).
fakesilent
fakesilentOP•3mo ago
Yeh is there anyway we can get someone who is more familiar with this? I would appreciate that, as I'm kind of at a breaking point here haha
Clever Tagline
Clever Tagline•3mo ago
Tagging @erquhart to see if he's got any ideas.
erquhart
erquhart•3mo ago
Merging Streams of Convex data
New convex-helpers are available now for fetching streams of documents, merging them together, filtering them them out, and paginating the results. Wi...
Clever Tagline
Clever Tagline•3mo ago
I've only used streams once, I set it up months ago, and it was in a use case quite different from this, so my mind didn't even go there.
erquhart
erquhart•3mo ago
I've only used it some, but it can do some pretty cool stuff
fakesilent
fakesilentOP•3mo ago
Awesome, I'll take a look - thanks.

Did you find this page helpful?