Ordered listings created within the last 6 days gives too many bytes read error
schema
This is giving me the following too many bytes read error:
I am genuinely puzzled on how to do this without running into this error. ~6k documents
12 Replies
Thanks for posting in <#1088161997662724167>.
Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets.
- Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.)
- Use search.convex.dev to search Docs, Stack, and Discord all at once.
- Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI.
- Avoid tagging staff unless specifically instructed.
Thank you!
Your
withIndex syntax is incomplete. You're not using it to make any comparison using the cashflow field (guessing the field name from the index name), so it's effectively pushing the full table read through to the order step. Review the docs if needed: https://docs.convex.dev/database/reading-data/indexes/#querying-documents-using-indexesIndexes | Convex Developer Hub
Speed up queries with database indexes
I wouldn't like to make a comparison on the cashflow field, I am merely using the index for sorting. I just need to get highest cashflow listings from the last 6 days.
Gotcha. I also overlooked that you'd included the table index in your original post, so I didn't need to guess the field name. 😂
While the index does help with sorting, by not applying any criteria it's going to return the entire table. Instead of doing the filter outside the index, do it in the index and remove the
.filter() section. Per the docs, you'll need to first make at least one .eq() comparison, and then you can do the date comparison with the _creationTime field (that field is included in every index by default). I feel that this might yield better results than filtering outside the index, though I could be mistaken.Well that is kind of the issue haha, i need to do one
eq but it has to be on the monthlyCashflow field which is just not possible because that is what I am trying to sort.
It seems so weird to not be able to do this normally. Granted of course I have wide documents but I don't mind if it takes longer, I just want to be able to do it lol.
The docs mention if you run into this limit, to use a index, but that is precisely what I am using... very frustrating.Yeah, I hear you. I'm still not familiar enough with the nuances of complex queries to wrap my head around how else to accomplish this.
And I just dug deeper into the index part of the docs and saw some examples following the pattern you used (
.withIndex but no range expression), which I somehow thought was less efficient, but there are some use cases mentioned there (though they aren't clicking as readily as I'd hoped).Yeh is there anyway we can get someone who is more familiar with this? I would appreciate that, as I'm kind of at a breaking point here haha
Tagging @erquhart to see if he's got any ideas.
Streams is what you want: https://stack.convex.dev/merging-streams-of-convex-data
Merging Streams of Convex data
New convex-helpers are available now for fetching streams of documents, merging them together, filtering them them out, and paginating the results. Wi...
I've only used streams once, I set it up months ago, and it was in a use case quite different from this, so my mind didn't even go there.
I've only used it some, but it can do some pretty cool stuff
Awesome, I'll take a look - thanks.