RJ
RJ•3y ago

filter vs withIndex

I'm encountering a very confusing querying bug, and I'm not sure whether it's an issue with Convex or my misunderstanding of the API 🤔
26 Replies
RJ
RJOP•3y ago
Given a schema that looks like this:
export default defineSchema({
docs: defineTable({
doc: s.string(),
}),
steps: defineTable({
docId: s.id("docs"),
position: s.number(),
step: s.string(),
clientId: s.string(),
}).index("by_doc_id_and_position", ["docId", "position"]),
});
export default defineSchema({
docs: defineTable({
doc: s.string(),
}),
steps: defineTable({
docId: s.id("docs"),
position: s.number(),
step: s.string(),
clientId: s.string(),
}).index("by_doc_id_and_position", ["docId", "position"]),
});
these two queries do not return the same results:
const stepsWithIndex: Document<"steps">[] = await db
.query("steps")
.withIndex("by_doc_id_and_position", (q) =>
q.eq("docId", docId).gt("position", version)
)
.collect();

const stepsFilter: Document<"steps">[] = await db
.query("steps")
.filter((q) =>
q.and(q.eq(q.field("docId"), docId), q.gt(q.field("position"), version))
)
.collect();
const stepsWithIndex: Document<"steps">[] = await db
.query("steps")
.withIndex("by_doc_id_and_position", (q) =>
q.eq("docId", docId).gt("position", version)
)
.collect();

const stepsFilter: Document<"steps">[] = await db
.query("steps")
.filter((q) =>
q.and(q.eq(q.field("docId"), docId), q.gt(q.field("position"), version))
)
.collect();
Should they?
ian
ian•3y ago
Is the ordering the difference? The former will be in position order, then creation time order. The second will be in creation time order
RJ
RJOP•3y ago
Interesting! That's good to know, but no, it's not. I'm getting a different number of documents between the two queries. In fact, unless I'm misreading something, it looks like I'm just getting steps for the wrong docId
ian
ian•3y ago
Which one is incorrect? Are they both running in the same query? Mutation? Or run separately? This is very surprising
RJ
RJOP•3y ago
withIndex is incorrect In a query which initially used the stepsWithIndex version I switched to the stepsFilter version to test whether this would fix the issues I'd been observing in the UI code, after narrowing the problem down to unexpected results from the particular Convex function which contained it So I've never tested running them at the same time, in the same query. But I will now! They produce different results (differing by number of documents) in the same query
ian
ian•3y ago
Is withIndex one returning more? There’s a max number of rows scanned currently, which includes rows scanned that don’t match the filter. So the filter query may be returning a subset of results. But I thought it’d fail in that case rather than silently return the subset. I haven’t seen the code on it though If the withIndex results have the wrong docId that would be very surprising
RJ
RJOP•3y ago
No, the difference is between 0 and n, where n seems, based on my observations so far, arbitrarily large I'll triple-check that the docId is wrong in the withIndex version of the query (I was also very surprised!)
ian
ian•3y ago
Some sample data that reproduces this would be awesome, so we could tease this out
RJ
RJOP•3y ago
Hmm, how best to do that? I could share an export of the tables and copy the full code of the query? I don't mind sharing access to the project, either And sorry—also triple-checked that the docId is wrong for the withIndex query, it is
ian
ian•3y ago
Either exporting the data or adding a teammate if the data is in prod should work.
lee
lee•3y ago
it could also help debug on our side if you share your deployment name (https://dashboard.convex.dev/ -> click project (and pick production or development) -> settings -> deployment url)
RJ
RJOP•3y ago
The data is in the dev environment, and here is the deployment name: https://healthy-opossum-607.convex.cloud May I have an email to add one (or both) of you to the project? Seems easier than copy+pasting a bunch of stuff(?) to the team*, I guess?
presley
presley•3y ago
presley@convex.dev
RJ
RJOP•3y ago
Sent
presley
presley•3y ago
I only see the prod deployment (currently dev deployments are private for each member). Can you repro this there? What query you are calling that is observing the bug?
RJ
RJOP•3y ago
Ah right Yeah I can getStepsSince One sec Actually, will be more than one sec; here are exports of the data in the meantime
RJ
RJOP•3y ago
RJ
RJOP•3y ago
There's a nice Parcel (the JS bundler I'm using) bug which is preventing my production deploys from working at the moment. Are the above exports sufficient for reproduction?
presley
presley•3y ago
We are working ion it Should be sufficient
RJ
RJOP•3y ago
Ok, let me know if not, I'm sure I can figure something out if necessary!
sujayakar
sujayakar•3y ago
@RJ, to confirm, the doc field on docs is a string that's a JSON encoded document?
RJ
RJOP•3y ago
Correct @sujayakar Same with the step field on steps
sujayakar
sujayakar•3y ago
okay! we've got a repro and have found the bug. we'll have a fix up shortly and will update the thread here when it's out. thanks for the detailed report @RJ 🙂
RJ
RJOP•3y ago
Fantastic! Thanks @sujayakar
presley
presley•3y ago
You can also kick me out of your project. We don't have a button to leave yet. Ok, issue should be fixed now. Sorry for the trouble @RJ and thank you for the report.
RJ
RJOP•3y ago
No worries, glad to have found that one sooner rather than later 🙂

Did you find this page helpful?