Tom Redman•8h ago

"Too many bytes read" when using an index + filter, but no issues when removing the filter

Hey all, My understanding is that I should use an index to minimize the number of returned results, and using a filter beyond that will only filter on that subset of documents (vs a whole table scan). I'm seeing slightly different behavior than expected. This query throws a "too many bytes" error (and the dashboard log shows it reads ~9k rows, even though my paginate function is limited in this case to just 1):

const novemberFirst = 1761955200000;
const { page, isDone, continueCursor } = await ctx.db
.query("emailEvents")
.withIndex("by_creation_time", (q) => q.gt("_creationTime", novemberFirst))
.filter((q) =>
  q.and(
    q.neq(q.field("executionId"), undefined),
    q.or(
      q.eq(q.field("templateId"), undefined),
      q.eq(q.field("templateScheduledFor"), undefined),
      q.eq(q.field("campaignType"), undefined),
      q.eq(q.field("campaignTypeId"), undefined),
      q.eq(q.field("contactId"), undefined)
    )
  )
)
.paginate({
  cursor: null,
  numItems: 1
});

const novemberFirst = 1761955200000;
const { page, isDone, continueCursor } = await ctx.db
.query("emailEvents")
.withIndex("by_creation_time", (q) => q.gt("_creationTime", novemberFirst))
.filter((q) =>
  q.and(
    q.neq(q.field("executionId"), undefined),
    q.or(
      q.eq(q.field("templateId"), undefined),
      q.eq(q.field("templateScheduledFor"), undefined),
      q.eq(q.field("campaignType"), undefined),
      q.eq(q.field("campaignTypeId"), undefined),
      q.eq(q.field("contactId"), undefined)
    )
  )
)
.paginate({
  cursor: null,
  numItems: 1
});

But this query does not:

const novemberFirst = 1761955200000;
const { page, isDone, continueCursor } = await ctx.db
.query("emailEvents")
.withIndex("by_creation_time", (q) => q.gt("_creationTime", novemberFirst))
.paginate({
  cursor: null,
  numItems: 1
});

const novemberFirst = 1761955200000;
const { page, isDone, continueCursor } = await ctx.db
.query("emailEvents")
.withIndex("by_creation_time", (q) => q.gt("_creationTime", novemberFirst))
.paginate({
  cursor: null,
  numItems: 1
});

Notice I'm also using paginate, which should limit the results no matter what, correct? I guess I can drop the filter() and manually filter the pages of events, but I'm just wondering where my gap in understanding might be.

1 Reply

Convex Bot•8h ago

Thanks for posting in <#1088161997662724167>. Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets. - Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.) - Use search.convex.dev to search Docs, Stack, and Discord all at once. - Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI. - Avoid tagging staff unless specifically instructed. Thank you!

"Too many bytes read" when using an index + filter, but no issues when removing the filter

Did you find this page helpful?