yes102098320•3mo ago

Time Series Data to User for graph

Hi everyone, I'm looking for some advice, I currently have a requirement that requires sending all time series data for an "entity" so that the user can view the data in a Time Series graph. Currently the time series data is stored as a nested array on a document (with hard coded limit of 8100 items) as:

[
  [1749379643, 835, 493],
  [1749381846, 469, 515],
  [1749383243, 705, 232],
]

[
  [1749379643, 835, 493],
  [1749381846, 469, 515],
  [1749383243, 705, 232],
]

Once the limit of 8100 is reached it creates a new document and starts a new nested array. However I've now hit the 8192 limit for a function return, I can see that there is paginated queries however I don't know how I could update my current implementation to support paginated queries. Below is current implementation:

export const getEntityData = query({
  args: { entityId: v.id("entities") },
  handler: async (ctx, args) => {
    // Retrieve all documents for the given entity
    const allEntityDocs = await ctx.db
      .query("entity_graph_data")
      .withIndex("by_entity_id", (q) => q.eq("entity_id", args.entityId))
      .collect();

    // Return an empty array if no data is found
    if (!allEntityDocs || allEntityDocs.length === 0) return [];

    // Sort the documents from oldest to newest using _creationTime
    allEntityDocs.sort(
      (a, b) => Number(a._creationTime) - Number(b._creationTime)
    );

    // Merge all the 'value' arrays into a single array.
    const mergedValues = allEntityDocs.reduce(
      (acc, doc) => acc.concat(doc.value),
      [] as number[][]
    );

    // Final sort by the timestamp within each tuple (assumed to be at index 0)
    mergedValues.sort((a, b) => a[0] - b[0]);

    return mergedValues;
  },
});

export const getEntityData = query({
  args: { entityId: v.id("entities") },
  handler: async (ctx, args) => {
    // Retrieve all documents for the given entity
    const allEntityDocs = await ctx.db
      .query("entity_graph_data")
      .withIndex("by_entity_id", (q) => q.eq("entity_id", args.entityId))
      .collect();

    // Return an empty array if no data is found
    if (!allEntityDocs || allEntityDocs.length === 0) return [];

    // Sort the documents from oldest to newest using _creationTime
    allEntityDocs.sort(
      (a, b) => Number(a._creationTime) - Number(b._creationTime)
    );

    // Merge all the 'value' arrays into a single array.
    const mergedValues = allEntityDocs.reduce(
      (acc, doc) => acc.concat(doc.value),
      [] as number[][]
    );

    // Final sort by the timestamp within each tuple (assumed to be at index 0)
    mergedValues.sort((a, b) => a[0] - b[0]);

    return mergedValues;
  },
});

Any advice would be very appericated.

2 Replies

Convex Bot•3mo ago

Thanks for posting in <#1088161997662724167>. Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets. - Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.) - Use search.convex.dev to search Docs, Stack, and Discord all at once. - Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI. - Avoid tagging staff unless specifically instructed. Thank you!

erquhart•2mo ago

If a single doc is all a function can return due to size limits, you'll need to paginate with a page size of 1. The initial sort on _creationTime isn't necessary, ascending by _creationTime is the default sort as long as no other fields are included in your by_entity_id index. The merging and sorting of merged values you can move to the client, stitch the pages together there.

Time Series Data to User for graph

Did you find this page helpful?