2 replies

Time Series Data to User for graph

Hi everyone,

I'm looking for some advice, I currently have a requirement that requires sending all time series data for an "entity" so that the user can view the data in a Time Series graph.

Currently the time series data is stored as a nested array on a document (with hard coded limit of 8100 items) as:

[
  [1749379643, 835, 493],
  [1749381846, 469, 515],
  [1749383243, 705, 232],
]

[
  [1749379643, 835, 493],
  [1749381846, 469, 515],
  [1749383243, 705, 232],
]

[
  [1749379643, 835, 493],
  [1749381846, 469, 515],
  [1749383243, 705, 232],
]

[
  [1749379643, 835, 493],
  [1749381846, 469, 515],
  [1749383243, 705, 232],
]

Once the limit of 8100 is reached it creates a new document and starts a new nested array.

However I've now hit the 8192 limit for a function return, I can see that there is paginated queries however I don't know how I could update my current implementation to support paginated queries.

Below is current implementation:

export const getEntityData = query({
  args: { entityId: v.id("entities") },
  handler: async (ctx, args) => {
    // Retrieve all documents for the given entity
    const allEntityDocs = await ctx.db
      .query("entity_graph_data")
      .withIndex("by_entity_id", (q) => q.eq("entity_id", args.entityId))
      .collect();

    // Return an empty array if no data is found
    if (!allEntityDocs || allEntityDocs.length === 0) return [];

    // Sort the documents from oldest to newest using _creationTime
    allEntityDocs.sort(
      (a, b) => Number(a._creationTime) - Number(b._creationTime)
    );

    // Merge all the 'value' arrays into a single array.
    const mergedValues = allEntityDocs.reduce(
      (acc, doc) => acc.concat(doc.value),
      [] as number[][]
    );

    // Final sort by the timestamp within each tuple (assumed to be at index 0)
    mergedValues.sort((a, b) => a[0] - b[0]);

    return mergedValues;
  },
});

export const getEntityData = query({
  args: { entityId: v.id("entities") },
  handler: async (ctx, args) => {
    // Retrieve all documents for the given entity
    const allEntityDocs = await ctx.db
      .query("entity_graph_data")
      .withIndex("by_entity_id", (q) => q.eq("entity_id", args.entityId))
      .collect();

    // Return an empty array if no data is found
    if (!allEntityDocs || allEntityDocs.length === 0) return [];

    // Sort the documents from oldest to newest using _creationTime
    allEntityDocs.sort(
      (a, b) => Number(a._creationTime) - Number(b._creationTime)
    );

    // Merge all the 'value' arrays into a single array.
    const mergedValues = allEntityDocs.reduce(
      (acc, doc) => acc.concat(doc.value),
      [] as number[][]
    );

    // Final sort by the timestamp within each tuple (assumed to be at index 0)
    mergedValues.sort((a, b) => a[0] - b[0]);

    return mergedValues;
  },
});

export const getEntityData = query({
  args: { entityId: v.id("entities") },
  handler: async (ctx, args) => {
    // Retrieve all documents for the given entity
    const allEntityDocs = await ctx.db
      .query("entity_graph_data")
      .withIndex("by_entity_id", (q) => q.eq("entity_id", args.entityId))
      .collect();

    // Return an empty array if no data is found
    if (!allEntityDocs || allEntityDocs.length === 0) return [];

    // Sort the documents from oldest to newest using _creationTime
    allEntityDocs.sort(
      (a, b) => Number(a._creationTime) - Number(b._creationTime)
    );

    // Merge all the 'value' arrays into a single array.
    const mergedValues = allEntityDocs.reduce(
      (acc, doc) => acc.concat(doc.value),
      [] as number[][]
    );

    // Final sort by the timestamp within each tuple (assumed to be at index 0)
    mergedValues.sort((a, b) => a[0] - b[0]);

    return mergedValues;
  },
});

export const getEntityData = query({
  args: { entityId: v.id("entities") },
  handler: async (ctx, args) => {
    // Retrieve all documents for the given entity
    const allEntityDocs = await ctx.db
      .query("entity_graph_data")
      .withIndex("by_entity_id", (q) => q.eq("entity_id", args.entityId))
      .collect();

    // Return an empty array if no data is found
    if (!allEntityDocs || allEntityDocs.length === 0) return [];

    // Sort the documents from oldest to newest using _creationTime
    allEntityDocs.sort(
      (a, b) => Number(a._creationTime) - Number(b._creationTime)
    );

    // Merge all the 'value' arrays into a single array.
    const mergedValues = allEntityDocs.reduce(
      (acc, doc) => acc.concat(doc.value),
      [] as number[][]
    );

    // Final sort by the timestamp within each tuple (assumed to be at index 0)
    mergedValues.sort((a, b) => a[0] - b[0]);

    return mergedValues;
  },
});

Any advice would be very appericated.

Time Series Data to User for graph

Similar Threads

Similar Threads

Similar Threads