sonandmjy•3w ago

filtering documents with getPage

I'm looking at the getPage function so I can control the pagination more. But I can't seem to figure out how the filtering works. For example, I have a messages table that stores messages all the messages from many companies. However, I'm not sure how I would only include those results in the getPage return since I don't see a param that allows me to filter my index 🤔 eg this is my function right now and I only want the results with only a particular channelId and destinationId. How would I achieve this? Thank you for any advice in advance!

 const { page: _page, ...response } = yield* Effect.promise(() =>
      getPage(ctx, {
        table: "messages",
        index: "by_destination_and_channel_id",
        schema: schema,
        order,
        startIndexKey: args.startIndexKey,
        endIndexKey: args.endIndexKey,
        endInclusive: args.endInclusive,
        startInclusive: args.startInclusive,
      }),
    );

 const { page: _page, ...response } = yield* Effect.promise(() =>
      getPage(ctx, {
        table: "messages",
        index: "by_destination_and_channel_id",
        schema: schema,
        order,
        startIndexKey: args.startIndexKey,
        endIndexKey: args.endIndexKey,
        endInclusive: args.endInclusive,
        startInclusive: args.startInclusive,
      }),
    );

5 Replies

Convex Bot•3w ago

Thanks for posting in <#1088161997662724167>. Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets. - Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.) - Use search.convex.dev to search Docs, Stack, and Discord all at once. - Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI. - Avoid tagging staff unless specifically instructed. Thank you!

lee•3w ago

Filtering is done with startIndexKey and endIndexKey. E.g. if you set startIndexKey: [destinationId, channelId] it will start at the first document with the given destinationId and channelId, giving the page starting there

sonandmjyOP•3w ago

oh I see, and it will only fetch results containing these two index keys if no endIndexKey is provided? thank you for the response!

lee•3w ago

If no endKey is provided, it will continue providing documents after those values. If you want to only include documents with destinationId, channelId matching, you would also provide endIndexKey Out of curiosity, what control were you aiming to get using getPage

sonandmjyOP•3w ago

export const replicateMessagesPromise = fromPromise<
  void,
  {
    db: KaolinRxDatabase;
    apiService: TApiServiceLayer;
    channelId: Id<"channels">;
    destinationId: Id<"destinations">;
  }
>(({ input: { db, apiService, channelId, destinationId } }) => {
  const program = Effect.gen(function* () {
    yield* Effect.promise(async () =>
      replicateRxCollection({
        collection: db.messages,
        live: false,
        replicationIdentifier: `messages-V2VGetMessagesByDestinationAndChannelIdHandlerMessage-${channelId}-${destinationId}`,
        pull: {
          async handler(
            lastCheckpoint: {
              messageId: Id<"messages">;
            } | null,
            batchSize,
          ) {
            const program = Effect.gen(function* () {
              const apiService = yield* ApiService;
              const response =
                yield* apiService.getMessagesByDestinationAndChannelId({
                  startIndexKey: lastCheckpoint
                    ? [channelId, destinationId, lastCheckpoint.messageId]
                    : undefined,
                });


              const lastElement = lastOfArray(response.page);
              return yield* Effect.succeed({
                documents: response.page.map((m) => ({
                  messageId: m._id,
                  text: m.searchField,
                  channelId: m.channelId,
                  destinationId: m.destinationId,
                  data: m,
                  creationTime: m._creationTime,
                })),
                checkpoint: lastElement
                  ? {
                      messageId: lastElement._id,
                    }
                  : lastCheckpoint,
              });
            }).pipe(Effect.provide(apiService));

            return program.pipe(Effect.runPromise);
          },
        },
      }),
    );
  });

  return program.pipe(Effect.runPromise);
});

export const replicateMessagesPromise = fromPromise<
  void,
  {
    db: KaolinRxDatabase;
    apiService: TApiServiceLayer;
    channelId: Id<"channels">;
    destinationId: Id<"destinations">;
  }
>(({ input: { db, apiService, channelId, destinationId } }) => {
  const program = Effect.gen(function* () {
    yield* Effect.promise(async () =>
      replicateRxCollection({
        collection: db.messages,
        live: false,
        replicationIdentifier: `messages-V2VGetMessagesByDestinationAndChannelIdHandlerMessage-${channelId}-${destinationId}`,
        pull: {
          async handler(
            lastCheckpoint: {
              messageId: Id<"messages">;
            } | null,
            batchSize,
          ) {
            const program = Effect.gen(function* () {
              const apiService = yield* ApiService;
              const response =
                yield* apiService.getMessagesByDestinationAndChannelId({
                  startIndexKey: lastCheckpoint
                    ? [channelId, destinationId, lastCheckpoint.messageId]
                    : undefined,
                });


              const lastElement = lastOfArray(response.page);
              return yield* Effect.succeed({
                documents: response.page.map((m) => ({
                  messageId: m._id,
                  text: m.searchField,
                  channelId: m.channelId,
                  destinationId: m.destinationId,
                  data: m,
                  creationTime: m._creationTime,
                })),
                checkpoint: lastElement
                  ? {
                      messageId: lastElement._id,
                    }
                  : lastCheckpoint,
              });
            }).pipe(Effect.provide(apiService));

            return program.pipe(Effect.runPromise);
          },
        },
      }),
    );
  });

  return program.pipe(Effect.runPromise);
});

Hi Lee, I'm actually trying to do data replication using rx-db, the above is some semi-working code I am doing right now. So I am making a chat app. I successfully did it using the paginate fn already but I am now experimenting with a local first approach using rx-db. The idea I am running with is that i want to try to save the messages inside indexedDB when the user first load the conversation by paginating through them and saving them (that's what getPage is for) so when they next visit the page load can be instant. rx-db provides the help on handling conflicts, data replication, live updates etc. And I am orchestrating this all through xstate the apiService.getMessagesByDestinationAndChannelId is basically just wrapping the convex client (underneath it is calling the endpoint with getPage) also idk if this is a good idea but I may also experiment with replacing the live updates for this particular use case using another service like liveblocks or sth even though convex has is realtime by default because seems kind of expensive if I am subscribing to every change to every message to keep my indexDB layer in sync 🤔

filtering documents with getPage

Did you find this page helpful?