Hal•3w ago

Concatenating Streams

Hi all! I’m working with the Convex stream helpers and I understand that mergedStream can merge multiple streams and interleave their results based on a specified order. However, I’m looking for a way to strictly concatenate two or more streams—so that all items from the first stream are yielded before any items from the second stream onwards (i.e., no interleaving). Is there a built-in helper or recommended pattern in Convex for this kind of strict concatenation? I couldn’t find this in the docs or the Merging Streams of Convex data article. Thanks in advance for any guidance!

4 Replies

Convex Bot•3w ago

Thanks for posting in <#1088161997662724167>. Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets. - Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.) - Use search.convex.dev to search Docs, Stack, and Discord all at once. - Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI. - Avoid tagging staff unless specifically instructed. Thank you!

ian•3w ago

It's a bit of a gap right now. However, you could do them sequentially somewhat manually, like

const results = await stream1.take(10);
if (results.length < 10) {
  results.push(...await stream1.take(10-results.length));
}

const results = await stream1.take(10);
if (results.length < 10) {
  results.push(...await stream1.take(10-results.length));
}

The trick is that for pagination, you'd need to return a cursor that encodes which stream you left off at (e.g. ${streamName}|${result.continueCursor}, and when the function get a cursor argument, decode it and skip the all queries until the target one, passing in that cursor to continue, then continue fetching results until you're at the end of the last stream. Handling endCursor and splitting streams would be tricky, less on the handling side, but more in terms of how to split results that span multiple streams... but most of the time you won't need to split streams, so maybe that's fine

HalOP•3w ago

hey thanks Ian. seems a bit too complicated. I was hoping that there is a trick in mergedStream to achieve this 😅 I'll try your suggestion later anyway

HalOP•2w ago

based on your suggestion, I made this: https://gist.github.com/halindraprakoso/72aa3103af2629427ca6e5953743a064 The issue is that, because a function can only have one .pagination() call per function, when the first key’s pagination finishes, it only returns the remaining items from the initial query before stopping, and then resumes again in the next page. But I actually like this behavior for my current scenario since it provides a clear separation. However, if in the future I want the items from the second pagination to fill in the remaining results, I’ll probably use the pagination helper.

Gist

simple pagination across multiple queries in convex

simple pagination across multiple queries in convex - paginateAcrossQueries.ts

Concatenating Streams

Did you find this page helpful?