[Module: server/TS]: Generic query pipelining
Hello, I am trying to build a custom react hook that contains a "pipeline" of Convex functions. The pipeline is simple: Query A returns an array of Id<Table>, and Query B accepts an array of Id<Table> and enriches them (basically performing joins with other tables):
() => getDocumentIds --> (Ids) => enrichDocuments(Ids)
I want to make this hook generic such that it can be used with any compatible getter/enrichment function pair, e.g. both getVideoIds --> enrichVideos and getChannelIds --> enrichChannels. Therefore, the return type of Query A must be compatible with the parameter type of Query B. I have taken inspiration from the implementation of useQuery and have actually made it work in practice, but not without ignoring a TS error - not very peace of mind-y.
I found that specifying Args in the FunctionReference of QueryB approximated the solution, but I had to manually specify the correct object key (e.g. "videoIds"). Is there a way to use some placeholder as such?: { PLACEHOLDER: FunctionReturnType<BaseQuery> }
I have attached screenshots of my code until now, the TS error, and usage in the front-end (which works).
I know this post may be confusing, and that it may not be clear what I am actually asking about or trying to achieve, and I apologize for that. I am a bit out of my depth TS-wise, but I hope someone understands what I'm trying to do and has some ideas! Thank you very much.
TL;DR
How can I write a generic custom hook which takes as parameters QueryA, QueryA['_args'] as well asQueryB in order to invoke in a generic way convex.query(query: QueryB, ...args: FunctionReturnType<QueryA> ? A simplifying concern is that I know as the "human" that any QueryB I want to pass to this hook will only have one argument of the same type as the return value of QueryA (an array of document Ids)
3 Replies
Consider doing this joining on the server with getEnrichedVideos. It's easier to protect videos a user shouldn't have access to and there's one less server roundtrip.
Thanks @ballingt , that's a good point, but the challenge I'm facing right now is a different one. I think that I can sum it up in this sentence: I am trying to build a custom hook/query function that: 1. invokes a query and returns the results shuffled; 2. paginates/batches the results from step 1, invokes some other Convex functions on each batch, and returns such an enriched batch
The reason we are doing this whole business is that we want to present randomized/shuffled feeds of different data to our user. All data must be enriched before presenting (database joins, Storage lookups), and since this is expensive, we defer this enrichment until the user scrolls to the end of the current feed.
This is pretty straightforward to implement, and I have done so successfully already. I use two queries: one that fetches all document ids to show and shuffles them, and another query that takes in an array of ids (a batch) and returns enriched versions of them (basically a custom paginator).
What I am hoping to achieve now is to generalize this approach since we apply it many different places in our app and for different tables (videos, videos in N categories, channels, images, .. etc) where each use case has its own fetcher and enricher queries. So I was trying to come up with a
useRandomizedBatchedQuery that I pasted the screenshots of above.
In my wildest dreams, this hook would be used like this:
const {enrichedDataBatch, loadMore} = useRandomizedBatchedQuery(fetchQuery: getVideoIdsByCategories, fetchQueryArgs: ["category1","category2"], enrichQuery: enrichVideosByIds)
And the hook would then enforce that only a compatible pair of queries could be passed as arguments (i.e. where the return value of the first query argument can be used to run the second query), such that it would show an error if I tried to call it like useRandomizedBatchedQuery(getVideoIds, ..., enrichChannels since getVideoIds does not return channelIds.
@Tom I'm sorry for writing so verbosely about this - I realize it's probably quite coupled to our use case, and that it's a lot of effort for you to not only understand the situation but to read all my walls of text!
Please don't spend more time on this issue unless you find the problem interesting and/or worthwhile - it's not a blocking issue for our development, but rather a very interesting personal learning opportunity for me and exercise in extending the logic you guys have implemented with Convex to build something abstract, generic, and "beautiful". I am spelling out my thoughts in this much detail on the off-chance that somebody else might have a few screws loose as well and find this an interesting problem!!
What I am aiming at may be an over-abstraction, and I have concluded that I will probably take one step down the abstraction ladder and just implement a useBatchedQuery hook instead of a useRandomizedBatchedQuery hook (I guess I am just re-inventing the paginatedQuery now?).
Anyway, thank you for being a receptacle for my disordered stream of consciousness, haha, it always helps to speak the problem and your thought processes "aloud"!You can also build these general patterns on the backend; you could write a helper that takes a list of IDs and fetches them all, randomized. Helper functions on the backend are just TypeScript functions. If you want to do this from the client instead that's fine, but consider that this makes implementing access control more involved. To get the types right I'd write a lookup table at the type level to map Id<"users"> to your enriched object; you'll still need if statements on the server side for how to enrich each possible Id type.