oscklm•11mo ago

Seeking Advice on Optimal Data Enrichment Practices as Project Becomes More Complex

Hey, I'm currently refining my approach to data enrichment in our applications and would appreciate your insights and recommendations. My functions for enriching query results are becoming quite extensive, and I'm considering whether to stick with granular, targeted enrichment functions or to use a more comprehensive single function that fetches all related data regardless of the specific needs of certain queries. Here’s an example of a more detailed enrichment function I use, mostly for a admin dashbord:

async function enrichWithProfileAndVideoAndActivity(
  ctx: QueryCtx,
  upload: Doc<'userUpload'>,
) {
  const [imageUrl, profile, video, activity] = await Promise.all([
    ctx.storage.getUrl(upload.imageStorageId),
    ctx.db.get(upload.userProfileId),
    ctx.db.get(upload.videoId),
    ctx.db.query('activity').withIndex('by_video', (q) => q.eq('videoId', upload.videoId)).first(),
  ]);

  return {
    ...upload,
    enrichments: {
      imageUrl,
      profileName: profile?.name,
      videoTitle: video?.title,
      activityTitle: activity?.title,
    },
  }
}

async function enrichWithProfileAndVideoAndActivity(
  ctx: QueryCtx,
  upload: Doc<'userUpload'>,
) {
  const [imageUrl, profile, video, activity] = await Promise.all([
    ctx.storage.getUrl(upload.imageStorageId),
    ctx.db.get(upload.userProfileId),
    ctx.db.get(upload.videoId),
    ctx.db.query('activity').withIndex('by_video', (q) => q.eq('videoId', upload.videoId)).first(),
  ]);

  return {
    ...upload,
    enrichments: {
      imageUrl,
      profileName: profile?.name,
      videoTitle: video?.title,
      activityTitle: activity?.title,
    },
  }
}

And here's a simpler function that's used mainly in the public app:

async function enrichWithProfile(ctx: QueryCtx, upload: Doc<'userUpload'>) {
  const [imageUrl, profile] = await Promise.all([
    ctx.storage.getUrl(upload.imageStorageId),
    ctx.db.get(upload.userProfileId),
  ]);

  return {
    ...upload,
    enrichments: {
      imageUrl,
      profileName: profile?.name,
    },
  }
}

async function enrichWithProfile(ctx: QueryCtx, upload: Doc<'userUpload'>) {
  const [imageUrl, profile] = await Promise.all([
    ctx.storage.getUrl(upload.imageStorageId),
    ctx.db.get(upload.userProfileId),
  ]);

  return {
    ...upload,
    enrichments: {
      imageUrl,
      profileName: profile?.name,
    },
  }
}

I'm also curious to what naming convention people like using for this? Do you recommend maintaining multiple specific enrichment functions, or should I consolidate them into a single, more comprehensive function? Looking forward to your thoughts and suggestions!

2 Replies

erquhart•11mo ago

Just my two cent working through similar needs: - multiple specific vs comprehensive: I think there's room for both in the same app, I've been calling this case by case - conventions: I'm using a similar approach (except calling it "compose" instead of "enrich"). Because these tend to be based on client needs, I've begun naming them based on the features/functionality they serve, which tends to lead to simpler and more durable names. These functions change over time, so naming based on purpose rather than exact content seems to work well.

oscklmOP•11mo ago

Thanks for sharing your take on this mate. Always appreciated. Giving some food for thought.

Seeking Advice on Optimal Data Enrichment Practices as Project Becomes More Complex

Did you find this page helpful?