pipsai•10mo ago

Bandwith help

Does anyone have tips to lessen bandwidth consumption for an LLM App? I feel like 953Mb for 20 conversations around 90 messages each session (10/mb each message) is too much or is this normal? I

4 Replies

CodingWithJamal•10mo ago

What kind of data are you storing? imagine the bandwidth will be high as it calculates reads and writes and your feeding the llm the chat history?

pipsaiOP•10mo ago

Its the responses from the llm which is costing a lot. It is the chat history, is that normal? This is the query and mutation that costs the most. Am I doing it right?

export const list = query({
  args: {
    sessionId: v.string(),
  },
  handler: async (ctx, args) => {
    return await ctx.db
      .query("LessonBotMessages")
      .withIndex("bySessionId", (q) => q.eq("SessionID", args.sessionId))
      .collect();
  },
});      

export const updateBotMessage = internalMutation(
  async (
    ctx,
    { messageId, text }: { messageId: Id<"LessonBotMessages">; text: string }
  ) => {
    await ctx.db.patch(messageId, { Text: text });
  }
);

export const list = query({
  args: {
    sessionId: v.string(),
  },
  handler: async (ctx, args) => {
    return await ctx.db
      .query("LessonBotMessages")
      .withIndex("bySessionId", (q) => q.eq("SessionID", args.sessionId))
      .collect();
  },
});      

export const updateBotMessage = internalMutation(
  async (
    ctx,
    { messageId, text }: { messageId: Id<"LessonBotMessages">; text: string }
  ) => {
    await ctx.db.patch(messageId, { Text: text });
  }
);

sshader•10mo ago

https://discord.com/channels/1019350475847499849/1228767529166569593/1228798306751090768 might be interesting to you Every time updateBotMessage gets called, you're loading all the LessonBotMessages for that session, so if you're calling updateBotMessage very frequently (like on every word), this could add up pretty quickly

pipsaiOP•10mo ago

This is spot on! Thank you so much!

Bandwith help

Did you find this page helpful?