zid•15mo ago

whats wrong with my index??

When I apply the index below, I get the following error:

forum.js:115 usePaginatedQuery hit error, resetting pagination state: [CONVEX Q(user:getDistrictThreads)] Uncaught Error: InvalidCursor: Tried to run a query starting from a cursor, but it looks like this cursor is from a different query.

forum.js:115 usePaginatedQuery hit error, resetting pagination state: [CONVEX Q(user:getDistrictThreads)] Uncaught Error: InvalidCursor: Tried to run a query starting from a cursor, but it looks like this cursor is from a different query.

My query

    threads = await db
      .query("threads")
      .withIndex(`by_districtsId_postDate_${filter}`, (q) =>
        q.eq("districtsId", districtsId).gte("postDate", timeFrameStart)
      )
      .order("desc")
      .paginate(paginationOpts);

    threads = await db
      .query("threads")
      .withIndex(`by_districtsId_postDate_${filter}`, (q) =>
        q.eq("districtsId", districtsId).gte("postDate", timeFrameStart)
      )
      .order("desc")
      .paginate(paginationOpts);

Here's my schema for reference

  threads: defineTable({
    districtsId: v.id("districts"),
    title: v.string(),
    username: v.string(),
    userElementalAvatarId: v.id("userElementalAvatars"), // fetch username, avatar
    message: v.string(), 
    commentsCount: v.number(),
    imageUrl: v.union(v.string(),v.null()),
    postDate: v.string(), // ISO string
    lastActiveAt: v.string(), // ISO string
    tags: v.optional(v.array(v.string())), // "discussion", "rant", etc
    points: v.number(), // sum of all reactions
    laughs: v.number(),
    hearts: v.number(),
    likes: v.number(),
    dislikes: v.number(),
    upvotes: v.number(),
    downvotes: v.number(),
  })
  .index("by_districtsId_postDate_points", ["districtsId", "postDate", "points"])
  .index("by_districtsId_postDate_laughs", ["districtsId", "postDate", "laughs"])
  .index("by_districtsId_postDate_hearts", ["districtsId", "postDate", "hearts"])
  .index("by_districtsId_postDate_likes", ["districtsId", "postDate", "likes"])
  .index("by_districtsId_postDate_dislikes", ["districtsId", "postDate", "dislikes"])
  // ...

  threads: defineTable({
    districtsId: v.id("districts"),
    title: v.string(),
    username: v.string(),
    userElementalAvatarId: v.id("userElementalAvatars"), // fetch username, avatar
    message: v.string(), 
    commentsCount: v.number(),
    imageUrl: v.union(v.string(),v.null()),
    postDate: v.string(), // ISO string
    lastActiveAt: v.string(), // ISO string
    tags: v.optional(v.array(v.string())), // "discussion", "rant", etc
    points: v.number(), // sum of all reactions
    laughs: v.number(),
    hearts: v.number(),
    likes: v.number(),
    dislikes: v.number(),
    upvotes: v.number(),
    downvotes: v.number(),
  })
  .index("by_districtsId_postDate_points", ["districtsId", "postDate", "points"])
  .index("by_districtsId_postDate_laughs", ["districtsId", "postDate", "laughs"])
  .index("by_districtsId_postDate_hearts", ["districtsId", "postDate", "hearts"])
  .index("by_districtsId_postDate_likes", ["districtsId", "postDate", "likes"])
  .index("by_districtsId_postDate_dislikes", ["districtsId", "postDate", "dislikes"])
  // ...

12 Replies

zidOP•15mo ago

I think it has to do with the gte, but not sure how that is causing the cursor issue. To note, the postdated is being stored as isostring Ah shoot, OK I think I read the api incorrectly. Will check once I get a chance nevermind, the api is correct...filter and withIndex api is different...bah, please help Also to note, I'm storing my custom dates as ISO strings on a string field. yea i think its due to timestamp format not being numerical. need to test still nope, still doesnt work...

sshader•15mo ago

Where does filter come from? This error usually indicates running a paginated query over one index, and then running it again (by calling loadMore or reacting to the data changing) with a different index

zidOP•15mo ago

Hey! Filter, and all other dynamic variables are defined from the beginning.

  const [district, setDistrict] = useState({
    districtsId: process.env.NEXT_PUBLIC_INITIAL_DISTRICTS_KEY,
    name: "Home",
  });
  const [timeFrame, setTimeFrame] = useState("month");
  const [filter, setFilter] = useState("points");

  const [district, setDistrict] = useState({
    districtsId: process.env.NEXT_PUBLIC_INITIAL_DISTRICTS_KEY,
    name: "Home",
  });
  const [timeFrame, setTimeFrame] = useState("month");
  const [filter, setFilter] = useState("points");

I think it may have to do with me calculating a timestamp at runtime, even though its not asynchronous, im wondering if convex does something to immediately run the query so that once my calculation is done, it re sets?

export const getThreads = query({
  args: {
    districtsId: v.union(v.id("districts"), v.null()),
    timeFrame: v.union(
      v.literal("week"),
      v.literal("month"),
      v.literal("year"),
      v.literal("allTime")
    ), // 'day', 'month', 'year', 'allTime'
    filter: v.union(
      v.literal("points"),
      v.literal("hearts"),
      v.literal("laughs"),
      v.literal("likes"),
      v.literal("dislikes")
    ),
    paginationOpts: paginationOptsValidator,
  },
  handler: async (
    { db },
    { districtsId, timeFrame, filter, paginationOpts }
  ) => {
    console.log("getThreads-----", {
      districtsId,
      timeFrame,
      filter,
      paginationOpts,
    });
    // const timeFrameStart = calculateTimeFrameStart(timeFrame);
    const timeFrameStart = calculateTimestamp(timeFrame);
    console.log("🚀 ~ timeFrameStart:", timeFrameStart);

    let threads;

    console.log("getThreads-----paginationOpts", {
      paginationOpts,
    });

    threads = await db
      .query("threads")
      // .withIndex(`by_districtsId_postDate_${filter}`, (q) =>
      //   q.eq("districtsId", districtsId).gte("postDate", timeFrameStart)
      // )
      .withIndex("by_districtsId_postDate_points", (q) =>
        q
          .eq("districtsId", "kd7ajxxpfssrr4wezmnvw3y1qd6fz6dv")
          // .gte("postDate", "1700763448295")
          .gte("postDate", calculateTimestamp(timeFrame))
      )
      .order("desc")
      .paginate(paginationOpts);
    // .collect();

    return threads;
  },
});

export const getThreads = query({
  args: {
    districtsId: v.union(v.id("districts"), v.null()),
    timeFrame: v.union(
      v.literal("week"),
      v.literal("month"),
      v.literal("year"),
      v.literal("allTime")
    ), // 'day', 'month', 'year', 'allTime'
    filter: v.union(
      v.literal("points"),
      v.literal("hearts"),
      v.literal("laughs"),
      v.literal("likes"),
      v.literal("dislikes")
    ),
    paginationOpts: paginationOptsValidator,
  },
  handler: async (
    { db },
    { districtsId, timeFrame, filter, paginationOpts }
  ) => {
    console.log("getThreads-----", {
      districtsId,
      timeFrame,
      filter,
      paginationOpts,
    });
    // const timeFrameStart = calculateTimeFrameStart(timeFrame);
    const timeFrameStart = calculateTimestamp(timeFrame);
    console.log("🚀 ~ timeFrameStart:", timeFrameStart);

    let threads;

    console.log("getThreads-----paginationOpts", {
      paginationOpts,
    });

    threads = await db
      .query("threads")
      // .withIndex(`by_districtsId_postDate_${filter}`, (q) =>
      //   q.eq("districtsId", districtsId).gte("postDate", timeFrameStart)
      // )
      .withIndex("by_districtsId_postDate_points", (q) =>
        q
          .eq("districtsId", "kd7ajxxpfssrr4wezmnvw3y1qd6fz6dv")
          // .gte("postDate", "1700763448295")
          .gte("postDate", calculateTimestamp(timeFrame))
      )
      .order("desc")
      .paginate(paginationOpts);
    // .collect();

    return threads;
  },
});

my calculateTimestamp looks like

function calculateTimestamp(timeFrame) {
  const now = new Date();

  switch (timeFrame) {
    // implement when we get more users and threads
    // case "24h":
    //   now.setHours(now.getHours() - 24);
    //   break;
    case "week":
      // Subtract 7 days for the past week
      now.setDate(now.getDate() - 7);
      return now.getTime();
    case "month":
      // Safely handle month boundaries
      let oneMonthAgo = new Date(
        now.getFullYear(),
        now.getMonth() - 1,
        Math.min(
          now.getDate(),
          daysInMonth(now.getFullYear(), now.getMonth() - 1)
        ),
        now.getHours(),
        now.getMinutes(),
        now.getSeconds(),
        now.getMilliseconds()
      );
      return oneMonthAgo.getTime();
    case "year":
      // Safely handle leap years
      let oneYearAgo = new Date(
        now.getFullYear() - 1,
        now.getMonth(),
        Math.min(
          now.getDate(),
          daysInMonth(now.getFullYear() - 1, now.getMonth())
        ),
        now.getHours(),
        now.getMinutes(),
        now.getSeconds(),
        now.getMilliseconds()
      );
      return oneYearAgo.getTime();
    // Add more cases as needed
  }
  return now.getTime();
}

function daysInMonth(year, month) {
  return new Date(year, month + 1, 0).getDate();
}

export default calculateTimestamp;

function calculateTimestamp(timeFrame) {
  const now = new Date();

  switch (timeFrame) {
    // implement when we get more users and threads
    // case "24h":
    //   now.setHours(now.getHours() - 24);
    //   break;
    case "week":
      // Subtract 7 days for the past week
      now.setDate(now.getDate() - 7);
      return now.getTime();
    case "month":
      // Safely handle month boundaries
      let oneMonthAgo = new Date(
        now.getFullYear(),
        now.getMonth() - 1,
        Math.min(
          now.getDate(),
          daysInMonth(now.getFullYear(), now.getMonth() - 1)
        ),
        now.getHours(),
        now.getMinutes(),
        now.getSeconds(),
        now.getMilliseconds()
      );
      return oneMonthAgo.getTime();
    case "year":
      // Safely handle leap years
      let oneYearAgo = new Date(
        now.getFullYear() - 1,
        now.getMonth(),
        Math.min(
          now.getDate(),
          daysInMonth(now.getFullYear() - 1, now.getMonth())
        ),
        now.getHours(),
        now.getMinutes(),
        now.getSeconds(),
        now.getMilliseconds()
      );
      return oneYearAgo.getTime();
    // Add more cases as needed
  }
  return now.getTime();
}

function daysInMonth(year, month) {
  return new Date(year, month + 1, 0).getDate();
}

export default calculateTimestamp;

Thats weird, when i try to use calculateTimestamp in the frontend, before passing it to convex function, i get an infinite loop...

  const { currentResults, diff, doSync, status, loadMore } =
    useBufferedPaginatedState(
      api.user.getThreads,
      {
        /* query args */
        districtsId: district.districtsId,
        // timeFrame: timeFrame,
        timeFrame: calculateTimestamp(timeFrame),
        filter: filter,
      },
      { initialNumItems: 2 },
      (oldVal, newVal) => {
        // Example differ function that calculates the difference between old and new data.
        return newVal.length - (oldVal?.length ?? 0);
      }
    );

  const { currentResults, diff, doSync, status, loadMore } =
    useBufferedPaginatedState(
      api.user.getThreads,
      {
        /* query args */
        districtsId: district.districtsId,
        // timeFrame: timeFrame,
        timeFrame: calculateTimestamp(timeFrame),
        filter: filter,
      },
      { initialNumItems: 2 },
      (oldVal, newVal) => {
        // Example differ function that calculates the difference between old and new data.
        return newVal.length - (oldVal?.length ?? 0);
      }
    );

memoizing seems to have done the trick. although, this feels a bit hacky

  const memoizedTimestamp = useMemo(
    () => calculateTimestamp(timeFrame),
    [timeFrame]
  );

  const memoizedTimestamp = useMemo(
    () => calculateTimestamp(timeFrame),
    [timeFrame]
  );

lee•15mo ago

it looks like every time getThreads runs, it picks a different timeFrameStart. in order to paginate over a consistent range of the data, the query needs to use the same index filters. the query is running multiple times, once for each page of the pagination, and each time it should have the same q.gte filters or it will reset memoization sounds like a good solution. with calculateTimestamp returning a different value each time, you will either get these errors from convex or an infinite loop in react or something like that.

zidOP•15mo ago

thank you for that...makes sense now. wdyt about the memoizedTimestamp workaround my goal is to select timeframes that are say a month old from today or a year old from today. may i ask what your approach would be, if not memoizing the timestamp?

lee•15mo ago

you could also change calculateTimestamp to zero out the hour/minute/second so it effectively rounds to the nearest day, which would return a consistent value unless you're on a day boundary (in which case it would throw error + reoload once which should be fine)

zidOP•15mo ago

hmm

lee•15mo ago

if you do that on the client side, convex may be able to cache more. but i'm not sure whether it's better than memoizing

zidOP•15mo ago

okay, i think i understand..let me make an attempt..

lee•15mo ago

you could also memoize new Date() on the client and pass it in as an argument. anything that doesn't cause a different index range each time should work.

zidOP•15mo ago

by zeroing out do you mean now.setHours(0, 0, 0, 0);? and hmm, okay okay, thank you. goodness glad to be over this hump at least, be able to see the horizon made a hook for this

function useMemoizedTimestamp(timeFrame) {
  const baseDate = useMemo(() => {
    const now = new Date();
    // This makes timestamp is always at the start of a day, which keeps it consistent throughout that day.
    now.setHours(0, 0, 0, 0);
    return now;
  }, []);

  // Memoize the timestamp calculation
  const memoizedTimestamp = useMemo(() => {
    return calculateTimestamp(baseDate, timeFrame);
  }, [baseDate, timeFrame]);

  return memoizedTimestamp;
}

function useMemoizedTimestamp(timeFrame) {
  const baseDate = useMemo(() => {
    const now = new Date();
    // This makes timestamp is always at the start of a day, which keeps it consistent throughout that day.
    now.setHours(0, 0, 0, 0);
    return now;
  }, []);

  // Memoize the timestamp calculation
  const memoizedTimestamp = useMemo(() => {
    return calculateTimestamp(baseDate, timeFrame);
  }, [baseDate, timeFrame]);

  return memoizedTimestamp;
}

Also, i have another question/issue related to pagination, should i create another post or can i ask here?

erquhart•15mo ago

Making another post is helpful for making the question/answer discoverable for the community.

whats wrong with my index??

Did you find this page helpful?