Clever Tagline•5w ago

Tracking document changes

I'm porting data and logic for our business systems from Airtable to a custom app powered by Convex. Part of what Airtable provides as a standard feature is record history. This history has proved invaluable in solving problems that come up on occasion, so I need to replicate this in the new system. I'm thinking that storing all changes in a separate table would be the way to go, with an ID field indexed to make searching for changes to a given document more efficient, and the app to use pagination to optimize the query process. The main thing that I'm struggling with is how to identify and collect the change data. We have dozens of tables, and we'll need this history tracked on each of their documents. I really don't relish the idea of building that many separate history trackers. I'm thinking that some kind of "diff" helper function would be useful. I could pass it any document along with the changes to be saved to that document, and it would return the difference between the two as an object that I could stringify and save to the history table. Is this doable? If so, I'd love any advice on how to make it work. My grasp of TypeScript is still pretty basic, so I can't quite wrap my head around how I'd approach something like this (if it's even possible with the current Convex design).

31 Replies

Convex Bot•5w ago

Thanks for posting in <#1088161997662724167>. Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets. - Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.) - Use search.convex.dev to search Docs, Stack, and Discord all at once. - Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI. - Avoid tagging staff unless specifically instructed. Thank you!

Matt Luo•5w ago

I think there's a component for this

Clever TaglineOP•5w ago

@Matt Luo I looked through the component list earlier and didn't find anything. Checked it again just now in case I'd overlooked something, but I still don't see anything for this use case.

Matt Luo•5w ago

https://www.npmjs.com/package/convex-table-history

npm

convex-table-history

A table history component for Convex.. Latest version: 0.1.2, last published: 2 months ago. Start using convex-table-history in your project by running npm i convex-table-history. There are no other projects in the npm registry using convex-table-history.

Matt Luo•5w ago

Is this helpful?

Clever TaglineOP•5w ago

Immensely! This looks like exactly what I need. My only question is: why isn't this listed on the official components page yet?

Matt Luo•5w ago

Um, im not sure, but worst case you can read the code to be inspired

Clever TaglineOP•5w ago

@lee Is this safe to use, or are there still bugs being worked out that would explain its absence from the official component list?

lee•5w ago

No known bugs, but it hasn't gone through the full review process

Clever TaglineOP•5w ago

Gotcha. I'll start adding it to my project tomorrow and see how things go. I've currently got only one trackable table implemented, but by the time I'm done it'll be a few dozen. Definitely curious to see how this affects my stats as things grow.

Matt Luo•5w ago

🙏 tell us what you learn!

Clever TaglineOP•5w ago

Absolutely!

jamwt•5w ago

we actually had another company just asking about this pattern too... ^ @james fyi

Clever TaglineOP•5w ago

Development is moving slowly as I'm the lone dev and there's a Titanic-sized boatload of stuff to do. It'll be many months before this thing is ready to release to the company. @lee Just submitted a PR to fix a minor issue I found in the docs. About to start adding this component to my project. Got the basics set up with a single table, and it appears to be working well so far. I'd also like to save the initial state of a document as the first "change," but I'm struggling to figure out how to do that. Taking a cue from the example "patch" function from the docs, I made the following:

export async function createContact(ctx: MutationCtx, document: Doc<"contacts">) {
  let documentId = await ctx.db.insert("contacts", document)
  await contactChanges.update(ctx, documentId, document);
}

export async function createContact(ctx: MutationCtx, document: Doc<"contacts">) {
  let documentId = await ctx.db.insert("contacts", document)
  await contactChanges.update(ctx, documentId, document);
}

I quickly found that this won't work because I don't have a document yet, so I can't type the document argument as a Doc of anything. If I type it as Partial<Doc<"contacts">>, the type checker indicates an issue with the insert method because some of the fields in my table schema are required, and they could possibly come in as undefined. Not sure how to fix this, or if I should take an entirely different approach. I suppose I could change the schema so that every field is optional, but that doesn't really feel like the best solution On a whim I switched the type of document to the partial doc as described above, then made every field optional in the table schema. While it got rid of the type error indicator on the insert function, the call to the table history updater still indicates a type error:

Types of property '_id' are incompatible.
    Type 'Id<"contacts"> | undefined' is not assignable to type 'Id<"contacts">'.
      Type 'undefined' is not assignable to type 'Id<"contacts">'.
        Type 'undefined' is not assignable to type 'string'.ts(2345)

Types of property '_id' are incompatible.
    Type 'Id<"contacts"> | undefined' is not assignable to type 'Id<"contacts">'.
      Type 'undefined' is not assignable to type 'Id<"contacts">'.
        Type 'undefined' is not assignable to type 'string'.ts(2345)

Is this because the document doesn't yet exist? I think I wasn't taking enough cues from the example for patching. Updated the creation function to the following, and there are no more type errors:

export async function createContact(ctx: MutationCtx, document: Partial<Doc<"contacts">>) {
  let documentId = await ctx.db.insert("contacts", document)
  const newDocument = await ctx.db.get(documentId);
  await contactChanges.update(ctx, documentId, newDocument);
}

export async function createContact(ctx: MutationCtx, document: Partial<Doc<"contacts">>) {
  let documentId = await ctx.db.insert("contacts", document)
  const newDocument = await ctx.db.get(documentId);
  await contactChanges.update(ctx, documentId, newDocument);
}

That said, I'm still wondering if there's a better way to approach this than making all fields optional in the table schema.

lee•5w ago

try WithoutSystemFields<Doc<"contacts">> ?

Clever TaglineOP•5w ago

It wasn't the system fields causing the problem. I had several fields as optional in the table schema, and that's what led to the type checking error on the insert method. By passing a partial doc, any field could theoretically be undefined, which clashed with the field requirements from the schema.

lee•5w ago

how are you inserting a partial doc if the fields are required

Clever TaglineOP•5w ago

The data I'm passing has those required fields, but because the function definition for this createContact helper function types the document as Partial<Doc<"contacts">> it doesn't know that.

lee•5w ago

yeah i'm suggesting you change the createContact function arg type

export async function createContact(ctx: MutationCtx, document: WithoutSystemFields<Doc<"contacts">>) {
  let documentId = await ctx.db.insert("contacts", document)
  const newDocument = await ctx.db.get(documentId);
  await contactChanges.update(ctx, documentId, newDocument);
}

export async function createContact(ctx: MutationCtx, document: WithoutSystemFields<Doc<"contacts">>) {
  let documentId = await ctx.db.insert("contacts", document)
  const newDocument = await ctx.db.get(documentId);
  await contactChanges.update(ctx, documentId, newDocument);
}

Clever TaglineOP•5w ago

I guess I'm confused on what WithoutSystemFields refers to. I thought it only ignored the _id and _creationTime fields.

lee•5w ago

right. it's used as the argument to insert https://github.com/get-convex/convex-js/blob/45f7c4196003a4d827f925d3781ebe64278c5546/src/server/database.ts#L160 because when you insert a document, you need all of the fields besides the _id and _creationTime

Clever TaglineOP•5w ago

Hmmm...I guess I don't understand how that will impact a case where non-system fields are required by the schema, but this function can't "trust" (for lack of a better term) that I'm going to pass data for all required fields.

lee•5w ago

it will behave in the same way as ctx.db.insert behaves (because it's the same type)

Clever TaglineOP•5w ago

Well, it worked, but I still don't fully understand the logic behind it. Maybe I'm focusing too much on the reference to system fields in the name. Thanks for the help!

lee•5w ago

https://github.com/get-convex/table-history/pull/2/files

Clever TaglineOP•5w ago

Thanks again. That definitely clears up my understanding of how all of this works. I'm hitting an error when trying to query the history. I'm trying to follow the doc examples, but it's indicating an error in my app component. Here's the query function:

export const getContactHistory = query({
  args: {
    id: v.id("contacts"),
    maxTs: v.number(),
    paginationOpts: paginationOptsValidator
  },
  handler: async (ctx, args) => {
    let { user } = await getCurrentUser(ctx);
    return await contactChanges.listDocumentHistory(ctx, args.id, args.maxTs, args.paginationOpts)
  }
})

export const getContactHistory = query({
  args: {
    id: v.id("contacts"),
    maxTs: v.number(),
    paginationOpts: paginationOptsValidator
  },
  handler: async (ctx, args) => {
    let { user } = await getCurrentUser(ctx);
    return await contactChanges.listDocumentHistory(ctx, args.id, args.maxTs, args.paginationOpts)
  }
})

Here's how I'm trying to use it in the app component:

export default function ViewContactHistory({ contact }: {contact: Doc<"contacts">}) {
  const { userToUse } = useOutletContext<AppContext>()
  const [opened, { open, close }] = useDisclosure(false)
  const [currentTs] = useState(Date.now())
  const contactHistory = usePaginatedQuery(
    api.sales.model.contacts.getContactHistory,
    userToUse && opened
    ? {
      id: contact._id,
      maxTs: currentTs,
    }
    : "skip",
    {
      initialNumItems: 5
    }
  );

  return (
    /* UI stuff here */
  )
}

export default function ViewContactHistory({ contact }: {contact: Doc<"contacts">}) {
  const { userToUse } = useOutletContext<AppContext>()
  const [opened, { open, close }] = useDisclosure(false)
  const [currentTs] = useState(Date.now())
  const contactHistory = usePaginatedQuery(
    api.sales.model.contacts.getContactHistory,
    userToUse && opened
    ? {
      id: contact._id,
      maxTs: currentTs,
    }
    : "skip",
    {
      initialNumItems: 5
    }
  );

  return (
    /* UI stuff here */
  )
}

Here's the error indicator for api.sales.model.contacts.getContactHistory:

Argument of type 'FunctionReference<"query", "public", { id: Id<"contacts">; paginationOpts: { id?: number | undefined; endCursor?: string | null | undefined; maximumRowsRead?: number | undefined; maximumBytesRead?: number | undefined; numItems: number; cursor: string | null; }; maxTs: number; }, { ...; }, string | undefined>' is not assignable to parameter of type 'PaginatedQueryReference'.
  The types of '_returnType.pageStatus' are incompatible between these types.
    Type 'string | null | undefined' is not assignable to type '"SplitRecommended" | "SplitRequired" | null | undefined'.
      Type 'string' is not assignable to type '"SplitRecommended" | "SplitRequired" | null | undefined'.ts(2345)

Argument of type 'FunctionReference<"query", "public", { id: Id<"contacts">; paginationOpts: { id?: number | undefined; endCursor?: string | null | undefined; maximumRowsRead?: number | undefined; maximumBytesRead?: number | undefined; numItems: number; cursor: string | null; }; maxTs: number; }, { ...; }, string | undefined>' is not assignable to parameter of type 'PaginatedQueryReference'.
  The types of '_returnType.pageStatus' are incompatible between these types.
    Type 'string | null | undefined' is not assignable to type '"SplitRecommended" | "SplitRequired" | null | undefined'.
      Type 'string' is not assignable to type '"SplitRecommended" | "SplitRequired" | null | undefined'.ts(2345)

lee•5w ago

Huh that's not expected. I wonder why that would happen Probably need some type annotations or as const in there somewhere

Clever TaglineOP•4w ago

Thanks for the hint. A type annotation on the function reference fixed it: api.sales.model.contacts.getContactHistory as PaginatedQueryReference

lee•4w ago

I have a fix for this; will publish new version tonight. Thanks for reporting

Clever TaglineOP•4w ago

@lee I've got a feature request for this component. Would it be okay to discuss it here, or would you prefer that I start a new thread?

lee•4w ago

New thread in #components ?

Tracking document changes

Did you find this page helpful?