beerman•4mo ago

struggling with mutation

I'm trying to infer a type from my schema in order to type locationData which I will pass to the mutation but I keep getting the following error:

src/convex/addCompleteProperty.ts|7 col 23-62 error| Type 'TableDefinition<VObject<{ district?: string | undefined; city?: string | undefined; area?: string | undefined; postal_code?: string | undefined; nearest_landmark?: string | undefined; latitude?: number | undefined; longitude?: number | undefined; country: string; province: string; address: string; }, { ...; }, "requ...' does not satisfy the constraint 'Validator<any, OptionalProperty, any>'.

import { mutation } from './_generated/server';
import schema from './schema';
import { Infer } from 'convex/values';

type Location = Infer<typeof schema.tables.property_locations>;

export default mutation(
    async ({ db }, { propertyData, locationData, features, amenities, views }) => {
        // Insert or find the location
        const location = await db
            .query('property_locations')
            .withIndex('by_province', (q) => q.eq('province', locationData.province))
            .first();
}))
...

import { mutation } from './_generated/server';
import schema from './schema';
import { Infer } from 'convex/values';

type Location = Infer<typeof schema.tables.property_locations>;

export default mutation(
    async ({ db }, { propertyData, locationData, features, amenities, views }) => {
        // Insert or find the location
        const location = await db
            .query('property_locations')
            .withIndex('by_province', (q) => q.eq('province', locationData.province))
            .first();
}))
...

Here's the relevant section of my schema

export default defineSchema({
    property_locations: defineTable({
        country: v.string(),
        province: v.string(),
        district: v.optional(v.string()),
        city: v.optional(v.string()),
        area: v.optional(v.string()),
        postal_code: v.optional(v.string()),
        address: v.string(),
        nearest_landmark: v.optional(v.string()),
        latitude: v.optional(v.number()),
        longitude: v.optional(v.number())
    })
        .index('by_province', ['province'])
        .index('by_district', ['district'])
        .index('by_city', ['city'])
        .index('by_city_area', ['city', 'area'])
        .index('by_country', ['country'])
});

export default defineSchema({
    property_locations: defineTable({
        country: v.string(),
        province: v.string(),
        district: v.optional(v.string()),
        city: v.optional(v.string()),
        area: v.optional(v.string()),
        postal_code: v.optional(v.string()),
        address: v.string(),
        nearest_landmark: v.optional(v.string()),
        latitude: v.optional(v.number()),
        longitude: v.optional(v.number())
    })
        .index('by_province', ['province'])
        .index('by_district', ['district'])
        .index('by_city', ['city'])
        .index('by_city_area', ['city', 'area'])
        .index('by_country', ['country'])
});

Where did I go wrong?

145 Replies

Convex Bot•4mo ago

Thanks for posting in <#1088161997662724167>. Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets. - Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.) - Use search.convex.dev to search Docs, Stack, and Discord all at once. - Additionally, you can post your questions in the Convex Community's <#1228095053885476985> channel to receive a response from AI. - Avoid tagging staff unless specifically instructed. Thank you!

ballingt•4mo ago

@beerman there's one more property access to do

Infer<typeof schema.tables.property_locations.validator>;

Infer<typeof schema.tables.property_locations.validator>;

property_locations is the whole table, .validator is the validator used to create the table. You could create a type that did it this way like type InferFromTable<T> = Infer<T["validator"]>, this was just a more consistent way to treat validators for Infer<> since it also works with argument and output validators

beermanOP•4mo ago

I see, yeah that got rid of the error. What's the optimal way to now let Convex know that locationData is of type Location? as Location produced another error

export default mutation(
    async (
        { db },
        {
            propertyData,
            locationData,
            features,
            amenities,
            views
        }: { propertyData: any; locationData: Location; features: any; amenities: any; views: any }
    ) => {
        // Insert or find the location
        const location = await db
            .query('property_locations')
            .withIndex('by_province', (q) => q.eq('province', locationData.province))
            .first();

export default mutation(
    async (
        { db },
        {
            propertyData,
            locationData,
            features,
            amenities,
            views
        }: { propertyData: any; locationData: Location; features: any; amenities: any; views: any }
    ) => {
        // Insert or find the location
        const location = await db
            .query('property_locations')
            .withIndex('by_province', (q) => q.eq('province', locationData.province))
            .first();

this worked but I was wondering if I can attach the type directly

ballingt•4mo ago

~~could you show more code? as Location sounds like a cast that would be a TypeScript error if these don't overlap enough~~

this worked but I was wondering if I can attach the type directly

could you say more? I'm not sure what attach the type directly means here

beermanOP•4mo ago

I meant, perhaps add the type where declaring locationData in the second async argument

beermanOP•4mo ago

beermanOP•4mo ago

This works though. Last question - say I want to add by_city and by_area to the indexing, do I just chain another .withIndex or can I add them all in the same one?

ballingt•4mo ago

are you saying you wish TypeScript syntax were nicer, and let you do

const foo = ({a: string}) => a

const foo = ({a: string}) => a

instead of

const foo = ({a}: {a: string}) => a

const foo = ({a}: {a: string}) => a

const foo = (args: {a: string}) => args.a

const foo = (args: {a: string}) => args.a

? If so I agree, but that's the syntax we're stuck with by using TypeScript

beermanOP•4mo ago

Yep, I thought so

ballingt•4mo ago

You can add them to the same one, as long as you access them in that order

beermanOP•4mo ago

Can you elaborate on what you mean by accessing them in that order? My understanding is that this indexing will allow my queries to execute faster. So i defined that these properties should be indexed in the db, but in order for that to take place, I also need to specify in the mutation. Is that right?

ballingt•4mo ago

some docs here, the gist is that an index is a single ordering of records. If you add a single index on name and then age, you can't use that to get all users from ages 12-14. But you can use it to find all the people named to from age 12-14.

beermanOP•4mo ago

Hmm, okay so I must have misunderstood how it works. I thought that indexing all fields would have been a good idea since the website gives users the options to filter properties by district, city, neighbourhood, etc so I wanted them all to load faster depending on what the user is filtering by

ballingt•4mo ago

That sounds true But when a user starts to compound those filters it gets more interesting The properties you have listed here don't really need to be compounted, right? You don't have to allow searches where city = seattle AND nearest landmark = "space needle"? Each index is an ordering of all of your properties. If the index is by city, then they're all like pages of a book in order by city name. If the index is on city and then closest landmark, then closest landmark is like a tiebreaker for properties in the same city.

beermanOP•4mo ago

Not yet, I was just trying to optimize basic filtering at the start. So if the user wanted to view all properties in the city center, they would get served those faster since i'm indexing the city_area

ballingt•4mo ago

A query is more efficient if there is a single span of pages of this book that have all the results you want. Yeah, if you write the query to use that index

beermanOP•4mo ago

I'm still a little confused but I don't want to bother you with stupid questions so I'm going to read the docs and try to get a better understanding of how indexing works

ballingt•4mo ago

Feel free to keep asking, might be slow on the responses especially helpful to hear after you've read the docs what is still confusing there's also this essay on what an index is https://docs.convex.dev/database/indexes/indexes-and-query-perf

beermanOP•4mo ago

Alright, il let you know if I'm still confused after taking all the info in Actually, it looks like I don't even need to define those indexes in the mutation since I already defined what needs to be indexed in my schema and i'm not concerned with speed while checking if a record exists while adding data to the db Hey there, quick question. I am trying to add complete data for a property and its relations at once. The problem is that a property has fields like property_location_id which can't be passed into the mutation in the propertyData as the records are not yet created. Here is a part of my schema:

import { defineSchema, defineTable } from 'convex/server';
import { v } from 'convex/values';

export default defineSchema({
    properties: defineTable({
        title: v.string(),
        property_type_id: v.id('property_type_options'),
        property_location_id: v.id('property_locations'),  
        ...
    })

    property_type_options: defineTable({
        name: v.string()
    }),

    property_locations: defineTable({
        country: v.string(),
        ...
    })
});

import { defineSchema, defineTable } from 'convex/server';
import { v } from 'convex/values';

export default defineSchema({
    properties: defineTable({
        title: v.string(),
        property_type_id: v.id('property_type_options'),
        property_location_id: v.id('property_locations'),  
        ...
    })

    property_type_options: defineTable({
        name: v.string()
    }),

    property_locations: defineTable({
        country: v.string(),
        ...
    })
});

I thought that the best way to handle this would be to infer the property type from my schema and omit those fields. I tried the following:

import { mutation } from './_generated/server';
import schema from './schema';
import { Infer, v } from "convex/values";

const propertyTypeMap = {
    'Historic Home': 'mh76wh4ekvhfr4vqh9cpq50aax70xnxh',
    ...
};

type PropertyTypeName = keyof typeof propertyTypeMap;
type PropertyData = Omit<
    Infer<typeof schema.tables.properties.validator>,
    'property_type_id' | 'property_location_id'
>;

export default mutation({
    args: {
        propertyData: PropertyData,
...

import { mutation } from './_generated/server';
import schema from './schema';
import { Infer, v } from "convex/values";

const propertyTypeMap = {
    'Historic Home': 'mh76wh4ekvhfr4vqh9cpq50aax70xnxh',
    ...
};

type PropertyTypeName = keyof typeof propertyTypeMap;
type PropertyData = Omit<
    Infer<typeof schema.tables.properties.validator>,
    'property_type_id' | 'property_location_id'
>;

export default mutation({
    args: {
        propertyData: PropertyData,
...

But It is now complaining that PropertyData refers to a type but is being used as a value because it wants me to give it a validator, but the validator includes these fields which I don't yet have values for since the records are not yet created. What is the optimal way to do this?

ballingt•4mo ago

You can also pick fields out of a validator instead of picking them out of the type, like

const { property_type_id, property_location_id, ...rest } = schema.tables.properties.validator.fields;
const propertyWithoutForeignKeys = v.object(rest);

const { property_type_id, property_location_id, ...rest } = schema.tables.properties.validator.fields;
const propertyWithoutForeignKeys = v.object(rest);

or you can build these up separately and compose them in the schema I think your approach sounds good, if you want an endpoint that accepts all this data at once then you'll need to accept these entities in a slightly different form than you store them Another option is using your own foreign key columns instead of _id if you already know those ahead of time but what you've got here looks good, you'll insert on piece first and then use its id as the foreign key on the other entities composing validators looks like

const a = v.object({a: v.string()});
const b = v.object({b: v.string()});
const combined = v.object(...a.fields, ...b.fields)

const a = v.object({a: v.string()});
const b = v.object({b: v.string()});
const combined = v.object(...a.fields, ...b.fields)

beermanOP•4mo ago

Interesting. That was pretty much what I needed. But here is another problem which I'm not sure how to tackle - so essentially, I have a table for property_type_options which contains all the different types of properties. The propertyTypeMap is just an object with the type names and their respective record ids (not sure if there is any way to do this without having this map here). Since I need to link a property type to the property, I'm passing it separately outside propertyData but I want to make sure that the string passed matches a valid property type option. I'm not sure how to achieve this with the validator bare in mind that i'm trying to minimize maintenance overhead in case schema changes. Without having the map there, I imagine the only other way would be to have some kind of constraint on that field?

ballingt•4mo ago

The propertyTypeMap is just an object with the type names and their respective record ids (not sure if there is any way to do this without having this map here)

You could do a db lookup to get the one you want by name, or you could use a literal like v.union(v.literal('duplex'), v.literal('apartment'), ...). The problem with this is that it'll be some trouble to make sure that in dev and prod you have the same IDs for these records. For this reason I'd basically never hardcode IDs, I want my codebase to work even if I cleared the data. You can do efficient lookups by any column that has an index, not just ID

Without having the map there, I imagine the only other way would be to have some kind of constraint on that field

Not quite sure I follow, but the validator language seems descriptive enough to do this (it's v.id("property_type") it's not descriptive enough to do everything, sometimes you'll still need to write TypeScript code that checks your own invariants. yeah passing up one of these 15 strings and then writing logic to make sure they match makes sense the v.string() etc. validator language is really just for the very low-level constraints that makes sense to express in the database and in any programming language If you want your mutation to accept data that isn't an ID, you can either validate it as a union of literals or just as a string, that you then run arbitrary code on and reject if it's not bad

beermanOP•4mo ago

I tried the followinf=g but got an error saying that pluck does not exist on type QueryInitializer

import { mutation } from './_generated/server';
import schema from './schema';
import { Infer, v } from 'convex/values';

const { property_type_id, property_location_id, ...rest } =
    schema.tables.properties.validator.fields;

const propertyDataWithoutForeignKeys = v.object(rest);

export default mutation({
    args: {
        propertyData: propertyDataWithoutForeignKeys,
        propertyTypeName: schema.tables.property_type_options.validator,
        locationData
        // features: schema.tables.features.validator.name
        // amenities: schema.tables.amenities.validator.name.array(),
        // views: schema.tables.views.validator.name.array()
    },
    handler: async (ctx, args) => {
        const { propertyData, propertyTypeName, locationData, features, amenities, views } = args;

        const propertyTypeId = await ctx.db
            .query('property_type_options')
            .filter((q) => q.eq(q.field('name'), propertyTypeName))
            .pluck('_id')
            .unique();

        if (!propertyTypeId) throw new Error(`Property type "${propertyTypeName}" does not exist.`);

        // Insert location
        const propertyLocationId = await ctx.db.insert('property_locations', locationData);

        // Insert the property
        const propertyId = await ctx.db.insert('properties', {
            ...propertyData,
            property_type_id: propertyTypeId,
            property_location_id: propertyLocationId
        });

        return { propertyId, locationId };
    }
});

import { mutation } from './_generated/server';
import schema from './schema';
import { Infer, v } from 'convex/values';

const { property_type_id, property_location_id, ...rest } =
    schema.tables.properties.validator.fields;

const propertyDataWithoutForeignKeys = v.object(rest);

export default mutation({
    args: {
        propertyData: propertyDataWithoutForeignKeys,
        propertyTypeName: schema.tables.property_type_options.validator,
        locationData
        // features: schema.tables.features.validator.name
        // amenities: schema.tables.amenities.validator.name.array(),
        // views: schema.tables.views.validator.name.array()
    },
    handler: async (ctx, args) => {
        const { propertyData, propertyTypeName, locationData, features, amenities, views } = args;

        const propertyTypeId = await ctx.db
            .query('property_type_options')
            .filter((q) => q.eq(q.field('name'), propertyTypeName))
            .pluck('_id')
            .unique();

        if (!propertyTypeId) throw new Error(`Property type "${propertyTypeName}" does not exist.`);

        // Insert location
        const propertyLocationId = await ctx.db.insert('property_locations', locationData);

        // Insert the property
        const propertyId = await ctx.db.insert('properties', {
            ...propertyData,
            property_type_id: propertyTypeId,
            property_location_id: propertyLocationId
        });

        return { propertyId, locationId };
    }
});

ballingt•4mo ago

If you use ai-generated code it's unfortunately going to make up methods like this the error message is correct, pluck does not exist on type QueryInitializer

beermanOP•4mo ago

Ouch, I thought that Kappa was trained on your docs

ballingt•4mo ago

unfortunately before that it was trained on the internet there's no select equivalent in Convex today, you'll have to grab the whole record

beermanOP•4mo ago

Yeah I was reading through the relevant documentation and couldn't find how to extract one field from a query

beermanOP•4mo ago

this should do it right?

beermanOP•4mo ago

i should probably query it withIndex by_name as well

ballingt•4mo ago

for property type that looks reasonable, yeah was going to say that but like if you only have 10 it's not that big a deal but you'll use less bandwidth if you can just pick the right one instead of doing a full table scan like would happen as written

beermanOP•4mo ago

Hey Tom, quick question

import { query } from './_generated/server';
import { getOneFrom, getAll, getManyFrom, getManyVia } from 'convex-helpers/server/relationships';

export const get = query({
    args: {},
    handler: async (ctx) => {
        // Fetch all properties
        const properties = await ctx.db.query('properties').collect();

        // Fetch related data for each property
        return await Promise.all(
            properties.map(async (property) => {
                // Fetch related data here. For example:
                const propertyType = await getOneFrom(
                    ctx.db,
                    'property_type_options',
                    'id',
                    property.property_type_id
                );
                const propertyLocation = await getOneFrom(
                    ctx.db,
                    'property_locations',
                    'id',
                    property.property_location_id
                );

                // Return the property with its related data
                return {
                    ...property,
                    propertyType,
                    propertyLocation
                };
            })
        );
    }
});

import { query } from './_generated/server';
import { getOneFrom, getAll, getManyFrom, getManyVia } from 'convex-helpers/server/relationships';

export const get = query({
    args: {},
    handler: async (ctx) => {
        // Fetch all properties
        const properties = await ctx.db.query('properties').collect();

        // Fetch related data for each property
        return await Promise.all(
            properties.map(async (property) => {
                // Fetch related data here. For example:
                const propertyType = await getOneFrom(
                    ctx.db,
                    'property_type_options',
                    'id',
                    property.property_type_id
                );
                const propertyLocation = await getOneFrom(
                    ctx.db,
                    'property_locations',
                    'id',
                    property.property_location_id
                );

                // Return the property with its related data
                return {
                    ...property,
                    propertyType,
                    propertyLocation
                };
            })
        );
    }
});

 const properties = useQuery(api.properties.get);
  $inspect(properties.data)

 const properties = useQuery(api.properties.get);
  $inspect(properties.data)

This gives me the following error. I also tried to use '_id' as the index, which also yields the same error.

Arguments for the rest parameter 'fieldArg' were not provided. src/convex/properties.ts:20:36 - error TS2554: Expected 5 arguments, but got 4. 20 const propertyLocation = await getOneFrom( ~~ node_modules/convex-helpers/server/relationships.d.ts:59:310 59 export declare function getOneFrom<DataModel extends GenericDataModel, TableName extends TablesWithLookups<DataModel>, IndexName extends UserIndexes<DataModel, TableName>>(db: GenericDatabaseReader<DataModel>, table: TableName, index: IndexName, value: TypeOfFirstIndexField<DataModel, TableName, IndexName>, ...fieldArg: FieldIfDoesntMatchIndex<DataModel, TableName, IndexName>): Promise<DocumentByName<DataModel, TableName> | null>; ~~~~~~~~~~~~~~~~~ Arguments for the rest parameter 'fieldArg' were not provided. Found 2 errors in the same file, starting at: src/convex/properties.ts:14

beermanOP•4mo ago

ballingt•4mo ago

Does this error make sense? It says that in

const propertyType = await getOneFrom(
  ctx.db,
  'property_type_options',
   'id',
   property.property_type_id
);

const propertyType = await getOneFrom(
  ctx.db,
  'property_type_options',
   'id',
   property.property_type_id
);

you're missing an argument

/**
 * Get a document matching the given value for a specified field.
 *
 * `null` if not found.
 * Useful for fetching a document with a one-to-one relationship via backref.
 * Requires the table to have an index on the field named the same as the field.
 * e.g. `defineTable({ fieldA: v.string() }).index("fieldA", ["fieldA"])`
 *
 * Getting 'string' is not assignable to parameter of type 'never'?
 * Make sure your index is named after your field.
 *
 * @param db DatabaseReader, passed in from the function ctx
 * @param table The table to fetch the target document from.
 * @param index The index on that table to look up the specified value by.
 * @param value The value to look up the document by, often an ID.
 * @param field The field on that table that should match the specified value.
 *   Optional if the index is named after the field.
 * @returns The document matching the value, or null if none found.
 */

/**
 * Get a document matching the given value for a specified field.
 *
 * `null` if not found.
 * Useful for fetching a document with a one-to-one relationship via backref.
 * Requires the table to have an index on the field named the same as the field.
 * e.g. `defineTable({ fieldA: v.string() }).index("fieldA", ["fieldA"])`
 *
 * Getting 'string' is not assignable to parameter of type 'never'?
 * Make sure your index is named after your field.
 *
 * @param db DatabaseReader, passed in from the function ctx
 * @param table The table to fetch the target document from.
 * @param index The index on that table to look up the specified value by.
 * @param value The value to look up the document by, often an ID.
 * @param field The field on that table that should match the specified value.
 *   Optional if the index is named after the field.
 * @returns The document matching the value, or null if none found.
 */

@beerman do you have a question about this?

beermanOP•4mo ago

I know that it says that i'm missing an argument, but this argument is optional. I found out that index is literally index, not a field. So that's not useful in my case as I am only indexing property types by name, with each property being given the property type id

beermanOP•4mo ago

so it's literally impossible to index the id as that table doesn't store those ids

beermanOP•4mo ago

Kapa suggested that this is the proper way to get these relations, but that doesn't work either

ballingt•4mo ago

This argument is only optional if the index is named after the field

beermanOP•4mo ago

Right, but there is no index for this and index is not an optional field

ballingt•4mo ago

ah got it, the index on id is automatic and doesn't have a name? What is it you're trying to do, just get by id?

beermanOP•4mo ago

No no, take a look at the schema above. property_type_options doesn't contain the id of the property record. It only has its internal id and the property type name The relevant property_type_id is stored on the property table so I'm just trying to use it to get that property type relation

ballingt•4mo ago

sorry could you repeat what the goal is?

beermanOP•4mo ago

property has a field called property_type_id which stores the id of the propertyType record in the property_type_options table. I'm trying to get that related data while querying the property so instead of getting the record id, i want to get the actual data of the record

ballingt•4mo ago

ok, and you could do this manually with ctx.db.get(property_type_id)? but you'd like to use this helper which combines the property with the proptery type? question marks because I'm asking, not because this doesn't make sense

beermanOP•4mo ago

ctx.db.get didn't work, i got this error

beermanOP•4mo ago

but yes, I was trying to use the helpers because it's neater imo

ballingt•4mo ago

What does that error say

beermanOP•4mo ago

I don't understand this error

ballingt•4mo ago

it's ugly, but it says this is a promise you need to await the ctx.db.get

beermanOP•4mo ago

ah, i should await it yep, I was being dumb

ballingt•4mo ago

Got it, so the ask here is how to get a foreign key when it's a simple id instead of based on a another column makes sense

beermanOP•4mo ago

so as for the helpers, is there no way to use them without an index?

ballingt•4mo ago

I don't see a helper for this, seems like it would just be called get?

beermanOP•4mo ago

I think I just misunderstood what these helpers were supposed to be used for. Yes, get works as long as i pass it the id of the record I want to pull. But i don't think that would work for one to many or many to many relations

ballingt•4mo ago

if there's a signature for the function you'd want would be helpful to see got it, you'd like something that assembled the object by combining these objects like a join? and these don't do that?

beermanOP•4mo ago

I have features which contains all the different features a property could have. And another table called property_features which contains the id of the property and the id of the feature. So each property could have multiple features. How would I get all the features for a property when i'm not indexing that manually? I mean, get worked. I got the data i needed. I just wanted to use these helpers because they're neat. The issue is that like i said, I am not indexing the ids on the property_type_options

beermanOP•4mo ago

Interesting. Looks like ids are indexed by the system by default. I got the data

ballingt•4mo ago

ah exactly, cool

beermanOP•4mo ago

another question if you don't mind - so since property_type_options, features, amenities, and views are tables that already contain all the predefined options that can be used, is there a good way to implement checking when filling in these fields? since i'm going to be passing an array of feature strings into this mutation, is it possible to implement type checking on the contents of the array so that all entries are valid features without having to query the db before inserting each feature? if i wanted to hardcode, I would do something like this and a would show as invalid

type feature = 'something' | 'somethingElse';
const features: feature[] = ['a', 'something'];

type feature = 'something' | 'somethingElse';
const features: feature[] = ['a', 'something'];

ballingt•4mo ago

What's your schema like, do you already have a validator that is the union of these literals

beermanOP•4mo ago

message.txt

ballingt•4mo ago

not at a computer atm, but you can use Infer<> on a validator to build the type or you can map over an array of these literals to build the validator and build the type

beermanOP•4mo ago

The property_type_options, features, amenities, views were prefilled in their repsective tables. Maybe that wasn't the best idea?

beermanOP•4mo ago

the validator would just validate against the name (string) field of each of those tables though, not the actual values right?

ballingt•4mo ago

Ah if there's no validator for then you need to generate code from these values I don't quite understand what you want here

beermanOP•4mo ago

The validator for the property_type_options just validates that the input for name is a string. It doesn't contain the possible values that are in the table

ballingt•4mo ago

You want a type for feature, which is just a union of string literals yeah? ah yeah you'd need to generate typescript code if you want to represent this in TypeScript

beermanOP•4mo ago

Sorry but I don't quite understand what you mean by that. Generate it from what?

ballingt•4mo ago

TypeScript isn't going to query your database everytime you typecheck right

beermanOP•4mo ago

yes

ballingt•4mo ago

so you need to query your database and generate this TypeScript code or change this to write a validator (or just a typescript type) with all of these "generate" meaning construct a string from the database values, this isn't a pretty somution

beermanOP•4mo ago

What is the best solution in your opinion? You clearly know all the best practices and it seems like i'm doing something outside the norm

ballingt•4mo ago

If these features are static, I'd stick that in a validator or at least a typescript type if you need typechecking on something you have to put it in typescript

beermanOP•4mo ago

right, so instead of sticking all these predefined values into tables, just stick them in a validator instead which would result in less tables, which is a good thing

ballingt•4mo ago

yeah, and you can still use records for them if you want! But the validator for that table is the union of all these literals less tables is whatever, use as many tables as you like

beermanOP•4mo ago

this would still introduce some maintenance overhead right since i would then need to edit the validator every time a new option was added to or removed from the table

ballingt•4mo ago

totally, that's just the normal typescript tradeoff typescript can't typecheck things that aren't in the code ase if it's really dynamic, then uou could give up on having a typescript type on these

beermanOP•4mo ago

I understand that, I just throught that convex might have some way to pull from the db to build a dynamic validator or something like that

ballingt•4mo ago

uou wouldn't expect typescript sutocompletion on eg username so you can totally dynamically validate it validation is about runtime but if you want autocompletion in vscode, that's a typescript thing

beermanOP•4mo ago

ideally, I would also like to get that kind of validation in the convex function dashboard as well i don't know, it seemed doable

ballingt•4mo ago

say more, like in the data editor?

beermanOP•4mo ago

yes it already knows that it wants an array of strings, so why not an array of specific strings?

beermanOP•4mo ago

here

ballingt•4mo ago

i think I see what you mean, like in Airtable this would work, but in Convex the dashboard interaction is determined by the schema. and the autocompletion here is TypeScript-based, like in your editor

beermanOP•4mo ago

here's the thing, i get my property listings from a couple different sources i am partnered with. Different agencies in my area, all using their own schemas and listing formats. So normalizing all this data is a pain in the backside hence, i need everything validated to make sure that it's all congruent

ballingt•4mo ago

to back up, you can validate this! You just can't autocomplete it with typescript

beermanOP•4mo ago

oh i don't care about the autocompletion

ballingt•4mo ago

oh sorry, big confusion validation is about writing javascript code to chevk donething convex mutations are arbitrary javascript just check the database for a record with this string

beermanOP•4mo ago

that's what i've been doing, but i didn't even want to let it fail incase there's a wrong string in the array

beermanOP•4mo ago

that's why i wanted to get type checking on the array i'm passing into the mutation

ballingt•4mo ago

sorry what do you mean by type chevking here

beermanOP•4mo ago

line 18,17 i'm going to be passing arrays of features, amenities, views if i wanted to hardcode, I would do something like this and a would show as invalid

type feature = 'something' | 'somethingElse';
const features: feature[] = ['a', 'something'];

type feature = 'something' | 'somethingElse';
const features: feature[] = ['a', 'something'];

but i don't want to hardcode this. I want to use the values from the features, amenities, views table these are all predefined

ballingt•4mo ago

see that's typescript, you're using types do you want the kind of chevking where it throws an error at runtime, or the kind where you get red squiggles in your editor?

beermanOP•4mo ago

red squiggles in the editor and convex function input would suffice since i only need it to tell me that one or more of the values provided are invalid

ballingt•4mo ago

v.string() convex validators do both

beermanOP•4mo ago

yes but that only checks that it's a string not that i'm passing it an invalid feature

ballingt•4mo ago

so red squiggles are a typescript thing, you'd need to write out the types or write a validator

beermanOP•4mo ago

ok, and with the validator, those values would still need to be hardcoded

ballingt•4mo ago

Right, Yeah if you want more runtime checking, you do what you're doing below, checking the db convex validators are static, a string or a union of strings or the right kind of Id

beermanOP•4mo ago

if i was to delete the features table and just write a validator, i would then have to create a duplicate of the same feature across all the different properties that have it instead of just reusing the same record and linking to it in the property_features table. this is probably a lot less efficient right?

ballingt•4mo ago

I don't understand this

beermanOP•4mo ago

features table contains all possible features property_features table contains records of all features (property_id, feature_id) for all properties I don't know, it might work out the same.

ballingt•4mo ago

I don't get it, could you show a code example?

beermanOP•4mo ago

say i scrap the features table and just insert property_id, feature into property_features, is that in any way less efficient? it would make it one to many instead of many to many

ballingt•4mo ago

Could you show these schemas? I'm confused about what property_id, feature means oh ok you're building a join table? Doesn't seem necessary if it's one to many

beermanOP•4mo ago

hey, here is the features example

beermanOP•4mo ago

So here's the dilemma: should I continue doing it this way and add a custom validator to validate all the values of the passed array, or perhaps add the custom validator and ditch the features table in favor of simply adding features (property_id, name) into property_features how would you with all your knowledge and experience personally handle all this?

beermanOP•4mo ago

this is nice

ballingt•4mo ago

I'd do it this way, and you can still have a table of these if you want, indexed by that string

beermanOP•4mo ago

Alrighty, one more thing. Do you know how to use TableNamesInDataModel? it's a generic and expects a parameter, I assume my dataModel from generated/dataModel.d.ts?

        const getId = async (table: TableNamesInDataModel, name: string) => {
            const record = await ctx.db
                .query(table)
                .withIndex('by_name', (q) => q.eq('name', name))
                .unique();
            if (!record) throw new Error(`${table} "${name}" does not exist.`);
            return record._id;
        };

        const getId = async (table: TableNamesInDataModel, name: string) => {
            const record = await ctx.db
                .query(table)
                .withIndex('by_name', (q) => q.eq('name', name))
                .unique();
            if (!record) throw new Error(`${table} "${name}" does not exist.`);
            return record._id;
        };

EDIT: I think rather than TableNamesInDataModel, TableNames from _generated/dataModel should work. EDIT2: No, now my index names are not being correctly inferred "Argument of type "by_name" is not assignable to parameter of type "by_id" | "by_creation_date" TableNamesInDataModel<DataModel> cannot use namespace 'TableNamesInDataModel' as a type

ballingt•4mo ago

How did you import that type?

beermanOP•4mo ago

import TableNamesInDataModel from 'convex/server'; import DataModel from './_generated/dataModel';

ballingt•4mo ago

that's not valid import syntax well it doesn't do what you want you want to use curly braces

beermanOP•4mo ago

i did try that, as well as importing them as types. Here's with curly braces though

ballingt•4mo ago

Are you using an IDE / vscode? it can be more reliable to auto-import

beermanOP•4mo ago

neovim, i do have auto imports enabled. I just didn't accept as i was trying different things

ballingt•4mo ago

Did you fix both if these imports?

beermanOP•4mo ago

looks like it's defaulting to system indexes rather than picking up on the user defined indexes per table yep

ballingt•4mo ago

Can you look at these types? In neovim i write const foo: never = DataModel and look at the error

beermanOP•4mo ago

it is there

ballingt•4mo ago

not the source code, the hover type

beermanOP•4mo ago

ballingt•4mo ago

you might try vscode if you don't have your neivim typescript set up yet

beermanOP•4mo ago

I have it set up

ballingt•4mo ago

you want to be able to expand a type I'm wondering what that type is, maybe it's broken somewhere

beermanOP•4mo ago

I think the error might be happening because i'm trying withIndex on table even though some tables don't have by_name set up as an index

ballingt•4mo ago

oh sorry, there it is

beermanOP•4mo ago

harmless but still annoying obviously i'm not going to pass in a table that doesn't have that index

ballingt•4mo ago

I don't understand this by name isn't an index thing, it's just tables having names I'm going offline for a bit, I'd look for TypeScript errors

beermanOP•4mo ago

it has to be

beermanOP•4mo ago

this helper is meant to replace all these getTypeId helpers

ballingt•4mo ago

oh sorry, yeah that makes sense yeah you have e some generics to wrangle here

beermanOP•4mo ago

should be resolved if I also pass index as an argument instead of specifying in the query

ballingt•4mo ago

yeah maybe, these are tricky ok signing off

beermanOP•4mo ago

I'm just going to // @ts-ignore that one as it's a waste of time

ballingt•4mo ago

the generic is easier if you pass in the table instead of passing in the table name

beermanOP•4mo ago

I couldn't get it to work

        const getRecordId = async (table: TableNamesInDataModel<DataModel>, name: string) => {
            const record = await ctx.db
                .query(table)
                .withIndex('by_name', (q) => q.eq('name', name))
                .unique();

            if (!record) throw new Error(`Record with name "${name}" does not exist in table.`);

            return record._id;
        };

        const getRecordId = async (table: TableNamesInDataModel<DataModel>, name: string) => {
            const record = await ctx.db
                .query(table)
                .withIndex('by_name', (q) => q.eq('name', name))
                .unique();

            if (!record) throw new Error(`Record with name "${name}" does not exist in table.`);

            return record._id;
        };

can you show me what you mean?

ballingt•4mo ago

For this you can't accept TableNamesInDataModel<DataModel> with TypeScript because some of those tables might not have a .name column or a .by_name index, as you pointed out. So you'd want to write a function that took in a subset of DataModel containing just the tables that do fit those constraints (this is a fancy mapped type you could write) and then type your ctx with that. This is pretty fancy, I wouldn't bother. I'd write separate getRecordId functions for each table you need it for.

beermanOP•4mo ago

Yep, I came to the same conclusion and stuck with a separate function for each type of record as needed I stumbled into a problem where some properties would have additions like a pool with water jets and waterfalls that didn't exactly match the general "Private Pool" predefined feature in my db and validator, and adding all these variations of features every time one popped up is a mess and a nightmare. So I added a "details" field to account for these variations on a per-property level while still fitting into my general predefined feature and amenity options. I ended up with something a little more complex than i'd have liked (in terms of having to add a new feature or amenity to both the db table as well as the validator - il probably write a helper function that accepts a feature and adds it to the features table and then appends it to the validator stored in convex/validators or something like that to make this simpler), but I didn't see a simpler way to do it. Below are the are the relevant extracts you need to get an idea. Let me know what you think schema

properties: defineTable({
  ...
})

features: defineTable({
        name: v.string()
})
.index('by_name', ['name']),

property_features: defineTable({
    feature_id: v.id('features'),
    property_id: v.id('properties'),
    details: v.optional(v.string())
})
.index('by_feature_id', ['feature_id'])
.index('by_property_id', ['property_id']),

properties: defineTable({
  ...
})

features: defineTable({
        name: v.string()
})
.index('by_name', ['name']),

property_features: defineTable({
    feature_id: v.id('features'),
    property_id: v.id('properties'),
    details: v.optional(v.string())
})
.index('by_feature_id', ['feature_id'])
.index('by_property_id', ['property_id']),

mutation

const featuresValidator = v.array(
    v.object({
        name: v.union(
            v.literal('Balcony'),
            v.literal('Deck'),
            v.literal('Terrace'),
            v.literal('Patio'),
            v.literal('Private Pool'),
            ...
        ),
        details: v.optional(v.string())
    })
);

export default mutation({
    args: {
        propertyData: propertyDataValidator,
        features: featuresValidator,
    },
    handler: async (ctx, args) => {
        const { propertyData, features } = args;

        const getFeatureId = async (name: string) => {
            const record = await ctx.db
                .query('features')
                .withIndex('by_name', (q) => q.eq('name', name))
                .unique();

            if (!record) throw new Error(`Feature "${name}" does not exist.`);

            return record._id;
        };


        try {
            // Insert property
            const propertyId = await ctx.db.insert('properties', {
                ...propertyData,
                ...
            });

            // Insert related data (features, amenities, views) in parallel
            await Promise.all([
                ...features.map((feature) =>
                    getFeatureId(feature.name).then((featureId) =>
                        ctx.db.insert('property_features', {
                            property_id: propertyId,
                            feature_id: featureId,
                            details: feature.details
                        })
                    )
                ),
            ]);

            return { propertyId, propertyTypeId, propertyLocationId };
        } catch (error) {
            console.error('Error during property mutation', error);
            throw new Error('Failed to insert property or related data');
        }
    }
});

const featuresValidator = v.array(
    v.object({
        name: v.union(
            v.literal('Balcony'),
            v.literal('Deck'),
            v.literal('Terrace'),
            v.literal('Patio'),
            v.literal('Private Pool'),
            ...
        ),
        details: v.optional(v.string())
    })
);

export default mutation({
    args: {
        propertyData: propertyDataValidator,
        features: featuresValidator,
    },
    handler: async (ctx, args) => {
        const { propertyData, features } = args;

        const getFeatureId = async (name: string) => {
            const record = await ctx.db
                .query('features')
                .withIndex('by_name', (q) => q.eq('name', name))
                .unique();

            if (!record) throw new Error(`Feature "${name}" does not exist.`);

            return record._id;
        };


        try {
            // Insert property
            const propertyId = await ctx.db.insert('properties', {
                ...propertyData,
                ...
            });

            // Insert related data (features, amenities, views) in parallel
            await Promise.all([
                ...features.map((feature) =>
                    getFeatureId(feature.name).then((featureId) =>
                        ctx.db.insert('property_features', {
                            property_id: propertyId,
                            feature_id: featureId,
                            details: feature.details
                        })
                    )
                ),
            ]);

            return { propertyId, propertyTypeId, propertyLocationId };
        } catch (error) {
            console.error('Error during property mutation', error);
            throw new Error('Failed to insert property or related data');
        }
    }
});

beermanOP•4mo ago

For somebody like me who had only used Supabase in most of my projects and played around with PocketBase on a few occasions, Convex is magical and such a pleasure to work with.

beermanOP•4mo ago

@ballingt sorry for another ping, As you know, I have a table features which stores different feature options and joiner table property_features which connects the features to properties and has an additional details field for property specific information per feature. I'm trying to get all the property features (name, as well as the details), so I thought that getManyVia would be the best option to use, but I'm only getting the names but not the details. Am I not using the right helper? extract from schema

properties: defineTable({
  ...
})

features: defineTable({
        name: v.string()
})
.index('by_name', ['name']),

property_features: defineTable({
    feature_id: v.id('features'),
    property_id: v.id('properties'),
    details: v.optional(v.string())
})
.index('by_feature_id', ['feature_id'])
.index('by_property_id', ['property_id']),

properties: defineTable({
  ...
})

features: defineTable({
        name: v.string()
})
.index('by_name', ['name']),

property_features: defineTable({
    feature_id: v.id('features'),
    property_id: v.id('properties'),
    details: v.optional(v.string())
})
.index('by_feature_id', ['feature_id'])
.index('by_property_id', ['property_id']),

query

export const get = query({
    args: {},
    handler: async (ctx) => {
        // Fetch all properties
        const properties = await ctx.db.query('properties').collect();

        // Fetch related data for each property
        return await Promise.all(
            properties.map(async (property) => {
                const propertyFeatures = await getManyVia(
                    ctx.db,
                    'property_features',
                    'feature_id',
                    'by_property_id',
                    property._id
                );

                return {
                    propertyFeatures
                };
            })
        );
    }
});

export const get = query({
    args: {},
    handler: async (ctx) => {
        // Fetch all properties
        const properties = await ctx.db.query('properties').collect();

        // Fetch related data for each property
        return await Promise.all(
            properties.map(async (property) => {
                const propertyFeatures = await getManyVia(
                    ctx.db,
                    'property_features',
                    'feature_id',
                    'by_property_id',
                    property._id
                );

                return {
                    propertyFeatures
                };
            })
        );
    }
});

I know that I can do it like this but is it optimal?

const propertyFeatures = await asyncMap(
     await getManyFrom(ctx.db, 'property_features', 'by_property_id', property._id),
     async (record) => {
         const feature = await ctx.db.get(record.feature_id);
         if (!feature) throw new Error(`Feature ${record.feature_id} not found.`);

         return {
             name: feature.name,
             details: record.details
         };
    }
);

const propertyFeatures = await asyncMap(
     await getManyFrom(ctx.db, 'property_features', 'by_property_id', property._id),
     async (record) => {
         const feature = await ctx.db.get(record.feature_id);
         if (!feature) throw new Error(`Feature ${record.feature_id} not found.`);

         return {
             name: feature.name,
             details: record.details
         };
    }
);

struggling with mutation

Did you find this page helpful?