sbkl
sbkl
PostsComments
CCConvex Community
Created by sbkl on 4/10/2025 in #support-community
Using 5MB excel file from convex file storage: Your request couldn't be completed. Try again later
yes everything works. From other posts, I understood it was supposed to be faster. The difference is quite important.
16 replies
CCConvex Community
Created by sbkl on 4/14/2025 in #support-community
streaming api upsert for json fields send bad request response
my bad. was getting an error with my schema/update. Used a sample dataset with regular mutation to identify it.
3 replies
CCConvex Community
Created by sbkl on 4/13/2025 in #support-community
Streaming import API takes 3 times longer than regular functions
Batched by 10k rows and all rows are uploaded to the table. So the jsonSchema, primaryKey and messages seems to work fine. Am I missing anything?
6 replies
CCConvex Community
Created by sbkl on 4/13/2025 in #support-community
Streaming import API takes 3 times longer than regular functions
// index/primary key was added with /api/streaming_import/add_primary_key_indexes and confirmed active with /api/streaming_import/primary_key_indexes_ready
  const primaryKey = [
    ["organisationId"],
    ["collectionId"],
    ["regionId"],
    ["marketId"],
    ["storeId"],
    ["materialExternalId"],
    ["articleExternalId"],
  ];
  
  for (let i = 0; i < batches.length; i += 10000) {
    const batch = batches.slice(i, i + 10000);
    const data = {
      tables: {
        collectionAllocations: {
          primaryKey,
          jsonSchema: {
            type: "object",
            properties: {
              organisationId: {
                type: "id",
                tableName: "organisations",
              },
              collectionId: { type: "id", tableName: "collections" },
              regionId: { type: "id", tableName: "regions" },
              marketId: { type: "id", tableName: "markets" },
              storeId: { type: "id", tableName: "stores" },
              materialExternalId: { type: "string" },
              articleExternalId: { type: "string" },
              units: {
                type: "union",
                value: [
                  { type: "number" },
                  { type: "null" },
                  { type: "literal", value: "x" },
                ],
              },
            },
          },
        },
      },
      messages: batch.map((row) => {
        return {
          tableName: "collectionAllocations",
          data: row,
        };
      }),
    };
  
    const response = await fetch(
      `${env.CONVEX_URL}/api/streaming_import/import_airbyte_records`,
      {
        method: "POST",
        headers: {
          Accept: "application/json",
          "Content-Type": "application/json",
          "Convex-Client": "streaming-import-0.1.0",
          Authorization: `Convex ${env.DEPLOY_KEY}`,
        },
        body: JSON.stringify(data),
      },
    );
  }  
// index/primary key was added with /api/streaming_import/add_primary_key_indexes and confirmed active with /api/streaming_import/primary_key_indexes_ready
  const primaryKey = [
    ["organisationId"],
    ["collectionId"],
    ["regionId"],
    ["marketId"],
    ["storeId"],
    ["materialExternalId"],
    ["articleExternalId"],
  ];
  
  for (let i = 0; i < batches.length; i += 10000) {
    const batch = batches.slice(i, i + 10000);
    const data = {
      tables: {
        collectionAllocations: {
          primaryKey,
          jsonSchema: {
            type: "object",
            properties: {
              organisationId: {
                type: "id",
                tableName: "organisations",
              },
              collectionId: { type: "id", tableName: "collections" },
              regionId: { type: "id", tableName: "regions" },
              marketId: { type: "id", tableName: "markets" },
              storeId: { type: "id", tableName: "stores" },
              materialExternalId: { type: "string" },
              articleExternalId: { type: "string" },
              units: {
                type: "union",
                value: [
                  { type: "number" },
                  { type: "null" },
                  { type: "literal", value: "x" },
                ],
              },
            },
          },
        },
      },
      messages: batch.map((row) => {
        return {
          tableName: "collectionAllocations",
          data: row,
        };
      }),
    };
  
    const response = await fetch(
      `${env.CONVEX_URL}/api/streaming_import/import_airbyte_records`,
      {
        method: "POST",
        headers: {
          Accept: "application/json",
          "Content-Type": "application/json",
          "Convex-Client": "streaming-import-0.1.0",
          Authorization: `Convex ${env.DEPLOY_KEY}`,
        },
        body: JSON.stringify(data),
      },
    );
  }
6 replies
CCConvex Community
Created by sbkl on 4/10/2025 in #support-community
Using 5MB excel file from convex file storage: Your request couldn't be completed. Try again later
I might create another post on this as even if related, it is another issue...
16 replies
CCConvex Community
Created by sbkl on 4/10/2025 in #support-community
Using 5MB excel file from convex file storage: Your request couldn't be completed. Try again later
// index/primary key was added with /api/streaming_import/add_primary_key_indexes and confirmed active with /api/streaming_import/primary_key_indexes_ready
  const primaryKey = [
    ["organisationId"],
    ["collectionId"],
    ["regionId"],
    ["marketId"],
    ["storeId"],
    ["materialExternalId"],
    ["articleExternalId"],
  ];
  
  for (let i = 0; i < batches.length; i += 10000) {
    const batch = batches.slice(i, i + 10000);
    const data = {
      tables: {
        collectionAllocations: {
          primaryKey,
          jsonSchema: {
            type: "object",
            properties: {
              organisationId: {
                type: "id",
                tableName: "organisations",
              },
              collectionId: { type: "id", tableName: "collections" },
              regionId: { type: "id", tableName: "regions" },
              marketId: { type: "id", tableName: "markets" },
              storeId: { type: "id", tableName: "stores" },
              materialExternalId: { type: "string" },
              articleExternalId: { type: "string" },
              units: {
                type: "union",
                value: [
                  { type: "number" },
                  { type: "null" },
                  { type: "literal", value: "x" },
                ],
              },
            },
          },
        },
      },
      messages: batch.map((row) => {
        return {
          tableName: "collectionAllocations",
          data: row,
        };
      }),
    };
  
    const response = await fetch(
      `${env.CONVEX_URL}/api/streaming_import/import_airbyte_records`,
      {
        method: "POST",
        headers: {
          Accept: "application/json",
          "Content-Type": "application/json",
          "Convex-Client": "streaming-import-0.1.0",
          Authorization: `Convex ${env.DEPLOY_KEY}`,
        },
        body: JSON.stringify(data),
      },
    );
  }        
// index/primary key was added with /api/streaming_import/add_primary_key_indexes and confirmed active with /api/streaming_import/primary_key_indexes_ready
  const primaryKey = [
    ["organisationId"],
    ["collectionId"],
    ["regionId"],
    ["marketId"],
    ["storeId"],
    ["materialExternalId"],
    ["articleExternalId"],
  ];
  
  for (let i = 0; i < batches.length; i += 10000) {
    const batch = batches.slice(i, i + 10000);
    const data = {
      tables: {
        collectionAllocations: {
          primaryKey,
          jsonSchema: {
            type: "object",
            properties: {
              organisationId: {
                type: "id",
                tableName: "organisations",
              },
              collectionId: { type: "id", tableName: "collections" },
              regionId: { type: "id", tableName: "regions" },
              marketId: { type: "id", tableName: "markets" },
              storeId: { type: "id", tableName: "stores" },
              materialExternalId: { type: "string" },
              articleExternalId: { type: "string" },
              units: {
                type: "union",
                value: [
                  { type: "number" },
                  { type: "null" },
                  { type: "literal", value: "x" },
                ],
              },
            },
          },
        },
      },
      messages: batch.map((row) => {
        return {
          tableName: "collectionAllocations",
          data: row,
        };
      }),
    };
  
    const response = await fetch(
      `${env.CONVEX_URL}/api/streaming_import/import_airbyte_records`,
      {
        method: "POST",
        headers: {
          Accept: "application/json",
          "Content-Type": "application/json",
          "Convex-Client": "streaming-import-0.1.0",
          Authorization: `Convex ${env.DEPLOY_KEY}`,
        },
        body: JSON.stringify(data),
      },
    );
  }
16 replies
CCConvex Community
Created by sbkl on 4/10/2025 in #support-community
Using 5MB excel file from convex file storage: Your request couldn't be completed. Try again later
Just tried the streaming import api and uploaded 100k rows for test. While it was taking 2 min or so with regular queries (for database consistency check) and mutations, it takes 8 min with this import api which is supposed to be faster and more efficient at uploading big dataset than the regular function api.
16 replies
CCConvex Community
Created by sbkl on 4/10/2025 in #support-community
Using 5MB excel file from convex file storage: Your request couldn't be completed. Try again later
@Eternal Mori this is what I am doing. I am using "use node" for memory purpose, streaming the content of the excel file not to hit the memory too hard, I am very careful not to exceed the limits for arguments for actions, queries and mutations, using workflows and batching to ensure no action time limit is passed, indexing and paginating to ensure not too many bytes read from database etc... Having this error message: "Your request couldn't be completed. Try again later." is not very helpful to understand what is wrong and would like to know if there is a way to investigate further the root cause of the error.
16 replies
CCConvex Community
Created by sbkl on 4/10/2025 in #support-community
Using 5MB excel file from convex file storage: Your request couldn't be completed. Try again later
also I am storing the file with convex file storage passing the storage id to the actions and then retrieve the worbook with XLSX package
16 replies
CCConvex Community
Created by sbkl on 4/10/2025 in #support-community
Using 5MB excel file from convex file storage: Your request couldn't be completed. Try again later
use case: got 8000 articles for 80 stores for their quantity allocations. So roughly 700k rows to upsert. No transaction argument limit, making sure I don't go above the 600 seconds action limits or byte limit for function args.
16 replies
CCConvex Community
Created by sbkl on 4/10/2025 in #support-community
Using 5MB excel file from convex file storage: Your request couldn't be completed. Try again later
yes this is what I am doing. I am also using the workpool component and various optimisations to stream the excel workbook and hitting a wall. The most rows I was able to upsert with various batching sizes (500,1000 records) is 200k rows. enqueuing actions for each batch. Then at some point I get this error. Any direction on this would be appreciated!
16 replies
CCConvex Community
Created by sbkl on 10/28/2024 in #support-community
convex-react-query state and plan on useInfiniteQuery hook support
Not perfect yet as it doesn't account various args types. Like array of string that can hold same values but in different order but the core mechanics works. Types can be improved too.
24 replies
CCConvex Community
Created by sbkl on 10/28/2024 in #support-community
convex-react-query state and plan on useInfiniteQuery hook support
Usage: 
  const { results, status, loadMore } = useInfiniteQuery(
    // change with your paginated convex query path
    api.myPaginatedQuery,
    // Put the related args of your query excluding the paginationOptions
    args,
    { initialNumItems: 24 },
  );
  const { results, status, loadMore } = useInfiniteQuery(
    // change with your paginated convex query path
    api.myPaginatedQuery,
    // Put the related args of your query excluding the paginationOptions
    args,
    { initialNumItems: 24 },
  );
24 replies
CCConvex Community
Created by sbkl on 10/28/2024 in #support-community
convex-react-query state and plan on useInfiniteQuery hook support
Ok I finally made it. The valtio store manages multiple paginated queries and query keys for a given query based on args. Placeholder data are there out of the box when changing query params. use-infinite-query.ts 
"use client";

import type {
  PaginatedQueryArgs,
  PaginatedQueryReference,
  UsePaginatedQueryReturnType,
} from "convex/react";
import * as React from "react";
import { usePaginatedQuery } from "convex/react";
import { getFunctionName } from "convex/server";
import { proxy, useSnapshot } from "valtio";

type InfiniteQueryStore<Query extends PaginatedQueryReference> = Record<
  string,
  {
    queries: Record<string, UsePaginatedQueryReturnType<Query>>;
    lastKey: string | null;
  }
>;

const store = proxy<InfiniteQueryStore<any>>({});

export function useInfiniteQuery<Query extends PaginatedQueryReference>(
  query: Query,
  args: PaginatedQueryArgs<Query>,
  options: { initialNumItems: number },
) {
  const queryName = getFunctionName(query);
  const queryKey = JSON.stringify(args);
  const paginatedQuery = usePaginatedQuery(query, args, options);
  const state = React.useRef(store).current;
  const snap = useSnapshot(state);
  React.useEffect(() => {
    if (
      paginatedQuery.status !== "LoadingFirstPage" &&
      paginatedQuery.status !== "LoadingMore"
    ) {
      state[queryName] = {
        queries: state[queryName]
          ? {
              ...state[queryName].queries,
              [queryKey]: paginatedQuery,
            }
          : {
              [queryKey]: paginatedQuery,
            },
        lastKey: queryKey,
      };
    }
  }, [paginatedQuery, queryKey, queryName]);
  return (snap[queryName]?.queries[queryKey] ??
    (snap[queryName]?.lastKey
      ? snap[queryName]?.queries[snap[queryName]?.lastKey as string]
      : {
          results: [],
          status: "LoadingFirstPage",
          isLoading: true,
          loadMore: (_numItems: number) => {},
        })) as UsePaginatedQueryReturnType<Query>;
}
"use client";

import type {
  PaginatedQueryArgs,
  PaginatedQueryReference,
  UsePaginatedQueryReturnType,
} from "convex/react";
import * as React from "react";
import { usePaginatedQuery } from "convex/react";
import { getFunctionName } from "convex/server";
import { proxy, useSnapshot } from "valtio";

type InfiniteQueryStore<Query extends PaginatedQueryReference> = Record<
  string,
  {
    queries: Record<string, UsePaginatedQueryReturnType<Query>>;
    lastKey: string | null;
  }
>;

const store = proxy<InfiniteQueryStore<any>>({});

export function useInfiniteQuery<Query extends PaginatedQueryReference>(
  query: Query,
  args: PaginatedQueryArgs<Query>,
  options: { initialNumItems: number },
) {
  const queryName = getFunctionName(query);
  const queryKey = JSON.stringify(args);
  const paginatedQuery = usePaginatedQuery(query, args, options);
  const state = React.useRef(store).current;
  const snap = useSnapshot(state);
  React.useEffect(() => {
    if (
      paginatedQuery.status !== "LoadingFirstPage" &&
      paginatedQuery.status !== "LoadingMore"
    ) {
      state[queryName] = {
        queries: state[queryName]
          ? {
              ...state[queryName].queries,
              [queryKey]: paginatedQuery,
            }
          : {
              [queryKey]: paginatedQuery,
            },
        lastKey: queryKey,
      };
    }
  }, [paginatedQuery, queryKey, queryName]);
  return (snap[queryName]?.queries[queryKey] ??
    (snap[queryName]?.lastKey
      ? snap[queryName]?.queries[snap[queryName]?.lastKey as string]
      : {
          results: [],
          status: "LoadingFirstPage",
          isLoading: true,
          loadMore: (_numItems: number) => {},
        })) as UsePaginatedQueryReturnType<Query>;
}
24 replies
CCConvex Community
Created by sbkl on 1/30/2025 in #support-community
convex auth session across subdomains
The use case is to provide a custom subdomain or domain feature in a multi-tenant app. Using vercel for this. Created an issue. Clerk is definitely a great product but I am at the beginning of the project and want to keep it simple. So if I can do it with convex-auth, I've got the current need pretty well covered.
5 replies
CCConvex Community
Created by sbkl on 11/20/2024 in #support-community
Using gemini with openai library working in action but not http action
Thanks. Could be a reason. Ended up building a simple class mocking the openai library api for gemini api. Works great.
7 replies
CCConvex Community
Created by sbkl on 10/28/2024 in #support-community
convex-react-query state and plan on useInfiniteQuery hook support
or I can simply use tanstack query to invalidate the queries related for each mutation which seems more natural since I have been using tanstack query but not really the convex way.
24 replies
CCConvex Community
Created by sbkl on 10/28/2024 in #support-community
convex-react-query state and plan on useInfiniteQuery hook support
Was thinking to create a pagination table with a documentType, documentCount, create a record each time there is an update on a specific table (add, update, delete) and have a useQuery convex query listening for changes and use refetch each there is a change. I know all that is really not what convex is trying to achieve but that's all I can come up with to get pagination cached and working as expected while trying to still keep reactivity...
24 replies
CCConvex Community
Created by sbkl on 10/28/2024 in #support-community
convex-react-query state and plan on useInfiniteQuery hook support
@ballingt Gave up on that solution. Works well for a single simple paginated query, but for more complex queries with filters, I quickly realised I need a queryKey mechanism just like tanstack query and was finding myself trying to recreate the way the cache works with tanstack query lol. So now I am able to use useInfiniteQuery hook from tanstack query like so: 
export const list = protectedQuery({
  args: {
    paginationOptions: v.object({
      cursor: v.union(v.string(), v.null()),
      numItems: v.optional(v.number()),
    }),
  },
  handler: async (ctx, { paginationOptions: { cursor, numItems } }) => {
    const user = await getCurrentUserOrThrow(ctx);
    const results = await ctx.db
      .query("documents")
      .withIndex("userId", (q) => q.eq("userId", user._id))
      .order("desc")
      .paginate({
        cursor,
        numItems: numItems ?? 24,
      });

    const { page, isDone, continueCursor } = results;

    return {
      items: page,
      nextCursor: isDone ? undefined : continueCursor,
    };
  },
});
export const list = protectedQuery({
  args: {
    paginationOptions: v.object({
      cursor: v.union(v.string(), v.null()),
      numItems: v.optional(v.number()),
    }),
  },
  handler: async (ctx, { paginationOptions: { cursor, numItems } }) => {
    const user = await getCurrentUserOrThrow(ctx);
    const results = await ctx.db
      .query("documents")
      .withIndex("userId", (q) => q.eq("userId", user._id))
      .order("desc")
      .paginate({
        cursor,
        numItems: numItems ?? 24,
      });

    const { page, isDone, continueCursor } = results;

    return {
      items: page,
      nextCursor: isDone ? undefined : continueCursor,
    };
  },
});
Then: 
export function useDocumentInfiniteQuery() {
  const convex = useConvex();

  const infiniteQuery = useInfiniteQuery({
    queryKey: ["documents"],
    async queryFn({ pageParam }: { pageParam: string | null }) {
      const result = await convex.query(api.documents.query.list, {
        paginationOptions: {
          cursor: pageParam,
        },
      });
      return result;
    },
    getNextPageParam: (lastPage) => lastPage.nextCursor,
    initialPageParam: null,
  });

  return {
    ...infiniteQuery,
    items: infiniteQuery.data?.pages.flatMap((page) => page.items) ?? [],
  };
}
export function useDocumentInfiniteQuery() {
  const convex = useConvex();

  const infiniteQuery = useInfiniteQuery({
    queryKey: ["documents"],
    async queryFn({ pageParam }: { pageParam: string | null }) {
      const result = await convex.query(api.documents.query.list, {
        paginationOptions: {
          cursor: pageParam,
        },
      });
      return result;
    },
    getNextPageParam: (lastPage) => lastPage.nextCursor,
    initialPageParam: null,
  });

  return {
    ...infiniteQuery,
    items: infiniteQuery.data?.pages.flatMap((page) => page.items) ?? [],
  };
}
So from a pagination and cache point of view, I get everything out of the box from tanstack query and the paginate method in the convex backend and it is working properly BUT I obviously lost the reactivity. What would the way to subscribe to changes on a specific table and refetch with tanstack query?
24 replies
CCConvex Community
Created by sbkl on 10/28/2024 in #support-community
convex-react-query state and plan on useInfiniteQuery hook support
Actually the scroll position is kept without additional effort and it works like a charm! Also tested the loadMore methods and it works great. Thank you for the suggestion
24 replies