Large Tables
Hi I'm trying to import a CSV that has 3M rows (crunchbase basic csv) for a lookup usecase I was looking forward to. Basically an Algolia/Redis type experience with the convenience of convex.
I chunked it into 100k rows each file but it still fails with a 408 :
Importing data from "../DATA/crunchbase_basic/01.csv" to table "cb_organizations" failed
408 Request Timeout
1. Is it a bad idea to have a 2-3Gb csv dataset as a single table for a algolia style experience?
2. What is the optimal batch size to avoid these fails?
3. I'm doing this in DEV but seems like an overkill to keep loading this data for every preview, I'd rather have a PROD table with this which I refer to in DEV but I realize that a convex app only can point to a single endpoint. Should I load the data to PROD and then "replicate" the environment to DEV to avoid having to load the data twice? how should I think about this?
4. Do I understand correctly that each row is considered at least 1k for the purpose of quotas (so I should try to keep long tables around but under 1k per row for maximum cost efficiency?)
Thanks
P.
I chunked it into 100k rows each file but it still fails with a 408 :
npx convex import --table cb_basic_org 01.csv --replace Importing data from "../DATA/crunchbase_basic/01.csv" to table "cb_organizations" failed408 Request Timeout
1. Is it a bad idea to have a 2-3Gb csv dataset as a single table for a algolia style experience?
2. What is the optimal batch size to avoid these fails?
3. I'm doing this in DEV but seems like an overkill to keep loading this data for every preview, I'd rather have a PROD table with this which I refer to in DEV but I realize that a convex app only can point to a single endpoint. Should I load the data to PROD and then "replicate" the environment to DEV to avoid having to load the data twice? how should I think about this?
4. Do I understand correctly that each row is considered at least 1k for the purpose of quotas (so I should try to keep long tables around but under 1k per row for maximum cost efficiency?)
Thanks
P.
