Not valid JSON: EOF

Getting this error when trying to import a file with CLI command

pnpx convex import --table temp_quotes ./data/temp_quotes_data.json

pnpx convex import --table temp_quotes ./data/temp_quotes_data.json

Importing data from "./data/temp_quotes_data.json" to table "temp_quotes" failed Not valid JSON: EOF while parsing a string at line 302 column 9

I have prettier as formatter and I see it adds a new line at the end of the file. Could that be affecting?

26 Replies

Gorka CesiumOP•10mo ago

this is what the file looks like at line 302 col 9

erquhart•10mo ago

Addresses in JSON parse errors are always post some sort of internal formatting, probably doesn’t map to your input file as is. More likely that there’s a legitimate syntax error somewhere, probably involving quotation marks. If you have jq installed you can see if you can get a better error message:

jq empty ./data/temp_quotes_data.json

jq empty ./data/temp_quotes_data.json

Gorka CesiumOP•10mo ago

@erquhart ran it but returns empty

Gorka CesiumOP•10mo ago

i'm extracting fields fine with jq no errors 🤔 i'm using MacOS

erquhart•10mo ago

The error says EOF while parsing a string, any chance the file is massive? If so I wonder if it's being truncated at some point in the import process. Yeah that command was just to check validity, sounds like it is. According to jq, anyway, but json validity is a moving target. Maybe the Convex team will have some insight.

Gorka CesiumOP•10mo ago

it is a couple of MB @ian any idea on what can be happening?

ian•10mo ago

A couple of MB shouldn't be an issue. @lee A couple thoughts: 1. In VSCode settings set "Render Whitespace" to "all" in case there are hidden characters like some null terminator. 2. Try running it with the first half of the file and see if it's an issue, or just the first row, and bisect to see if it's a certain entry. 3. You're on the latest Convex? @lee improved import in the last few weeks to handle huge imports/exports. It could have previously been terminating after some limit - maybe even as low as 4MB but don't quote me on that. It should be more scalable post 1.9 iirc. 4. Run jq -c '.[]' ./data/temp_quotes_data.json > temp_quotes_data.jsonl and try importing the jsonl format, which can scan line by line rather than needing to load it all at once as an array.

lee•10mo ago

converting to jsonl will probably work. i should look into where this error is coming from, but it definitely looks like the json is being truncated, which wouldn't be a problem with jsonl

Gorka CesiumOP•10mo ago

i'll try this. that jq command looks awesome

Gorka CesiumOP•10mo ago

i converted the file to jsonl and now i get this error

Gorka CesiumOP•10mo ago

✖ Importing data from "./data/temp_quotes_data.jsonl" to table "temp_quotes" failed
Hit an error while importing:
Could not complete import because schema changed. Avoid modifying schema.ts while importing tables

✖ Importing data from "./data/temp_quotes_data.jsonl" to table "temp_quotes" failed
Hit an error while importing:
Could not complete import because schema changed. Avoid modifying schema.ts while importing tables

is this import command importing to Dev or Prod? my intention is to Dev

erquhart•10mo ago

dev is the default Assuming you tried more than once and got the same error?

Gorka CesiumOP•10mo ago

i see this error in dev

Gorka CesiumOP•10mo ago

i'll try with the optional for supplier field running "convex": "^1.10.0",

erquhart•10mo ago

Yeah looks like the imported data is legitimately failing schema validation. You could also just turn off schema validation before the import if you know it doesn't matter. Might be nice to have a flag for that actually

Gorka CesiumOP•10mo ago

that optional worked! so it was helpful to run convex dev side by side with the convex import so i could see the errors

erquhart•10mo ago

Nice! The error from the import command was basically incorrect, though, so that could be improved. cc/ @lee

Gorka CesiumOP•10mo ago

yeah, that would've been very quick to debug

lee•10mo ago

i don't understand why npx convex import would throw that error unless you're actively changing the schema. if the data doesn't match the schema, it throws a different error

Gorka CesiumOP•10mo ago

i think it was throwing the "schema change error" because the json was not matching 100% my schema but it wasn't saying what field was mismatching until i ran convex dev i saw exactly what was the mismatch

erquhart•10mo ago

If dev wasn't running then schema change wouldn't really be happening

Gorka CesiumOP•10mo ago

i ran the schema change with dev and then i stopped it so i would run the import

lee•10mo ago

when i try an import that doesn't match the schema, i get

1 - $ npx convex import --table messages data.jsonl
Import change summary:
table    | create | delete |
----------------------------
messages | 4      | 0 of 0 |
Once the import has started, it will run in the background.
Interrupting `npx convex import` will not cancel it.
✖ Importing data from "data.jsonl" to table "messages" failed
Hit an error while importing:
Failed to insert or update a document in table "messages" because it does not match the schema: Object is missing the required field `body`. Consider wrapping the field validator in `v.optional(...)` if this is expected.

Object: {}
Validator: v.object({body: v.string()})

1 - $ npx convex import --table messages data.jsonl
Import change summary:
table    | create | delete |
----------------------------
messages | 4      | 0 of 0 |
Once the import has started, it will run in the background.
Interrupting `npx convex import` will not cancel it.
✖ Importing data from "data.jsonl" to table "messages" failed
Hit an error while importing:
Failed to insert or update a document in table "messages" because it does not match the schema: Object is missing the required field `body`. Consider wrapping the field validator in `v.optional(...)` if this is expected.

Object: {}
Validator: v.object({body: v.string()})

i ran the schema change with dev and then i stopped it so i would run the import

ah that's probably it. if the dev command doesn't finish, the schema might still be trying to commit on the server i'm surprised retries wouldn't work though

Gorka CesiumOP•10mo ago

yeah maybe it was not done. i've been having a lot of delays when running dev in this project i'll remove all the heavy files from the convex directory that seems to help

lee•10mo ago

cool. we're working on performance of npx convex dev, so hopefully that won't be necessary for long btw i found the Not valid JSON: EOF while parsing a string at line 302 column 9 error and improved the error message to recommend the jq command to create JSONLines

Gorka CesiumOP•10mo ago

even when i had dev running, updated schema and imported, i noticed the schema failed. but dev said all was in sync before throwing the import error. then it synced again and the next import worked. so seems like dev saved changes after each import attempt instead of after saving the schema file

Not valid JSON: EOF

Did you find this page helpful?