Airbyte Convex source connector failure
I'm using Airbyte (OSS version as the source connector was not available on Airbyte Cloud at the time) to sync my Convex data to Snowflake, and started seeing the following error today when I syncing:
Any clue what might be going on here?
53 Replies
👋 This looks like an issue on our end, which I'm trying to fix now! Sorry for the trouble and thank you for raising. I'll update back when this is fixed on our side.
Thank you @sshader!
I believe this is fixed now, but let us know if that's not the case!
It looks fixed now on my end, thanks again for your help @sshader 🙇♂️
Actually, I'm seeing that one very tiny table (containing four documents) is not syncing any of its records to Snowflake. I don't see any errors in the Airbyte syncing logs. I also tried changing the syncing method (for just that table) to see if that made a difference, from "Incremental | Deduped + history" to "Full refresh | Overwrite", and neither synced any data
@sshader, is there any chance the above might also be a Convex issue?
This is actually also the case for another table with only one document in it (which I have been meaning to delete)
I guess I might as well check all of them, one sec
I have a table with 10 documents that is syncing only 9 documents to Snowflake, and many appear to be duplicates
Another table with 10 documents is not syncing any, and other tables with fewer than 10 are not syncing any, without exception
Definitely possible, and sorry for the continued issues -- we're trying to come up with ideas on our end. Is it possible to try the "Reset your data" button on the connection and see if this resolves the issues? Also if you could send us the airbyte logs (here or over DM) that would be helpful for investigating
I've tried resetting the data a few times, but with the same results. I'll DM you the logs in a moment
We've been able to repro, and we're looking into the issue now (it's weird, when we curl the convex api directly it works, but the records aren't arriving at airbyte). Thanks for reporting!
Awesome, thank you!
Any clue when this might be fixed? My team is eager to be able to use Metabase again (which is what we're using Airbyte/Snowflake for), as they haven't been able to since Tuesday
From what we can tell, it's a bug in the airbyte convex connector, as opposed to being on the convex API side. We should have a patch today, but open source review and deployment means it won't be available on airbyte's repo for a while. If we gave you a git patch, would you be able to apply it to your airbyte clone/fork?
Technically I should be able to somehow, since I'm not using Airbyte Cloud (yet—they've made the Convex source connector available there now, though, so I'd like to switch from an EC2 instance running Docker to that).
Do you know why this only just started occurring? I did recently upgrade to the latest patch version of the Convex source connector—could that have been the cause? (It didn't look like it to me, based on the diff I saw, but what do I know.) I haven't upgraded Airbyte, either.
Great question. We'll get back to you later today. Quick question: have you always been using "Incremental deduped history" until this week?
More generally, FWIW, the vibe I've gotten from using Airbyte is that it's pretty unstable software (just based on the issues I've run into while setting it up, my attempts at troubleshooting those issues/the support available in those scenarios, the solutions it seems people have to resort to—things like that). And that kind of sucks because it's the only reasonable way to sync data from Convex, for all the reasons you might want to do that.
Do y'all have any interest in supporting other ETL (or whatever people call them these days—I hear the Airbyte folks like "ELT") software, like Vector (https://vector.dev/ )? Even if only so that folks have other options for accomplishing this kind of task? My sense of the data engineering landscape is that a lot of the tools in it are pretty brittle and unstable, but at least having multiple options would make things feel a bit safer, should scenarios like this one arise.
Yeah, I've been using that syncing strategy the whole time
We are looking into alternatives to Airbyte, yes (such as Fivetran). I'll pass the Vector idea along, thanks 😊.
(also y'all could build your own OLAP DB, that'd be cool too 😄)
We are also looking into that and will keep you posted 😊
One more data point in case it's helpful: Vector was recommended to me by ClickHouse folks—originally I tried to set up Airbyte with ClickHouse as the source (rather than Snowflake), but ran into issues there. The ClickHouse folks mentioned that they found that people tended to have a better experience using Vector than Airbyte—with ClickHouse specifically, I assume. But still, maybe means something 🤷♂️
I shouldn't say "than Airbyte"—rather I think they meant "in general, compared to similar types of tools" (again "for ClickHouse specifically" I assume)
@RJ the tool in this world that works really, really well is fivetran, and we have a relationship going with fivetran to get a source connector up soon. I'm pretty confident this will be an excellent solution to the "convex-as-a-source" side of things
the destination side, we're still searching
I do want to apologize for the quality of the airbyte and convex experience, we're not satisfied with it either. our team has really worked very hard over the last 6 months to try to make this work well (shout out to @Emma ), but it has been a real struggle
re the OLAP side, we are strongly considering having a SQL interface using some sort of OLAP engine for your data in convex, one that would execute without a transactional context and provide powerful querying on slightly stale data--and it wouldn't interfere with your live traffic
this is going to be really really useful! but I also want to fully acknowledge it's a great addition, but not a substition for a really good stream out to a cloud data warehouse
eventually most companies will truly need to get some of their convex data intermingled with their other CDW data and will need a true streaming egress solution. It isn't in our strategic plans at all to take on being a wholesale replacement for all the things snowflake or databricks do for you, so we will still need a great solution for the Convex -> CDW path eventually even if we (also) have a lightweight OLAP-esque engine within convex
cc @casado who is on the board of convex, fivetran, dbt, and knows a lot about this world. dunno if you have any thoughts
(also, internally, we do use vector and fivetran to drive convex, so we're confident with both of these tools and what they're good at)
re clickhouse recommending vector, to be blunt, it's because they're not on this list: https://fivetran.com/docs/destinations . and getting onto that list has been very, very competitive in the last few years because fivetran adds connectors quite slowly but with very high quality. the recent addition of "s3 data lake" starts to open things up more to open-format OLAP systems like DuckDB (which can operate directly on that data lake). vector is great, but it's not nearly as capable at this particular job as fivetran is. it's better for unstructured best-effort metrics data than 100% complete, typed, transactional replication with change-data-capture type semantics
Thanks for all the thoughts @jamwt! I can sympathize with the challenges in perfecting the Airbyte/Convex experience; I would be terrified to have to develop a connector for Airbyte, after what I've seen of it so far! 😅 So big thanks for sure to @Emma and the rest of the team for your efforts there 🙏 I'm also glad to hear that there is actually a good solution out there (Fivetran), and that you all are working to get integrated with it.
Haha, that definitely makes sense. This is obviously not an area of expertise for me, so I'm just gathering insights where I can! But much prefer y'all's take in any case.
cool. if you're looking for something lightweight right now, my personal recommendation would be ([whatever] -> fivetran -> s3 data lake), and then use DuckDB, which is really fun for "scale down analytics"
snowflake / databricks / bigquery and friends really are overkill for almost everyone's dataset
if you end up needing some sort of transforms on your raw data, in a single-machine world like duckdb, fun tools like https://dagster.io/ or dagworks.io are great ways to express transforms in a little bit of python
Dagster
The cloud-native open source orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability.
What are some use cases y'all are imagining this serving, given the assumption (which I agree with) that streaming to a CDW will be necessary in any case?
1. I want to quickly (one-off) do some digging in my convex data for a one-time exploratory purpose -- this is too hard right now
SQL is great for this as long as the expensive queries don't impact your live traffic. our own OLAP thing can solve this problem
2. I want to periodically compute aggregates (and probably store those aggregates back out into some table using a mutation) -- in an action
3. I want to grab a bunch of records to stream to e.g. AI service, in an action
a sql query on a replica sort of interface is realy useful for these things. especially as convex projects get bigger and have millions of records in them
My immediate reactions, for what they're worth:
1. I imagine I would almost immediately still want to just sync all the data to some service which provides a decent querying interface. Unless you're considering also providing some SQL querying UI in the dev dashboard, maybe with facilities to render the results and export them as CSVs or something?
2/3. I imagine that this would be most useful to me if I were able to still query using some kind of TS (or other language client) querying DSL. Is that also a part of what y'all have in mind for this sort of thing?
yeah, (1) would be with a little dashboard support to easily get to this query engine
@sujayakar has even built a little prototype of this awhile ago
Thank you for the recommendations!
(2/3) -- yeah, there might be some (I'm spitballing here, but)
runSQL function available in action contexts that you can just pass it queries and it returns some stream of rows or whatever
{runQuery, runSQL, ..}, {yourArgs, areHere}) => { // action body }
(it would be next to "Data", "Function" -- a "SQL" view or something that you can type a query and it gives you a basic row view with simple export, exactly)I think Fivetran -> DuckDB is the ultimate right answer for most projects.
Neat, I think those would both be really awesome! I do really love the philosophy of supporting the basic versions of features like this that would otherwise require a lot of effort and franken-stitching of different cloud services to have any answer to (like what you all did with Convex search). It matches the path of growth of software engineering team's needs really nicely—by the time many teams would need particularly sophisticated or specialized versions of what you offer in any given category, they're ideally better resourced to take on that additional work—or else what they need is at least significant enough in impact for that team that the extra work feels better justified.
this is our philosophy exactly, great to hear it's communicating through the product choices! thank you
Still curious for an answer to this question, if you’ve yet found one
my best guess is the behavior regressed in https://github.com/airbytehq/airbyte/pull/21924 . When it started affecting you depends on when you updated to Airbyte 0.27.0 . The bug symptom: a single page of up to 100 records is skipped at the beginning of each airbyte sync, because airbyte does an "availability check" which consumes records from the iterator. We're discussing with Airbyte support here https://airbytehq.slack.com/archives/C027KKE4BCZ/p1686329017267509
GitHub
Restore HttpAvailabilityStrategy as default for HttpStreams by eroh...
Set HttpAvailabilityStrategy as default for HttpStreams
Reverts #21488 and updates documentation accordingly
Closes #20401
Originally was going to be part of #21888, but want to avoid race conditio...
I had definitely never updated Airbyte since I set it up, unless it somehow did so automatically (I understand that's not the case). It had just been sitting on an EC2 instance, running on Docker, since I first set it up, untouched on the command line (until today, when I did update it).
The bug symptom: a single page of up to 100 records is skipped at the beginning of each airbyte sync, because airbyte does an "availability check" which consumes records from the iterator.Does this somehow explain the duplicate records I was seeing as well? Have you been able to reproduce that yourself? Now, since upgrading Airbyte to the latest version, I'm seeing this error when I try to sync:
We have disabled the connector entirely until we have a fix that doesn't drop hundreds of records
We have not been able to repro the duplicate records issue 🤔
I see, thanks
Sorry about this. Working on resolving ASAP
what version of airbyte were you using before upgrading today?
0.44.1, I believeinteresting. i would guess that you have always had this issue then 🫣 . it's difficult to notice if you're not looking for it. thanks so much for reporting!
I think that's pretty unlikely, for a few different reasons
One is that I had a Metabase query (so a query against Snowflake) which was summing up all of the values in one column of one table, and I also had a Convex query which was doing the same thing, and they had always agreed with each other (returned the same result) prior to this issue surfacing (I verified this many times)
good to know. you were using Incremental + dedupe history?
Yeah, the whole time
another hypothesis is that the "Bad Request for url" error which you originally noticed (which was a bug on the convex side that we fixed) reset the airbyte stream. it may be that the records only get skipped when the stream starts from scratch.
Hmm, interesting
I did reset all of the data many times over the course of my Airbyte instance's life, but I don't remember well enough exactly how recently I had done so prior to running into that issue
When this issue arose it was immediately apparent because it broke all of the Metabase queries that my team relies on
Actually, thinking about it more, I'm pretty sure that I'd reset the data recently enough that this issue would have been pretty apparent then
Well it's good to know it was working until recently, even if I don't understand why.
Unless that means y'all haven't actually found the root cause of the problem 😅 I don't care about the reasons that it is or was or wasn't working, as long as it does work going forward—I really just want to make sure you've got the most helpful/accurate info so you can debug successfully
agreed. i'm digging through airbyte code now to see if i can understand the root cause
Thanks @lee. I'm sorry this is such a bear to figure out! If I can provide any other info to be helpful, please let me know
Sorry to keep bothering about this, but any updates?
Hi. No worries, sorry for the delay. I'm waiting on a support response / code review from Airbyte. I've been going to their office hours to bump the issue each day. Once they have confirmed the fix, I can describe to you how to patch it into your repo.
Ok sounds good, thank you Lee!
hi! sorry for the delay. the fix has landed in airbyte's repo. i can either walk you through compiling it yourself, or you can wait for airbyte's next release
Do you have any idea how Airbyte determines when to cut new releases? I browsed around a bit and couldn't find any clues. It would obviously be much easier to wait for a new Airbyte release, and since at this point it's been >2 weeks since this has been broken, I'm ok waiting a few more days if that's the timeline we're talking about.
Well, I shouldn't say "obviously"—do you have a sense of how much time/effort it would be to pull in the updated connector manually, before a new Airbyte release?
i don't know how they cut releases, but it looks like they release every 2 weeks or so https://github.com/airbytehq/airbyte/tags. as for running it yourself, it '
it's pretty quick for me, but that may be because i've set up the build dependencies. the instructions are here https://docs.airbyte.com/contributing-to-airbyte/developing-locally/#add-a-connector-under-development-to-airbyte
basically it's
and then changing the Settings -> Source -> version number to "dev"
Developing Locally | Airbyte Documentation
The following technologies are required to build Airbyte locally.
They cut a release yesterday; I've upgraded the connector and, at least so far, everything looks to be working again. Thanks for getting this fixed @lee 🙇♂️