Airbyte connector not syncing records created after 9/10
The sync executes successfully right now, but doesn't seem to be syncing any records created after September 10th 🤔
9 Replies
Deployment name? I can check up on it to see what's going on
giant-ant-595I poked around at it and it seems like the connector had a traffic change on 09/19 and another traffic change ~09/16 - but not seeing evidence of not syncing records. That is pretty unusual. It's running and saying it completed sync, but isn't getting new records?
May be a good opportunity to give fivetran a try.
I'd also suggest recreating the airbyte connector.
It seems like the airbyte connector is making requests, but at a lower frequency to before.
Is it possible you have more than one airbyte connector set up, or changed the frequency somehow?
@RJ we have a lot more faith in the fivetran connector in the long run. might be worth giving it a shot. we've been developing it with the fivetran for weeks now. what's the destination? bigquery?
I did, I messaged you about it 🙂
Snowflake
I did just recreate the Airbyte connector (prior to starting this thread), but I'll do some more investigating to see if something funky is going on
But yeah, that's the behavior I'm seeing
I should also note that I’m open to suggestions if you have any regarding the destination, I’m not at all wed to Snowflake!
So I was running the OSS version of Airbyte on an EC2 instance—I just shut that instance down and set up a new connection using Airbyte's cloud service, and am observing the same behavior 😦
Separately, I'm also noticing that syncing fails if I try to replicate empty tables/streams. That's not a blocker for me though, simply not syncing said streams solves that problem.
Or at least prevents the syncing from failing with an error.
Cool - thanks for the info. I'm going to try and repro and figure out what's going on
I have a fix pending for the Airbyte empty tables issue! That should be fixed in the next day or two.
I'm now looking into not syncing issue in the title
The airbyte empty tables issue has been fixed! You should now be able to sync empty tables.
I have not been able to repro the records after 9/10 issue to airbyte. With permission from @RJ - I set up a convex connector to import from their instance and was able to sync documents after 9/10
Final update on this—we did sort of identify what the issue was, but it was not clear whether it was a Convex or Snowflake issue, and seemed too hard to pin down to be worth investigating further. So I've decided to just make the full-time switch now over to Fivetran! And that's been a great experience so far.
great! @RJ keep us apprised on how that continues to work out. we're putting a lot of eggs into the fivetran basket moving forward for customers who want a streaming export, so knowing that's a high quality experience is really important to us
Gladly!