fix(source-google-drive): dedupe files with identical paths to prevent file-transfer staging collision by dallasinman · Pull Request #83345 · airbytehq/airbyte

dallasinman · 2026-08-04T01:15:31Z

What

In file transfer mode, two Drive files that resolve to the same folder-relative path are staged to the same local path (/staging/files/<relative path>). The destination reads and then deletes each staged file (FileChunkTask.kt:81-107), so the second record referencing that path throws java.io.FileNotFoundException and the destination exits 1 — killing the whole sync, permanently, on every attempt.

Real customer impact (org 92102db1-f952-4087-895f-196a3daee3ef): zero successful syncs since 2026-07-31, 5/5 attempts of every hourly job failing on the same staged path, blocking their Context Store entirely. 433 other files transfer fine; two duplicate-named pairs break everything.

This revives the fix from #79696 (open, conflicted) rebased onto current master.

How

SourceGoogleDriveStreamReader.get_matching_files now collects matching files keyed by their folder-relative URI instead of yielding immediately, keeps the most recently modified file on a collision, and logs a warning naming the dropped Drive file ID, the colliding path, and the retained ID:

Dropping duplicate Google Drive file id '<old>' for colliding path '<path>'; keeping newer file id '<new>'

Dedupe is keyed on path + name, not the basename, so identically-named files in different folders are unaffected.

Review guide

source_google_drive/stream_reader.py — dedupe by URI, newest wins.
unit_tests/test_reader.py — two Drive IDs, one filename, one emitted record.
metadata.yaml / pyproject.toml / docs/integrations/sources/google-drive.md — 0.5.27 → 0.5.28.

Two things worth a reviewer's opinion:

Laziness tradeoff. Records are now buffered until folder traversal completes rather than streamed as discovered. This matches the approach in fix(source-google-drive): deduplicate files by URI to prevent ENOENT errors in file transfer mode #79696 but does change memory/first-record latency for very large drives. A Set[str] of seen paths with first-wins semantics would preserve streaming at the cost of not being able to prefer the newest duplicate.
Scope. This fixes the source side only. The generic CDK fix (per-file UUID staging dir, fix(file-based): give each transferred file a unique staging path airbyte-python-cdk#1078 + test(source-google-drive, source-microsoft-sharepoint): allow unique staging subdirectory #82754) is still draft and would prevent the crash for all file-based sources, but leaves duplicates overwriting each other at the destination object key. Separately, FileChunkTask arguably should fail soft on a missing staged file rather than aborting the sync.

User Impact

Syncs no longer fail when a Drive contains multiple files with the same name in the same folder. Tradeoff: only one of a duplicate-named pair is synced (the most recently modified); previously neither was, because the sync failed outright.

Can this PR be safely reverted and rolled back?

YES 💚
NO ❌

Requested by Dallas Inman to unblock a customer evaluation.

Link to Devin session: https://app.devin.ai/sessions/f1935df080e0485e9f80f7c2c9c70290
Requested by: dallasinman

devin-ai-integration · 2026-08-04T01:15:34Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment, CI, and merge conflict monitoring

github-actions · 2026-08-04T01:16:00Z

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

🛠️ Quick Fixes
- /format-fix - Fixes most formatting issues.
- /bump-version - Bumps connector versions, scraping changelog description from the PR title.
  - Bump types: patch (default), minor, major, major_rc, rc, promote.
  - The rc type is a smart default: applies minor_rc if stable, or bumps the RC number if already RC.
  - The promote type strips the RC suffix to finalize a release.
  - Example: /bump-version type=rc or /bump-version type=minor
- /bump-progressive-rollout-version - Alias for /bump-version type=rc. Bumps with an RC suffix and enables progressive rollout.
❇️ AI Testing and Review (internal link: AI-SDLC Docs):
- /ai-prove-fix - Runs prerelease readiness checks, including testing against customer connections.
- /ai-canary-prerelease - Rolls out prerelease to 5-10 connections for canary testing.
- /ai-review - AI-powered PR review for connector safety and quality gates.
📝 AI Documentation:
- /ai-docs-review - AI-powered documentation review for PRs with connector changes.
- /ai-create-docs-pr - Creates a documentation PR for connector changes, stacked on the current PR.
🚀 Connector Releases:
- /publish-connectors-prerelease - Publishes pre-release connector builds (tagged as {version}-preview.{git-sha}) for all modified connectors in the PR.
- /enable-autopilot-rollouts - Enables autopilot progressive rollouts for the modified connector(s) in the PR, remediating "autopilot rollouts not enabled for {connector-name}" auto-merge blockers. Sets defaultRolloutMode: autopilot and enableProgressiveRollout: true, preserving any existing autopilotConfig.
  - Optional args: connector=<CONNECTOR_NAME> (defaults to the modified connectors in the PR), strategy=fast|slow|default (defaults to fast).
  - Example: /enable-autopilot-rollouts or /enable-autopilot-rollouts connector=source-faker strategy=slow
☕️ JVM connectors:
- /update-connector-cdk-version connector=<CONNECTOR_NAME> - Updates the specified connector to the latest CDK version.
  Example: /update-connector-cdk-version connector=destination-bigquery
🐍 Python connectors:
- /poe connector source-example lock - Run the Poe lock task on the source-example connector, committing the results back to the branch.
- /poe source example lock - Alias for /poe connector source-example lock.
- /poe source example use-cdk-branch my/branch - Pin the source-example CDK reference to the branch name specified.
- /poe source example use-cdk-latest - Update the source-example CDK dependency to the latest available version.
⚙️ Admin commands:
- /force-merge reason="<REASON>" - Force merges the PR using admin privileges, bypassing CI checks. Requires a reason.
  Example: /force-merge reason="CI is flaky, tests pass locally"

📚 Show Repo Guidance

Helpful Resources

Breaking Changes Guide - Breaking changes, migration guides, and upgrade deadlines
Developing Connectors Locally
Managing Connector Secrets
On-Demand Regression Tests
#connector-ci-issues
#connector-publish-updates
#connector-build-statuses

📝 Edit this welcome message.

github-actions · 2026-08-04T01:16:21Z

Note

Autopilot progressive rollouts are not enabled for the following modified connector(s):

source-google-drive

This is a courtesy heads-up only — it does not block merge or fail any check.
To enable automatic progressive rollouts for the connector(s) above, comment
/enable-autopilot-rollouts on this PR. This sets defaultRolloutMode: autopilot
and enableProgressiveRollout: true in each connector's metadata.yaml,
preserving any existing autopilotConfig.

github-actions · 2026-08-04T01:20:09Z

Deploy preview for airbyte-docs ready!

Project:	`airbyte-docs`
Status:	✅ Deploy successful!
Preview URL:	https://airbyte-docs-64ulbrgia-airbyte-growth.vercel.app
Latest Commit:	`f3a1b4d`

Deployed with vercel-action

github-actions · 2026-08-04T01:20:47Z

`source-google-drive` Connector Test Results

45 tests 41 ✅ 33s ⏱️
2 suites 4 💤
2 files 0 ❌

Results for commit 766c445.

♻️ This comment has been updated with latest results.

github-actions · 2026-08-04T01:24:17Z

Airbyte Support Bot (airbyte-support-bot) · 2026-08-04T11:34:12Z

↪️ Triggering /ai-resolve-conflicts per Hands-Free AI Triage Project triage next step.

Reason: The branch now conflicts with master (mergeStateStatus: DIRTY), which blocks every downstream pipeline stage. Resolving the conflict lets the PR resume at its correct stage on the next pass.

Devin session

octavia-bot · 2026-08-04T11:34:36Z

…t staging path collision Co-Authored-By: dallas.inman <dallas.inman@airbyte.io>

devin-ai-integration · 2026-08-04T11:35:36Z

☑️ Conflict resolved and pushed (766c445d406) — no separate conflict-resolution pass needed.

master published 0.5.27 as a dependency update (#83492) while this was open, so the only conflict was the changelog row. Rebased onto master and re-bumped this change to 0.5.28 in metadata.yaml, pyproject.toml, and the docs changelog. No change to the dedupe logic.

Devin session

devin-ai-integration · 2026-08-04T11:36:24Z

⚠️ Could not push a conflict resolution: the PR branch changed while I was resolving.

I checked out f3a1b4d and merged origin/master (only conflict: the docs/integrations/sources/google-drive.md changelog, where master released 0.5.27 via #83492 — resolved by keeping both entries and bumping this connector to 0.5.28 in metadata.yaml / pyproject.toml). Before pushing, the branch had been force-updated to 766c445 (the same commit rebased onto a newer master), so git push --force-with-lease was correctly rejected rather than clobber that work.

Nothing was pushed. A fresh /ai-resolve-conflicts run against the current HEAD is needed. Note that master already ships google-drive 0.5.27, so whatever lands here must bump to 0.5.28.

Devin session

Airbyte Support Bot (airbyte-support-bot) · 2026-08-04T16:16:31Z

↪️ Triggering /ai-prove-fix per Hands-Free AI Triage Project triage next step.

Reason: /ai-resolve-conflicts pushed a resolved branch (HEAD 766c445d) and all connector CI checks are now green on it, with only the Vercel docs preview still outstanding. There is no prove-fix on record, so validating the path-dedupe behaviour is the next gate.

Devin session

octavia-bot · 2026-08-04T16:17:07Z

Airbyte Support Bot (airbyte-support-bot) · 2026-08-04T16:19:31Z

🟡 No Regression Detected — Fix Not Exercised (live test proposed, awaiting human approval)

source-google-drive PR #83345 (head 766c445d) · Pre-release: airbyte/source-google-drive:0.5.28-preview.766c445 · Baseline: latest (0.5.27)

Summary: Comparison regression tests pass cleanly with zero behavioral deltas, but the standard integration-test drive contains no duplicate-path files, so the dedupe code path was never exercised. Every currently-affected customer connection runs in a managed workspace whose source secrets the regression harness cannot retrieve (infrastructure limitation), so the fix could not be exercised via regression testing. A live prove-fix test (pin one affected connection to the pre-release + one sync) is proposed and awaiting human approval in Slack; this comment will be updated with the result if approved.

📋 Evidence log

Attempt 1 — GSM creds, all streams, control=0.5.27 (run): SPEC/CHECK/DISCOVER/READ all ✅, zero deltas (9 vs 9 records across test and test_unstructured). Regression-safety confirmed; dedupe path not exercised (no colliding paths in dataset).
Attempts against 3 affected customer connections (1, 2, 3): all failed on the same harness infrastructure error (workspace source secret not found in the secret store the harness reads) — inconclusive, not counted as test evidence.
The unit test added in this PR directly covers the duplicate-path scenario (two files, same relative path → one record, newest kept).
Detailed connection-level results (private): https://github.com/airbytehq/airbyte-internal-issues/issues/16894

🔬 Evidence plan & pre-flight checks

Proving criteria: on a drive with ≥2 files resolving to the same folder-relative path, target emits one record per path (newest kept) + "Dropping duplicate…" warning; control emits duplicates (the condition that crashes file-transfer destinations with FileNotFoundException).
Disproving criteria: duplicates still emitted by target, or regressions on SPEC/CHECK/DISCOVER/READ.

Pre-flight: viability ✅ · safety ✅ · not breaking ✅ (patch bump, no schema/spec/state changes) · reversible ✅. Note: files are now buffered until folder traversal completes (memory/latency tradeoff on very large drives), acknowledged in the PR.

Live test proposal (pending approval): pin one Tier-2, unpinned, US connection currently failing every sync with the exact staging-collision signature to 0.5.28-preview.766c445, trigger one sync; unpin immediately if disproven/inconclusive.

Next steps

If the live test is approved and passes, this will be upgraded to ✅ Fix Proven; consider /ai-canary-prerelease before broad rollout.
After merge, hands-free daily triage monitoring can confirm the affected connections recover.

Devin session

github-actions · 2026-08-04T16:21:42Z

github-actions · 2026-08-04T16:40:15Z

github-actions · 2026-08-04T16:56:22Z

github-actions · 2026-08-04T17:13:33Z

github-actions · 2026-08-04T17:17:27Z

github-actions · 2026-08-04T17:42:24Z

Airbyte Support Bot (airbyte-support-bot) · 2026-08-04T18:07:54Z

🙋 Escalated to #human-in-the-loop per Hands-Free AI Triage Project triage next step.

Reason: /ai-prove-fix cannot complete. It is waiting on approval to pivot to live connection testing, and every attempt against the three affected customer connections failed with the same infrastructure error: the regression harness could not retrieve those workspaces' source config secrets, because all three are Airbyte Hosted Data workspaces whose secrets live outside the GSM project the harness reads. The GSM test drive has no duplicate-path files, so the fix cannot be exercised there either — it showed no regression but did not validate the change. Someone needs to approve live connection testing or point the harness at a workspace where duplicate paths exist. The secret-store gap is worth fixing on its own, since it will block prove-fix for every Airbyte Hosted Data workspace.

Devin session

devin-ai-integration · 2026-08-04T18:08:27Z

Context from the author on the prove-fix escalation, so whoever picks this up doesn't re-derive it:

The regression harness can't validate this one by design. The failure mode is a destination-side crash (FileNotFoundException in FileChunkTask after the first record consumes and deletes the shared staged file), so it only reproduces in a real file-transfer sync against a Drive containing duplicate relative paths. A source-only READ comparison against the GSM test drive can't produce it — correctly, that run showed no regression and no signal.
The change is covered by a unit test asserting one emitted record for two Drive IDs sharing a filename, and the connector test suite is green on 766c445d.
The pre-release airbyte/source-google-drive:0.5.28-preview.766c445 is published and verified on Docker Hub. The intended live validation is pinning it for the reporting customer's actor, which is pending owner approval — that will be the end-to-end proof, and I'll post the result here.

The secret-store gap the bot flagged is real and separate: Airbyte Hosted Data workspaces keep their source config secrets outside the GSM project the harness reads, so prove-fix will fail the same way for every such workspace. Worth its own issue against the harness rather than blocking this PR.

Devin session

github-actions · 2026-08-04T18:11:47Z

dallasinman self-assigned this Aug 4, 2026

Octavia Squidington III (octavia-squidington-iii) added the connectors/source/google-drive label Aug 4, 2026

fix(source-google-drive): dedupe files with identical names to preven…

766c445

…t staging path collision Co-Authored-By: dallas.inman <dallas.inman@airbyte.io>

devin-ai-integration Bot force-pushed the devin/google-drive-dedupe-0.5.27 branch from f3a1b4d to 766c445 Compare August 4, 2026 11:35

Airbyte Support Bot (airbyte-support-bot) added the hyd-prove Hydra: ai-prove-fix stage has run label Aug 4, 2026

Uh oh!

Conversation

dallasinman commented Aug 4, 2026 • edited by devin-ai-integration Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Review guide

User Impact

Can this PR be safely reverted and rolled back?

Uh oh!

devin-ai-integration Bot commented Aug 4, 2026

🤖 Devin AI Engineer

Uh oh!

github-actions Bot commented Aug 4, 2026

👋 Greetings, Airbyte Team Member!

PR Slash Commands

Helpful Resources

Uh oh!

github-actions Bot commented Aug 4, 2026

Uh oh!

github-actions Bot commented Aug 4, 2026

Uh oh!

github-actions Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

source-google-drive Connector Test Results

Uh oh!

github-actions Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Airbyte Support Bot (airbyte-support-bot) commented Aug 4, 2026

Uh oh!

octavia-bot Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devin-ai-integration Bot commented Aug 4, 2026

Uh oh!

devin-ai-integration Bot commented Aug 4, 2026

Uh oh!

Airbyte Support Bot (airbyte-support-bot) commented Aug 4, 2026

Uh oh!

octavia-bot Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Airbyte Support Bot (airbyte-support-bot) commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Airbyte Support Bot (airbyte-support-bot) commented Aug 4, 2026

Uh oh!

devin-ai-integration Bot commented Aug 4, 2026

Uh oh!

github-actions Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dallasinman commented Aug 4, 2026 •

edited by devin-ai-integration Bot

Loading

github-actions Bot commented Aug 4, 2026 •

edited

Loading

`source-google-drive` Connector Test Results

github-actions Bot commented Aug 4, 2026 •

edited

Loading

octavia-bot Bot commented Aug 4, 2026 •

edited

Loading

octavia-bot Bot commented Aug 4, 2026 •

edited

Loading

Airbyte Support Bot (airbyte-support-bot) commented Aug 4, 2026 •

edited

Loading

github-actions Bot commented Aug 4, 2026 •

edited

Loading

github-actions Bot commented Aug 4, 2026 •

edited

Loading

github-actions Bot commented Aug 4, 2026 •

edited

Loading

github-actions Bot commented Aug 4, 2026 •

edited

Loading

github-actions Bot commented Aug 4, 2026 •

edited

Loading

github-actions Bot commented Aug 4, 2026 •

edited

Loading

github-actions Bot commented Aug 4, 2026 •

edited

Loading