feat(source-facebook-marketing): add incremental sync to ad_creatives_from_ads by devin-ai-integration[bot] · Pull Request #83704 · airbytehq/airbyte

devin-ai-integration · 2026-08-04T22:47:46Z

What

ad_creatives_from_ads is full refresh. Every sync pages the entire ads edge for an account, then issues one AdCreative.api_get() per unique creative ID — so the per-creative fan-out is proportional to the account's whole creative catalogue on every run.

For one multi-tenant customer this stream is 59% of total Facebook Marketing stream time across 30 days, and ~99.9% of their worst attempts (one job: 19.5h wall clock, of which this stream is 70,055s and every other stream combined is 44s). They're moving to a 3-hour cadence, which the current behaviour can't fit.

This makes the parent ads read incremental, so the fan-out scales with recently changed ads instead of all ads.

How

AdCreativesFromAds is reparented onto FBMarketingIncrementalStream. It already declared entity_prefix = "ad", so the inherited _state_filter emits the same request the Ads stream has used in production for years:

{"filtering": [{"field": "ad.updated_time", "operator": "GREATER_THAN", "value": <unix ts>}]}

request_params merges via deep_merge, so this clause is appended to the existing ad.effective_status IN [...] filter rather than replacing it — asserted by a new test.

The awkward part is that the cursor belongs to the parent ad but the emitted record is a creative, and AdCreative exposes no timestamp of its own (Meta's AdCreative reference has no created_time/updated_time field). So:

the parent read requests ["id", "creative", "updated_time"];
state advances from the parent ad, and read_records deliberately bypasses the incremental base's read_records — that implementation would take the cursor from whatever it yields, which here is a creative;
the parent's updated_time is copied onto the emitted creative, so the declared cursor_field actually exists on the record and in the schema;
a parent ad arriving without updated_time skips the state update instead of raising KeyError.

updated_time is added to ad_creatives.json but excluded from the fields requested from the AdCreative API in both AdCreativesFromAds._get_creative_fields() and AdCreatives.fields(), since it isn't a valid field on that object.

Review guide

source_facebook_marketing/streams/streams.py — the only behavioural change. Worth a close look at the FBMarketingStream.read_records(self, ...) call and the state handling around it.
source_facebook_marketing/schemas/ad_creatives.json — additive updated_time, plus the two exclusion sites that keep it out of the outbound field list.
unit_tests/test_streams.py — filter emitted with state; status and cursor clauses coexisting; no date filter on an empty first sync without start_date; cursor copied onto the creative; state advancing to the max parent cursor; missing parent updated_time not crashing.
integration_tests/configured_catalog.json / future_state.json — the stream was previously absent from the configured catalog entirely, so CAT never exercised it. It now runs and passes in CI.

User Impact

Opt-in. Existing connections keep running full refresh until a user switches the stream's sync mode; the first incremental run still does a full pass and seeds state from it.

Known limitation, worth calling out in review: the cursor is the parent ad's. Meta doesn't document ad.updated_time's semantics, but creatives are near-immutable — POST /{ad_creative_id} accepts only account_id, adlabels, name and status, so any content change requires creating a new creative and repointing the ad, which necessarily moves ad.updated_time. The residual gaps are an in-place rename/status/label change on a creative, and an edit to the page post behind effective_object_story_id — neither is guaranteed to move the ad's timestamp. Full refresh catches those today; incremental would not.

One adjacent improvement left for a separate PR: _get_creative_fields() reads the full schema rather than the configured catalog, so deselecting fields in the UI doesn't shrink the per-creative requests.

Version bumped 6.0.2 → 6.1.0. Additive, so no breakingChanges entry and no migration guide.

Can this PR be safely reverted and rolled back?

YES 💚

Link to Devin session: https://app.devin.ai/sessions/887ed96fb29b4b439476888f89ff39be

Co-Authored-By: mark.grivnin <mark.grivnin@airbyte.io>

devin-ai-integration · 2026-08-04T22:47:47Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment, CI, and merge conflict monitoring

github-actions · 2026-08-04T22:48:23Z

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

🛠️ Quick Fixes
- /format-fix - Fixes most formatting issues.
- /bump-version - Bumps connector versions, scraping changelog description from the PR title.
  - Bump types: patch (default), minor, major, major_rc, rc, promote.
  - The rc type is a smart default: applies minor_rc if stable, or bumps the RC number if already RC.
  - The promote type strips the RC suffix to finalize a release.
  - Example: /bump-version type=rc or /bump-version type=minor
- /bump-progressive-rollout-version - Alias for /bump-version type=rc. Bumps with an RC suffix and enables progressive rollout.
❇️ AI Testing and Review (internal link: AI-SDLC Docs):
- /ai-prove-fix - Runs prerelease readiness checks, including testing against customer connections.
- /ai-canary-prerelease - Rolls out prerelease to 5-10 connections for canary testing.
- /ai-review - AI-powered PR review for connector safety and quality gates.
📝 AI Documentation:
- /ai-docs-review - AI-powered documentation review for PRs with connector changes.
- /ai-create-docs-pr - Creates a documentation PR for connector changes, stacked on the current PR.
🚀 Connector Releases:
- /publish-connectors-prerelease - Publishes pre-release connector builds (tagged as {version}-preview.{git-sha}) for all modified connectors in the PR.
- /enable-autopilot-rollouts - Enables autopilot progressive rollouts for the modified connector(s) in the PR, remediating "autopilot rollouts not enabled for {connector-name}" auto-merge blockers. Sets defaultRolloutMode: autopilot and enableProgressiveRollout: true, preserving any existing autopilotConfig.
  - Optional args: connector=<CONNECTOR_NAME> (defaults to the modified connectors in the PR), strategy=fast|slow|default (defaults to fast).
  - Example: /enable-autopilot-rollouts or /enable-autopilot-rollouts connector=source-faker strategy=slow
☕️ JVM connectors:
- /update-connector-cdk-version connector=<CONNECTOR_NAME> - Updates the specified connector to the latest CDK version.
  Example: /update-connector-cdk-version connector=destination-bigquery
🐍 Python connectors:
- /poe connector source-example lock - Run the Poe lock task on the source-example connector, committing the results back to the branch.
- /poe source example lock - Alias for /poe connector source-example lock.
- /poe source example use-cdk-branch my/branch - Pin the source-example CDK reference to the branch name specified.
- /poe source example use-cdk-latest - Update the source-example CDK dependency to the latest available version.
⚙️ Admin commands:
- /force-merge reason="<REASON>" - Force merges the PR using admin privileges, bypassing CI checks. Requires a reason.
  Example: /force-merge reason="CI is flaky, tests pass locally"

📚 Show Repo Guidance

Helpful Resources

Breaking Changes Guide - Breaking changes, migration guides, and upgrade deadlines
Developing Connectors Locally
Managing Connector Secrets
On-Demand Regression Tests
#connector-ci-issues
#connector-publish-updates
#connector-build-statuses

📝 Edit this welcome message.

Co-Authored-By: mark.grivnin <mark.grivnin@airbyte.io>

github-actions · 2026-08-04T22:49:20Z

Note

Autopilot progressive rollouts are not enabled for the following modified connector(s):

source-facebook-marketing

This is a courtesy heads-up only — it does not block merge or fail any check.
To enable automatic progressive rollouts for the connector(s) above, comment
/enable-autopilot-rollouts on this PR. This sets defaultRolloutMode: autopilot
and enableProgressiveRollout: true in each connector's metadata.yaml,
preserving any existing autopilotConfig.

github-actions · 2026-08-04T22:51:25Z

`source-facebook-marketing` Connector Test Results

451 tests 447 ✅ 15s ⏱️
2 suites 4 💤
2 files 0 ❌

Results for commit ff233a0.

♻️ This comment has been updated with latest results.

github-actions · 2026-08-04T22:52:13Z

Deploy preview for airbyte-docs ready!

Project:	`airbyte-docs`
Status:	✅ Deploy successful!
Preview URL:	https://airbyte-docs-cf2jzw9n3-airbyte-growth.vercel.app
Latest Commit:	`ff233a0`

Deployed with vercel-action

github-actions · 2026-08-04T23:47:58Z

Connector CI Tests Started

These tests will leverage Airbyte's integration test credentials.

Check job output.
✅ Connector CI Tests job completed successfully. See logs for details.

Airbyte Support Bot (airbyte-support-bot) · 2026-08-05T11:37:32Z

↪️ Triggering /ai-review per Hands-Free AI Triage Project triage next step.

Reason: PR is ready for review and no /ai-review has run yet — this is the next pipeline stage for the incremental-sync addition to ad_creatives_from_ads.

Devin session

octavia-bot · 2026-08-05T11:37:56Z

AI PR Review starting...

Reviewing PR for connector safety and quality.
View playbook

🔗 Session URL: https://app.devin.ai/sessions/2cbae92ddc79456b94692b4a2305e3ee

✅ Devin AI session created successfully!

Airbyte Support Bot (airbyte-support-bot) · 2026-08-05T11:38:39Z

🛡️ AI PR Review Report

🟡 Review Action: NO ACTION (INCONCLUSIVE)

Gate	Status
Live / E2E Tests	UNKNOWN

🔶 Risk Level: 3/5

Cursor/state logic added to an existing stream (streams.py) with unit tests; opt-in and additive, but it changes how the parent ads edge is read.

🔧 Remediation Required

Live / E2E Tests (UNKNOWN) — no /ai-prove-fix run was found on this PR. This is not a failure; the change simply isn't validated yet.

Run /ai-prove-fix on this PR. It publishes a pre-release and validates the change (isolated regression by default; live-connection pinning only with explicit human approval).
Note: source-facebook-marketing Pre-Release Checks is currently failing on this HEAD SHA. That check is excluded from the CI Checks gate by design, but it will likely need to be resolved before a pre-release can be published for validation.

📋 PR Details

Connector(s): source-facebook-marketing
PR: #83704
HEAD SHA: ff233a0dff403c5b1ce30575778f6288d1e6ee8d
Session: https://app.devin.ai/sessions/2cbae92ddc79456b94692b4a2305e3ee

🔍 Gate Evaluation Details

Gate-by-Gate Analysis

Gate	Status	Enforced?	Details
PR Hygiene	PASS	Yes	Description is ~3.5k chars of substantive content (What / How / Review guide / User Impact); `docs/integrations/sources/facebook-marketing.md` changelog updated with a `6.1.0` row; no unresolved human review comments.
Code Hygiene	PASS	WARNING	Source changed (`streams.py`, `source.py`, `schemas/ad_creatives.json`) and `unit_tests/test_streams.py` changed (+120/-2).
Test Coverage	PASS	Yes	Behavioral change (title `feat(...)`, `add`). Four new test functions with new assertions added in `unit_tests/test_streams.py`.
Code Security	PASS	Yes	No changed file matches auth/oauth/token/credential/secret/encrypt path patterns. `metadata.yaml` diff is `dockerImageTag: 6.0.2 → 6.1.0` only — no `allowedHosts`, `connectorBuildOptions`, or `dockerRepository` keywords.
Per-Record Performance	PASS	WARNING	The per-record additions in `read_records` are a dict lookup, an assignment, and a datetime comparison in `_get_updated_state`. No new allocations, network calls, or regex compilation in the record loop. The PR's stated intent is to reduce per-sync fan-out.
Breaking Dependencies	PASS	WARNING	`pyproject.toml` diff is the connector's own `version` bump only; no dependency versions changed.
Backwards Compatibility	PASS	Yes (enforced)	See "Backwards compatibility analysis" below — additive only.
Forwards Compatibility	PASS	Warning (elevates Risk Level)	See "Forwards compatibility analysis" below.
Behavioral Changes	PASS	Warning (elevates Risk Level)	See "Behavioral changes analysis" below.
Out-of-Scope Changes	PASS	Skip	All 9 changed files are under `airbyte-integrations/connectors/source-facebook-marketing/` or `docs/`. In scope.
CI Checks	PASS	Yes	See "CI analysis" below.
Live / E2E Tests	UNKNOWN	Yes	Validation is required (sync-behavior change: `streams.py`, `cursor_field`, `state`, `incremental`). No `/ai-prove-fix` conclusion found on the PR → UNKNOWN, not FAIL.

Backwards compatibility analysis

Changed files matching breaking-change patterns: schemas/ad_creatives.json, metadata.yaml, integration_tests/configured_catalog.json, integration_tests/future_state.json.

Schema: the only schema edit is the addition of a new property:
```
"updated_time": { "type": ["null", "string"], "format": "date-time" }
```
No existing property was removed, and no existing property's type or format was changed. Adding format: date-time to a new field is additive; the enforced format-change rule applies to existing fields, which are untouched here.
Streams: no stream removed or renamed. ad_creatives_from_ads gains incremental as a supported sync mode; existing connections stay on full refresh until a user opts in.
Primary key / cursor: the stream's primary key is unchanged and its record values are not re-keyed — _fetch_creative_details still keys on the creative id. A cursor_field is introduced where none existed; no prior cursor value is being redefined.
Version: 6.0.2 → 6.1.0 (minor). Consistent with an additive change; correspondingly no releases.breakingChanges entry and no migration guide, which is correct here.

Forwards compatibility analysis

State/cursor keywords are present in the diff (state, cursor_field, updated_time, FBMarketingIncrementalStream), so this gate was evaluated in full rather than short-circuited.

Partition-key sub-check: not triggered. The partition/slicing shape ({"account_id": ...}) is unchanged; no partition router, partition_field, or parent-stream config was modified.
Rollback: rolling back to 6.0.2 returns the stream to full refresh, where any state written by 6.1.0 is simply ignored — no state-format collision. future_state.json was updated so the CAT future-state test exercises the new state shape.
Tests: test_ad_creatives_from_ads_emits_parent_updated_time_and_advances_state asserts state advances to the max parent cursor, and test_ad_creatives_from_ads_parent_without_updated_time_does_not_crash asserts a missing parent updated_time leaves state untouched instead of raising.
Residual (non-blocking) risk, already disclosed in the PR description: the cursor is the parent ad's updated_time, not the creative's, so an in-place creative rename/status/label edit or an edit to the page post behind effective_object_story_id may not move the cursor and could be missed in incremental mode. This is inherent to the design (AdCreative exposes no timestamp) and is opt-in, but it is the item most worth a human's judgement.

Behavioral changes analysis

Grepping the diff hunks for operational-risk keywords, the only literal match is "limit": 100 inside the new test test_ad_creatives_from_ads_request_params_skip_cursor_on_first_sync_without_start_date, which asserts the existing default page limit rather than changing it. No rate_limit, retry, backoff, timeout, sleep, error_handler, response_filters, or http_codes changes appear in the diff. The request-shape change is the added ad.updated_time GREATER_THAN filtering clause, which is appended (via deep_merge) to the existing ad.effective_status IN [...] clause — asserted by test_ad_creatives_from_ads_request_params_include_status_and_cursor_filters.

CI analysis

The single failing check-run on this HEAD SHA is source-facebook-marketing Pre-Release Checks, which matches the *Pre-Release Checks* exclusion pattern and is owned by the Live / E2E Tests gate, not CI Checks. All core checks are green:

Test source-facebook-marketing Connector — success (451 tests, 447 ✅, 0 ❌)
Lint source-facebook-marketing Connector — success
Build and Verify Artifacts (source-facebook-marketing) — success
Connector CI Checks Summary — success
Format Check, Check Changelog Updated, Docs / MarkDownLint, Docs / Vale, Build Airbyte Docs — success

📚 Evidence Consulted

Evidence

Changed files: 9 files (+174 / −14)
- source_facebook_marketing/streams/streams.py (+20 / −7)
- source_facebook_marketing/schemas/ad_creatives.json (+5)
- source_facebook_marketing/source.py (+2)
- unit_tests/test_streams.py (+120 / −2)
- integration_tests/configured_catalog.json (+9), integration_tests/future_state.json (+12)
- metadata.yaml (+1 / −1), pyproject.toml (+1 / −1)
- docs/integrations/sources/facebook-marketing.md (+4 / −3)
CI checks: 1 failure (source-facebook-marketing Pre-Release Checks — excluded, pre-release); all core build/lint/test checks success; combined commit status success
PR labels: connectors/source/facebook-marketing, hyd-review. No validation/override labels (live-tests-passed, prerelease-validated) present.
PR description: present (~3.5k chars, structured)
/ai-prove-fix: no run found on this PR
Existing bot reviews: none (no reviews of any kind on this PR)

❓ How to Respond

Resolving a Failing Gate

Each non-PASS gate above lists concrete remediation. Apply it, push, then re-run /ai-review — the bot re-evaluates from the diff, CI check-runs, and /ai-prove-fix results.

Written explanations do not change a gate verdict. If you believe a gate is a false positive or it cannot be remediated as written, request review from a human maintainer and explain the situation to them in a PR comment — a human reviewer can approve or merge over a bot FAIL. Never edit the PR description to argue with the bot; the description is how reviewers learn what the PR does.

github-actions · 2026-08-05T11:42:24Z

Pre-release Connector Publish Started

Publishing pre-release build for connector source-facebook-marketing.
PR: #83704

Pre-release versions will be tagged as {version}-preview.ff233a0
and are available for version pinning via the scoped_configuration API.

View workflow run
❌ Pre-release Publish FAILED for source-facebook-marketing.

github-actions · 2026-08-05T11:47:06Z

Pre-release Connector Publish Started

Publishing pre-release build for connector source-facebook-marketing.
PR: #83704

Pre-release versions will be tagged as {version}-preview.ff233a0
and are available for version pinning via the scoped_configuration API.

View workflow run
❌ Pre-release Publish FAILED for source-facebook-marketing.

devin-ai-integration Bot and others added 2 commits August 4, 2026 22:44

feat(source-facebook-marketing): add incremental ad creatives from ads

cf77d13

Co-Authored-By: mark.grivnin <mark.grivnin@airbyte.io>

fix(source-facebook-marketing): address incremental review feedback

2e1ee06

Co-Authored-By: mark.grivnin <mark.grivnin@airbyte.io>

Octavia Squidington III (octavia-squidington-iii) added the connectors/source/facebook-marketing label Aug 4, 2026

docs(source-facebook-marketing): link incremental sync PR

ff233a0

Co-Authored-By: mark.grivnin <mark.grivnin@airbyte.io>

devin-ai-integration Bot marked this pull request as ready for review August 4, 2026 23:24

Airbyte Support Bot (airbyte-support-bot) added the hyd-review label Aug 5, 2026

Uh oh!

Conversation

devin-ai-integration Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Review guide

User Impact

Can this PR be safely reverted and rolled back?

Uh oh!

devin-ai-integration Bot commented Aug 4, 2026

🤖 Devin AI Engineer

Uh oh!

github-actions Bot commented Aug 4, 2026

👋 Greetings, Airbyte Team Member!

PR Slash Commands

Helpful Resources

Uh oh!

github-actions Bot commented Aug 4, 2026

Uh oh!

github-actions Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

source-facebook-marketing Connector Test Results

Uh oh!

github-actions Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Airbyte Support Bot (airbyte-support-bot) commented Aug 5, 2026

Uh oh!

octavia-bot Bot commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Airbyte Support Bot (airbyte-support-bot) commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🛡️ AI PR Review Report

🟡 Review Action: NO ACTION (INCONCLUSIVE)

🔶 Risk Level: 3/5

Gate-by-Gate Analysis

Backwards compatibility analysis

Forwards compatibility analysis

Behavioral changes analysis

CI analysis

Evidence

Resolving a Failing Gate

Uh oh!

github-actions Bot commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

devin-ai-integration Bot commented Aug 4, 2026 •

edited

Loading

github-actions Bot commented Aug 4, 2026 •

edited

Loading

`source-facebook-marketing` Connector Test Results

github-actions Bot commented Aug 4, 2026 •

edited

Loading

github-actions Bot commented Aug 4, 2026 •

edited

Loading

octavia-bot Bot commented Aug 5, 2026 •

edited

Loading

Airbyte Support Bot (airbyte-support-bot) commented Aug 5, 2026 •

edited

Loading

github-actions Bot commented Aug 5, 2026 •

edited

Loading

github-actions Bot commented Aug 5, 2026 •

edited

Loading