Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
272 changes: 75 additions & 197 deletions skills/ads/data-manager-api/data-manager-api-audience-ingestion/SKILL.md

Large diffs are not rendered by default.

262 changes: 65 additions & 197 deletions skills/ads/data-manager-api/data-manager-api-event-ingestion/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,194 +14,32 @@ metadata:
---
# Data Manager API Event Ingestion

## Core Directives
## Implementation Workflow

* [IMPORTANT] When reading API documentation from developers.google.com, read
the Markdown equivalent of the page by appending `.md.txt` to the URL.
### Prerequisites

## Sequential Implementation Workflow

For simple informational, configuration, or setup questions, skip any irrelevant
steps and answer using only the relevant guidelines or documentation provided
below.
- **Authentication & Library Installation**: If you need to set up access to
the Data Manager API or install the client and utility libraries, refer to
the `data-manager-api-setup` skill.

### Step 1: Identify Use Case & Read Documentation

- **Determine Destination Account Type**: [CRITICAL] If it's not
explicitly stated, STOP and CLARIFY with the user where the data is being
sent (e.g., Google Ads, Floodlight, Google Analytics)
BEFORE generating any code. Do NOT assume Google Ads by default. This maps
BEFORE generating any code. Do not assume Google Ads by default. This maps
to the `account_type` field of the `operating_account` in the `Destination`,
and also determines valid event identifiers and requirements.
- **Read Documentation**: [CRITICAL] You MUST follow the
the [Send events guide](https://developers.google.com/data-manager/api/devguides/events/send-events)
to understand implementation steps, user/event identifier requirements, and
how to configure the destination object.

### Step 2: Setup Auth

1. **Enable API (Prerequisite)**: Check that the user has enabled the Data
Manager API in their Google Cloud project.
2. **Generate ADC**: Authenticate the local workspace using Application
Default Credentials (ADC) via `gcloud auth application-default login`.
* **Required Scopes**: Include scopes
`https://www.googleapis.com/auth/datamanager` and
`https://www.googleapis.com/auth/cloud-platform`.
* **Multi-API Scopes**: If using the same credentials for other APIs,
append their scopes (e.g.,
`https://www.googleapis.com/auth/adwords`).
* **Service Accounts**: Ensure the Service Account has the
`Service Usage Consumer` IAM role, and the user executing `gcloud`
has the Token Creator role
(`roles/iam.serviceAccountTokenCreator`) on that Service Account
for impersonation.
3. **Reference**: Refer to [Set up API access](https://developers.google.com/data-manager/api/devguides/quickstart/set-up-access)
for a walkthrough of the `gcloud` CLI auth setup.

### Step 3: Install Client Library and Utilities

Refer to [Install a client
library](https://developers.google.com/data-manager/api/devguides/quickstart/install-library)
for detailed installation instructions.

| Language | Installation Instructions |
| :--- | :--- |
| **Python** | `pip install google-ads-datamanager` |
| **Java** | Follow the [quickstart](https://github.com/googleapis/google-cloud-java/tree/main/java-datamanager#quickstart) instructions to install the Maven/Gradle dependency. |
| **Node** | `npm install @google-ads/datamanager` |
| **PHP** | Install the `googleads/data-manager` component using Composer. |
| **.NET** | Install the `Google.Ads.DataManager.V1` NuGet package. |

[IMPORTANT] The utility library is NOT available on public package managers
(such as PyPI or Maven). Follow the below instructions to install:

1. Clone the repository from GitHub using the commands from the table below.
2. Always use the latest available version of the library. Determine the
actual version identifier (`VERSION`) from the cloned repository metadata.
- **Python**: Find the version in `pyproject.toml`.
- **Java**: Find the version in `data-manager-util/build.gradle`.
- **Node**: Find the version in `util/package.json` (as the `version`
field).
3. Follow the language-specific instructions in the next section to build
and install the utility dependency, replacing `VERSION` with the version
identifier you found.

| Language | Git Clone Command |
| :--- | :--- |
| **Python** | `git clone https://github.com/googleads/data-manager-python.git` |
| **Java** | `git clone https://github.com/googleads/data-manager-java.git` |
| **Node** | `git clone https://github.com/googleads/data-manager-node.git` |
| **PHP** | `git clone https://github.com/googleads/data-manager-php.git` |
| **.NET** | `git clone https://github.com/googleads/data-manager-dotnet.git` |
- **Read Documentation**: [CRITICAL] Follow the
[Send events guide](https://developers.google.com/data-manager/api/devguides/events/send-events.md.txt)
to implement the integration, as steps for configuring and sending the
request may vary between destinations.

#### Install Utility Library
### Step 2: Retrieve Code Sample

##### Python

1. Navigate to the `data-manager-python` directory and install the utility
library:

```shell
pip install .
```
2. Declare a dependency in your project's `requirements.txt` file (replacing
`VERSION` with the identified version):

```
google-ads-datamanager-util==VERSION
```

##### Java

1. Navigate to the `data-manager-java` directory.
2. Build and publish the utility library to your local Maven repository:

```shell
./gradlew data-manager-util:install
```
3. Declare a dependency on the utility library in your project (replacing
`VERSION` with the identified version):
* **Gradle**:
```none
implementation 'com.google.api-ads:data-manager-util:VERSION'
```

* **Maven**:
```xml
<dependency>
<groupId>com.google.api-ads</groupId>
<artifactId>data-manager-util</artifactId>
<version>VERSION</version>
</dependency>
```

##### Node

1. Navigate to the `data-manager-node` directory and install dependencies:

```shell
npm install
```
2. Navigate to the `util` directory:

```shell
cd util
```
3. Pack the utility library into a `.tgz` archive:

```shell
npm pack
```
4. Declare a dependency in your Node.js project's `package.json` pointing to
the path of the generated `.tgz` archive (replacing `VERSION` with the
identified version):

```json
{
"dependencies": {
"@google-ads/data-manager-util": "file:/path/to/google-ads-datamanager-util-VERSION.tgz"
}
}
```

##### PHP

1. Navigate to the `data-manager-php` directory.
2. Resolve dependencies for the library:

```shell
composer update --prefer-dist
```
3. Update your project's `composer.json` to declare a dependency on the utility
library using a path repository:

```json
{
"repositories": [
{
"type": "path",
"url": "/path/to/cloned/data-manager-php"
}
],
"require": {
"googleads/data-manager-util": "@dev"
}
}
```

##### .NET

In your .NET project, declare a `ProjectReference` dependency pointing to the
cloned library's `.csproj` path:

```xml
<ProjectReference Include="\path\to\cloned\Google.Ads.DataManager.Util\src\Google.Ads.DataManager.Util.csproj" />
```

### Step 4: Retrieve Code Sample

[IMPORTANT] If writing or updating an ingestion script, ALWAYS retrieve the
relevant code sample to use as a reference:
> [!IMPORTANT]
> If writing or updating an ingestion script, ALWAYS retrieve the
> relevant code sample to use as a reference:

| Language | Sample |
| :--- | :--- |
Expand All @@ -211,29 +49,30 @@ relevant code sample to use as a reference:
| **Node** | [`ingest_events.ts`](https://github.com/googleads/data-manager-node/blob/main/samples/events/ingest_events.ts) |
| **.NET**| [`IngestEvents.cs`](https://github.com/googleads/data-manager-dotnet/blob/main/samples/IngestEvents.cs) |

### Step 5: Retrieve migration guides
### Step 3: Retrieve Migration Guides

[CRITICAL] If refactoring code to upgrade from another Google API, ALWAYS
extract the full contents of the relevant field mapping guide.
> [!CRITICAL]
> If refactoring code to upgrade from another Google API, ALWAYS
> extract the full contents of the relevant field mapping guide.

#### Google Ads

* **Google Ads API Offline Conversions**:
[Google Ads Offline Conversions Migration Field Mappings](https://developers.google.com/data-manager/api/devguides/events/google-ads/offline/upgrade/field-mappings)
[Google Ads Offline Conversions Migration Field Mappings](https://developers.google.com/data-manager/api/devguides/events/google-ads/offline/upgrade/field-mappings.md.txt)
* **Google Ads API Store Sales**:
[Google Ads Store Sales Migration Field Mappings](https://developers.google.com/data-manager/api/devguides/events/google-ads/store-sales/upgrade/field-mappings)
[Google Ads Store Sales Migration Field Mappings](https://developers.google.com/data-manager/api/devguides/events/google-ads/store-sales/upgrade/field-mappings.md.txt)

#### Google Analytics

* **Measurement Protocol (Google Analytics)**:
[Google Analytics Measurement Protocol Migration Field Mappings](https://developers.google.com/data-manager/api/devguides/events/analytics/measurement-protocol/upgrade/field-mappings)
[Google Analytics Measurement Protocol Migration Field Mappings](https://developers.google.com/data-manager/api/devguides/events/analytics/measurement-protocol/upgrade/field-mappings.md.txt)

#### Campaign Manager 360 (CM360)

* **Campaign Manager 360 API Offline Conversions**:
[Campaign Manager 360 Offline Conversions Migration Field Mappings](https://developers.google.com/data-manager/api/devguides/events/cm360/offline/upgrade/field-mappings)
[Campaign Manager 360 Offline Conversions Migration Field Mappings](https://developers.google.com/data-manager/api/devguides/events/cm360/offline/upgrade/field-mappings.md.txt)

### Step 6: Implementation
### Step 4: Implementation

Implement the ingestion logic using the following checkpoints:

Expand All @@ -245,27 +84,37 @@ Implement the ingestion logic using the following checkpoints:
authenticating using a manager account or a data partner account), and
`linked_account` (if you're a data partner accessing the account via a
partner link to a manager account). **STRONGLY RECOMMENDED**: Refer to the
[Configure destinations and headers](https://developers.google.com/data-manager/api/devguides/concepts/destinations)
[Configure destinations and headers](https://developers.google.com/data-manager/api/devguides/concepts/destinations.md.txt)
guide for more details on configuring destinations.
- [ ] **Prepare Event Data**: Use the utility library helpers to format and
normalize user identifiers correctly.
- [ ] **Construct Payload**: Build the request payload
(`IngestEventsRequest`) containing the destinations, event records, and
consent permissions.
- [ ] **Support Validation**: Support sending the `validate_only` boolean
option on the `IngestEventsRequest` to allow developers to validate schemas
without actually uploading data.
- [ ] **Send Request**: Execute `ingest_events` and record the returned
request ID for logging/troubleshooting.
- [ ] **Retrieve Request Status**: Check the status of the ingestion request
using diagnostics. Since request processing is asynchronous, a
successful ingestion response (HTTP 200 OK returning a `request_id`) only
indicates the payload was received. To check if the records actually
succeeded, partially succeeded, or failed to process, query the
`client.retrieve_request_status` endpoint using the `request_id`. Skipping
this step is a common user mistake.

## Formatting

* Fetch the [Format user data](https://developers.google.com/data-manager/api/devguides/concepts/formatting)
* Fetch the [Format user data](https://developers.google.com/data-manager/api/devguides/concepts/formatting.md.txt)
guide and use that as the source of truth for formatting and
normalization rules.

* Use the utility library to format, hash, and encrypt user data
(emails, phone numbers, addresses).

**Python Example:**

```python
from google.ads.datamanager_util import Formatter
from google.ads.datamanager_util.format import Encoding
Expand All @@ -290,24 +139,43 @@ Implement the ingestion logic using the following checkpoints:
* Note that `consent` can be set globally on the `IngestEventsRequest` or on
individual `Event`s.
* Verify that `UserIdentifier` uses `email_address` and `phone_number`.
Do NOT use the Google Ads API fields `hashed_email` and
Do not use the Google Ads API fields `hashed_email` and
`hashed_phone_number`.
* Ensure the currency field on the event is named `currency`, not
`currency_code`.
* Do not call the diagnostics endpoint (`retrieve_request_status`) if
`validate_only` is set to `true`.

## Error Handling & Troubleshooting

### Inspecting Error Payloads

[IMPORTANT] Refer to
[Understand API Errors](https://developers.google.com/data-manager/api/devguides/concepts/understand-errors)
for a detailed guide on how to understand the structure of errors returned by
the API.
> [!IMPORTANT]
> Refer to [Understand API Errors](https://developers.google.com/data-manager/api/devguides/concepts/understand-errors.md.txt)
> for a detailed guide on how to understand the structure of errors returned by
> the API.

### Retrieving Request Status (Diagnostics)

Periodically poll for status using exponential backoff, starting at least
30 minutes after sending the `IngestEventsRequest`.

1. Call `client.retrieve_request_status` using
`RetrieveRequestStatusRequest(request_id=...)`.
2. Loop through `request_status_per_destination` in the response to inspect
each target's `request_status`.
3. If processing is complete and `request_status` is `SUCCESS`,
`PARTIAL_SUCCESS`, or `FAILED`, inspect diagnostic values:
* **Event Record Counts**: Check `events_ingestion_status.record_count`
(includes both success and failure).
* **Error Details**: If status is `FAILED` or `PARTIAL_SUCCESS`, inspect
each error's `reason` and `record_count` under
`error_info.error_counts`.
* **Warning Details**: Inspect each warning's `reason` and `record_count`
under `warning_info.warning_counts` (even if the destination status is
`SUCCESS`).

## API Reference

When implementing or debugging API integrations, use the API reference to lookup
field names, types, and acceptable values. DO NOT guess values.

* **REST API Reference:**
https://developers.google.com/data-manager/api/reference/rest/v1/events/ingest
* [REST API Reference](https://developers.google.com/data-manager/api/reference/rest/v1/events/ingest.md.txt)
* [Diagnostics Guide](https://developers.google.com/data-manager/api/devguides/diagnostics.md.txt)
54 changes: 54 additions & 0 deletions skills/ads/data-manager-api/data-manager-api-setup/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
---
name: data-manager-api-setup
description: >-
Guides developers through client library installation and authentication setup
steps for the Data Manager API. Use this skill when a user is getting started
with the Data Manager API and needs to setup their local environment, install
the client library, or setup access to the API. Don't use for implementing
audience or event ingestion logic (use the data-manager-api-audience-ingestion
or data-manager-api-event-ingestion skills instead).
metadata:
version: 1.0
category: GoogleAds
---
# Data Manager API Setup

## Setup Authentication

Refer to [Set up API access](https://developers.google.com/data-manager/api/devguides/quickstart/set-up-access.md.txt)
for more details.

1. **Enable API (Prerequisite)**: Check that the user has enabled the Data
Manager API in their Google Cloud project.
2. **Generate ADC**: Authenticate the local workspace using Application
Default Credentials (ADC) via `gcloud auth application-default login`.
* **Required Scopes**: Include scopes
`https://www.googleapis.com/auth/datamanager` and
`https://www.googleapis.com/auth/cloud-platform`.
* **Multi-API Scopes**: If using the same credentials for other APIs,
append their scopes (e.g.,
`https://www.googleapis.com/auth/adwords`).
* **Service Accounts**: Ensure the Service Account has the
`Service Usage Consumer` IAM role, and the user executing `gcloud`
has the Token Creator role
(`roles/iam.serviceAccountTokenCreator`) on that Service Account
for impersonation.

## Install Client & Utility Libraries

Refer to [Install a client library](https://developers.google.com/data-manager/api/devguides/quickstart/install-library.md.txt)
for more details.

The companion utility libraries provide pre-built helper classes
and functions to correctly format, hash, and encrypt user identifiers (such as
emails, phone numbers, and physical addresses) prior to API ingestion. Use of
these libraries is highly recommended to ensure that user identifier formatting
matches the API's specifications.

Select the language-specific installation guide below:

* [Python Setup Reference](references/python.md) (packages: `google-ads-datamanager` and `google-ads-datamanager-util`)
* [Java Setup Reference](references/java.md) (packages: `com.google.api-ads:data-manager` and `data-manager-util`)
* [Node Setup Reference](references/node.md) (packages: `@google-ads/datamanager` and `@google-ads/data-manager-util`)
* [PHP Setup Reference](references/php.md) (packages: `googleads/data-manager` and `googleads/data-manager-util`)
* [.NET Setup Reference](references/dotnet.md) (packages: `Google.Ads.DataManager.V1` and `Google.Ads.DataManager.Util.csproj`)
Loading