diff --git a/content/develop/ai/context-engine/_index.md b/content/develop/ai/context-engine/_index.md index 413ae2e1e8..a897de9401 100644 --- a/content/develop/ai/context-engine/_index.md +++ b/content/develop/ai/context-engine/_index.md @@ -65,7 +65,7 @@ You define your data model once, specifying the entities that matter (such as cu ## Data integration -AI agents are only as reliable as the data they work with. [Redis Data Integration (RDI)]({{< relref "/operate/rc/databases/rdi" >}}) keeps your Redis Cloud database in sync with your existing relational databases, including Oracle, MySQL, PostgreSQL, and SQL Server, so agents always have access to current, accurate business data without querying slow primary databases directly. +AI agents are only as reliable as the data they work with. [Redis Data Integration (RDI)]({{< relref "/operate/rc/rdi" >}}) keeps your Redis Cloud database in sync with your existing relational databases, including Oracle, MySQL, PostgreSQL, and SQL Server, so agents always have access to current, accurate business data without querying slow primary databases directly. RDI uses a data pipeline that performs an initial sync of your source data into Redis, then captures changes in real time. Updates from your primary database appear in Redis within seconds, eliminating stale data and cache misses. Your agents interact only with Redis, which provides fast and predictable query performance. diff --git a/content/develop/ai/context-engine/data-integration/_index.md b/content/develop/ai/context-engine/data-integration/_index.md index f2d6677396..a2cdaa0db4 100644 --- a/content/develop/ai/context-engine/data-integration/_index.md +++ b/content/develop/ai/context-engine/data-integration/_index.md @@ -13,7 +13,7 @@ weight: 40 Redis Data Integration (RDI) is a fully-managed data pipeline service that keeps your Redis Cloud database in sync with your existing relational databases in near real time. By streaming live data from your primary databases into Redis, RDI ensures that AI agents always have access to accurate, up-to-date business data without querying slow source databases directly. -[Get started](#get-started) with RDI on [Redis Cloud]({{< relref "/operate/rc/databases/rdi" >}}) or explore the full [Redis Data Integration documentation]({{< relref "/integrate/redis-data-integration" >}}). +[Get started](#get-started) with RDI on [Redis Cloud]({{< relref "/operate/rc/rdi" >}}) or explore the full [Redis Data Integration documentation]({{< relref "/integrate/redis-data-integration" >}}). ## Redis Data Integration overview @@ -62,11 +62,11 @@ RDI on Redis Cloud is available in preview for Redis Cloud Pro databases hosted To get started: -1. [Prepare your source database]({{< relref "/operate/rc/databases/rdi/setup" >}}) and configure credentials and connectivity. -2. [Define your data pipeline]({{< relref "/operate/rc/databases/rdi/define" >}}) by selecting which tables to sync and how to map them. -3. [View and manage your pipeline]({{< relref "/operate/rc/databases/rdi/view-edit" >}}) once it's running. +1. [Prepare your source database]({{< relref "/operate/rc/rdi/setup" >}}) and configure credentials and connectivity. +2. [Define your data pipeline]({{< relref "/operate/rc/rdi/define" >}}) by selecting which tables to sync and how to map them. +3. [View and manage your pipeline]({{< relref "/operate/rc/rdi/view-edit" >}}) once it's running. -See the [RDI Cloud quick start]({{< relref "/operate/rc/databases/rdi/quick-start" >}}) to get up and running quickly with a PostgreSQL source database. +See the [RDI Cloud quick start]({{< relref "/operate/rc/rdi/quick-start" >}}) to get up and running quickly with a PostgreSQL source database. -tab-sep- diff --git a/content/embeds/rc-context-retriever-get-started.md b/content/embeds/rc-context-retriever-get-started.md index 0edf71243f..1c574cde0c 100644 --- a/content/embeds/rc-context-retriever-get-started.md +++ b/content/embeds/rc-context-retriever-get-started.md @@ -1,4 +1,4 @@ -To set up a Redis Context Retriever on Redis Cloud, you need a database on Redis Cloud that already has relevant data. If you use a relational database, use [Redis Data Integration (RDI)]({{< relref "/operate/rc/databases/rdi" >}}) to ingest data into a Redis Cloud database. +To set up a Redis Context Retriever on Redis Cloud, you need a database on Redis Cloud that already has relevant data. If you use a relational database, use [Redis Data Integration (RDI)]({{< relref "/operate/rc/rdi" >}}) to ingest data into a Redis Cloud database. When you have a database, [Create a context retriever service]({{< relref "/operate/rc/context-engine/context-retriever/create-service" >}}) for your database on Redis Cloud. diff --git a/content/embeds/rc-rdi-create-rdi-workspace.md b/content/embeds/rc-rdi-create-rdi-workspace.md new file mode 100644 index 0000000000..26e39186ab --- /dev/null +++ b/content/embeds/rc-rdi-create-rdi-workspace.md @@ -0,0 +1,27 @@ +To create a Data Integration workspace for an existing [Pro subscription]({{< relref "/operate/rc/databases/create-database/create-pro-database-new" >}}): + +1. From the Redis Cloud console, select **Pipelines** from the left-hand menu. If you don't have any workspaces yet, select **Create workspace** to go to the **Create workspace** page. + + {{The create workspace button.}} + + If you already have a workspace deployed, you'll see your current workspaces. Select **New workspace** to go to the **Create workspace** page. + + {{The new workspace button.}} + + You can also go to the **Pipelines** tab from your subscription page and select **Create workspace** to go to the **Create workspace** page for your subscription. + + {{The create workspace button.}} + +2. Select your Pro subscription from the list if it's not already selected. + + {{The select pro subscription drop down.}} + +3. A **Data Integration subnet (CIDR)** is automatically generated for you. If, for any reason, a CIDR is not generated, enter a valid CIDR that does not conflict with your applications or other databases. + + {{The select pro subscription drop down.}} + +4. Select **Create workspace** to create your workspace. + + {{The create workspace button.}} + +Your workspace will be created in the background. You can select **Create pipeline** to [create your pipeline]({{}}) while the workspace is provisioning, or you can select **Create pipeline later** to go back to the Redis Cloud console. \ No newline at end of file diff --git a/content/embeds/rc-rdi-secrets-permissions.md b/content/embeds/rc-rdi-secrets-permissions.md index 6ca05645d3..ff70bd2699 100644 --- a/content/embeds/rc-rdi-secrets-permissions.md +++ b/content/embeds/rc-rdi-secrets-permissions.md @@ -16,4 +16,4 @@ } ``` -After you store this secret, you can view and copy the [Amazon Resource Name (ARN)](https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_iam-permissions.html#iam-resources) of your secret on the secret details page. Save the secret ARN to use when you [define your source database]({{}}). \ No newline at end of file +After you store this secret, you can view and copy the [Amazon Resource Name (ARN)](https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_iam-permissions.html#iam-resources) of your secret on the secret details page. Save the secret ARN to use when you [define your source database]({{}}). \ No newline at end of file diff --git a/content/integrate/redis-data-integration/_index.md b/content/integrate/redis-data-integration/_index.md index 27435f6da8..be74f76189 100644 --- a/content/integrate/redis-data-integration/_index.md +++ b/content/integrate/redis-data-integration/_index.md @@ -42,7 +42,7 @@ and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is necessary. {{}} -RDI is supported with Redis databases or [CRDB](https://redis.io/active-active/) (Active Active Replication) targets, and is also available on [Redis Cloud]({{< relref "/operate/rc/databases/rdi" >}}). +RDI is supported with Redis databases or [CRDB](https://redis.io/active-active/) (Active Active Replication) targets, and is also available on [Redis Cloud]({{< relref "/operate/rc/rdi" >}}). {{}} ## Features diff --git a/content/integrate/redis-data-integration/installation/_index.md b/content/integrate/redis-data-integration/installation/_index.md index 93bb54e337..c5e7bdee75 100644 --- a/content/integrate/redis-data-integration/installation/_index.md +++ b/content/integrate/redis-data-integration/installation/_index.md @@ -17,7 +17,7 @@ type: integration weight: 20 --- -The guides in this section explain the options you have for installing and upgrading RDI on your own servers. See the [Redis Cloud RDI guide]({{< relref "/operate/rc/databases/rdi" >}}) to +The guides in this section explain the options you have for installing and upgrading RDI on your own servers. See the [Redis Cloud RDI guide]({{< relref "/operate/rc/rdi" >}}) to learn how to set up RDI for a cloud database. Before you use RDI, you must also configure your source database to enable CDC. See the [Prepare source databases]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs" >}}) diff --git a/content/operate/rc/changelog/october-2025.md b/content/operate/rc/changelog/october-2025.md index ef2ae23f1f..572a8b0f66 100644 --- a/content/operate/rc/changelog/october-2025.md +++ b/content/operate/rc/changelog/october-2025.md @@ -18,11 +18,11 @@ tags: ### Data Integration -Redis Cloud now supports [Redis Data Integration (RDI)]({{< relref "/operate/rc/databases/rdi" >}}) to create data pipelines that ingest data from a supported primary database to Redis. +Redis Cloud now supports [Redis Data Integration (RDI)]({{< relref "/operate/rc/rdi" >}}) to create data pipelines that ingest data from a supported primary database to Redis. Using a data pipeline lets you have a cache that is always ready for queries. RDI Data pipelines ensure that any changes made to your primary database are captured in your Redis cache within a few seconds, preventing cache misses and stale data within the cache. -See [Data Integration]({{< relref "/operate/rc/databases/rdi" >}}) to learn how to set up data pipelines with Redis Cloud. +See [Data Integration]({{< relref "/operate/rc/rdi" >}}) to learn how to set up data pipelines with Redis Cloud. ## Security fixes diff --git a/content/operate/rc/context-engine/context-retriever/create-service.md b/content/operate/rc/context-engine/context-retriever/create-service.md index b41dfe2c0c..c4e7e810c9 100644 --- a/content/operate/rc/context-engine/context-retriever/create-service.md +++ b/content/operate/rc/context-engine/context-retriever/create-service.md @@ -15,7 +15,7 @@ Redis Context Retriever helps teams expose operational context to AI agents thro ## Prerequisites and limitations -To create a Redis Context Retriever service, you will need a Redis Cloud database that already has relevant data. If you don't have one, see [Create a database]({{< relref "/operate/rc/databases/create-database" >}}). If your source data lives in a relational database, use [Redis Data Integration (RDI)]({{< relref "/operate/rc/databases/rdi" >}}) to ingest it into a Redis Cloud database first. +To create a Redis Context Retriever service, you will need a Redis Cloud database that already has relevant data. If you don't have one, see [Create a database]({{< relref "/operate/rc/databases/create-database" >}}). If your source data lives in a relational database, use [Redis Data Integration (RDI)]({{< relref "/operate/rc/rdi" >}}) to ingest it into a Redis Cloud database first. {{< note >}} Agent Memory does not support the following databases during public preview: diff --git a/content/operate/rc/databases/rdi/define.md b/content/operate/rc/databases/rdi/define.md deleted file mode 100644 index 36a20fb0e7..0000000000 --- a/content/operate/rc/databases/rdi/define.md +++ /dev/null @@ -1,122 +0,0 @@ ---- -Title: Define data pipeline -alwaysopen: false -categories: -- docs -- operate -- rc -description: Define the source connection and data pipeline. -hideListLinks: true -weight: 3 ---- - -After you have [prepared your source database]({{}}) and connection information, you can set up your new pipeline. To do this: - -1. [Define the source connection](#define-source-connection) by entering all required source database information. -2. [Define the data pipeline](#define-data-pipeline) by selecting the data that you want to sync from your source database to the target database. - -## Define source connection - -1. In the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Data Pipeline** tab. -1. Select **Define source database**. - {{The define source database button.}} -1. Enter a **Pipeline name**. - {{The pipeline name and deployment CIDR fields.}} -1. A **Deployment CIDR** is automatically generated for you. If, for any reason, a CIDR is not generated, enter a valid CIDR that does not conflict with your applications or other databases. -1. In the **Source database connectivity** section, enter the **PrivateLink service name** of the [PrivateLink connected to your source database]({{< relref "/operate/rc/databases/rdi/setup#set-up-connectivity" >}}). - {{The Source database connectivity section, with database connection details and connectivity options.}} -1. Enter your database details. This depends on your database type, and includes: - - **Port**: The database's port - - **Database**: Your database's name, or the root database *(PostgreSQL, Oracle only)*, or a comma-separated list of one or more databases you want to connect to *(SQL Server only)* - - **Database Server ID**: Unique ID for the replication client. Enter a number that is not used by any existing replication clients *(mySQL and mariaDB only)* - - **PDB**: Name of the Oracle pluggable database *(Oracle only)* -1. Enter the ARN of your [database credentials secret]({{< relref "/operate/rc/databases/rdi/setup#create-database-credentials-secrets" >}}) in the **Source database secrets ARN** field. -1. If your database requires TLS, select **Use TLS**. Enter the ARN of your [CA certificate secret]({{< relref "/operate/rc/databases/rdi/setup#create-database-credentials-secrets" >}}) in the **CA Cert Secret ARN** field. - {{The Source database connectivity section, with Use TLS selected and the CA Cert Secret ARN field.}} -1. If your database requires mTLS, select **Use mTLS**. Enter the ARN of your [Client certificate secret]({{< relref "/operate/rc/databases/rdi/setup#create-database-credentials-secrets" >}}) in the **Client Certificate Secret ARN** field and the ARN of your [Client key secret]({{< relref "/operate/rc/databases/rdi/setup#create-database-credentials-secrets" >}}) in the **Client Key Secret ARN** field. - {{The Source database connectivity section, with Use TLS selected and the Client Certificate Secret ARN and Client Key Secret ARN fields.}} -1. If your database requires mTLS with a client key passphrase, enter the ARN of your [Client key passphrase secret]({{< relref "/operate/rc/databases/rdi/setup#create-database-credentials-secrets" >}}) in the **Please add a secret ARN for the password to use with the secret store** field. -1. Select **Advanced properties** to configure additional optional properties for your pipeline. - {{The advanced properties section.}} - You can add any [Debezium source property](https://debezium.io/documentation/reference/stable/connectors/) for your source database type in the **Collector source properties** section and any [Redis server Debezium sink property](https://debezium.io/documentation/reference/stable/operations/debezium-server.html#_redis_stream) in the **Collector sink properties** section. -1. Select **Start pipeline setup**. - {{The start pipeline setup button.}} -1. Redis Cloud will attempt to connect to PrivateLink. If your PrivateLink does not allow automatic acceptance of incoming connections, accept the incoming connection on AWS PrivateLink to proceed. See [Accept or Reject PrivateLink connection requests](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#accept-reject-connection-requests). - - If Redis Cloud can't find your PrivateLink connection, make sure that the PrivateLink service name is correct and that Redis Cloud is listed as an Allowed Principal for your VPC. See [Set up connectivity]({{}}) for more info. - -At this point, Redis Cloud will provision the pipeline infrastructure that will allow you to define your data pipeline. - -{{The Pipeline setup in progress screen.}} - -Pipelines are provisioned in the background. You aren't allowed to make changes to your data pipeline or to your database during provisioning. This process will take about an hour, so you can close the window and come back later. - -When your pipeline is provisioned, select **Complete setup**. You will then [define your data pipeline](#define-data-pipeline). - -{{The complete setup button.}} - -## Define data pipeline - -After your pipeline is provisioned, you will be able to define your pipeline. You will select the database schemas, tables, and columns that you want to import and synchronize with your primary database. - -### Configure a new pipeline - -1. In the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Data Pipeline** tab. If your pipeline is already provisioned, select **Complete setup** to go to the **Data modeling** section. - {{The complete setup button.}} -1. Select the Schema and Tables you want to migrate to the target database from the list. - {{The data modeling section. }} - - Select **Manage Columns** to choose which columns you want to import. - - {{The manage columns button.}} - - You can select any number of columns from a table. - - {{The manage columns screen, with a few columns selected from one table}} - - If any tables are missing a unique constraint, a warning will appear in the **Data modeling** section. Select **Manage columns** to select the columns that define a unique constraint for those tables. - - Select **Save** to save your column changes and go back to schema selection. - - {{The save button.}} - - Select **Add schema** to add more database schemas. - - {{The add schema button.}} - - Select **Delete** to delete a schema. You must have at least one schema to continue. - - {{The delete schema button.}} - - After you've selected the schemas and tables you want to sync, select **Continue**. - - {{The continue button.}} - -1. Select the Redis data type to write keys to the target. You can choose **Hash key** or **JSON key** if the target database supports JSON. - {{The data modeling section, with Hash Key selected.}} - - You can also supply one or more [transformation job files]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}) that specify how you want to transform the captured data before writing it to the target. Select **Upload jobs** to upload your job files. - - {{The transformation jobs section. Select Upload jobs to upload transformation jobs.}} - - When you upload job files, Redis Cloud will validate the job files to check for errors. - - Select **Continue**. - {{The continue button.}} - -1. In the **Advanced properties** section, you can add any processor properties to control how the data is processed. See the [RDI configuration file reference]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors" >}}) for all available processor properties. - - {{The advanced properties section with processor properties.}} - - Select **Continue**. - {{The continue button.}} - -1. Review the tables you selected in the **Review and deploy** section. If everything looks correct, select **Confirm & Deploy** to start ingesting data from your source database. - - {{The Confirm & Deploy button.}} - -At this point, the data pipeline will ingest data from the source database to your target Redis database. This process will take time, especially if you have a lot of records in your source database. - -After this initial sync is complete, the data pipeline enters the *change streaming* phase, where changes are captured as they happen. Changes in the source database are added to the target within a few seconds of capture. - -You can view the status of your data pipeline in the **Data pipeline** tab of your database. See [View and edit data pipeline]({{}}) to learn more. \ No newline at end of file diff --git a/content/operate/rc/databases/rdi/quick-start.md b/content/operate/rc/databases/rdi/quick-start.md deleted file mode 100644 index 373e2a6d56..0000000000 --- a/content/operate/rc/databases/rdi/quick-start.md +++ /dev/null @@ -1,144 +0,0 @@ ---- -Title: RDI on Redis Cloud quick start -linkTitle: Quick start -alwaysopen: false -categories: -- docs -- operate -- rc -description: Learn how to create a data pipeline between a PostgreSQL source database created with Terraform and a Redis Cloud target database. -hideListLinks: true -weight: 1 ---- - -The [`rdi-cloud-automation` GitHub repository](https://github.com/redis/rdi-cloud-automation) contains a Terraform script that quickly sets up a PostgreSQL source database on an EC2 instance and all required permissions and network setup to connect it to a Redis Cloud target database. - -{{< note >}} -This guide is for demonstration purposes only. It is not recommended for production use. -{{< /note >}} - -## Prerequisites - -To follow this guide, you need to: - -1. Create a [Redis Cloud Pro database]({{< relref "/operate/rc/databases/create-database/create-pro-database-new" >}}) hosted on Amazon Web Services (AWS). - - Turn on Multi-AZ replication and [manually select the availability zones]({{< relref "/operate/rc/databases/configuration/high-availability#availability-zones" >}}) when creating the database. - -1. Install the [AWS CLI](https://aws.amazon.com/cli/) and set up [credentials for the CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html). - -1. Install [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli). - -## Create a data pipeline - -1. On the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Data Pipeline** tab. -1. Select **Create pipeline**. - {{The create pipeline button.}} -1. Select **PostgreSQL** as the source database type. - {{The select source database type list.}} -1. Under **Source database credentials and certificates**, save the provided ARN. This will be the `redis_secrets_arn` you will need later. - - {{The setup connectivity section containing the credentials ARN.}} - -1. Under **Setup connectivity**, save the provided ARN. This will be the `redis_privatelink_arn` you will need later. - - {{The setup connectivity section containing the private link ARN.}} - -## Create the source database and network resources - -1. Clone or download the [`rdi-cloud-automation` GitHub repository](https://github.com/redis/rdi-cloud-automation). - -1. In a terminal window, go to the `examples/aws-ec2-privatelink` directory. - -1. Run `terraform init` to initialize the Terraform working directory. - -1. Open the `example.tfvars` file and edit the following variables: - - - `region`: The AWS region where your Redis Cloud database is deployed. - - `azs`: The availability zone IDs where your Redis Cloud database is deployed. - - `port`: The port number for the new PostgreSQL source database. - - `name`: A prefix for all of the created AWS resources. - - `redis_secrets_arn`: The source database credentials and certificates ARN from the Redis Cloud console. - - `redis_privatelink_arn`: The PrivateLink ARN from the Redis Cloud console. - -1. To view the configuration, run: - - ```sh - terraform plan -var-file=example.tfvars - ``` - -1. To create the AWS resources, run: - - ```sh - terraform apply -var-file=example.tfvars - ``` - - This example creates the following resources on your AWS account: - - - An AWS KMS key with the required permissions for RDI - - A VPC with a public and private subnet and all necessary route tables - - An EC2 instance running a PostgreSQL database with a security group that allows access from Redis Cloud - - An AWS Secrets Manager secret for the PostgreSQL database credentials - - A Network Load Balancer (NLB), a listener, and target group to route traffic to the EC2 instance with AWS PrivateLink - - An AWS PrivateLink endpoint service for the PostgreSQL database - -Creating the AWS resources will take some time. After the resources are created, you'll be able to view them in the AWS management console. - -Save the following outputs: - -- `database`: The name of the PostgreSQL database. -- `port`: The port number for the PostgreSQL database. -- `secret_arn`: The ARN of the AWS Secrets Manager secret for the PostgreSQL database credentials. -- `vpc_endpoint_service_name`: The name of the AWS PrivateLink endpoint service for the PostgreSQL database. - -If you lose any outputs, run `terraform output` to view them again. - -## Define source connection - -1. Return to the [Redis Cloud console](https://cloud.redis.io/). Go to your target database and select the **Data Pipeline** tab. -1. Select **Define source database**. - {{The define source database button.}} -1. Enter a **Pipeline name**. - {{The pipeline name and deployment CIDR fields.}} -1. A **Deployment CIDR** is automatically generated for you. If, for any reason, a CIDR is not generated, enter a valid CIDR that does not conflict with your applications or other databases. -1. Enter the terraform outputs in the following fields: - - **PrivateLink service name**: `vpc_endpoint_service_name` - - **Database**: `database` - - **Port**: `port` - - **Source database secrets ARN**: `secret_arn` -1. Select **Start pipeline setup**. - {{The start pipeline setup button.}} - -At this point, Redis Cloud will provision the pipeline infrastructure that will allow you to define your data pipeline. - -{{The Pipeline setup in progress screen.}} - -Pipelines are provisioned in the background. You aren't allowed to make changes to your data pipeline or to your database during provisioning. This process will take about an hour, so you can close the window and come back later. - -When your pipeline is provisioned, select **Complete setup**. - -{{The complete setup button.}} - -## Define data pipeline - -After your pipeline is provisioned, you will be able to define your pipeline. You will select the database schemas, tables, and columns that you want to import and synchronize with your primary database. - -See [Define data pipeline]({{}}) for detailed steps on defining your data pipeline. - -After you define your data pipeline, it will ingest data from the source database to your target Redis database. This process will take time, especially if you have a lot of records in your source database. - -After this initial sync is complete, the data pipeline enters the *change streaming* phase, where changes are captured as they happen. Changes in the source database are added to the target within a few seconds of capture. You can see this by connecting to your source database and making changes to the data, and then connecting to your target Redis database and verifying that the changes are reflected there. - -You can view the status of your data pipeline in the **Data pipeline** tab of your database. See [View and edit data pipeline]({{}}) to learn more. - -## Delete sample resources - -{{< warning >}} -Make sure to [delete your data pipeline]({{}}) before deleting the sample resources. -{{< /warning >}} - -To delete the sample resources created by Terraform, run: - -```sh -terraform destroy -var-file=example.tfvars -``` diff --git a/content/operate/rc/databases/rdi/_index.md b/content/operate/rc/rdi/_index.md similarity index 91% rename from content/operate/rc/databases/rdi/_index.md rename to content/operate/rc/rdi/_index.md index b83a9f1b17..6042ec0003 100644 --- a/content/operate/rc/databases/rdi/_index.md +++ b/content/operate/rc/rdi/_index.md @@ -1,5 +1,8 @@ --- Title: Data Integration +aliases: + - /operate/rc/databases/rdi/ + - /operate/rc/databases/rdi alwaysopen: false categories: - docs @@ -7,7 +10,7 @@ categories: - rc description: Use Redis Data Integration with Redis Cloud. hideListLinks: true -weight: 99 +weight: 38 tocEmbedHeaders: true --- @@ -94,9 +97,9 @@ For more info on how RDI works, see [RDI Architecture]({{}}) to learn how to share your source database credentials and TLS certificates with Redis Cloud. +Data pipelines are set up to ensure a high level of data security. Source database credentials and TLS secrets are stored in AWS secret manager and shared using the Kubernetes CSI driver for secrets. See [Share source database credentials]({{}}) to learn how to share your source database credentials and TLS certificates with Redis Cloud. -Connections to the source database use Java Database Connectivity (JDBC) through [AWS PrivateLink](https://aws.amazon.com/privatelink/), ensuring that the data pipeline is only exposed to the specific database endpoint. See [Set up connectivity]({{}}) to learn how to connect your PrivateLink to the Redis Cloud VPC. +Connections to the source database use Java Database Connectivity (JDBC) through [AWS PrivateLink](https://aws.amazon.com/privatelink/), ensuring that the data pipeline is only exposed to the specific database endpoint. See [Set up connectivity]({{}}) to learn how to connect your PrivateLink to the Redis Cloud VPC. RDI encrypts all network connections with TLS. The pipeline will process data from the source database in-memory and write it to the target database using a TLS connection. There are no external connections to your data pipeline except from Redis Cloud management services. @@ -133,11 +136,12 @@ Please be aware of the following limitations: ## Get started -To get started fast with RDI on Redis Cloud, see the [RDI Cloud quick start]({{}}) to create a data pipeline between a PostgreSQL source database and a Redis Cloud target database. +To get started fast with RDI on Redis Cloud, see the [RDI Cloud quick start]({{}}) to create a data pipeline between a PostgreSQL source database and a Redis Cloud target database. To create a new data pipeline, you need to: -1. [Prepare your source database]({{}}) and any associated credentials. -2. [Define the source connection and data pipeline]({{}}) by selecting which tables to sync. +1. [Create a Data Integration workspace]({{}}) for your Pro subscription. +1. [Prepare your source database]({{}}) and any associated credentials. +1. [Define the source connection and data pipeline]({{}}) by selecting which tables to sync. -Once your data pipeline is defined, you can [view and edit]({{}}) it. +Once your data pipeline is defined, you can [view and edit]({{}}) it. diff --git a/content/operate/rc/rdi/create-workspace.md b/content/operate/rc/rdi/create-workspace.md new file mode 100644 index 0000000000..5abc0d40f3 --- /dev/null +++ b/content/operate/rc/rdi/create-workspace.md @@ -0,0 +1,28 @@ +--- +Title: Create and manage Data Integration workspace +linkTitle: Create workspace +aliases: + - /operate/rc/databases/rdi/create-workspace + - /operate/rc/databases/rdi/create-workspace/ +alwaysopen: false +categories: +- docs +- operate +- rc +description: Create and manage the infrastructure for your Data Integration pipelines. +hideListLinks: true +weight: 2 +tocEmbedHeaders: true +--- + +Before you can create your first Data Integration pipeline for a Redis Cloud subscription, you must first deploy the cloud infrastructure needed to host the pipeline and run the workers associated with the pipeline. In Redis Cloud, this is called a **Workspace**. Each Pro subscription can have one Data integration workspace. You only need to set up the workspace once - any pipelines you create for your subscription will run on the workspace until you delete it. You won't be charged for a workspace until you start running your Data Integration pipeline. + +## Create a Data Integration workspace + +{{< embed-md "rc-rdi-create-rdi-workspace.md" >}} + +## View workspace status and details + + + +## Delete workspace \ No newline at end of file diff --git a/content/operate/rc/rdi/define.md b/content/operate/rc/rdi/define.md new file mode 100644 index 0000000000..d393aa2269 --- /dev/null +++ b/content/operate/rc/rdi/define.md @@ -0,0 +1,177 @@ +--- +Title: Create data pipeline +aliases: + - /operate/rc/databases/rdi/define/ + - /operate/rc/databases/rdi/define +alwaysopen: false +categories: +- docs +- operate +- rc +description: Define the source connection and data pipeline. +hideListLinks: true +weight: 4 +--- + +After you have [prepared your source database]({{}}) and connection information, and [created the workspace for your pipeline]({{}}), you can set up your new pipeline. + +In the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Pipelines** tab. You'll see your subscription's workspace. Select **Add pipeline** to add a pipeline to the workspace. + +You can also go to the **Pipelines** page from the left-hand menu and select **Add pipeline** from your workspace. Or, you can go to the **Pipelines** tab of your subscription and select **Add pipeline** from your workspace. + +{{The workspace section of the Pipelines tab for a database. Select Add pipeline to add a pipeline.}} + +If you've started to create a pipeline, you'll see a draft pipeline. Select **More actions > Resume pipeline setup** to continue with pipeline setup. + +{{The workspace section of the Pipelines tab for a database with a draft pipeline. Select Resume pipeline setup to continue.}} + +Creating a pipeline is split into the following steps: + +1. [**Pipeline setup**](#pipeline-setup): Defines the source database type and the target database. +2. [**Source configuration**](#source-configuration): Defines and tests the source database connectivity and credentials. +3. [**Dataset**](#dataset): Defines the data to import to your Redis database. +4. [**Transformations**](#transformations): Defines how records are stored in Redis. +5. [**Review & deploy**](#review-and-deploy): Shows your pipeline details and deploys it. + +## Pipeline setup + +In the **Pipeline setup** step: + +1. Select your source database type. The following database types are supported: + - MySQL + - mariaDB + - Oracle + - SQL Server + - PostgreSQL + {{The select source database type list and source name field.}} + +1. Enter a name for your source database in the **Source name** field. This is a name for the source database that will appear on Redis Cloud. +1. Select the target Redis Cloud database from the **Target database** list. + + {{The target database list.}} + +Select **Continue to source** to move to the **Source configuration** step. + +{{The Continue to source button.}} + +## Source configuration + +During the **Source configuration** step, you'll share the connectivity information and credentials you created when you prepared your source database. + +This step is separated into three expandable sections: + +1. [Source connectivity](#source-connectivity) defines the connectivity method for your source database. +2. [Secrets](#secrets) defines the authentication needed for your source database. +3. [Source configuration](#source-configuration-section) defines the source-specific settings for your database connection. + +When you've completed all three sections, select **Test source** to test Redis Cloud's connection with the source database. After the test completes, select **Continue to dataset** to move to the [**Dataset**](#dataset) step. + +{{The Continue to dataset button.}} + +### Source connectivity + +Select whether you want your pipeline to connect to your database using **AWS Private Link** or using the **Public endpoint**. + +- If your pipeline uses AWS PrivateLink, enter the **Private Link service name** of the [PrivateLink connected to your source database]({{< relref "/operate/rc/rdi/setup#set-up-connectivity" >}}). + + {{The Source database connectivity section for PrivateLink connection.}} + + Select **Connect to Private Link** to test your Private Link connectivity. This will take a few minutes. + + If Redis Cloud can't find your PrivateLink connection, make sure that the PrivateLink service name is correct and that Redis Cloud is listed as an Allowed Principal for your VPC. See [Set up connectivity]({{}}) for more info. + +- If your pipeline uses the public endpoint, enter the source IP address or hostname in the **Source IP address / Hostname** field. + + {{The Source database connectivity section for Public endpoint connection.}} + +### Secrets + +{{The Secrets section.}} + +Enter the ARN of your [database credentials secret]({{< relref "/operate/rc/rdi/setup#create-database-credentials-secrets" >}}) in the **Credentials secret ARN** field. + +Under **Transit security**: + +- If your database requires TLS, select **TLS**. Enter the ARN of your [CA certificate secret]({{< relref "/operate/rc/rdi/setup#create-database-credentials-secrets" >}}) in the **CA Certificate Secret ARN** field. + {{The Source database connectivity section, with TLS selected and the CA Cert Secret ARN field.}} +- If your database requires mTLS, select **mTLS**. + {{The Source database connectivity section, with mTLS selected and the Client Certificate Secret ARN and Client Key Secret ARN fields.}} + + Enter the following secrets in the fields: + - **CA Certificate Secret ARN**: [CA certificate secret]({{< relref "/operate/rc/rdi/setup#create-database-credentials-secrets" >}}) + - **Client Certificate Secret ARN**: [Client certificate secret]({{< relref "/operate/rc/rdi/setup#create-database-credentials-secrets" >}}) + - **Client Private Key Secret ARN**: [Client key secret]({{< relref "/operate/rc/rdi/setup#create-database-credentials-secrets" >}}) + - **Password secret ARN for secret store** (_Optional_): [Client key passphrase secret]({{< relref "/operate/rc/rdi/setup#create-database-credentials-secrets" >}}) + +Select **Validate** to check that Redis Cloud can access your secrets. + +### Source configuration {#source-configuration-section} + +In this section, you'll enter your database details. This depends on your database type, and includes: + +- **Port**: The database's port +- **Database**: Your database's name, or the root database *(PostgreSQL, Oracle only)*, or a comma-separated list of one or more databases you want to connect to *(SQL Server only)* +- **Database Server ID**: Unique ID for the replication client. Enter a number that is not used by any existing replication clients *(mySQL and mariaDB only)* +- **PDB**: Name of the Oracle pluggable database *(Oracle only)* + +Under **Collector properties**, Select **Edit advanced properties** to configure additional optional properties for your pipeline. + +{{The collector properties section.}} + +{{The advanced properties section.}} + +You can add any [Debezium source property](https://debezium.io/documentation/reference/stable/connectors/) for your source database type in the **Collector source properties** section and any [Redis server Debezium sink property](https://debezium.io/documentation/reference/stable/operations/debezium-server.html#_redis_stream) in the **Collector sink properties** section. Select **Save properties** to return to Source configuration. + +## Dataset + +In this step, you will select the database schemas, tables, and columns that you want to import and synchronize with your primary database. + +{{The dataset step.}} + +1. In the **Schemas** section, select the schema(s) you want to migrate to the target database from the list. + + {{The dataset step with a schema selected.}} + +1. When you select a schema, you will see its tables in the **Tables** section. Redis Cloud will automatically select all tables for import. You can de-select any columns you do not wish to import to your Redis database. + +1. Select a table to view its columns in the **Columns** section. You can de-select any columns you do not wish to import. + + {{The columns section, with a few columns selected from one table}} + + If any tables are missing a unique key, a warning will appear in the **Data modeling** section. Select **Show affected** to filter the **Tables** section to the tables without a unique key. + + {{The dataset step filtered to show tables that are missing a unique key.}} + + For these tables, select the key icon next to the column that defines a unique key. + + {{The unique key icon.}} + +Select **Continue to transformations** to move to the **Transformations** step. + +{{The Continue to dataset button.}} + +## Transformations + +{{The Transformations step.}} + +1. Select how your records will be stored in Redis. You can choose **Hash** or **JSON**. + +1. Under **Transformation jobs**, you can supply one or more [transformation job files]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}) that specify how you want to transform the captured data before writing it to the target. Select **Upload jobs** to upload your job files. When you upload job files, Redis Cloud will validate the job files to check for errors. + +1. Select **Edit advanced properties** to add any processor properties to control how the data is processed. + + {{The Advanced Processor properties.}} + + See the [RDI configuration file reference]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors" >}}) for all available processor properties. + +## Review and deploy + +Review the tables you selected in the **Review and deploy** step. If everything looks correct, select **Deploy pipeline** to start ingesting data from your source database. + +{{The Deploy pipeline button.}} + +At this point, the data pipeline will ingest data from the source database to your target Redis database. This process will take time, especially if you have a lot of records in your source database. + +After this initial sync is complete, the data pipeline enters the *change streaming* phase, where changes are captured as they happen. Changes in the source database are added to the target within a few seconds of capture. + +You can view the status of your data pipeline in the **Data pipeline** tab of your database. See [View and edit data pipeline]({{}}) to learn more. \ No newline at end of file diff --git a/content/operate/rc/rdi/quick-start.md b/content/operate/rc/rdi/quick-start.md new file mode 100644 index 0000000000..72d6f91a77 --- /dev/null +++ b/content/operate/rc/rdi/quick-start.md @@ -0,0 +1,176 @@ +--- +Title: RDI on Redis Cloud quick start +linkTitle: Quick start +aliases: + - /operate/rc/databases/rdi/quick-start/ + - /operate/rc/databases/rdi/quick-start +alwaysopen: false +categories: +- docs +- operate +- rc +description: Learn how to create a data pipeline between a PostgreSQL source database created with Terraform and a Redis Cloud target database. +hideListLinks: true +weight: 1 +--- + +The [`rdi-cloud-automation` GitHub repository](https://github.com/redis/rdi-cloud-automation) contains a Terraform script that quickly sets up a PostgreSQL source database on an EC2 instance and all required permissions and network setup to connect it to a Redis Cloud target database. + +{{< note >}} +This guide is for demonstration purposes only. It is not recommended for production use. +{{< /note >}} + +## Prerequisites + +To follow this guide, you need to: + +1. Create a [Redis Cloud Pro database]({{< relref "/operate/rc/databases/create-database/create-pro-database-new" >}}) hosted on Amazon Web Services (AWS). + + Turn on Multi-AZ replication and [manually select the availability zones]({{< relref "/operate/rc/databases/configuration/high-availability#availability-zones" >}}) when creating the database. + +1. Install the [AWS CLI](https://aws.amazon.com/cli/) and set up [credentials for the CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html). + +1. Install [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli). + +## Create a data integration workspace + +Before you can create your first Data Integration pipeline for a Redis Cloud subscription, you must first deploy the cloud infrastructure needed to host the pipeline and run the workers associated with the pipeline. In Redis Cloud, this is called a **Workspace**. See [Create and manage Data Integration workspace]({{}}) for more information. + +{{< embed-md "rc-rdi-create-rdi-workspace.md" >}} + +## Get required ARNs + +1. On the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Pipelines** tab. +1. Select **Add pipeline**. + {{The workspace section of the Pipelines tab for a database. Select Add pipeline to add a pipeline.}} +1. Select **PostgreSQL** as the source database type. + {{The select source database type list.}} +1. Enter a name for your source database in the **Source name** field. This is a name for the source database that will appear on Redis Cloud. +1. Select **Continue to source** to move to the **Source configuration** step. + + {{The select source database type list.}} +1. Under **Source connectivity**, save the provided ARN. This will be the `redis_privatelink_arn` you will need later. + + {{The setup connectivity section containing the private link ARN.}} + +1. Under **Secrets**, save the provided ARN. This will be the `redis_secrets_arn` you will need later. + + {{The setup connectivity section containing the credentials ARN.}} + +## Create the source database and network resources + +1. Clone or download the [`rdi-cloud-automation` GitHub repository](https://github.com/redis/rdi-cloud-automation). + +1. In a terminal window, go to the `examples/aws-ec2-privatelink` directory. + +1. Run `terraform init` to initialize the Terraform working directory. + +1. Open the `example.tfvars` file and edit the following variables: + + - `region`: The AWS region where your Redis Cloud database is deployed. + - `azs`: The availability zone IDs where your Redis Cloud database is deployed. + - `port`: The port number for the new PostgreSQL source database. + - `name`: A prefix for all of the created AWS resources. + - `redis_secrets_arn`: The source database credentials and certificates ARN from the Redis Cloud console. + - `redis_privatelink_arn`: The PrivateLink ARN from the Redis Cloud console. + +1. To view the configuration, run: + + ```sh + terraform plan -var-file=example.tfvars + ``` + +1. To create the AWS resources, run: + + ```sh + terraform apply -var-file=example.tfvars + ``` + + This example creates the following resources on your AWS account: + + - An AWS KMS key with the required permissions for RDI + - A VPC with a public and private subnet and all necessary route tables + - An EC2 instance running a PostgreSQL database with a security group that allows access from Redis Cloud + - An AWS Secrets Manager secret for the PostgreSQL database credentials + - A Network Load Balancer (NLB), a listener, and target group to route traffic to the EC2 instance with AWS PrivateLink + - An AWS PrivateLink endpoint service for the PostgreSQL database + +Creating the AWS resources will take some time. After the resources are created, you'll be able to view them in the AWS management console. + +Save the following outputs: + +- `database`: The name of the PostgreSQL database. +- `port`: The port number for the PostgreSQL database. +- `secret_arn`: The ARN of the AWS Secrets Manager secret for the PostgreSQL database credentials. +- `vpc_endpoint_service_name`: The name of the AWS PrivateLink endpoint service for the PostgreSQL database. + +If you lose any outputs, run `terraform output` to view them again. + +## Resume pipeline setup + +1. Return to the [Redis Cloud console](https://cloud.redis.io/). Go to your target database and select the **Pipelines** tab. +1. You'll see a draft pipeline in the workspace you created. Select **More actions > Resume pipeline setup** to continue with pipeline setup. + + {{The workspace section of the Pipelines tab for a database with a draft pipeline. Select Resume pipeline setup to continue.}} + +1. Continue to the **Source configuration** step. + +1. In the **Source connectivity** section, enter the `vpc_endpoint_service_name` output in the **PrivateLink service name** field. + + {{The Source database connectivity section for PrivateLink connection.}} + +1. Select **Connect to Private Link** to test your Private Link connectivity. This will take a few minutes, but you can continue while it's testing. + +1. In the **Secrets** section, enter the `secret_arn` output in the **Credentials secret ARN** field. + + {{The Secrets section.}} + +1. Select **Validate** to check that Redis Cloud can access your secrets. + +1. In the **Source configuration** section, enter the terraform outputs in the following fields. + - **Database**: `database` + - **Port**: `port` + +1. Select **Test source** to test Redis Cloud's connection with the source database. After the test completes, select **Continue to dataset**. + + {{The Continue to dataset button.}} + +1. In the **Schemas** section, select the schema(s) you want to migrate to the target database from the list. + + {{The dataset step with a schema selected.}} + +1. When you select a schema, you will see its tables in the **Tables** section. Redis Cloud will automatically select all tables for import. You can de-select any columns you do not wish to import to your Redis database. + +1. Select a table to view its columns in the **Columns** section. You can de-select any columns you do not wish to import. + + {{The columns section, with a few columns selected from one table}} + +1. Select **Continue to transformations** to move to the **Transformations** step. + + {{The Continue to dataset button.}} + +1. Select how your records will be stored in Redis. You can choose **Hash** or **JSON**. + + {{The Transformations step.}} + +1. Review the tables you selected in the **Review and deploy** step. If everything looks correct, select **Deploy pipeline** to start ingesting data from your source database. + + {{The Deploy pipeline button.}} + +At this point, the data pipeline will ingest data from the source database to your target Redis database. This process will take time, especially if you have a lot of records in your source database. + +After this initial sync is complete, the data pipeline enters the *change streaming* phase, where changes are captured as they happen. Changes in the source database are added to the target within a few seconds of capture. + +You can view the status of your data pipeline in the **Data pipeline** tab of your database. See [View and edit data pipeline]({{}}) to learn more. + +## Delete sample resources + +{{< warning >}} +Make sure to [delete your data pipeline]({{}}) before deleting the sample resources. +{{< /warning >}} + +To delete the sample resources created by Terraform, run: + +```sh +terraform destroy -var-file=example.tfvars +``` diff --git a/content/operate/rc/databases/rdi/rds-proxy.md b/content/operate/rc/rdi/rds-proxy.md similarity index 84% rename from content/operate/rc/databases/rdi/rds-proxy.md rename to content/operate/rc/rdi/rds-proxy.md index 0f286d5138..64ef8ca50b 100644 --- a/content/operate/rc/databases/rdi/rds-proxy.md +++ b/content/operate/rc/rdi/rds-proxy.md @@ -1,5 +1,8 @@ --- Title: RDS Proxy setup for RDI +aliases: + - /operate/rc/databases/rdi/rds-proxy/ + - /operate/rc/databases/rdi/rds-proxy alwaysopen: false categories: - docs @@ -12,7 +15,7 @@ weight: 99 --- {{}} -We do not recommend using RDS Proxy for RDI connections. The [Lambda function approach]({{< relref "/operate/rc/databases/rdi/setup#setup-lambda-function" >}}) provides better failover handling and is the recommended solution for production environments. +We do not recommend using RDS Proxy for RDI connections. The [Lambda function approach]({{< relref "/operate/rc/rdi/setup#setup-lambda-function" >}}) provides better failover handling and is the recommended solution for production environments. Additionally, RDS Proxy does not work with RDS PostgreSQL and Aurora PostgreSQL because it does not support PostgreSQL logical replication. @@ -91,20 +94,20 @@ Replace `` with the endpoint of your RDS Proxy. Save this IP add ## Configure the Network Load Balancer -When you [create the Network Load Balancer]({{< relref "/operate/rc/databases/rdi/setup#create-network-load-balancer-rds" >}}), use the RDS Proxy IP address instead of the database IP address: +When you [create the Network Load Balancer]({{< relref "/operate/rc/rdi/setup#create-network-load-balancer-rds" >}}), use the RDS Proxy IP address instead of the database IP address: 1. In **Register targets**, enter the static IP address of your RDS Proxy (obtained in the previous step). 2. Enter the port number where your RDS Proxy is exposed. 3. Select **Include as pending below**. -4. Complete the remaining Network Load Balancer setup as described in the [main setup guide]({{< relref "/operate/rc/databases/rdi/setup#create-network-load-balancer-rds" >}}). +4. Complete the remaining Network Load Balancer setup as described in the [main setup guide]({{< relref "/operate/rc/rdi/setup#create-network-load-balancer-rds" >}}). ## Next steps After setting up RDS Proxy and the Network Load Balancer: -1. [Create an endpoint service]({{< relref "/operate/rc/databases/rdi/setup#create-endpoint-service-rds" >}}) through AWS PrivateLink. -2. [Share your source database credentials]({{< relref "/operate/rc/databases/rdi/setup#share-source-database-credentials" >}}) with Redis Cloud. -3. Continue with the [RDI pipeline configuration]({{< relref "/operate/rc/databases/rdi/define" >}}). +1. [Create an endpoint service]({{< relref "/operate/rc/rdi/setup#create-endpoint-service-rds" >}}) through AWS PrivateLink. +2. [Share your source database credentials]({{< relref "/operate/rc/rdi/setup#share-source-database-credentials" >}}) with Redis Cloud. +3. Continue with the [RDI pipeline configuration]({{< relref "/operate/rc/rdi/define" >}}). {{}} When using RDS Proxy, you do not need to set up the Lambda function for failover handling, as the proxy provides a static endpoint. diff --git a/content/operate/rc/databases/rdi/setup.md b/content/operate/rc/rdi/setup.md similarity index 90% rename from content/operate/rc/databases/rdi/setup.md rename to content/operate/rc/rdi/setup.md index 45cb3efbdb..ba10ece2f2 100644 --- a/content/operate/rc/databases/rdi/setup.md +++ b/content/operate/rc/rdi/setup.md @@ -1,5 +1,8 @@ --- Title: Prepare source database +aliases: + - /operate/rc/databases/rdi/setup/ + - /operate/rc/databases/rdi/setup alwaysopen: false categories: - docs @@ -7,32 +10,12 @@ categories: - rc description: Prepare your source database, network setup, and database credentials for Data integration. hideListLinks: true -weight: 2 +weight: 3 --- -## Create new data pipeline - -1. In the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Data Pipeline** tab. -1. Select **Create pipeline**. - {{The create pipeline button.}} -1. Select your source database type. The following database types are supported: - - MySQL - - mariaDB - - Oracle - - SQL Server - - PostgreSQL - {{The select source database type list.}} -1. If you know the size of your source database, enter it into the **Source dataset size** field. - {{Enter the amount of source data you plan to ingest.}} -1. Under **Setup connectivity**, save the provided ARN and extract the AWS account ID for the account associated with your Redis Cloud cluster from it. - - {{The select source database type list.}} - - The AWS account ID is the string of numbers after `arn:aws:iam::` in the ARN. For example, if the ARN is `arn:aws:iam::123456789012:role/redis-data-pipeline`, the AWS account ID is `123456789012`. - ## Prepare source database -Before using the pipeline, you must first prepare your source database to use the Debezium connector for change data capture (CDC). See [Prerequisites]({{}}) to find a list of supported source databases and database versions. +Before using the pipeline, you must first prepare your source database to use the Debezium connector for change data capture (CDC). See [Prerequisites]({{}}) to find a list of supported source databases and database versions. See [Prepare source databases]({{}}) to find steps for your database type: - Hosted on an AWS EC2 instance: @@ -47,9 +30,42 @@ See [Prepare source databases]({{}}) for more information about CDC. -## Set up connectivity +## Get cluster account ID + +Before you can set up your source connectivity and secrets, you need the AWS Account ID for your Redis Cloud cluster so that you can give it access to your connectivity and secrets. + +1. On the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Pipelines** tab. +1. Select **Add pipeline**. + {{The workspace section of the Pipelines tab for a database. Select Add pipeline to add a pipeline.}} +1. Select your source database type. The following database types are supported: + - MySQL + - mariaDB + - Oracle + - SQL Server + - PostgreSQL + {{The select source database type list.}} +1. Enter a name for your source database in the **Source name** field. This is a name for the source database that will appear on Redis Cloud. +1. Select **Continue to source** to move to the **Source configuration** step. + + {{The select source database type list.}} + +1. Under **Source connectivity**, save the provided ARN and extract the AWS account ID for the account associated with your Redis Cloud cluster from it. + + {{The Private Link Role ARN and availability zones.}} + + The AWS account ID is the string of numbers after `arn:aws:iam::` in the ARN. For example, if the ARN is `arn:aws:iam::123456789012:role/redis-data-pipeline`, the AWS account ID is `123456789012`. + +1. If your source database is accessible via the public endpoint and you want to use public connectivity for your data pipeline, select **Public endpoint** and save the **Redis Cloud outbound IP address** to add to your source database's allow list. + +Select **Save & exit** to exit pipeline setup. You'll come back here when you [define your source connection and data pipeline]({{}}). + +## Set up AWS Private Link connectivity + +{{< note >}} +If your source database is accessible via a public endpoint and you want to use public connectivity for your data pipeline, proceed to [Share source database credentials](#share-source-database-credentials). +{{< /note >}} -To ensure that you can connect your Redis Cloud database to the source database, you need to set up an endpoint service through AWS PrivateLink. +If your source database is not accessible via a public endpoint, you need to set up an endpoint service through AWS PrivateLink to be able to connect to it. The following diagrams show the network setup for the different database setups: @@ -137,7 +153,7 @@ To connect to your RDS or Aurora database, we recommend using a Lambda function 1. [Set up Lambda function connectivity](#setup-lambda-function) to route requests to your database. {{}} -If you have specific requirements that necessitate using RDS Proxy instead of the recommended Lambda function approach, see the [RDS Proxy setup guide]({{< relref "/operate/rc/databases/rdi/rds-proxy" >}}). Note that RDS Proxy is not recommended and does not work with PostgreSQL. +If you have specific requirements that necessitate using RDS Proxy instead of the recommended Lambda function approach, see the [RDS Proxy setup guide]({{< relref "/operate/rc/rdi/rds-proxy" >}}). Note that RDS Proxy is not recommended and does not work with PostgreSQL. {{}} ### Create network load balancer {#create-network-load-balancer-rds} @@ -412,6 +428,6 @@ In the [AWS Management Console](https://console.aws.amazon.com/), use the **Serv ## Next steps -After you have set up your source database and prepared connectivity and credentials, select **Define source database** to [define your source connection and data pipeline]({{}}). +After you have set up your source database and prepared connectivity and credentials, select **Define source database** to [define your source connection and data pipeline]({{}}). {{The define source database button.}} \ No newline at end of file diff --git a/content/operate/rc/databases/rdi/view-edit.md b/content/operate/rc/rdi/view-edit.md similarity index 99% rename from content/operate/rc/databases/rdi/view-edit.md rename to content/operate/rc/rdi/view-edit.md index 267b638df0..bba0e279ea 100644 --- a/content/operate/rc/databases/rdi/view-edit.md +++ b/content/operate/rc/rdi/view-edit.md @@ -1,5 +1,8 @@ --- Title: View and edit data pipeline +aliases: + - /operate/rc/databases/rdi/view-edit/ + - /operate/rc/databases/rdi/view-edit alwaysopen: false categories: - docs diff --git a/content/operate/rdi.md b/content/operate/rdi.md index 4f95186a95..0c257620a3 100644 --- a/content/operate/rdi.md +++ b/content/operate/rdi.md @@ -13,4 +13,4 @@ Redis Data Integration (RDI) is a [change data capture](https://en.wikipedia.org See the main [RDI docs section]({{< relref "/integrate/redis-data-integration" >}}) under [Libraries and tools]({{< relref "/integrate" >}}) to learn how to install and use RDI on your own servers. See the -[Redis Cloud RDI guide]({{< relref "/operate/rc/databases/rdi" >}}) to learn how to set up RDI for a cloud database. +[Redis Cloud RDI guide]({{< relref "/operate/rc/rdi" >}}) to learn how to set up RDI for a cloud database. diff --git a/static/images/rc/rdi/rdi-advanced-properties.png b/static/images/rc/rdi/rdi-advanced-properties.png index 256b1b431a..63515a8236 100644 Binary files a/static/images/rc/rdi/rdi-advanced-properties.png and b/static/images/rc/rdi/rdi-advanced-properties.png differ diff --git a/static/images/rc/rdi/rdi-choose-target.png b/static/images/rc/rdi/rdi-choose-target.png new file mode 100644 index 0000000000..ed883937e3 Binary files /dev/null and b/static/images/rc/rdi/rdi-choose-target.png differ diff --git a/static/images/rc/rdi/rdi-confirm-deploy.png b/static/images/rc/rdi/rdi-confirm-deploy.png index 58efc43712..1536750fed 100644 Binary files a/static/images/rc/rdi/rdi-confirm-deploy.png and b/static/images/rc/rdi/rdi-confirm-deploy.png differ diff --git a/static/images/rc/rdi/rdi-continue-to-dataset-button.png b/static/images/rc/rdi/rdi-continue-to-dataset-button.png new file mode 100644 index 0000000000..9369ff0ce0 Binary files /dev/null and b/static/images/rc/rdi/rdi-continue-to-dataset-button.png differ diff --git a/static/images/rc/rdi/rdi-continue-to-source-button.png b/static/images/rc/rdi/rdi-continue-to-source-button.png new file mode 100644 index 0000000000..803fc67e72 Binary files /dev/null and b/static/images/rc/rdi/rdi-continue-to-source-button.png differ diff --git a/static/images/rc/rdi/rdi-continue-to-transformations-button.png b/static/images/rc/rdi/rdi-continue-to-transformations-button.png new file mode 100644 index 0000000000..f009b386f9 Binary files /dev/null and b/static/images/rc/rdi/rdi-continue-to-transformations-button.png differ diff --git a/static/images/rc/rdi/rdi-create-workspace-button.png b/static/images/rc/rdi/rdi-create-workspace-button.png new file mode 100644 index 0000000000..a167a6458a Binary files /dev/null and b/static/images/rc/rdi/rdi-create-workspace-button.png differ diff --git a/static/images/rc/rdi/rdi-create-workspace-cidr.png b/static/images/rc/rdi/rdi-create-workspace-cidr.png new file mode 100644 index 0000000000..b9d71ae7d5 Binary files /dev/null and b/static/images/rc/rdi/rdi-create-workspace-cidr.png differ diff --git a/static/images/rc/rdi/rdi-create-workspace-select-subscription.png b/static/images/rc/rdi/rdi-create-workspace-select-subscription.png new file mode 100644 index 0000000000..7c8ded3209 Binary files /dev/null and b/static/images/rc/rdi/rdi-create-workspace-select-subscription.png differ diff --git a/static/images/rc/rdi/rdi-credentials-arn.png b/static/images/rc/rdi/rdi-credentials-arn.png index 2dec65e610..72b2d50813 100644 Binary files a/static/images/rc/rdi/rdi-credentials-arn.png and b/static/images/rc/rdi/rdi-credentials-arn.png differ diff --git a/static/images/rc/rdi/rdi-dataset-empty.png b/static/images/rc/rdi/rdi-dataset-empty.png new file mode 100644 index 0000000000..0f9e3f9725 Binary files /dev/null and b/static/images/rc/rdi/rdi-dataset-empty.png differ diff --git a/static/images/rc/rdi/rdi-dataset-missing-unique-key.png b/static/images/rc/rdi/rdi-dataset-missing-unique-key.png new file mode 100644 index 0000000000..ff3d806532 Binary files /dev/null and b/static/images/rc/rdi/rdi-dataset-missing-unique-key.png differ diff --git a/static/images/rc/rdi/rdi-dataset-schema-selected.png b/static/images/rc/rdi/rdi-dataset-schema-selected.png new file mode 100644 index 0000000000..f90d054eb7 Binary files /dev/null and b/static/images/rc/rdi/rdi-dataset-schema-selected.png differ diff --git a/static/images/rc/rdi/rdi-define-mtls.png b/static/images/rc/rdi/rdi-define-mtls.png index d048605e5e..fb76fdb378 100644 Binary files a/static/images/rc/rdi/rdi-define-mtls.png and b/static/images/rc/rdi/rdi-define-mtls.png differ diff --git a/static/images/rc/rdi/rdi-define-tls.png b/static/images/rc/rdi/rdi-define-tls.png index e4890da162..ffa4ce9dab 100644 Binary files a/static/images/rc/rdi/rdi-define-tls.png and b/static/images/rc/rdi/rdi-define-tls.png differ diff --git a/static/images/rc/rdi/rdi-new-workspace-button.png b/static/images/rc/rdi/rdi-new-workspace-button.png new file mode 100644 index 0000000000..148121b4f9 Binary files /dev/null and b/static/images/rc/rdi/rdi-new-workspace-button.png differ diff --git a/static/images/rc/rdi/rdi-pipeline-setup.png b/static/images/rc/rdi/rdi-pipeline-setup.png new file mode 100644 index 0000000000..75ac87f5f5 Binary files /dev/null and b/static/images/rc/rdi/rdi-pipeline-setup.png differ diff --git a/static/images/rc/rdi/rdi-processor-advanced-properties.png b/static/images/rc/rdi/rdi-processor-advanced-properties.png new file mode 100644 index 0000000000..7809bd91c4 Binary files /dev/null and b/static/images/rc/rdi/rdi-processor-advanced-properties.png differ diff --git a/static/images/rc/rdi/rdi-select-columns.png b/static/images/rc/rdi/rdi-select-columns.png index 764a5c4089..d0e4229690 100644 Binary files a/static/images/rc/rdi/rdi-select-columns.png and b/static/images/rc/rdi/rdi-select-columns.png differ diff --git a/static/images/rc/rdi/rdi-select-source-db.png b/static/images/rc/rdi/rdi-select-source-db.png index 3001356276..5274f0d101 100644 Binary files a/static/images/rc/rdi/rdi-select-source-db.png and b/static/images/rc/rdi/rdi-select-source-db.png differ diff --git a/static/images/rc/rdi/rdi-setup-connectivity-arn.png b/static/images/rc/rdi/rdi-setup-connectivity-arn.png index 913516d577..13481258ea 100644 Binary files a/static/images/rc/rdi/rdi-setup-connectivity-arn.png and b/static/images/rc/rdi/rdi-setup-connectivity-arn.png differ diff --git a/static/images/rc/rdi/rdi-source-configuration-collector-properties.png b/static/images/rc/rdi/rdi-source-configuration-collector-properties.png new file mode 100644 index 0000000000..b19cfeccaa Binary files /dev/null and b/static/images/rc/rdi/rdi-source-configuration-collector-properties.png differ diff --git a/static/images/rc/rdi/rdi-source-configuration-secrets.png b/static/images/rc/rdi/rdi-source-configuration-secrets.png new file mode 100644 index 0000000000..df5e3784d0 Binary files /dev/null and b/static/images/rc/rdi/rdi-source-configuration-secrets.png differ diff --git a/static/images/rc/rdi/rdi-source-configuration-source-connectivity-privatelink.png b/static/images/rc/rdi/rdi-source-configuration-source-connectivity-privatelink.png new file mode 100644 index 0000000000..c68abf7ea9 Binary files /dev/null and b/static/images/rc/rdi/rdi-source-configuration-source-connectivity-privatelink.png differ diff --git a/static/images/rc/rdi/rdi-source-configuration-source-connectivity-public.png b/static/images/rc/rdi/rdi-source-configuration-source-connectivity-public.png new file mode 100644 index 0000000000..7ab8f7c21a Binary files /dev/null and b/static/images/rc/rdi/rdi-source-configuration-source-connectivity-public.png differ diff --git a/static/images/rc/rdi/rdi-transformations.png b/static/images/rc/rdi/rdi-transformations.png new file mode 100644 index 0000000000..d43bdc6b14 Binary files /dev/null and b/static/images/rc/rdi/rdi-transformations.png differ diff --git a/static/images/rc/rdi/rdi-unique-key-selected.png b/static/images/rc/rdi/rdi-unique-key-selected.png new file mode 100644 index 0000000000..0bfdaeb1eb Binary files /dev/null and b/static/images/rc/rdi/rdi-unique-key-selected.png differ diff --git a/static/images/rc/rdi/rdi-workspace-add-pipeline.png b/static/images/rc/rdi/rdi-workspace-add-pipeline.png new file mode 100644 index 0000000000..2c3054973e Binary files /dev/null and b/static/images/rc/rdi/rdi-workspace-add-pipeline.png differ diff --git a/static/images/rc/rdi/rdi-workspace-resume-setup.png b/static/images/rc/rdi/rdi-workspace-resume-setup.png new file mode 100644 index 0000000000..66f2dde7f0 Binary files /dev/null and b/static/images/rc/rdi/rdi-workspace-resume-setup.png differ