Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ With the **Spice.ai Cloud Platform** you can:
| Use Case | Description |
| --------------------------------------------------- | ---------------------------------------------------------- |
| [Agentic AI Apps](use-cases/agentic-ai-apps.md) | Build AI agent backends with unified data and model access |
| [Analytics Replica](use-cases/analytics-replica.md) | Run analytics on operational data without ETL or migration |
| [Database CDN](use-cases/database-cdn.md) | Cache and accelerate hot data for low-latency applications |
| [Data Lakehouse](use-cases/data-lakehouse.md) | Federated queries across warehouses, lakes, and databases |
| [Enterprise Search](use-cases/enterprise-search.md) | Semantic search across enterprise data sources |
Expand Down
2 changes: 2 additions & 0 deletions SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
* [DuckDB Data Accelerator](features/data-acceleration/duckdb-data-accelerator.md)
* [PostgreSQL Data Accelerator](features/data-acceleration/postgresql-data-accelerator.md)
* [SQLite Data Accelerator](features/data-acceleration/sqlite-data-accelerator.md)
* [Database Replication and CDC](features/database-replication-and-cdc.md)
* [Search & Retrieval](features/search-and-retrieval.md)
* [AI Gateway](features/ai-gateway.md)
* [Semantic Models](features/semantic-models.md)
Expand Down Expand Up @@ -79,6 +80,7 @@
## Use-Cases

* [Agentic AI Apps](use-cases/agentic-ai-apps.md)
* [Analytics Replica](use-cases/analytics-replica.md)
* [Database CDN](use-cases/database-cdn.md)
* [Data Lakehouse](use-cases/data-lakehouse.md)
* [Enterprise Search](use-cases/enterprise-search.md)
Expand Down
2 changes: 2 additions & 0 deletions features/data-acceleration/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@ Spice supports three modes to refresh/update locally accelerated data from a con
| `append` | Append/add data to the dataset on each refresh | Append-only, immutable datasets, such as time-series or log data |
| `changes` | Apply incremental changes | Customer order lifecycle table |

`refresh_mode: changes` streams committed inserts, updates, and deletes from the source's own changelog. See [Database Replication and CDC](../database-replication-and-cdc.md) for supported sources and configuration.

#### Example - Accelerate with arrow accelerator under full refresh mode <a href="#example" id="example"></a>

```yaml
Expand Down
106 changes: 106 additions & 0 deletions features/database-replication-and-cdc.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
---
icon: arrows-rotate
description: Replicate committed changes from operational databases into an accelerated, query-ready replica using change data capture
---

# Database Replication and CDC

**Database replication** keeps an accelerated dataset continuously in step with its source by reading the source database's own changelog. Committed inserts, updates, and deletes are applied to the local replica within seconds, with no batch window and no external pipeline.

The mechanism is **change data capture (CDC)**: rather than re-reading the source table on a schedule, Spice consumes the stream of changes the database already produces for its own recovery and replication — the PostgreSQL write-ahead log, a MongoDB change stream, a DynamoDB stream — and applies each change to the accelerator as it commits.

Replication is enabled by setting `refresh_mode: changes` on an accelerated dataset.

{% hint style="info" %}
`changes` is one of three [refresh modes](data-acceleration/README.md#refresh-modes). Use `full` to replace a dataset on each refresh, `append` for immutable or time-series data, and `changes` to mirror a mutable source that emits a change feed.
{% endhint %}

### Why replicate

Running analytical queries against a production database competes with transaction processing for the same connections, buffer pool, and CPU. The usual alternatives each carry a cost:

* **ETL pipelines** add latency measured in minutes or hours, plus the infrastructure to build, schedule, and monitor them.
* **Read replicas** relieve the primary but run the same row-oriented engine, so analytical scans remain slow.
* **HTAP databases** require migrating off the existing system and couple transactional and analytical failure domains.

CDC-based replication into a columnar accelerator avoids all three. The operational database keeps serving transactions, analytical load lands on separate storage and compute, and the replica stays seconds behind rather than hours.

For the architecture built on this capability, see [Analytics Replica](../use-cases/analytics-replica.md).

### Supported sources

| Source | Mechanism | Configuration |
| ------ | --------- | ------------- |
| [PostgreSQL](../building-blocks/data-connectors/postgres.md) | Logical replication from the write-ahead log | `refresh_mode: changes` |
| [MongoDB](../building-blocks/data-connectors/mongodb.md) | Change streams on the source collection | `refresh_mode: changes` |
| [DynamoDB](../building-blocks/data-connectors/dynamodb.md) | DynamoDB Streams | `refresh_mode: changes` |
| [Apache Kafka](../building-blocks/data-connectors/kafka.md) | Event stream consumption | `refresh_mode: append` |
| [Debezium](../building-blocks/data-connectors/debezium.md) | Debezium change events over Kafka | `refresh_mode: changes` |

{% hint style="info" %}
Sources without a native Spice change feed — including MySQL and SQL Server — replicate through [Debezium](../building-blocks/data-connectors/debezium.md) over Kafka.
{% endhint %}

### Configuration

A replicated dataset needs a `primary_key` so that updates and deletes can be matched to existing rows, and an `on_conflict` rule so that repeated keys upsert rather than duplicate.

```yaml
datasets:
- from: postgres:public.orders
name: orders
params:
pg_host: postgres.example-org.com
pg_port: '5432'
pg_user: spice
pg_pass: ${secrets:pg_pass}
pg_db: myapp
pg_sslmode: verify-full
acceleration:
enabled: true
engine: cayenne
mode: file
refresh_mode: changes
primary_key: id
on_conflict:
id: upsert
```

On startup Spice loads an initial snapshot of the table, then switches to streaming changes. No `refresh_check_interval` is required — changes are applied as they arrive rather than on a poll.

Any accelerator engine can back a replicated dataset. [Cayenne](../building-blocks/data-accelerators/cayenne.md) is built for this workload, sustaining a high-throughput change feed while serving analytical scans from the same table.

### PostgreSQL prerequisites

Logical replication must be enabled on the source server:

```
wal_level = logical
max_replication_slots = 10
max_wal_senders = 10
```

Each replicated table needs a primary key, or `REPLICA IDENTITY FULL`, so that updates and deletes carry enough information to identify the affected row:

```sql
ALTER TABLE public.orders REPLICA IDENTITY FULL;
```

The connecting role needs the `REPLICATION` attribute, plus `SELECT` on the replicated tables. Spice creates and manages its own replication slot and publication.

{% hint style="warning" %}
An inactive replication slot causes the source server to retain write-ahead log segments indefinitely, which can exhaust disk on the primary. Drop the slot on the source if a replicated dataset is removed permanently.
{% endhint %}

### Handling deletes

CDC propagates hard deletes, which sets replication apart from incremental ingestion. A `DELETE` on the source removes the row from the replica on the next change event, with no reconciling full refresh and no soft-delete convention in the source schema.

Sources that expose no change feed at all — HTTP APIs, for example — use [incremental ingestion](data-acceleration/README.md#incremental-ingestion) with `refresh_mode: append` instead, where deletes are handled by soft-delete tombstones or a periodic full refresh.

### Related

* [Analytics Replica](../use-cases/analytics-replica.md) — the deployment pattern built on replication
* [Data Acceleration](data-acceleration/README.md) — refresh modes, incremental ingestion, and retention
* [Database CDN](../use-cases/database-cdn.md) — colocating a hot working set with an application
* [Change data capture](https://spiceai.org/docs/features/cdc) in the Spice.ai OSS documentation
80 changes: 80 additions & 0 deletions use-cases/analytics-replica.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
---
description: Attach a columnar analytics replica to an operational database without ETL or migration
icon: database
---

# Analytics Replica

The **analytics replica pattern** attaches a dedicated columnar node to an operational database and lets it absorb the analytical query load. The primary keeps serving transactions. Analytical questions — which customers churned today, which orders are stuck, how revenue moved this hour — run against a replica that stays seconds behind.

It is the shortest path from an operational database to analytical and AI workloads, because nothing has to be migrated and no pipeline has to be built.

### The problem

Analytical queries and transactional queries want opposite things from a database. Transactions want narrow row lookups and short-lived locks. Analytics wants wide scans over large ranges. Running both on one primary means the reporting query and the checkout path compete for the same buffer pool and CPU.

The established workarounds each trade one problem for another:

| Approach | What it costs |
| -------- | ------------- |
| **ETL pipeline** | Data arrives minutes or hours late, and the pipeline itself becomes infrastructure to build, schedule, monitor, and repair |
| **Read replica** | Removes load from the primary, but runs the same row-oriented engine — analytical scans are just as slow |
| **HTAP database** | Requires migrating off the current database, and recouples transactional and analytical failure domains |
| **Data warehouse** | Strong for analytics, but reached through a pipeline, so it inherits the latency and the operational burden |

### How it works

An analytics replica connects to the operational database, loads an initial snapshot, then uses change data capture (CDC) to apply committed changes continuously from the source's native changelog — the PostgreSQL write-ahead log, a MongoDB change stream, a DynamoDB stream. There is no batch interval; a committed change is queryable within seconds.

The replica stores data in a columnar format on its own storage and compute, so scans are fast and the load never reaches the primary. Queries run through the same federated SQL interface as the rest of the platform, which means a replicated table can be joined against a data lake, a warehouse, or another operational system in a single query.

```yaml
datasets:
- from: postgres:public.orders
name: orders
params:
pg_host: postgres.example-org.com
pg_user: spice
pg_pass: ${secrets:pg_pass}
pg_db: myapp
acceleration:
enabled: true
engine: cayenne
mode: file
refresh_mode: changes
primary_key: id
on_conflict:
id: upsert
```

See [Database Replication and CDC](../features/database-replication-and-cdc.md) for the supported sources, source prerequisites, and full configuration reference.

### Why it differs from a read replica

A read replica and an analytics replica solve different halves of the problem:

* A **read replica** moves load off the primary but keeps the row-oriented storage and execution engine, so a query scanning millions of rows is no faster than it was.
* An **analytics replica** changes the storage layout and the execution engine. Data is columnar, scans read only the columns a query touches, and segment statistics skip data that cannot match.

Both keep the primary healthy. Only one makes the analytical query fast.

### Incremental adoption

The pattern is adopted table by table. Replicate one table, point one dashboard or one agent at it, and leave everything else untouched. There is no cutover, no schema migration, and no change to how the application writes.

That property matters most when the destination is an AI agent. Grounding an agent in production data usually stalls on the pipeline needed to expose that data safely. Replicating the handful of tables the agent needs is a smaller commitment than building a warehouse feed, and the agent queries data that is seconds old rather than a day stale.

### Governing access

Because queries reach the replica rather than the operational database, access is governed at the replica:

* Agents and applications authenticate to Spice instead of holding database credentials, so the primary's credentials never leave the infrastructure that owns them.
* [Policy](../enterprise/features/policy.md) rules apply row-level filters and column masking at query time, based on the identity of the caller.
* Queries are recorded, so what an agent read is auditable after the fact.

### Related

* [Database Replication and CDC](../features/database-replication-and-cdc.md) — configuration, supported sources, and prerequisites
* [Federated SQL Query](../features/federated-sql-query.md) — joining a replica against other systems
* [Database CDN](database-cdn.md) — colocating a hot working set with an application
* [Agentic AI Apps](agentic-ai-apps.md) — grounding agents in operational data