Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 66 additions & 3 deletions docs/guides/message-brokers/kafka.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
---
title: Replaying Kafka
description: "Replay Kafka traffic seamlessly using Speedscale by following our guide to utilize the replay wizard or ProxyMock for efficient traffic simulation."
title: Kafka
description: "Record, mock, and replay Kafka traffic using Speedscale and ProxyMock for efficient traffic simulation and testing."
sidebar_position: 12
---

## Background

In this guide we will show you how to replay Kafka traffic using data captured by Speedscale. The network level modelling of Kafka does not match most people's mental model which leads to confusion and an undesirable replay scenario. Most people view their app interacting with Kafka like this.
This guide covers how to record, mock, and replay Kafka traffic using Speedscale and proxymock. Kafka's network level modelling does not match most people's mental model which leads to confusion and an undesirable replay scenario. Most people view their app interacting with Kafka like this.

For more information about Apache Kafka, see the [official Kafka documentation](https://kafka.apache.org/).

Expand Down Expand Up @@ -566,3 +566,66 @@ python main.py --csv out.csv --topic my-topic --brokers localhost:9092 --respect

</TabItem>
</Tabs>

## Recording Kafka Traffic with proxymock {#recording}

You can record Kafka traffic locally using proxymock's `--map` flag. This creates a reverse proxy that captures the Kafka wire protocol between your application and the broker.

### Start the Recorder

```bash
proxymock record --map 19092=localhost:9092
```

This tells proxymock to listen on port 19092 and forward Kafka traffic to the real broker on port 9092.

### Configure Your Application

Point your Kafka client at the mapped port:

```bash
export KAFKA_BOOTSTRAP_SERVERS=localhost:19092
./my-app
```

Or in your application configuration (Spring Boot example):

```yaml
spring:
kafka:
bootstrap-servers: ${KAFKA_BROKERS:localhost:9092}
```

Then start with the mapped port:

```bash
KAFKA_BROKERS=localhost:19092 java -jar target/my-app.jar
```

### What Gets Recorded

proxymock captures the Kafka binary wire protocol including `ApiVersions`, `Metadata`, `Produce`, `Fetch`, `FindCoordinator`, `JoinGroup`, `SyncGroup`, and other API operations. Each request/response pair is stored as an RRPair file.

## Mocking Kafka with proxymock {#mocking}

After recording Kafka traffic, you can mock the broker for local development and testing. This lets your application run without a real Kafka broker.

```bash
proxymock mock
```

proxymock starts a Kafka mock provider on the port from the recording (typically 9092). Your application connects normally and receives recorded responses.

To map to a different port:

```bash
proxymock mock --map 19092=localhost:9092
```

## Kubernetes and eBPF {#kubernetes}

In Kubernetes environments, Speedscale's eBPF collector (`nettap`) captures Kafka traffic automatically at the kernel level — no `--map` configuration, proxy settings, or application changes required. The collector uses kprobes and uprobes to observe TCP and TLS traffic directly, including Kafka's binary wire protocol.

This means the `--map` workflow described above applies to **local development** only. When your application is deployed in Kubernetes with the Speedscale operator, Kafka traffic is captured transparently alongside all other protocols.

For details on eBPF-based traffic collection, see the [eBPF Traffic Collection](/reference/ebpf-traffic-collection/) documentation.
8 changes: 6 additions & 2 deletions docs/proxymock/getting-started/language-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ proxymock record --map 65432=postgres://localhost:5432
proxymock record --map 1443=https://httpbin.org:443
```

For more database examples, see the [MySQL guide](../guides/mysql.md) and [PostgreSQL guide](../guides/postgres.md).
For more examples, see the [MongoDB guide](../guides/mongodb.md), [MySQL guide](../guides/mysql.md), [PostgreSQL guide](../guides/postgres.md), and [Kafka guide](/guides/message-brokers/kafka).

<Tabs groupId="language">
<TabItem value="golang" label="Go">
Expand All @@ -81,11 +81,15 @@ Java supports `-D` flags to set system properties, which can be set in an enviro
export JAVA_TOOL_OPTIONS="-Dhttp.proxyHost=localhost -Dhttp.proxyPort=4140 -Dhttps.proxyHost=localhost -Dhttps.proxyPort=4140"
```

Use the SOCKS proxy to capture database traffic:
Use the SOCKS proxy to capture JDBC database traffic (MySQL, PostgreSQL):
```shell
export JAVA_TOOL_OPTIONS="-DsocksProxyHost=localhost -DsocksProxyPort=4140"
```

:::caution
The **MongoDB Java driver** uses its own NIO/Netty transport that bypasses `java.net.Socket` entirely. JVM SOCKS flags (`-DsocksProxyHost`, `-DsocksProxyPort`) have no effect. Use `--map` instead for MongoDB traffic — see the [MongoDB guide](../guides/mongodb.md).
:::

With authentication and TLS certificates:
```shell
export JAVA_TOOL_OPTIONS="-Dhttp.proxyHost=localhost -Dhttp.proxyPort=4140 -Dhttps.proxyHost=localhost -Dhttps.proxyPort=4140 -Djavax.net.ssl.trustStore=$HOME/.speedscale/certs/cacerts.jks -Djavax.net.ssl.trustStorePassword=changeit"
Expand Down
139 changes: 139 additions & 0 deletions docs/proxymock/guides/mongodb.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
---
description: "Mock MongoDB database connections and queries for local development using ProxyMock to record and replay real MongoDB wire protocol traffic."
sidebar_position: 7
---

# MongoDB Mocking

This guide covers how to use proxymock to mock MongoDB database connections and queries for local development and testing.

## Introduction to MongoDB {#introduction}

MongoDB is one of the world's most popular NoSQL document databases, using a binary wire protocol (OP_MSG) for client-server communication. **proxymock** is able to record and mock MongoDB databases. This allows you to mock a MongoDB database, including real data, without running a MongoDB instance or populating it with data. To do this, we record your app talking to a MongoDB database and simulate the database in subsequent tests. To learn more about proxymock recording and architecture, check out the [quick start](../getting-started/quickstart/index.md).

## Why `--map` Is Required {#why-map}

Unlike HTTP-based services, MongoDB clients generally cannot be proxied using environment variables like `ALL_PROXY` or `http_proxy`. This is especially true for the **Java MongoDB driver**, which uses its own NIO/Netty-based transport layer that bypasses `java.net.Socket` entirely. JVM-level SOCKS flags (`-DsocksProxyHost`, `-DsocksProxyPort`) have no effect because the driver never goes through the standard socket path.

Python and Node.js MongoDB drivers may partially support SOCKS proxying, but behavior varies across versions and is fragile.

The `--map` flag is the reliable, language-agnostic approach for MongoDB. It tells proxymock to listen on a local port and forward traffic to the real MongoDB server, requiring only a port change in your application configuration.

## Recording MongoDB Traffic {#recording-intro}

The `proxymock record` command creates RRPair files from real MongoDB interactions. Each request will contain a MongoDB wire protocol command (like a `find` or `insert`) and the response will contain the corresponding documents or acknowledgment returned by the database.

### Start the Recorder {#start-recording}

Start a dedicated terminal window to run the proxymock recorder:

```bash
proxymock record --map 37017=localhost:27017
```

This tells the recorder to listen on port 37017 for MongoDB traffic and forward it to the real MongoDB server at 27017. Your can learn more about the how *proxymock* records on the [architecture page](../how-it-works/architecture.md).

### MongoDB Connection Configuration {#configure-mongodb-client}

For proxymock to capture MongoDB traffic, point your application at the mapped port. The exact mechanism depends on your language and framework.

**Java (Spring Boot / Micronaut)**

Externalize the MongoDB host and port so you can switch between the real and mapped ports without code changes. In `application.properties` or `application.yml`:

```yaml
# application.yml
spring:
data:
mongodb:
host: ${MONGO_HOST:localhost}
port: ${MONGO_PORT:27017}
database: mydb
```

Then start your app pointing at the mapped port:

```bash
MONGO_HOST=localhost MONGO_PORT=37017 java -jar target/my-app.jar
```

**Node.js**

```bash
MONGO_URL=mongodb://localhost:37017/mydb node app.js
```

**Python**

```bash
MONGO_URL=mongodb://localhost:37017/mydb python app.py
```

The key ingredient is redirecting your application's MongoDB connection to `localhost:37017` so traffic flows through the proxymock recorder via `--map`.

### What Gets Recorded

You can inspect the recording using the inspect command:

```bash
proxymock inspect
```

![MongoDB RRPairs](./mongodb/mongodb-rrpairs.png)

proxymock captures MongoDB wire protocol traffic as RRPair files containing:

- **Request Data**: MongoDB commands — `hello`/`isMaster` handshake, SASL authentication (SCRAM-SHA-1/256), `find`, `insert`, `update`, `delete`, `aggregate`, `getMore`, and others
- **Response Data**: Document results, cursor data, write acknowledgments, error responses
- **Timing Information**: Command execution times and connection latency

The actual wire protocol is binary but proxymock displays request and response data as JSON. The underlying files can be modified if you want your mock to return different values. You can learn more about the structure of the underlying recording by looking at the `proxymock` directory containing the recording files and the [docs](../how-it-works/rrpair-format.md).

### Troubleshooting Recording

- SOCKS proxy does not work with the Java MongoDB driver — use `--map` instead
- Python/Node.js drivers have inconsistent SOCKS support — `--map` is simpler and more reliable across all languages
- Verify MongoDB server is accessible from proxymock
- If using authentication, make sure to exercise the full connection lifecycle so the SCRAM handshake is captured

## Starting the Mock Server {#start-mocks}

Make sure to stop your local MongoDB server to prevent port conflicts. You can run your app against the normal MongoDB port 27017 now and proxymock will simulate the database.

Start the *proxymock* mock server:

```bash
proxymock mock
```

You can now run your MongoDB client normally and it will connect to proxymock on port 27017 like a normal database.

:::tip
If the mock server fails to start or your app times out connecting, verify that the recorded RRPairs include the MongoDB handshake (`hello` or `isMaster` command). Without the handshake, the driver cannot establish a connection to the mock.
:::

## Modifying Responses

To modify the responses manually, you can find the appropriate markdown file and edit the contents. However, to automate data transformation you can use the transform system provided by [Speedscale enterprise](https://app.speedscale.com). To edit your snapshot, upload it to the cloud:

```sh
proxymock cloud push snapshot
```

A link to your snapshot will be provided. In the Speedscale UI, add your transforms to modify response data as needed.

Remember to click Save. Now download the modified snapshot:

```sh
proxymock cloud pull snapshot <id>
```

You will notice a new `.metadata` directory containing your transform definitions. When you run `proxymock mock` again the transforms will be applied to your mock.

## Kubernetes and eBPF {#kubernetes}

In Kubernetes environments, Speedscale's eBPF collector (`nettap`) captures MongoDB traffic automatically at the kernel level — no `--map` configuration, SOCKS proxying, or application changes required. The collector uses kprobes and uprobes to observe TCP and TLS traffic directly, including MongoDB's binary wire protocol.

This means the `--map` workflow described above applies to **local development** only. When your application is deployed in Kubernetes with the Speedscale operator, MongoDB traffic is captured transparently alongside all other protocols.

For details on eBPF-based traffic collection, see the [eBPF Traffic Collection](/reference/ebpf-traffic-collection/) documentation.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
12 changes: 10 additions & 2 deletions docs/proxymock/guides/mysql.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,15 @@ A link to your snapshot will be provided. In the Speedscale UI, add your transfo
Remember to click Save. Now download the modified snapshot:

```sh
proyxmock cloud pull snapshot <id>
proxymock cloud pull snapshot <id>
```

You will notice a new `.metadata` directory containing your transform definitions. When you run `proxymock mock` again the transforms will be applied to your mock.
You will notice a new `.metadata` directory containing your transform definitions. When you run `proxymock mock` again the transforms will be applied to your mock.

## Kubernetes and eBPF {#kubernetes}

In Kubernetes environments, Speedscale's eBPF collector (`nettap`) captures MySQL traffic automatically at the kernel level — no `--map` configuration, SOCKS proxying, or application changes required. The collector uses kprobes and uprobes to observe TCP and TLS traffic directly, including the MySQL wire protocol.

This means the `--map` workflow described above applies to **local development** only. When your application is deployed in Kubernetes with the Speedscale operator, MySQL traffic is captured transparently alongside all other protocols.

For details on eBPF-based traffic collection, see the [eBPF Traffic Collection](/reference/ebpf-traffic-collection/) documentation.
10 changes: 9 additions & 1 deletion docs/proxymock/guides/postgres.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,4 +107,12 @@ Remember to click Save. Now download the modified snapshot:
proxymock cloud pull snapshot <id>
```

You will notice a new `.metadata` directory containing your transform definitions. When you run `proxymock mock` again the transforms will be applied to your mock.
You will notice a new `.metadata` directory containing your transform definitions. When you run `proxymock mock` again the transforms will be applied to your mock.

## Kubernetes and eBPF {#kubernetes}

In Kubernetes environments, Speedscale's eBPF collector (`nettap`) captures PostgreSQL traffic automatically at the kernel level — no `--map` configuration, SOCKS proxying, or application changes required. The collector uses kprobes and uprobes to observe TCP and TLS traffic directly, including the PostgreSQL wire protocol.

This means the `--map` workflow described above applies to **local development** only. When your application is deployed in Kubernetes with the Speedscale operator, PostgreSQL traffic is captured transparently alongside all other protocols.

For details on eBPF-based traffic collection, see the [eBPF Traffic Collection](/reference/ebpf-traffic-collection/) documentation.
Loading
Loading