diff --git a/docs/guides/message-brokers/kafka.md b/docs/guides/message-brokers/kafka.md index c58515b2..fe80dbaa 100644 --- a/docs/guides/message-brokers/kafka.md +++ b/docs/guides/message-brokers/kafka.md @@ -1,12 +1,12 @@ --- -title: Replaying Kafka -description: "Replay Kafka traffic seamlessly using Speedscale by following our guide to utilize the replay wizard or ProxyMock for efficient traffic simulation." +title: Kafka +description: "Record, mock, and replay Kafka traffic using Speedscale and ProxyMock for efficient traffic simulation and testing." sidebar_position: 12 --- ## Background -In this guide we will show you how to replay Kafka traffic using data captured by Speedscale. The network level modelling of Kafka does not match most people's mental model which leads to confusion and an undesirable replay scenario. Most people view their app interacting with Kafka like this. +This guide covers how to record, mock, and replay Kafka traffic using Speedscale and proxymock. Kafka's network level modelling does not match most people's mental model which leads to confusion and an undesirable replay scenario. Most people view their app interacting with Kafka like this. For more information about Apache Kafka, see the [official Kafka documentation](https://kafka.apache.org/). @@ -566,3 +566,66 @@ python main.py --csv out.csv --topic my-topic --brokers localhost:9092 --respect + +## Recording Kafka Traffic with proxymock {#recording} + +You can record Kafka traffic locally using proxymock's `--map` flag. This creates a reverse proxy that captures the Kafka wire protocol between your application and the broker. + +### Start the Recorder + +```bash +proxymock record --map 19092=localhost:9092 +``` + +This tells proxymock to listen on port 19092 and forward Kafka traffic to the real broker on port 9092. + +### Configure Your Application + +Point your Kafka client at the mapped port: + +```bash +export KAFKA_BOOTSTRAP_SERVERS=localhost:19092 +./my-app +``` + +Or in your application configuration (Spring Boot example): + +```yaml +spring: + kafka: + bootstrap-servers: ${KAFKA_BROKERS:localhost:9092} +``` + +Then start with the mapped port: + +```bash +KAFKA_BROKERS=localhost:19092 java -jar target/my-app.jar +``` + +### What Gets Recorded + +proxymock captures the Kafka binary wire protocol including `ApiVersions`, `Metadata`, `Produce`, `Fetch`, `FindCoordinator`, `JoinGroup`, `SyncGroup`, and other API operations. Each request/response pair is stored as an RRPair file. + +## Mocking Kafka with proxymock {#mocking} + +After recording Kafka traffic, you can mock the broker for local development and testing. This lets your application run without a real Kafka broker. + +```bash +proxymock mock +``` + +proxymock starts a Kafka mock provider on the port from the recording (typically 9092). Your application connects normally and receives recorded responses. + +To map to a different port: + +```bash +proxymock mock --map 19092=localhost:9092 +``` + +## Kubernetes and eBPF {#kubernetes} + +In Kubernetes environments, Speedscale's eBPF collector (`nettap`) captures Kafka traffic automatically at the kernel level — no `--map` configuration, proxy settings, or application changes required. The collector uses kprobes and uprobes to observe TCP and TLS traffic directly, including Kafka's binary wire protocol. + +This means the `--map` workflow described above applies to **local development** only. When your application is deployed in Kubernetes with the Speedscale operator, Kafka traffic is captured transparently alongside all other protocols. + +For details on eBPF-based traffic collection, see the [eBPF Traffic Collection](/reference/ebpf-traffic-collection/) documentation. diff --git a/docs/proxymock/getting-started/language-reference.md b/docs/proxymock/getting-started/language-reference.md index cf13e6b0..e5a479c9 100644 --- a/docs/proxymock/getting-started/language-reference.md +++ b/docs/proxymock/getting-started/language-reference.md @@ -54,7 +54,7 @@ proxymock record --map 65432=postgres://localhost:5432 proxymock record --map 1443=https://httpbin.org:443 ``` -For more database examples, see the [MySQL guide](../guides/mysql.md) and [PostgreSQL guide](../guides/postgres.md). +For more examples, see the [MongoDB guide](../guides/mongodb.md), [MySQL guide](../guides/mysql.md), [PostgreSQL guide](../guides/postgres.md), and [Kafka guide](/guides/message-brokers/kafka). @@ -81,11 +81,15 @@ Java supports `-D` flags to set system properties, which can be set in an enviro export JAVA_TOOL_OPTIONS="-Dhttp.proxyHost=localhost -Dhttp.proxyPort=4140 -Dhttps.proxyHost=localhost -Dhttps.proxyPort=4140" ``` -Use the SOCKS proxy to capture database traffic: +Use the SOCKS proxy to capture JDBC database traffic (MySQL, PostgreSQL): ```shell export JAVA_TOOL_OPTIONS="-DsocksProxyHost=localhost -DsocksProxyPort=4140" ``` +:::caution +The **MongoDB Java driver** uses its own NIO/Netty transport that bypasses `java.net.Socket` entirely. JVM SOCKS flags (`-DsocksProxyHost`, `-DsocksProxyPort`) have no effect. Use `--map` instead for MongoDB traffic — see the [MongoDB guide](../guides/mongodb.md). +::: + With authentication and TLS certificates: ```shell export JAVA_TOOL_OPTIONS="-Dhttp.proxyHost=localhost -Dhttp.proxyPort=4140 -Dhttps.proxyHost=localhost -Dhttps.proxyPort=4140 -Djavax.net.ssl.trustStore=$HOME/.speedscale/certs/cacerts.jks -Djavax.net.ssl.trustStorePassword=changeit" diff --git a/docs/proxymock/guides/mongodb.md b/docs/proxymock/guides/mongodb.md new file mode 100644 index 00000000..36d8261b --- /dev/null +++ b/docs/proxymock/guides/mongodb.md @@ -0,0 +1,139 @@ +--- +description: "Mock MongoDB database connections and queries for local development using ProxyMock to record and replay real MongoDB wire protocol traffic." +sidebar_position: 7 +--- + +# MongoDB Mocking + +This guide covers how to use proxymock to mock MongoDB database connections and queries for local development and testing. + +## Introduction to MongoDB {#introduction} + +MongoDB is one of the world's most popular NoSQL document databases, using a binary wire protocol (OP_MSG) for client-server communication. **proxymock** is able to record and mock MongoDB databases. This allows you to mock a MongoDB database, including real data, without running a MongoDB instance or populating it with data. To do this, we record your app talking to a MongoDB database and simulate the database in subsequent tests. To learn more about proxymock recording and architecture, check out the [quick start](../getting-started/quickstart/index.md). + +## Why `--map` Is Required {#why-map} + +Unlike HTTP-based services, MongoDB clients generally cannot be proxied using environment variables like `ALL_PROXY` or `http_proxy`. This is especially true for the **Java MongoDB driver**, which uses its own NIO/Netty-based transport layer that bypasses `java.net.Socket` entirely. JVM-level SOCKS flags (`-DsocksProxyHost`, `-DsocksProxyPort`) have no effect because the driver never goes through the standard socket path. + +Python and Node.js MongoDB drivers may partially support SOCKS proxying, but behavior varies across versions and is fragile. + +The `--map` flag is the reliable, language-agnostic approach for MongoDB. It tells proxymock to listen on a local port and forward traffic to the real MongoDB server, requiring only a port change in your application configuration. + +## Recording MongoDB Traffic {#recording-intro} + +The `proxymock record` command creates RRPair files from real MongoDB interactions. Each request will contain a MongoDB wire protocol command (like a `find` or `insert`) and the response will contain the corresponding documents or acknowledgment returned by the database. + +### Start the Recorder {#start-recording} + +Start a dedicated terminal window to run the proxymock recorder: + +```bash +proxymock record --map 37017=localhost:27017 +``` + +This tells the recorder to listen on port 37017 for MongoDB traffic and forward it to the real MongoDB server at 27017. Your can learn more about the how *proxymock* records on the [architecture page](../how-it-works/architecture.md). + +### MongoDB Connection Configuration {#configure-mongodb-client} + +For proxymock to capture MongoDB traffic, point your application at the mapped port. The exact mechanism depends on your language and framework. + +**Java (Spring Boot / Micronaut)** + +Externalize the MongoDB host and port so you can switch between the real and mapped ports without code changes. In `application.properties` or `application.yml`: + +```yaml +# application.yml +spring: + data: + mongodb: + host: ${MONGO_HOST:localhost} + port: ${MONGO_PORT:27017} + database: mydb +``` + +Then start your app pointing at the mapped port: + +```bash +MONGO_HOST=localhost MONGO_PORT=37017 java -jar target/my-app.jar +``` + +**Node.js** + +```bash +MONGO_URL=mongodb://localhost:37017/mydb node app.js +``` + +**Python** + +```bash +MONGO_URL=mongodb://localhost:37017/mydb python app.py +``` + +The key ingredient is redirecting your application's MongoDB connection to `localhost:37017` so traffic flows through the proxymock recorder via `--map`. + +### What Gets Recorded + +You can inspect the recording using the inspect command: + +```bash +proxymock inspect +``` + +![MongoDB RRPairs](./mongodb/mongodb-rrpairs.png) + +proxymock captures MongoDB wire protocol traffic as RRPair files containing: + +- **Request Data**: MongoDB commands — `hello`/`isMaster` handshake, SASL authentication (SCRAM-SHA-1/256), `find`, `insert`, `update`, `delete`, `aggregate`, `getMore`, and others +- **Response Data**: Document results, cursor data, write acknowledgments, error responses +- **Timing Information**: Command execution times and connection latency + +The actual wire protocol is binary but proxymock displays request and response data as JSON. The underlying files can be modified if you want your mock to return different values. You can learn more about the structure of the underlying recording by looking at the `proxymock` directory containing the recording files and the [docs](../how-it-works/rrpair-format.md). + +### Troubleshooting Recording + +- SOCKS proxy does not work with the Java MongoDB driver — use `--map` instead +- Python/Node.js drivers have inconsistent SOCKS support — `--map` is simpler and more reliable across all languages +- Verify MongoDB server is accessible from proxymock +- If using authentication, make sure to exercise the full connection lifecycle so the SCRAM handshake is captured + +## Starting the Mock Server {#start-mocks} + +Make sure to stop your local MongoDB server to prevent port conflicts. You can run your app against the normal MongoDB port 27017 now and proxymock will simulate the database. + +Start the *proxymock* mock server: + +```bash +proxymock mock +``` + +You can now run your MongoDB client normally and it will connect to proxymock on port 27017 like a normal database. + +:::tip +If the mock server fails to start or your app times out connecting, verify that the recorded RRPairs include the MongoDB handshake (`hello` or `isMaster` command). Without the handshake, the driver cannot establish a connection to the mock. +::: + +## Modifying Responses + +To modify the responses manually, you can find the appropriate markdown file and edit the contents. However, to automate data transformation you can use the transform system provided by [Speedscale enterprise](https://app.speedscale.com). To edit your snapshot, upload it to the cloud: + +```sh +proxymock cloud push snapshot +``` + +A link to your snapshot will be provided. In the Speedscale UI, add your transforms to modify response data as needed. + +Remember to click Save. Now download the modified snapshot: + +```sh +proxymock cloud pull snapshot +``` + +You will notice a new `.metadata` directory containing your transform definitions. When you run `proxymock mock` again the transforms will be applied to your mock. + +## Kubernetes and eBPF {#kubernetes} + +In Kubernetes environments, Speedscale's eBPF collector (`nettap`) captures MongoDB traffic automatically at the kernel level — no `--map` configuration, SOCKS proxying, or application changes required. The collector uses kprobes and uprobes to observe TCP and TLS traffic directly, including MongoDB's binary wire protocol. + +This means the `--map` workflow described above applies to **local development** only. When your application is deployed in Kubernetes with the Speedscale operator, MongoDB traffic is captured transparently alongside all other protocols. + +For details on eBPF-based traffic collection, see the [eBPF Traffic Collection](/reference/ebpf-traffic-collection/) documentation. diff --git a/docs/proxymock/guides/mongodb/mongodb-rrpair-detail.png b/docs/proxymock/guides/mongodb/mongodb-rrpair-detail.png new file mode 100644 index 00000000..70161d67 Binary files /dev/null and b/docs/proxymock/guides/mongodb/mongodb-rrpair-detail.png differ diff --git a/docs/proxymock/guides/mongodb/mongodb-rrpairs.png b/docs/proxymock/guides/mongodb/mongodb-rrpairs.png new file mode 100644 index 00000000..d09b516b Binary files /dev/null and b/docs/proxymock/guides/mongodb/mongodb-rrpairs.png differ diff --git a/docs/proxymock/guides/mysql.md b/docs/proxymock/guides/mysql.md index 09a368b2..102829bd 100644 --- a/docs/proxymock/guides/mysql.md +++ b/docs/proxymock/guides/mysql.md @@ -108,7 +108,15 @@ A link to your snapshot will be provided. In the Speedscale UI, add your transfo Remember to click Save. Now download the modified snapshot: ```sh -proyxmock cloud pull snapshot +proxymock cloud pull snapshot ``` -You will notice a new `.metadata` directory containing your transform definitions. When you run `proxymock mock` again the transforms will be applied to your mock. \ No newline at end of file +You will notice a new `.metadata` directory containing your transform definitions. When you run `proxymock mock` again the transforms will be applied to your mock. + +## Kubernetes and eBPF {#kubernetes} + +In Kubernetes environments, Speedscale's eBPF collector (`nettap`) captures MySQL traffic automatically at the kernel level — no `--map` configuration, SOCKS proxying, or application changes required. The collector uses kprobes and uprobes to observe TCP and TLS traffic directly, including the MySQL wire protocol. + +This means the `--map` workflow described above applies to **local development** only. When your application is deployed in Kubernetes with the Speedscale operator, MySQL traffic is captured transparently alongside all other protocols. + +For details on eBPF-based traffic collection, see the [eBPF Traffic Collection](/reference/ebpf-traffic-collection/) documentation. \ No newline at end of file diff --git a/docs/proxymock/guides/postgres.md b/docs/proxymock/guides/postgres.md index a742c895..846f326a 100644 --- a/docs/proxymock/guides/postgres.md +++ b/docs/proxymock/guides/postgres.md @@ -107,4 +107,12 @@ Remember to click Save. Now download the modified snapshot: proxymock cloud pull snapshot ``` -You will notice a new `.metadata` directory containing your transform definitions. When you run `proxymock mock` again the transforms will be applied to your mock. \ No newline at end of file +You will notice a new `.metadata` directory containing your transform definitions. When you run `proxymock mock` again the transforms will be applied to your mock. + +## Kubernetes and eBPF {#kubernetes} + +In Kubernetes environments, Speedscale's eBPF collector (`nettap`) captures PostgreSQL traffic automatically at the kernel level — no `--map` configuration, SOCKS proxying, or application changes required. The collector uses kprobes and uprobes to observe TCP and TLS traffic directly, including the PostgreSQL wire protocol. + +This means the `--map` workflow described above applies to **local development** only. When your application is deployed in Kubernetes with the Speedscale operator, PostgreSQL traffic is captured transparently alongside all other protocols. + +For details on eBPF-based traffic collection, see the [eBPF Traffic Collection](/reference/ebpf-traffic-collection/) documentation. \ No newline at end of file diff --git a/docs/proxymock/guides/secrets.md b/docs/proxymock/guides/secrets.md new file mode 100644 index 00000000..44626123 --- /dev/null +++ b/docs/proxymock/guides/secrets.md @@ -0,0 +1,121 @@ +--- +description: "Use the ${{secret:name/key}} syntax to inject sensitive values like database credentials, JWT signing keys, and API tokens into proxymock transforms without storing them in your recording." +sidebar_position: 8 +--- + +# Secrets + +Recorded traffic often contains credentials that differ between environments — database passwords, JWT signing keys, AWS access keys. The `${{secret:name/key}}` syntax lets proxymock resolve these values from local files at runtime, so transforms can re-sign tokens or re-authenticate connections without hardcoding secrets into your recording or blueprint. + +## Directory layout + +proxymock discovers secrets automatically by walking up from your `--in` directory. Place a `secrets/` directory alongside (or inside) your `proxymock/` workspace: + +``` +my-project/ +├── proxymock/ +│ ├── secrets/ ← preferred location +│ │ ├── mongo-creds/ +│ │ │ ├── host +│ │ │ ├── username +│ │ │ └── password +│ │ └── jwt-signing/ +│ │ └── key +│ └── recorded-2026-06-01T120000Z/ +│ └── ... +``` + +Each secret is a plain file whose contents are the value. For example, `secrets/mongo-creds/password` might contain `s3cret123` with no trailing newline. + +An alternative layout places `secrets/` one level up from `proxymock/`: + +``` +my-project/ +├── secrets/ +│ └── mongo-creds/ +│ └── ... +└── proxymock/ + └── recorded-.../ +``` + +Both layouts work. proxymock checks `/proxymock/secrets/` first, then `/secrets/`. + +## Referencing secrets in transforms + +Use the `${{secret:name/key}}` syntax in transform configuration fields that accept secret references. The `name` maps to the directory and `key` maps to the filename inside it. + +### MongoDB SASL authentication + +The `sasl_auth` transform re-authenticates MongoDB connections using SCRAM. Point each field at the appropriate secret file: + +```json +{ + "type": "sasl_auth", + "config": { + "host": "${{secret:mongo-creds/host}}", + "username": "${{secret:mongo-creds/username}}", + "password": "${{secret:mongo-creds/password}}" + } +} +``` + +This resolves to `secrets/mongo-creds/host`, `secrets/mongo-creds/username`, and `secrets/mongo-creds/password` on disk. + +For multiple MongoDB hosts, add one `sasl_auth` transform per host, each referencing its own secret directory. + +### JWT re-signing + +The `jwt_resign` transform re-signs JWT tokens using a local key: + +```json +{ + "type": "jwt_resign", + "config": { + "secretPath": "${{secret:jwt-signing/key}}" + } +} +``` + +### AWS Signature V4 + +The `aws_auth` transform re-signs AWS requests: + +```json +{ + "type": "aws_auth", + "config": { + "secretPath": "${{secret:aws-creds/secret}}", + "idPath": "${{secret:aws-creds/id}}" + } +} +``` + +## CLI usage + +No extra flags are needed. When you run `proxymock mock` or `proxymock replay`, the secrets directory is discovered automatically from the `--in` path: + +```bash +proxymock replay --in ./proxymock/recorded-2026-06-01T120000Z +``` + +proxymock walks up from the `--in` directory looking for a `secrets/` folder. If it finds one, all `${{secret:name/key}}` references in your transforms resolve against it. + +## Security notes + +- **Never commit secrets to version control.** Add `secrets/` to your `.gitignore`. +- Path traversal is blocked — references like `${{secret:../etc/passwd}}` are rejected and treated as a missing file. +- In Kubernetes, the Speedscale operator mounts secrets from the cluster's secret store. The local `secrets/` directory is only used for out-of-cluster (CLI) runs. + +## Troubleshooting + +**Transform fails with "file not found" for a secret path** +The `secrets/` directory wasn't discovered. Verify the directory exists and is reachable by walking up from your `--in` path. The directory must be named exactly `secrets` (not `.secrets` or `_secrets`). + +**Secret value has unexpected whitespace** +Secret files are read as-is. Make sure the file doesn't contain a trailing newline. On macOS/Linux: + +```bash +printf 's3cret123' > secrets/mongo-creds/password +``` + +Using `echo` adds a trailing newline by default — use `printf` or `echo -n` instead. diff --git a/docs/reference/technology-support.md b/docs/reference/technology-support.md index b90c7bf9..bf81b9b5 100644 --- a/docs/reference/technology-support.md +++ b/docs/reference/technology-support.md @@ -65,7 +65,7 @@ Speedscale replays involve three distinct steps that are supported separately: * | Google BigTable | DBMS | Full | If using the Google SDK with standard pagination | | Google Spanner | DBMS | Full | If using the Google SDK with standard pagination | | Microsoft Outlook 365 | API | Full | | -| MongoDB | DBMS | Capture Only | | +| MongoDB | DBMS | Full | See [proxymock guide](/proxymock/guides/mongodb) | | MySQL | DBMS | Full | | | Postgres | DBMS | Full | See [details](/guides/capture/bodies#postgres) | | Redis | DBMS | Capture Only | See [details](/guides/capture/bodies#redis) |