Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 23 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,15 +13,18 @@ Currently supported:

- Ruby with RSpec or Minitest.
- Python with pytest.
- JavaScript with Jest.

## Prerequisites

Before using DDTest, you must have **Datadog Test Optimization** already set up and enabled with a Datadog Test Optimization library for your language and framework. DDTest relies on this integration to discover your tests and plan test execution accordingly.

Minimum supported library versions:
Minimum supported library and runtime requirements:

- Ruby requires the `datadog-ci` gem **1.31.0** or higher.
- Python requires the `ddtrace` package **4.10.3** or higher and `pytest`.
- JavaScript requires the `dd-trace` package **6.0.0** or higher, Node.js, and
Jest.

For instructions on setting up Test Optimization, see the [Datadog Test Optimization documentation](https://docs.datadoghq.com/tests/setup/).

Expand Down Expand Up @@ -66,6 +69,16 @@ ddtest plan \
--max-parallelism 32
```

For JavaScript/Jest:

```bash
ddtest plan \
--platform javascript \
--framework jest \
--min-parallelism 8 \
--max-parallelism 32
```

This prepares the plan and writes it to `.testoptimization/` folder for later reuse.
Copy `.testoptimization/` to any CI job that runs `ddtest run` or reads DDTest's
plan file lists. For the full file layout and formats, see
Expand All @@ -87,16 +100,22 @@ For Python/pytest:
ddtest run --platform python --framework pytest
```

For JavaScript/Jest:

```bash
ddtest run --platform javascript --framework jest
```

For CI-node mode, worker environment variables, custom commands, and
parallelism details, see [Running DDTest](docs/running.md).

### Common settings

| CLI flag | What it does |
| --- | --- |
| `--platform` | Language/platform. Currently supported: `ruby`, `python`. |
| `--framework` | Test framework. Currently supported: `rspec`, `minitest`, `pytest`. |
| `--command` | Override the default test command used by Ruby frameworks. For pytest, use `PYTEST_ADDOPTS` for pytest flags. |
| `--platform` | Language/platform. Currently supported: `ruby`, `python`, `javascript`. |
| `--framework` | Test framework. Currently supported: `rspec`, `minitest`, `pytest`, `jest`. |
| `--command` | Override the default base command for supported framework modes. Currently used by RSpec and Minitest run/discovery, and Jest run/discovery. For pytest, use `PYTEST_ADDOPTS` for pytest flags. |
| `--min-parallelism` | Minimum CI node or worker count DDTest considers when planning. |
| `--max-parallelism` | Maximum CI node or worker count DDTest considers when planning. |
| `--target-time` | Target wall time DDTest tries to satisfy when selecting parallelism. |
Expand Down
29 changes: 26 additions & 3 deletions docs/best_practices.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,10 @@
## Optimize Planning Step

When using ddtest, you need to add a planning step that performs test discovery
(for example, RSpec dry-run or pytest collection) before execution. This stage
adds overhead: you can optimize it with the practices below.
before execution. For Ruby and Python, this can involve full framework
discovery such as RSpec dry-run or pytest collection. For Jest, DDTest uses
Jest's `--listTests` command to discover test files. This planning stage adds
overhead: you can optimize it with the practices below.

### Preinstall System Dependencies Via Docker

Expand Down Expand Up @@ -40,6 +42,15 @@ For GitHub Actions + pip:
cache: pip
```

For GitHub Actions + npm:

```yaml
- uses: actions/setup-node@v4
with:
node-version: "22"
cache: npm
```

### Disable Seeds/Fixtures During Discovery

Discovery (planning) does not execute tests; you don't have to setup DB,
Expand Down Expand Up @@ -123,7 +134,8 @@ corrupt, produced for a different platform/framework/test location/exclude
pattern, or based on a commit that is not available locally. It also invalidates
the cache when files under the current project's test root changed. For example,
the default RSpec root is `spec/**`, the default Minitest root is `test/**`,
and pytest uses `testpaths` from pytest config when available. With
pytest uses `testpaths` from pytest config when available, and Jest uses
DDTest's built-in Jest test file pattern unless `--tests-location` is set. With
`--tests-location custom/spec/**/*_spec.rb`, the root is `custom/**`.

In monorepos, run DDTest from the project subdirectory whose tests you are
Expand All @@ -141,6 +153,17 @@ For discovery, DDTest reads `testpaths` and `python_files` from `pytest.ini`,
`pyproject.toml`, `tox.ini`, or `setup.cfg`. If no pytest config defines those
settings, DDTest uses `**/{test_*,*_test}.py`.

## Jest Support
Comment thread
cbasitodx marked this conversation as resolved.

Use `--command` when your project runs Jest through a package manager or wrapper:

```bash
ddtest run --platform javascript --framework jest --command "pnpm jest --runInBand"
```

Do not include test files or a `--` separator in the command; DDTest appends the
file list and Jest flags itself.

## Minitest Support In Non-Rails Projects

We use `bundle exec rake test` command when we don't detect `rails` command to
Expand Down
40 changes: 40 additions & 0 deletions docs/examples/circleci.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,3 +170,43 @@ CircleCI node index to DDTest:
NODE_INDEX=${CIRCLE_NODE_INDEX:-0}
./bin/ddtest run --platform python --framework pytest --ci-node "${NODE_INDEX}"
```

## JavaScript / Jest Variant

Use the same setup workflow pattern for JavaScript projects, but use a Node.js
image, install project dependencies, and configure the runner for Jest:

```yaml
jobs:
plan:
docker:
- image: cimg/node:22.14
environment:
DD_ENV: ci
DD_TEST_OPTIMIZATION_RUNNER_PLATFORM: javascript
DD_TEST_OPTIMIZATION_RUNNER_FRAMEWORK: jest
DD_TEST_OPTIMIZATION_RUNNER_MIN_PARALLELISM: 1
DD_TEST_OPTIMIZATION_RUNNER_MAX_PARALLELISM: 4
steps:
- checkout
- run:
name: Install JavaScript dependencies
command: npm ci
- test-optimization-circleci-orb/autoinstrument:
languages: js
site: datadoghq.eu
- run:
name: Plan tests with ddtest
command: ./bin/ddtest plan
```

In the test job, keep the restored `.testoptimization/` plan, install
dependencies, and pass the CircleCI node index to DDTest:

```yaml
- run:
name: Run tests with ddtest
command: |
NODE_INDEX=${CIRCLE_NODE_INDEX:-0}
./bin/ddtest run --platform javascript --framework jest --ci-node "${NODE_INDEX}"
```
42 changes: 42 additions & 0 deletions docs/examples/github-actions.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,3 +133,45 @@ Configure Datadog Test Optimization for Python:
The `ddtest plan` and `ddtest run --ci-node ${{ matrix.ci_node_index }}`
commands can stay the same when the platform and framework are provided through
the environment.

## JavaScript / Jest Variant

Use the same plan/test job structure for JavaScript projects, but configure the
runner and setup steps for Jest:

```yaml
env:
DD_TEST_OPTIMIZATION_RUNNER_PLATFORM: javascript
DD_TEST_OPTIMIZATION_RUNNER_FRAMEWORK: jest
DD_TEST_OPTIMIZATION_RUNNER_MIN_PARALLELISM: 1
DD_TEST_OPTIMIZATION_RUNNER_MAX_PARALLELISM: 8
```

Replace each Ruby setup step with Node.js dependency installation:

```yaml
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: "22"
cache: npm
- name: Install JavaScript dependencies
run: npm ci
```

Configure Datadog Test Optimization for JavaScript:

```yaml
- name: Configure Datadog Test Optimization
uses: datadog/test-visibility-github-action@v2
with:
languages: js
api_key: ${{ secrets.DD_API_KEY }}
site: datadoghq.com
```

DDTest sets `NODE_OPTIONS=-r dd-trace/ci/init` for Jest worker processes, so the
project dependencies installed before `ddtest plan` must include `dd-trace`.
The `ddtest plan` and `ddtest run --ci-node ${{ matrix.ci_node_index }}`
commands can stay the same when the platform and framework are provided through
the environment.
2 changes: 1 addition & 1 deletion docs/layout.md
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,7 @@ Fields:
| --- | --- |
| `name` | Test name reported by the framework. |
| `suite` | Test suite name reported by the framework. |
| `module` | Framework module name, such as `rspec`, `minitest`, or `pytest`. |
| `module` | Framework module name, such as `rspec`, `minitest`, `pytest`, or `jest`. |
| `parameters` | Serialized test parameters. |
| `suiteSourceFile` | Source file containing the suite. |

Expand Down
5 changes: 4 additions & 1 deletion docs/local-ci-skippable-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,10 @@ Use `--runtime-tags` to override your local runtime tags with the tags from CI.
For Python projects, `runtime.name` is the Python implementation name (for
example `CPython` for standard Python), and `runtime.version` is the
Python interpreter version used in CI.
Python interpreter version used in CI.

For JavaScript projects, `runtime.name` is `node`, and `runtime.version` is
the Node.js version used in CI, including the leading `v` (for example,
`v22.16.0`).

2. Create the runtime tags JSON.

Expand Down
38 changes: 37 additions & 1 deletion docs/running.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,12 @@ For Python/pytest:
ddtest run --platform python --framework pytest
```

For JavaScript/Jest:

```bash
ddtest run --platform javascript --framework jest
```

On one CI node, the default `--min-parallelism` and `--max-parallelism` equal
the available physical CPU core count, so DDTest can start one worker per
physical core without defaulting to one worker per hyperthread.
Expand All @@ -45,6 +51,12 @@ For Python/pytest:
ddtest run --platform python --framework pytest --ci-node <CI_NODE_INDEX>
```

For JavaScript/Jest:

```bash
ddtest run --platform javascript --framework jest --ci-node <CI_NODE_INDEX>
```

In CI-node mode, DDTest uses one local worker by default so database and other
per-worker resources stay easy to isolate. To fan out within each CI node, set
`--ci-node-workers` to a positive integer, or use `--ci-node-workers ncpu` to
Expand Down Expand Up @@ -78,12 +90,19 @@ starting each worker.

Use `--command` to override the framework's default base test command where
supported. DDTest currently applies this override to RSpec run and full
discovery, Minitest run and full discovery, and Jest run:
discovery, Minitest run and full discovery, and Jest run and file discovery:

```bash
ddtest run --platform ruby --framework rspec --command "bundle exec rspec --profile"
```

For JavaScript/Jest, DDTest automatically appends `--listTests` during planning
and `--runTestsByPath <files>` during execution:

```bash
ddtest run --platform javascript --framework jest --command "pnpm jest --runInBand"
```

When using `--command`, do not include the `--` separator or test files in your
command. DDTest automatically appends selected tests and framework-specific
flags to the command you provide.
Expand Down Expand Up @@ -118,6 +137,23 @@ For Python/pytest, DDTest discovers test files using this priority:
Pytest does not have an equivalent to RSpec's pattern flag, so DDTest resolves
the pattern to explicit file paths before invoking `python -m pytest`.

## Jest Discovery And Instrumentation

For JavaScript/Jest, DDTest discovers test files with Jest's own `--listTests`
command. It uses this priority:

1. `--command` when set, with `--listTests` appended.
2. The local executable `node_modules/.bin/jest` when present.
3. `npx jest`.

Jest uses its own configuration and default test matching for `--listTests`.
When `--tests-location` or `--tests-exclude-pattern` is set, DDTest filters the
file list returned by Jest after discovery; it does not pass `--tests-location`
as Jest's `--testMatch`.

DDTest prepends `-r dd-trace/ci/init` to `NODE_OPTIONS` for worker processes
unless `NODE_OPTIONS` already loads `dd-trace/ci/init`.

## Parallelism Selection

DDTest chooses parallelism by estimating the runnable duration of each test file,
Expand Down
8 changes: 4 additions & 4 deletions docs/settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,17 @@ CLI flags take precedence over environment variables.

| CLI flag | Environment variable | Env alias | Default | What it does |
| --- | --- | --- | ---: | --- |
| `--platform` | `DD_TEST_OPTIMIZATION_RUNNER_PLATFORM` | | `ruby` | Language/platform. Currently supported: `ruby`, `python`. |
| `--framework` | `DD_TEST_OPTIMIZATION_RUNNER_FRAMEWORK` | | `rspec` | Test framework. Currently supported: `rspec`, `minitest`, `pytest`. |
| `--command` | `DD_TEST_OPTIMIZATION_RUNNER_COMMAND` | | `""` | Override the default base test command for supported framework modes. Currently used by RSpec and Minitest run/discovery, and Jest run; pytest ignores it. DDTest appends selected tests and framework-specific flags. For pytest, use `PYTEST_ADDOPTS` for pytest flags. |
| `--platform` | `DD_TEST_OPTIMIZATION_RUNNER_PLATFORM` | | `ruby` | Language/platform. Currently supported: `ruby`, `python`, `javascript`. |
| `--framework` | `DD_TEST_OPTIMIZATION_RUNNER_FRAMEWORK` | | `rspec` | Test framework. Currently supported: `rspec`, `minitest`, `pytest`, `jest`. |
| `--command` | `DD_TEST_OPTIMIZATION_RUNNER_COMMAND` | | `""` | Override the default base test command for supported framework modes. Currently used by RSpec and Minitest run/discovery, and Jest run/discovery; pytest ignores it. DDTest appends selected tests and framework-specific flags. For pytest, use `PYTEST_ADDOPTS` for pytest flags. |
| `--min-parallelism` | `DD_TEST_OPTIMIZATION_RUNNER_MIN_PARALLELISM` | | physical CPU count | Minimum count DDTest considers when planning. Interpret it as CI nodes in CI-node mode, or workers in a single-node run. |
| `--max-parallelism` | `DD_TEST_OPTIMIZATION_RUNNER_MAX_PARALLELISM` | | physical CPU count | Maximum count DDTest considers when planning. Interpret it as CI nodes in CI-node mode, or workers in a single-node run. |
| `--ci-job-overhead` | `DD_TEST_OPTIMIZATION_RUNNER_CI_JOB_OVERHEAD` | | `25s` | Modeled overhead for adding one more CI node. Accepts durations such as `25s`, `1m`, `1500ms`, or `0s` to disable this bias. Increase it to use fewer CI nodes; decrease it to prefer faster wall time. |
| `--target-time` | `DD_TEST_OPTIMIZATION_RUNNER_TARGET_TIME` | | `0s` | Target wall time for the selected split. Accepts durations such as `10m`, `300s`, `1500ms`, or `0s` to disable the target. DDTest first considers splits at or below this wall time; if none are possible within the min/max parallelism range, it warns and selects the split with the lowest expected wall time, ignoring CI job overhead, to get as close as possible to the target. |
| `--ci-node` | `DD_TEST_OPTIMIZATION_RUNNER_CI_NODE` | | `-1` (off) | Restrict this run to files assigned to CI node **N** (0-indexed). |
| `--ci-node-workers` | `DD_TEST_OPTIMIZATION_RUNNER_CI_NODE_WORKERS` | | `1` | Number of workers to start on this CI node. Use a positive integer, or `ncpu` to use the node's available physical CPU cores. |
| `--worker-env` | `DD_TEST_OPTIMIZATION_RUNNER_WORKER_ENV` | | `""` | Template env vars per worker: `--worker-env "DATABASE_NAME_TEST=app_test{{nodeIndex}}_{{workerIndex}}"`. `{{nodeIndex}}` is the CI node index (`0` for single-node runs); `{{workerIndex}}` is the worker process index within that CI node. |
| `--tests-location` | `DD_TEST_OPTIMIZATION_RUNNER_TESTS_LOCATION` | `KNAPSACK_PRO_TEST_FILE_PATTERN` | `""` | Custom glob pattern to discover test files, such as `--tests-location "custom/spec/**/*_spec.rb"` or `--tests-location "tests/**/*_test.py"`. Defaults to `spec/**/*_spec.rb` for RSpec, `test/**/*_test.rb` for Minitest, and pytest config or `**/{test_*,*_test}.py` for pytest. |
| `--tests-location` | `DD_TEST_OPTIMIZATION_RUNNER_TESTS_LOCATION` | `KNAPSACK_PRO_TEST_FILE_PATTERN` | `""` | Custom glob pattern to discover test files, such as `--tests-location "custom/spec/**/*_spec.rb"`, `--tests-location "tests/**/*_test.py"`, or `--tests-location "packages/**/__tests__/**/*.test.ts"`. Defaults to `spec/**/*_spec.rb` for RSpec, `test/**/*_test.rb` for Minitest, pytest config or `**/{test_*,*_test}.py` for pytest, and Jest's configured/default test matching for Jest. |
| `--tests-exclude-pattern` | `DD_TEST_OPTIMIZATION_RUNNER_TESTS_EXCLUDE_PATTERN` | `KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN` | `""` | Glob pattern to exclude test files from discovery, such as `--tests-exclude-pattern "spec/system/**/*_spec.rb"`. |
| `--test-discovery-cache` | `DD_TEST_OPTIMIZATION_RUNNER_TEST_DISCOVERY_CACHE` | | `""` | Path to a restored test discovery cache file. DDTest imports it before planning and refreshes the internal discovery cache after successful full discovery. |
| `--force-full-test-discovery` | `DD_TEST_OPTIMIZATION_RUNNER_FORCE_FULL_TEST_DISCOVERY` | | `false` | Force full test discovery when the framework supports it, including in suite-level skipping mode. |
Expand Down
15 changes: 15 additions & 0 deletions docs/third-party-runners.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,21 @@ if [ -s .testoptimization/runner/test-files.txt ]; then
fi
```

## Jest

When another runner consumes DDTest's file list for Jest, make sure the
`dd-trace` Test Optimization init is loaded the same way `ddtest run` loads it:

```bash
case "${NODE_OPTIONS:-}" in
*dd-trace/ci/init*) ;;
*) export NODE_OPTIONS="-r dd-trace/ci/init${NODE_OPTIONS:+ $NODE_OPTIONS}" ;;
esac
if [ -s .testoptimization/runner/test-files.txt ]; then
xargs ./node_modules/.bin/jest --runTestsByPath < .testoptimization/runner/test-files.txt
fi
```

## Custom Runners

Read `.testoptimization/runner/test-files.txt` when your runner should handle
Expand Down
6 changes: 4 additions & 2 deletions docs/upgrade-1.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,12 @@ Before upgrading to 1.0, update CI jobs and custom integrations to consume the

## Required Changes

Minimum supported library versions:
Minimum supported library and runtime requirements:

- Ruby requires the `datadog-ci` gem 1.31.0 or higher.
- Python requires the `ddtrace` package 4.10.3 or higher and `pytest`.
- JavaScript requires the `dd-trace` package 6.0.0 or higher, Node.js, and
Comment thread
cbasitodx marked this conversation as resolved.
Jest.

DDTest 1.0 writes plan files under `.testoptimization/runner/*`. If your
scripts, CI jobs, or custom test runners read DDTest plan files directly, update
Expand All @@ -26,7 +28,7 @@ these paths:

1. Verify the Datadog Test Optimization library version for your platform:
`datadog-ci` 1.31.0 or higher for Ruby, or `ddtrace` 4.10.3 or higher for
Python.
Python, or `dd-trace` 6.0.0 or higher for JavaScript.
2. Remove references to legacy root plan paths from CI templates and custom scripts.
3. Run `ddtest plan` in CI.
4. Run one CI shard with `DD_TEST_OPTIMIZATION_RUNNER_CI_NODE=0 ddtest run`.
Expand Down
Loading