Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/howto/set-up-continuous-integration-for-a-charm.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ If your charm is a Kubernetes charm, add the following job to `.github/workflows
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v7
with:
name: juju-dump-logs
name: integration-test-logs
path: logs
```

Expand Down
149 changes: 143 additions & 6 deletions docs/howto/write-integration-tests-for-a-charm.md
Original file line number Diff line number Diff line change
Expand Up @@ -460,22 +460,159 @@ tox -e integration -- tests/integration/test_charm.py -k "not test_one"
> - [`pytest | How to invoke pytest`](https://docs.pytest.org/en/7.1.x/how-to/usage.html)
> - [](#validate-your-charm-with-every-change)

(write-integration-tests-for-a-charm-configuring-jubilant-logs)=
## Configuring Jubilant logs

(write-integration-tests-for-a-charm-view-juju-logs)=
## View Juju logs
Jubilant emits logs during the integration tests. These logs are handled by pytest if you are using `pytest-jubilant`. We can configure this behavior in `pyproject.toml`.

If you initialised the charm with `charmcraft init`, the `[tool.pytest.ini_options]` table already exist in `pyproject.toml`. For example:

```toml
[tool.pytest.ini_options]
...
```

You can freely configure it to your needs.

> See more:
> - [`pytest | How to manage loggings`](https://docs.pytest.org/en/stable/reference/reference.html#logging)
> - [`pytest` | pyproject.toml](https://docs.pytest.org/en/stable/reference/customize.html#pyproject-toml)

The sections below cover the configuration that we suggest.

### Brief mode

```
[tool.pytest.ini_options]
...
log_cli = true
log_cli_level = "INFO"
log_cli_format = "%(levelname)s %(name)s %(message)s"
```

While the test is running, logs with level `INFO` and above are emitted live to the console.

### Verbose mode

This mode shows whatever is in Brief, and the verbose `DEBUG` level logs. Lowering the log level is all it needs to switch from Brief to Verbose mode.

```toml
[tool.pytest.ini_options]
...
log_cli = true
log_cli_level = "DEBUG"
log_cli_format = "%(asctime)s %(levelname)s %(name)s %(message)s"
log_cli_date_format = "%Y-%m-%dT%H:%M:%SZ"
```

We define the timestamps format with the `log_cli_date_format` key, following ISO 8601.

> See more:
> - [`datetime | strftime() and strptime() format codes`](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes)

You can remove the timestamps without changing the mode, but we recommend leaving it in.

### Error mode

Jubilant emits logs at `ERROR` level when an application or unit transition into `error` during `juju.wait()`. This mode is useful if you only care if any charm turns into error status.

```toml
[tool.pytest.ini_options]
...
log_cli = true
log_cli_level = "ERROR"
log_cli_format = "%(levelname)s %(name)s %(message)s"
```

(write-integration-tests-log-to-a-file-mode)=
### Log to a file

This is an ideal configuration for long-running integration tests (for example, in CI). It uses Brief mode for logging to the console, and Verbose mode for logging to a local file.

```toml
[tool.pytest.ini_options]
...

log_level = "INFO"

log_cli = true
log_cli_level = "INFO"
log_cli_format = "%(levelname)s %(name)s %(message)s"

log_file = "logs/verbose.log"
log_file_level = "DEBUG"
log_file_format = "%(asctime)s %(levelname)s %(name)s %(message)s"
log_file_date_format = "%Y-%m-%dT%H:%M:%SZ"
```

When a test item fails, pytest prints all logs captured during the test to the terminal:

```
------- Captured log call -------
...
```

`log_level = "INFO"` is intentional to keep the `Captured log call` consistent with Brief mode. If you want Verbose mode in `Captured log call`, you can set `log_level` to `DEBUG`.

The local file where Verbose mode logs land is `logs/verbose.log`. It contains logs from one pytest session. If you want to run multiple test suites, and store Verbose logs in separated files. You can override `log_file` for each pytest invocation:

If any tests fail, `pytest-jubilant` automatically prints the last 1000 lines of `juju debug-log` to stderr. You can also save the complete logs to disk with the `--juju-dump-logs` option.
```
pytest --log-file "run1.log" ...
pytest --log-file "run2.log" ...
```

Alternatively, you can set `log-file-mode` to `append`. Pytest will put verbose logs from all invocations into one file:

```
[tool.pytest.ini_options]
log-file-mode = "a"
...
```

Use `--juju-dump-logs` in CI with `actions/upload-artifact` to make debug logs available as build artifacts:
Use this mode in CI with `actions/upload-artifact` to make `logs/verbose.log` available as build artifacts:

```yaml
# In your integration test job
- run: tox -e integration -- --juju-dump-logs logs
- run: tox -e integration
- name: Upload logs
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v7
with:
name: integration-test-logs
path: logs
```

### Behavior when no pytest's logging config is set

This section covers the behavior when no logging configuration is set, either in config files, or from CLI arguments. In this case, pytest captures all log messages at `WARNING` level or above, and there is no logging to a file. You can still see messages from:

- `pytest`. For example: `tests/integration/test_charm.py::test_deploy PASSED`
- Logs from other modules if they are at `WARNING` or above.
- Usage hints from `pytest-jubilant`. For example: `Models were torn down. To keep models available for subsequent test runs or manual debugging, pass the following:
--no-juju-teardown`

(write-integration-tests-for-a-charm-view-juju-logs)=
## View `juju debug-log` logs

If any tests fail, `pytest-jubilant` automatically logs the last 1000 lines of `juju debug-log` at `DEBUG` level. Log messages from `pytest-jubilant` are handled depending on how you configure pytest (see [](write-integration-tests-for-a-charm-configuring-jubilant-logs):

- For Brief mode and Error mode: They don't appear anywhere.
- For Verbose mode: They appear in the terminal.
- For Log to a file: They appear in `log_file`, and don't appear in the terminal.

You can also save the complete `juju debug-log` to disk with the `--juju-dump-logs` option.

Use `--juju-dump-logs` in CI with `actions/upload-artifact` to make `juju debug-log` files available as build artifacts. We can modify the GitHub Actions from [](write-integration-tests-log-to-a-file-mode):

```diff
# In your integration test job
- - run: tox -e integration
+ - run: tox -e integration -- --juju-dump-logs logs
- name: Upload logs
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v7
with:
name: juju-dump-logs
name: integration-test-logs

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I rename to keep it general, as this artifact now contain both the log file generated by pytest, and the files generated by pytest-jubilant.

path: logs
```

Expand Down
Loading