Benchmark network

benchmark/README.md

@@ -1,3 +1,90 @@
 # Benchmark
 
 This folder contains AosEdge services for benchmark tests.
+
+Every item follows the same shape, which `services/template/py` and `services/template/cpp` define: a `config.yaml`
+and a `src/`, one image for any architecture, and results reported to VictoriaMetrics as `benchmark_result` samples
+bracketed by `checkpoint_event` Start/Stop, so that everything lands in the same Grafana tables regardless of which
+benchmark produced it.
+
+## Network performance
+
+`services/network/` covers the network chapter of the benchmark plan: what a service actually gets out of the
+container network AosCore builds for it — `veth`, a bridge, nftables, `tc` and `dnsmasq`.
+
+Three groups, one folder each:
+
+| Group                     | Measures                                                | Tool       |
+| ------------------------- | ------------------------------------------------------- | ---------- |
+| [`bandwidth`](services/network/bandwidth) | Throughput, TCP and UDP, both directions, plus UDP jitter and loss | `iperf3`   |
+| [`latency`](services/network/latency)     | Round trip time as percentiles, TCP and UDP             | `sockperf` |
+| [`dns`](services/network/dns)             | Name resolution time as percentiles                     | built in   |
+
+### Scenarios
+
+Each group is exercised in the same three scenarios. A group is one folder with one `config.yaml`, and the scenarios
+are items inside it — the client code is written once and pointed at different things:
+
+```
+services/network/<group>/src/client/     the client, one copy
+services/network/<group>/src/server/     the server, where the scenario needs one in a container
+services/network/<group>/config.yaml     one item per scenario, plus the server
+```
+
+The container is always the client; only the server side moves. A scenario is a different value of one environment
+variable, not different code, which is why the items share the sources rather than duplicating them.
+
+Install only the items a run needs. Every installed client generates traffic, so leaving all three in place would
+have them measure each other's interference.
+
+| Scenario           | Server side                                         | Measures                                     |
+| ------------------ | --------------------------------------------------- | -------------------------------------------- |
+| service -> service | a second item in the same group                     | two containers on one node and one bridge     |
+| service -> unit    | a plain process on the node                         | the container to node/gateway path            |
+| service -> external| a process on a machine outside the unit             | egress through masquerade to a LAN host        |
+
+The two scenarios whose server lives outside the container need it started by hand — each group's README gives the
+exact commands. One rule spans all of them: **bind the server to the address the client dials**. A node has more than
+one address on the path to a container, and without an explicit bind the reply is routed back over the service bridge
+and carries the bridge address as its source. Both `iperf3` and `sockperf` connect their sockets, so the kernel drops
+datagrams arriving from another address, and the symptom is characteristic — UDP fails while TCP passes.
+
+### Results
+
+Every client reports twice, and the split is deliberate.
+
+The **service log** gets the full result of each test: the tool's own output, error text, counters and per-interval
+detail. That is what makes a failed or surprising run explicable afterwards, and none of it belongs in a time series.
+
+**VictoriaMetrics** gets only what is worth charting — one `benchmark_result` sample per measured value, labeled by
+`name`, with `source` set to the instance's `AOS_INSTANCE_ID` so instances are told apart once a scenario is run at
+scale.
+
+| Group     | Samples pushed                                                         |
+| --------- | ---------------------------------------------------------------------- |
+| bandwidth | `<test> throughput, Mbps`, and for UDP `<test> loss, %` and `<test> jitter, ms` |
+| latency   | `<test> p50, us`, `<test> p99, us`, `<test> p999, us`                  |
+| dns       | `resolve p50, us`, `resolve p99, us`, `resolve p999, us`               |
+
+Latency and DNS report percentiles rather than averages on purpose. Their distributions are skewed: most samples sit
+near the minimum and a thin tail runs orders of magnitude longer, so an average hides exactly the behaviour that
+real-time and RPC traffic feel. A percentile is only worth the samples behind it, though — `p999` needs thousands of
+samples before it means anything.
+
+### Interpreting the figures
+
+On a unit that runs as a VM, none of the three scenarios crosses a physical wire: traffic moves through `veth`, a
+bridge and `tap`, in RAM. Throughput figures there are bounded by how fast one core can move data rather than by any
+link, which the `cpu_utilization_percent` section in `iperf3`'s output shows directly, so they should be read as a
+floor. Latency and DNS figures on the same paths mostly characterise the node's scheduling behaviour. The numbers
+start describing a network once the external scenario points at a host reached over a real interface.
+
+### Running at scale
+
+The benchmark plan repeats every measurement at 1, 16, 256 and 1024 instances. `config.yaml` keeps `minInstances: 1`
+throughout — instance count is a deploy time decision, not a property of the item — and every client tags its samples
+with `source`, so results from many instances stay separable.
+
+One caveat worth knowing before scaling up: every deployed client generates traffic, so a large deployment is the
+plan's *active-load* mode. The *idle-density* mode, where one pair measures while the rest merely exist, needs
+something the current items do not have — a way to keep most instances silent.

benchmark/services/network/bandwidth/README.md

@@ -0,0 +1,137 @@
+# Bandwidth
+
+Throughput through the container network, measured with [`iperf3`](https://iperf.fr/) for TCP and UDP in both
+directions, together with UDP jitter and packet loss.
+
+`config.yaml` holds four items: one server, and one client per scenario. The client code is the same for all three —
+a scenario is a different value of one environment variable, not different code.
+
+| Item                                          | Scenario           | `TARGET`           | Server side           |
+| --------------------------------------------- | ------------------ | ------------------ | --------------------- |
+| `benchmark-network-bandwidth-server`           | —                  | —                  | this bundle           |
+| `benchmark-network-bandwidth-client-service`   | service -> service | `bandwidth-server` | the server item above |
+| `benchmark-network-bandwidth-client-unit`      | service -> unit    | `10.0.0.100`       | `iperf3` on the node  |
+| `benchmark-network-bandwidth-client-external`  | service -> external| `10.0.0.1`         | `iperf3` on the host  |
+
+Install only the items a run needs: three clients installed at once would measure each other's interference.
+
+## What the client measures
+
+Four tests in a row against `TARGET`, each for `DURATION` seconds:
+
+| Test       | `iperf3` arguments        | Direction        |
+| ---------- | ------------------------- | ---------------- |
+| `tcp_up`   | (none)                    | client -> server |
+| `tcp_down` | `-R`                      | server -> client |
+| `udp_up`   | `-u -b $UDP_BANDWIDTH`    | client -> server |
+| `udp_down` | `-u -b $UDP_BANDWIDTH -R` | server -> client |
+
+`up` is the container sending, `down` is it receiving. Both are worth measuring because the stack shapes the two
+directions separately, and because the sending side is the one that saturates a core, which the
+`cpu_utilization_percent` section of the log shows directly.
+
+The first test is retried for up to 60 seconds while `iperf3` reports a connection or name resolution failure, so the
+run survives being started before its server. Later tests are not retried.
+
+Tests are spaced 3 seconds apart. An `iperf3` server runs one test at a time and needs a moment to reset between
+them: starting the next test the instant the previous one ends makes it fail on the control connection, most often
+right after an unlimited UDP test, which leaves the server draining its buffers. Measured on a unit, back to back
+tests failed in two runs out of three, and none failed with the gap.
+
+After the last test the instance stays alive idling, which keeps its logs available instead of having the unit
+restart it in a loop.
+
+## Environment
+
+| Variable        | Default  | Meaning                                          |
+| --------------- | -------- | ------------------------------------------------ |
+| `TARGET`        | per item | Server hostname or IP. Required.                  |
+| `DURATION`      | `5`      | Length of every single test, in seconds.          |
+| `PORT`          | `5201`   | `iperf3` port, the same for client and server.    |
+| `UDP_BANDWIDTH` | `0`      | Target rate for the UDP tests (`0` is unlimited). |
+
+## Results
+
+Two destinations, on purpose.
+
+The **log** gets every test in full, `iperf3`'s own JSON document included, as one line per test: the error text, the
+retransmit counts, the per-second intervals and the CPU utilisation of both sides. That is what makes a failed or
+surprising run explicable afterwards, and none of it belongs in a time series.
+
+**VictoriaMetrics** gets only what is worth charting, pushed as `benchmark_result` samples bracketed by
+`checkpoint_event` Start/Stop, in the same shape as `services/template/py`:
+
+| Sample name                | From          |
+| -------------------------- | ------------- |
+| `<test> throughput, Mbps`  | every test    |
+| `<test> loss, %`           | the UDP tests |
+| `<test> jitter, ms`        | the UDP tests |
+
+The `source` label is the instance's `AOS_INSTANCE_ID`, which is what tells instances apart once a scenario is run at
+scale.
+
+## Setting up each scenario
+
+**service -> service** needs nothing beyond installing both items: the server sets `hostname: bandwidth-server` and
+the client reaches it by that name.
+
+**service -> unit** needs an `iperf3` server on the node, bound to the address the client dials:
+
+```console
+setsid iperf3 -s -p 5201 -B 10.0.0.100 </dev/null >/tmp/iperf3-unit.log 2>&1 &
+```
+
+**service -> external** needs the same on the host outside the unit:
+
+```console
+setsid iperf3 -s -p 5201 -B 10.0.0.1 </dev/null >/tmp/iperf3-external.log 2>&1 &
+```
+
+`10.0.0.1` is the host's address on the bridge carrying the unit's network — the address that faces the unit, and a
+stable one, unlike the Aos service bridge that is recreated with a new subnet on every deployment.
+
+### Why the bind matters
+
+A node has more than one address on the path to a container, and without an explicit bind the reply is routed back
+over the service bridge and carries the bridge address as its source. `iperf3` connects its UDP socket, so the kernel
+drops datagrams arriving from another address — and the symptom is characteristic: the UDP tests fail while the TCP
+ones pass. Verified on a unit, where UDP failed against `10.0.0.100` without `-B` and worked with it.
+
+Binding the bridge address instead would also work, but it is a worse choice: that bridge does not exist at boot and
+changes between deployments.
+
+`setsid` matters when starting a server over SSH: a plain `&` leaves it attached to the session and it dies on
+logout, which then looks like the benchmark failing to reach the far side.
+
+### Before deploying
+
+The server item's own `iperf3` listens on 5201 too, but inside a container namespace, so it does not occupy the
+node's port; a leftover from an earlier run does. On Debian and Ubuntu the `iperf3` package also ships an enabled
+`iperf3.service` on `*:5201` — either use it as is, since it answers on every address, or take the port over with
+`sudo systemctl disable --now iperf3`.
+
+```console
+ss -lntu | grep 5201
+iperf3 -c 10.0.0.100 -p 5201 -t 2
+```
+
+## Requirements
+
+`iperf3` and `python3` must be in the container rootfs, which they are: service containers run on the node rootfs,
+and `aos-image-vm` installs both.
+
+`tmpLimit` is required rather than decorative. `iperf3` creates a temporary buffer file under `/tmp` for every
+stream, and the container rootfs is read-only, so without that quota every test fails on both sides with
+`unable to create a new stream: Read-only file system`.
+
+## Reading the numbers
+
+Between two instances on one node there is no wire: the traffic goes through `veth` and a bridge, in RAM. TCP lands
+in the tens of Gbit/s with the sending side pinned at ~100% of a core, which makes the figure a floor set by CPU
+rather than the capacity of a link. Raising `cpuLimit` raises the number, which is the clearest sign of what is
+actually being measured.
+
+UDP with `UDP_BANDWIDTH=0` measures something narrower still: `iperf3` sends 1448 byte datagrams, so the test becomes
+a syscall rate benchmark and reports several times less than TCP on the same path. That is not UDP being slower; it
+is the datagram size. Loss reported under those conditions is the receiver failing to keep up, not the network
+dropping traffic.

benchmark/services/network/bandwidth/config.yaml

@@ -0,0 +1,114 @@
+schemaVersion: 2
+publisher:
+  author: Developer Name
+  company: Company Name
+publish:
+  tlsKey: aos-user-sp.p12
+items:
+- identity:
+    type: service
+    codename: benchmark-network-bandwidth-server
+    title: Benchmark network bandwidth server
+    description: iperf3 server for the network bandwidth benchmark
+  version: 1.0.0-beta.1
+  images:
+  - sourceFolder: src/server
+    archInfo:
+      architecture: any
+  configuration:
+    workingDir: /
+    cmd: /usr/bin/python3 -u bandwidth_server.py
+    instances:
+      minInstances: 1
+    skipResourceLimits: true
+    quotas:
+      cpuLimit: 10000
+      ramLimit: 256MiB
+      tmpLimit: 64MiB
+    hostname: bandwidth-server
+    env:
+    - PORT=5201
+- identity:
+    type: service
+    codename: benchmark-network-bandwidth-client-service
+    title: Benchmark network bandwidth client (service to service)
+    description: Throughput between two services on the same unit
+  version: 1.0.0-beta.1
+  images:
+  - sourceFolder: src/client
+    archInfo:
+      architecture: any
+  configuration:
+    workingDir: /
+    cmd: /usr/bin/python3 -u bandwidth_client.py
+    instances:
+      minInstances: 1
+    skipResourceLimits: true
+    quotas:
+      cpuLimit: 10000
+      ramLimit: 256MiB
+      tmpLimit: 64MiB
+    resources:
+    - name: victoria-metrics
+      mode: rw
+    env:
+    - TARGET=bandwidth-server
+    - DURATION=5
+    - PORT=5201
+    - UDP_BANDWIDTH=0
+- identity:
+    type: service
+    codename: benchmark-network-bandwidth-client-unit
+    title: Benchmark network bandwidth client (service to unit)
+    description: Throughput between a service and the node it runs on
+  version: 1.0.0-beta.1
+  images:
+  - sourceFolder: src/client
+    archInfo:
+      architecture: any
+  configuration:
+    workingDir: /
+    cmd: /usr/bin/python3 -u bandwidth_client.py
+    instances:
+      minInstances: 1
+    skipResourceLimits: true
+    quotas:
+      cpuLimit: 10000
+      ramLimit: 256MiB
+      tmpLimit: 64MiB
+    resources:
+    - name: victoria-metrics
+      mode: rw
+    env:
+    - TARGET=10.0.0.100
+    - DURATION=5
+    - PORT=5201
+    - UDP_BANDWIDTH=0
+- identity:
+    type: service
+    codename: benchmark-network-bandwidth-client-external
+    title: Benchmark network bandwidth client (service to external)
+    description: Throughput between a service and a host outside the unit
+  version: 1.0.0-beta.1
+  images:
+  - sourceFolder: src/client
+    archInfo:
+      architecture: any
+  configuration:
+    workingDir: /
+    cmd: /usr/bin/python3 -u bandwidth_client.py
+    instances:
+      minInstances: 1
+    skipResourceLimits: true
+    quotas:
+      cpuLimit: 10000
+      ramLimit: 256MiB
+      tmpLimit: 64MiB
+    resources:
+    - name: victoria-metrics
+      mode: rw
+    env:
+    - TARGET=10.0.0.1
+    - DURATION=5
+    - PORT=5201
+    - UDP_BANDWIDTH=0