From eaa637a0e9d49a5f2e4015f45af9e8f01aeb5828 Mon Sep 17 00:00:00 2001 From: Mykola Solianko Date: Wed, 12 Aug 2026 11:46:44 +0300 Subject: [PATCH 1/4] benchmark: add network bandwidth services Move the iperf3 throughput benchmark into the new structure: one folder per benchmark group, a single src/ built for any architecture rather than a copy per architecture, and one item per scenario differing only in the TARGET environment variable, so nothing about a scenario is baked into the code. Results now go two ways. The log keeps every test in full, iperf3's own JSON document included, which is what makes a failed run diagnosable afterwards. VictoriaMetrics gets only what is worth a time series - throughput per test, plus loss and jitter for the UDP ones - pushed as benchmark_result samples bracketed by checkpoint_event Start/Stop, in the same shape as services/template/py. Both scenarios whose server runs outside the container need it bound to the address the client dials. Without that the reply carries the service bridge address as source, iperf3 connects its UDP socket, and the kernel drops the mismatched datagrams: the UDP tests fail while the TCP ones pass. Signed-off-by: Mykola Solianko --- .../services/network/bandwidth/README.md | 137 ++++++++ .../services/network/bandwidth/config.yaml | 114 +++++++ .../bandwidth/src/client/bandwidth_client.py | 301 ++++++++++++++++++ .../bandwidth/src/server/bandwidth_server.py | 23 ++ 4 files changed, 575 insertions(+) create mode 100644 benchmark/services/network/bandwidth/README.md create mode 100644 benchmark/services/network/bandwidth/config.yaml create mode 100644 benchmark/services/network/bandwidth/src/client/bandwidth_client.py create mode 100644 benchmark/services/network/bandwidth/src/server/bandwidth_server.py diff --git a/benchmark/services/network/bandwidth/README.md b/benchmark/services/network/bandwidth/README.md new file mode 100644 index 0000000..c1bdaa7 --- /dev/null +++ b/benchmark/services/network/bandwidth/README.md @@ -0,0 +1,137 @@ +# Bandwidth + +Throughput through the container network, measured with [`iperf3`](https://iperf.fr/) for TCP and UDP in both +directions, together with UDP jitter and packet loss. + +`config.yaml` holds four items: one server, and one client per scenario. The client code is the same for all three — +a scenario is a different value of one environment variable, not different code. + +| Item | Scenario | `TARGET` | Server side | +| --------------------------------------------- | ------------------ | ------------------ | --------------------- | +| `benchmark-network-bandwidth-server` | — | — | this bundle | +| `benchmark-network-bandwidth-client-service` | service -> service | `bandwidth-server` | the server item above | +| `benchmark-network-bandwidth-client-unit` | service -> unit | `10.0.0.100` | `iperf3` on the node | +| `benchmark-network-bandwidth-client-external` | service -> external| `10.0.0.1` | `iperf3` on the host | + +Install only the items a run needs: three clients installed at once would measure each other's interference. + +## What the client measures + +Four tests in a row against `TARGET`, each for `DURATION` seconds: + +| Test | `iperf3` arguments | Direction | +| ---------- | ------------------------- | ---------------- | +| `tcp_up` | (none) | client -> server | +| `tcp_down` | `-R` | server -> client | +| `udp_up` | `-u -b $UDP_BANDWIDTH` | client -> server | +| `udp_down` | `-u -b $UDP_BANDWIDTH -R` | server -> client | + +`up` is the container sending, `down` is it receiving. Both are worth measuring because the stack shapes the two +directions separately, and because the sending side is the one that saturates a core, which the +`cpu_utilization_percent` section of the log shows directly. + +The first test is retried for up to 60 seconds while `iperf3` reports a connection or name resolution failure, so the +run survives being started before its server. Later tests are not retried. + +Tests are spaced 3 seconds apart. An `iperf3` server runs one test at a time and needs a moment to reset between +them: starting the next test the instant the previous one ends makes it fail on the control connection, most often +right after an unlimited UDP test, which leaves the server draining its buffers. Measured on a unit, back to back +tests failed in two runs out of three, and none failed with the gap. + +After the last test the instance stays alive idling, which keeps its logs available instead of having the unit +restart it in a loop. + +## Environment + +| Variable | Default | Meaning | +| --------------- | -------- | ------------------------------------------------ | +| `TARGET` | per item | Server hostname or IP. Required. | +| `DURATION` | `5` | Length of every single test, in seconds. | +| `PORT` | `5201` | `iperf3` port, the same for client and server. | +| `UDP_BANDWIDTH` | `0` | Target rate for the UDP tests (`0` is unlimited). | + +## Results + +Two destinations, on purpose. + +The **log** gets every test in full, `iperf3`'s own JSON document included, as one line per test: the error text, the +retransmit counts, the per-second intervals and the CPU utilisation of both sides. That is what makes a failed or +surprising run explicable afterwards, and none of it belongs in a time series. + +**VictoriaMetrics** gets only what is worth charting, pushed as `benchmark_result` samples bracketed by +`checkpoint_event` Start/Stop, in the same shape as `services/template/py`: + +| Sample name | From | +| -------------------------- | ------------- | +| ` throughput, Mbps` | every test | +| ` loss, %` | the UDP tests | +| ` jitter, ms` | the UDP tests | + +The `source` label is the instance's `AOS_INSTANCE_ID`, which is what tells instances apart once a scenario is run at +scale. + +## Setting up each scenario + +**service -> service** needs nothing beyond installing both items: the server sets `hostname: bandwidth-server` and +the client reaches it by that name. + +**service -> unit** needs an `iperf3` server on the node, bound to the address the client dials: + +```console +setsid iperf3 -s -p 5201 -B 10.0.0.100 /tmp/iperf3-unit.log 2>&1 & +``` + +**service -> external** needs the same on the host outside the unit: + +```console +setsid iperf3 -s -p 5201 -B 10.0.0.1 /tmp/iperf3-external.log 2>&1 & +``` + +`10.0.0.1` is the host's address on the bridge carrying the unit's network — the address that faces the unit, and a +stable one, unlike the Aos service bridge that is recreated with a new subnet on every deployment. + +### Why the bind matters + +A node has more than one address on the path to a container, and without an explicit bind the reply is routed back +over the service bridge and carries the bridge address as its source. `iperf3` connects its UDP socket, so the kernel +drops datagrams arriving from another address — and the symptom is characteristic: the UDP tests fail while the TCP +ones pass. Verified on a unit, where UDP failed against `10.0.0.100` without `-B` and worked with it. + +Binding the bridge address instead would also work, but it is a worse choice: that bridge does not exist at boot and +changes between deployments. + +`setsid` matters when starting a server over SSH: a plain `&` leaves it attached to the session and it dies on +logout, which then looks like the benchmark failing to reach the far side. + +### Before deploying + +The server item's own `iperf3` listens on 5201 too, but inside a container namespace, so it does not occupy the +node's port; a leftover from an earlier run does. On Debian and Ubuntu the `iperf3` package also ships an enabled +`iperf3.service` on `*:5201` — either use it as is, since it answers on every address, or take the port over with +`sudo systemctl disable --now iperf3`. + +```console +ss -lntu | grep 5201 +iperf3 -c 10.0.0.100 -p 5201 -t 2 +``` + +## Requirements + +`iperf3` and `python3` must be in the container rootfs, which they are: service containers run on the node rootfs, +and `aos-image-vm` installs both. + +`tmpLimit` is required rather than decorative. `iperf3` creates a temporary buffer file under `/tmp` for every +stream, and the container rootfs is read-only, so without that quota every test fails on both sides with +`unable to create a new stream: Read-only file system`. + +## Reading the numbers + +Between two instances on one node there is no wire: the traffic goes through `veth` and a bridge, in RAM. TCP lands +in the tens of Gbit/s with the sending side pinned at ~100% of a core, which makes the figure a floor set by CPU +rather than the capacity of a link. Raising `cpuLimit` raises the number, which is the clearest sign of what is +actually being measured. + +UDP with `UDP_BANDWIDTH=0` measures something narrower still: `iperf3` sends 1448 byte datagrams, so the test becomes +a syscall rate benchmark and reports several times less than TCP on the same path. That is not UDP being slower; it +is the datagram size. Loss reported under those conditions is the receiver failing to keep up, not the network +dropping traffic. diff --git a/benchmark/services/network/bandwidth/config.yaml b/benchmark/services/network/bandwidth/config.yaml new file mode 100644 index 0000000..0383838 --- /dev/null +++ b/benchmark/services/network/bandwidth/config.yaml @@ -0,0 +1,114 @@ +schemaVersion: 2 +publisher: + author: Developer Name + company: Company Name +publish: + tlsKey: aos-user-sp.p12 +items: +- identity: + type: service + codename: benchmark-network-bandwidth-server + title: Benchmark network bandwidth server + description: iperf3 server for the network bandwidth benchmark + version: 1.0.0-beta.1 + images: + - sourceFolder: src/server + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u bandwidth_server.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 256MiB + tmpLimit: 64MiB + hostname: bandwidth-server + env: + - PORT=5201 +- identity: + type: service + codename: benchmark-network-bandwidth-client-service + title: Benchmark network bandwidth client (service to service) + description: Throughput between two services on the same unit + version: 1.0.0-beta.1 + images: + - sourceFolder: src/client + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u bandwidth_client.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 256MiB + tmpLimit: 64MiB + resources: + - name: victoria-metrics + mode: rw + env: + - TARGET=bandwidth-server + - DURATION=5 + - PORT=5201 + - UDP_BANDWIDTH=0 +- identity: + type: service + codename: benchmark-network-bandwidth-client-unit + title: Benchmark network bandwidth client (service to unit) + description: Throughput between a service and the node it runs on + version: 1.0.0-beta.1 + images: + - sourceFolder: src/client + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u bandwidth_client.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 256MiB + tmpLimit: 64MiB + resources: + - name: victoria-metrics + mode: rw + env: + - TARGET=10.0.0.100 + - DURATION=5 + - PORT=5201 + - UDP_BANDWIDTH=0 +- identity: + type: service + codename: benchmark-network-bandwidth-client-external + title: Benchmark network bandwidth client (service to external) + description: Throughput between a service and a host outside the unit + version: 1.0.0-beta.1 + images: + - sourceFolder: src/client + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u bandwidth_client.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 256MiB + tmpLimit: 64MiB + resources: + - name: victoria-metrics + mode: rw + env: + - TARGET=10.0.0.1 + - DURATION=5 + - PORT=5201 + - UDP_BANDWIDTH=0 diff --git a/benchmark/services/network/bandwidth/src/client/bandwidth_client.py b/benchmark/services/network/bandwidth/src/client/bandwidth_client.py new file mode 100644 index 0000000..6342906 --- /dev/null +++ b/benchmark/services/network/bandwidth/src/client/bandwidth_client.py @@ -0,0 +1,301 @@ +#!/usr/bin/env python3 +"""Bandwidth benchmark item: measures throughput with iperf3 and reports it. + +Runs four tests in a row against TARGET, each for DURATION seconds: TCP and +UDP, in both directions. iperf3 does the measuring; this script only starts it, +parses its JSON and reports the result. + +Results go two ways. Every test's full result, iperf3's own JSON document +included, is printed to the service log, which is what makes a failed run +diagnosable afterwards. Only the few numbers worth charting are pushed to +VictoriaMetrics as benchmark_result samples, bracketed by checkpoint_event +Start/Stop, exactly as services/template/py does. + +Configuration comes from the environment, so one image serves every scenario +and only the deployment differs: + TARGET server hostname or IP (required) + DURATION seconds per test (default 5) + PORT iperf3 port (default 5201) + UDP_BANDWIDTH target rate for the UDP tests, 0 is unlimited (default 0) + +Usage: + bandwidth_client.py [--victoria-url http://victoriametrics:8428] +""" + +import argparse +import datetime +import json +import os +import subprocess +import sys +import time +import urllib.error +import urllib.request + +TARGET = os.environ.get("TARGET", "") +DURATION = os.environ.get("DURATION", "5") +PORT = os.environ.get("PORT", "5201") +UDP_BANDWIDTH = os.environ.get("UDP_BANDWIDTH", "0") + +CONNECT_ATTEMPTS = 30 +CONNECT_DELAY = 2 +TEST_GAP = 3 +IDLE_DELAY = 60 + +NODE = "main" # No mechanism yet for an instance to learn which node it's actually running on. + +# Test name -> extra iperf3 arguments. "up" is client -> server, "down" is +# server -> client (iperf3 -R, reverse mode). +TESTS = [ + ("tcp_up", "tcp", []), + ("tcp_down", "tcp", ["-R"]), + ("udp_up", "udp", ["-u", "-b", UDP_BANDWIDTH]), + ("udp_down", "udp", ["-u", "-b", UDP_BANDWIDTH, "-R"]), +] + +# iperf3 error fragments that mean the server is not up yet, as opposed to the +# test itself having failed. Only these are worth retrying. +CONNECT_ERRORS = ( + "unable to connect to server", + "unable to send control message", + "unable to receive control message", + "Connection refused", + "No route to host", + "Name or service not known", + "Temporary failure in name resolution", +) + + +def parse_args(): + """Parse --victoria-url command-line option.""" + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--victoria-url", + default="http://victoriametrics:8428", + help="main node's VictoriaMetrics base URL (default: %(default)s)", + ) + return parser.parse_args() + + +def escape_label_value(value): + """Escape a string for safe embedding inside a Prometheus exposition-format label value.""" + return value.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n") + + +def format_precise_time(timestamp_us): + """Format a microsecond epoch timestamp as "YYYY-MM-DD HH:MM:SS.ffffff" (UTC).""" + seconds, microseconds = divmod(timestamp_us, 1_000_000) + dt = datetime.datetime.fromtimestamp(seconds, tz=datetime.timezone.utc) + dt += datetime.timedelta(microseconds=microseconds) + + return dt.strftime("%Y-%m-%d %H:%M:%S.%f") + + +def push_line(victoria_url, line): + """POST a single Prometheus exposition-format line to VictoriaMetrics.""" + request = urllib.request.Request( + f"{victoria_url.rstrip('/')}/api/v1/import/prometheus", + data=line.encode(), + method="POST", + ) + + try: + with urllib.request.urlopen(request, timeout=5) as response: + response.read() + except urllib.error.URLError as err: + print(f"failed to push to VictoriaMetrics: {err}", file=sys.stderr) + + +def push_event(victoria_url, node, source, event): + """Push a checkpoint_event sample (the same metric event_exporter.py produces).""" + timestamp_us = int(time.time() * 1_000_000) + labels = ",".join( + f'{name}="{escape_label_value(value)}"' + for name, value in ( + ("node", node), + ("source", source), + ("event", event), + ("time_us", format_precise_time(timestamp_us)), + ) + ) + time_s = timestamp_us / 1_000_000 + push_line(victoria_url, f"checkpoint_event{{{labels}}} 1 {time_s:.3f}") + + +def push_result(victoria_url, node, source, name, value): + """Push a single benchmark_result sample for one measured value.""" + timestamp_us = int(time.time() * 1_000_000) + labels = ",".join( + f'{label}="{escape_label_value(text)}"' + for label, text in ( + ("node", node), + ("source", source), + ("name", name), + ("time_us", format_precise_time(timestamp_us)), + ) + ) + time_s = timestamp_us / 1_000_000 + push_line(victoria_url, f"benchmark_result{{{labels}}} {value} {time_s:.3f}") + + +def throughput_bps(result, protocol): + """Extract the throughput iperf3 measured, in bits per second. + + TCP results are reported per direction in `sum_sent` / `sum_received`; the + receiver side is what actually arrived, so it is preferred. UDP results + carry the interesting numbers (including loss and jitter) in `sum`. + """ + end = result.get("end", {}) + + if protocol == "udp": + sections = ("sum", "sum_received", "sum_sent") + else: + sections = ("sum_received", "sum_sent", "sum") + + for name in sections: + section = end.get(name) + if isinstance(section, dict) and "bits_per_second" in section: + return section["bits_per_second"] + + return None + + +def udp_stats(result): + """Loss and jitter counters reported for a UDP test, if present.""" + summary = result.get("end", {}).get("sum") + if not isinstance(summary, dict): + return {} + + keys = ("jitter_ms", "lost_packets", "packets", "lost_percent") + + return {key: summary[key] for key in keys if key in summary} + + +def is_connect_error(message): + return any(fragment in message for fragment in CONNECT_ERRORS) + + +def run_iperf(extra_args): + """Run one iperf3 test and return its parsed result and error message.""" + cmd = ["iperf3", "-c", TARGET, "-p", PORT, "-t", DURATION, "-J"] + extra_args + + print(f"Running: {' '.join(cmd)}") + + process = subprocess.run(cmd, capture_output=True, text=True) + + try: + result = json.loads(process.stdout) + except ValueError: + error = process.stderr.strip() or f"iperf3 exited with code {process.returncode}" + return None, error + + # With -J iperf3 reports failures inside the JSON document itself. + return result, result.get("error", "") + + +def run_test(name, protocol, extra_args, attempts=1): + """Run one test, log its whole result, and return the values worth charting. + + The server instance may still be starting when the client starts, so the + first test is given several attempts: otherwise its result would say more + about instance start order than about the network. + """ + for attempt in range(1, attempts + 1): + result, error = run_iperf(extra_args) + + if not error or not is_connect_error(error) or attempt == attempts: + break + + print(f"Server {TARGET}:{PORT} not ready ({attempt}/{attempts}): {error}") + time.sleep(CONNECT_DELAY) + + metric = { + "test": name, + "protocol": protocol, + "target": TARGET, + "port": int(PORT), + "duration_s": int(DURATION), + } + + if error: + metric["error"] = error + else: + metric["throughput_bps"] = throughput_bps(result, protocol) + + if protocol == "udp": + metric.update(udp_stats(result)) + + if result is not None: + metric["raw"] = result + + # The log keeps everything, iperf3's own document included; only a handful + # of numbers are worth a time series. + print(json.dumps(metric)) + + values = {} + + if metric.get("throughput_bps") is not None: + values[f"{name} throughput, Mbps"] = round(metric["throughput_bps"] / 1e6, 3) + + if metric.get("lost_percent") is not None: + values[f"{name} loss, %"] = metric["lost_percent"] + + if metric.get("jitter_ms") is not None: + values[f"{name} jitter, ms"] = metric["jitter_ms"] + + return values + + +def run_benchmark(): + """Run the four tests and return their results as {value_name: value}.""" + results = {} + attempts = CONNECT_ATTEMPTS + + for index, (name, protocol, extra_args) in enumerate(TESTS): + # An iperf3 server runs one test at a time and needs a moment to reset + # between them. Starting the next test the instant the previous one + # ends makes it fail on the control connection, most often right after + # an unlimited UDP test, which leaves the server draining its buffers. + if index: + time.sleep(TEST_GAP) + + results.update(run_test(name, protocol, extra_args, attempts)) + attempts = 1 + + return results + + +def main(): + """Push a start event, run the benchmark, push its results, then push a stop event.""" + args = parse_args() + source = os.environ["AOS_INSTANCE_ID"] + + if not TARGET: + print("TARGET environment variable is required", file=sys.stderr) + return 1 + + print( + f"Bandwidth benchmark: target={TARGET} port={PORT} " + f"duration={DURATION}s udp_bandwidth={UDP_BANDWIDTH}" + ) + + push_event(args.victoria_url, NODE, source, "Start") + + try: + results = run_benchmark() + + for name, value in results.items(): + push_result(args.victoria_url, NODE, source, name, value) + finally: + push_event(args.victoria_url, NODE, source, "Stop") + + print("All tests finished") + + # The benchmark is a one shot run, but the instance keeps running so its + # logs stay available and the unit does not restart it in a loop. + while True: + time.sleep(IDLE_DELAY) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/benchmark/services/network/bandwidth/src/server/bandwidth_server.py b/benchmark/services/network/bandwidth/src/server/bandwidth_server.py new file mode 100644 index 0000000..038fda0 --- /dev/null +++ b/benchmark/services/network/bandwidth/src/server/bandwidth_server.py @@ -0,0 +1,23 @@ +import os +import subprocess +import time + +PORT = os.environ.get("PORT", "5201") + +RESTART_DELAY = 5 + + +def main(): + cmd = ["iperf3", "-s", "-p", PORT] + + # iperf3 -s never exits on its own, so this loop only matters if it dies: + # the service instance stays alive and the server comes back. + while True: + print(f"Starting iperf3 server: {' '.join(cmd)}") + code = subprocess.call(cmd) + print(f"iperf3 server exited with code {code}, restarting in {RESTART_DELAY}s") + time.sleep(RESTART_DELAY) + + +if __name__ == "__main__": + main() From 67cd8e322d617f18f926c1b304bb43903bfc2914 Mon Sep 17 00:00:00 2001 From: Mykola Solianko Date: Wed, 12 Aug 2026 11:46:44 +0300 Subject: [PATCH 2/4] benchmark: add network latency services Move the sockperf round trip time benchmark into the new structure, alongside bandwidth: one folder per benchmark group, a single src/ built for any architecture, and one item per scenario differing only in the TARGET environment variable. The client reports percentiles rather than an average, since latency distributions are skewed and an average hides the tail that real-time and RPC traffic feel. The log keeps every test in full, sockperf's own report included, which matters because the tool has no machine readable output and everything is parsed out of that text. VictoriaMetrics gets the three percentiles the benchmark plan names, pushed as benchmark_result samples bracketed by checkpoint_event Start/Stop. sockperf listens either on UDP or, with --tcp, on TCP, never on both, so the server side runs one of each; TCP and UDP port numbers are independent, so both use the same port. Signed-off-by: Mykola Solianko --- benchmark/services/network/latency/README.md | 142 ++++++++ .../services/network/latency/config.yaml | 114 +++++++ .../latency/src/client/latency_client.py | 315 ++++++++++++++++++ .../latency/src/server/latency_server.py | 40 +++ 4 files changed, 611 insertions(+) create mode 100644 benchmark/services/network/latency/README.md create mode 100644 benchmark/services/network/latency/config.yaml create mode 100644 benchmark/services/network/latency/src/client/latency_client.py create mode 100644 benchmark/services/network/latency/src/server/latency_server.py diff --git a/benchmark/services/network/latency/README.md b/benchmark/services/network/latency/README.md new file mode 100644 index 0000000..cda34b4 --- /dev/null +++ b/benchmark/services/network/latency/README.md @@ -0,0 +1,142 @@ +# Latency + +Round trip time through the container network, measured with +[`sockperf`](https://github.com/Mellanox/sockperf) as a ping-pong request/response test over TCP and UDP, and +reported as percentiles. + +`config.yaml` holds four items: one server, and one client per scenario. The client code is the same for all three — +a scenario is a different value of one environment variable, not different code. + +| Item | Scenario | `TARGET` | Server side | +| ------------------------------------------- | ------------------ | ---------------- | ---------------------- | +| `benchmark-network-latency-server` | — | — | this bundle | +| `benchmark-network-latency-client-service` | service -> service | `latency-server` | the server item above | +| `benchmark-network-latency-client-unit` | service -> unit | `10.0.0.100` | `sockperf` on the node | +| `benchmark-network-latency-client-external` | service -> external| `10.0.0.1` | `sockperf` on the host | + +Install only the items a run needs: three clients installed at once would measure each other's interference. + +## Why percentiles, not an average + +Latency distributions are skewed: most round trips sit close to the minimum, and a thin tail of rare ones runs orders +of magnitude longer. An average hides that tail — one round trip of 50 ms among ten thousand of 40 µs moves the +average by five microseconds and disappears. The tail is what real-time and RPC traffic feel: an operation making a +hundred sequential calls has roughly a 63% chance of hitting at least one p99 event. + +On this stack the tail comes from scheduling, from conntrack and nftables handling a new flow, and from softirq work +on `veth` competing with neighbouring instances — which is why the figures are expected to change with instance +density even when the median does not. + +A percentile is only worth the samples behind it. A five second ping-pong run normally produces a few hundred +thousand round trips, which is enough for `p999`; if a run reports only a few hundred observations, treat `p999` as +noise. + +## What the client measures + +| Test | `sockperf` arguments | Protocol | +| --------- | -------------------- | -------- | +| `udp_rtt` | (none) | UDP | +| `tcp_rtt` | `--tcp` | TCP | + +Both run with `--full-rtt`, so every figure is a full round trip. Without that flag `sockperf` halves its numbers and +reports one way latency instead, which is not what the benchmark plan asks for. + +The first test is retried for up to 60 seconds while `sockperf` fails to reach the server; a run that exits cleanly +but produced no observations counts as a failure too. Later tests are not retried, and tests are spaced 3 seconds +apart so the server has a moment to reset between them. + +After the last test the instance stays alive idling, which keeps its logs available instead of having the unit +restart it in a loop. + +## Environment + +| Variable | Default | Meaning | +| ---------- | -------- | ------------------------------------------------ | +| `TARGET` | per item | Server hostname or IP. Required. | +| `DURATION` | `5` | Length of every single test, in seconds. | +| `PORT` | `11111` | `sockperf` port, the same for client and server. | +| `MSG_SIZE` | `64` | Payload size in bytes passed to `sockperf -m`. | + +## Results + +Two destinations, on purpose. + +The **log** gets every test in full, `sockperf`'s own report included: all five percentiles it prints, min, max, +average, standard deviation, the observation count and the dropped message count, plus the raw text. The tool has no +machine readable output, so everything is parsed out of that report, and keeping it makes a surprising number +checkable afterwards. + +**VictoriaMetrics** gets the three percentiles the benchmark plan names, pushed as `benchmark_result` samples +bracketed by `checkpoint_event` Start/Stop, in the same shape as `services/template/py`: + +| Sample name | From | +| ------------------ | ---------- | +| ` p50, us` | both tests | +| ` p99, us` | both tests | +| ` p999, us` | both tests | + +The `source` label is the instance's `AOS_INSTANCE_ID`, which is what tells instances apart once a scenario is run at +scale. + +## Setting up each scenario + +`sockperf` listens either on UDP or, with `--tcp`, on TCP, never on both, so every server side is two processes. TCP +and UDP port numbers are independent, so both use the same port. + +**service -> service** needs nothing beyond installing both items: the server item runs both processes itself and +sets `hostname: latency-server`. + +**service -> unit** needs them on the node, bound to the address the client dials: + +```console +setsid sockperf server -i 10.0.0.100 -p 11111 /tmp/sockperf-udp.log 2>&1 & +setsid sockperf server -i 10.0.0.100 -p 11111 --tcp /tmp/sockperf-tcp.log 2>&1 & +``` + +**service -> external** needs the same on the host, with `10.0.0.1`. Install `sockperf` there first — Debian and +Ubuntu package it: + +```console +sudo apt-get install -y sockperf +``` + +The versions on the two sides then differ, since the `meta-aos-vm` recipe pins `3.10+git` while Ubuntu ships `3.7`. +That pairing has been exercised without trouble; a failure during the handshake rather than during the test is the +sign to look here, and the fix is to build the newer one on the host rather than to downgrade the unit. + +### Why the bind matters + +A node has more than one address on the path to a container, and without an explicit bind the reply is routed back +over the service bridge and carries the bridge address as its source. The client's socket is connected to the address +it dialed, so the kernel drops the mismatch and UDP fails while TCP passes. This was measured directly with `iperf3` +on the same paths. + +`setsid` matters when starting servers over SSH: a plain `&` leaves them attached to the session and they die on +logout, which then looks like the benchmark failing to reach the far side. + +### Before deploying + +The server item's own `sockperf` listens on 11111 too, but inside a container namespace, so it does not occupy the +node's port; a leftover from an earlier run does. + +```console +ss -lntu | grep 11111 +sockperf ping-pong -i 10.0.0.100 -p 11111 -t 2 --full-rtt +sockperf ping-pong -i 10.0.0.100 -p 11111 -t 2 --full-rtt --tcp +``` + +## Requirements + +`sockperf` and `python3` must be in the container rootfs. `sockperf` is not in a stock image: it comes from the +`sockperf` recipe in `meta-aos-vm`, installed into `aos-image-vm`, which covers both the node and the service +containers that run on the node rootfs. The external host needs its own copy, from its distribution. + +## Reading the numbers + +The CPU quota matters more here than for throughput. A container that exhausts its `cpuLimit` is throttled by the +cgroup, and throttling shows up directly as spikes in the tail: `p999` degrades while `p50` stays flat. If the tail +looks implausible, check the quota before blaming the network. + +Between two instances on one node, and between an instance and its node, there is no wire — the traffic goes through +`veth` and a bridge, in RAM — so what is characterised is the node's scheduling and softirq behaviour. Only the +external scenario crosses a real boundary, and its figures sit well above the other two, tail included. diff --git a/benchmark/services/network/latency/config.yaml b/benchmark/services/network/latency/config.yaml new file mode 100644 index 0000000..1e8ad43 --- /dev/null +++ b/benchmark/services/network/latency/config.yaml @@ -0,0 +1,114 @@ +schemaVersion: 2 +publisher: + author: Developer Name + company: Company Name +publish: + tlsKey: aos-user-sp.p12 +items: +- identity: + type: service + codename: benchmark-network-latency-server + title: Benchmark network latency server + description: sockperf servers for the network latency benchmark + version: 1.0.0-beta.1 + images: + - sourceFolder: src/server + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u latency_server.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 256MiB + tmpLimit: 64MiB + hostname: latency-server + env: + - PORT=11111 +- identity: + type: service + codename: benchmark-network-latency-client-service + title: Benchmark network latency client (service to service) + description: Round trip time between two services on the same unit + version: 1.0.0-beta.1 + images: + - sourceFolder: src/client + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u latency_client.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 256MiB + tmpLimit: 64MiB + resources: + - name: victoria-metrics + mode: rw + env: + - TARGET=latency-server + - DURATION=5 + - PORT=11111 + - MSG_SIZE=64 +- identity: + type: service + codename: benchmark-network-latency-client-unit + title: Benchmark network latency client (service to unit) + description: Round trip time between a service and the node it runs on + version: 1.0.0-beta.1 + images: + - sourceFolder: src/client + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u latency_client.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 256MiB + tmpLimit: 64MiB + resources: + - name: victoria-metrics + mode: rw + env: + - TARGET=10.0.0.100 + - DURATION=5 + - PORT=11111 + - MSG_SIZE=64 +- identity: + type: service + codename: benchmark-network-latency-client-external + title: Benchmark network latency client (service to external) + description: Round trip time between a service and a host outside the unit + version: 1.0.0-beta.1 + images: + - sourceFolder: src/client + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u latency_client.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 256MiB + tmpLimit: 64MiB + resources: + - name: victoria-metrics + mode: rw + env: + - TARGET=10.0.0.1 + - DURATION=5 + - PORT=11111 + - MSG_SIZE=64 diff --git a/benchmark/services/network/latency/src/client/latency_client.py b/benchmark/services/network/latency/src/client/latency_client.py new file mode 100644 index 0000000..41cc412 --- /dev/null +++ b/benchmark/services/network/latency/src/client/latency_client.py @@ -0,0 +1,315 @@ +#!/usr/bin/env python3 +"""Latency benchmark item: measures round trip time with sockperf and reports it. + +Runs a ping-pong test against TARGET over UDP and over TCP, each for DURATION +seconds, and reports the round trip time as percentiles rather than an average: +latency distributions are skewed, and an average hides the tail that real-time +and RPC traffic actually feel. + +Results go two ways. Every test's full result, sockperf's own report included, +is printed to the service log, which is what makes a failed run diagnosable +afterwards. Only the percentiles worth charting are pushed to VictoriaMetrics +as benchmark_result samples, bracketed by checkpoint_event Start/Stop, exactly +as services/template/py does. + +Configuration comes from the environment, so one image serves every scenario +and only the deployment differs: + TARGET server hostname or IP (required) + DURATION seconds per test (default 5) + PORT sockperf port (default 11111) + MSG_SIZE payload size in bytes (default 64) + +Usage: + latency_client.py [--victoria-url http://victoriametrics:8428] +""" + +import argparse +import datetime +import json +import os +import re +import subprocess +import sys +import time +import urllib.error +import urllib.request + +TARGET = os.environ.get("TARGET", "") +DURATION = os.environ.get("DURATION", "5") +PORT = os.environ.get("PORT", "11111") +MSG_SIZE = os.environ.get("MSG_SIZE", "64") + +CONNECT_ATTEMPTS = 30 +CONNECT_DELAY = 2 +TEST_GAP = 3 +IDLE_DELAY = 60 + +NODE = "main" # No mechanism yet for an instance to learn which node it's actually running on. + +# Test name -> extra sockperf arguments. sockperf speaks UDP unless told +# otherwise, so only the TCP test needs a flag. +TESTS = [ + ("udp_rtt", "udp", []), + ("tcp_rtt", "tcp", ["--tcp"]), +] + +# sockperf reports percentiles by their exact label; these are the ones the +# benchmark plan asks for, plus the neighbours that make the tail readable. +PERCENTILES = { + "50.000": "p50_us", + "90.000": "p90_us", + "99.000": "p99_us", + "99.900": "p999_us", + "99.990": "p9999_us", +} + +# Of those, the ones worth a time series. +CHARTED = (("p50_us", "p50"), ("p99_us", "p99"), ("p999_us", "p999")) + +PERCENTILE_RE = re.compile(r"percentile\s+([\d.]+)\s*=\s*([\d.]+)") +MIN_RE = re.compile(r"\s+observation\s*=\s*([\d.]+)") +MAX_RE = re.compile(r"\s+observation\s*=\s*([\d.]+)") +# sockperf labels the average avg-rtt with --full-rtt and avg-latency without. +AVG_RE = re.compile(r"avg-(?:rtt|latency)\s*=\s*([\d.]+)") +STDDEV_RE = re.compile(r"std-dev\s*=\s*([\d.]+)") +OBSERVATIONS_RE = re.compile(r"Total\s+(\d+)\s+observations") +DROPPED_RE = re.compile(r"dropped messages\s*=\s*(\d+)") +ANSI_RE = re.compile(r"\x1b\[[0-9;]*m") + + +def parse_args(): + """Parse --victoria-url command-line option.""" + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--victoria-url", + default="http://victoriametrics:8428", + help="main node's VictoriaMetrics base URL (default: %(default)s)", + ) + return parser.parse_args() + + +def escape_label_value(value): + """Escape a string for safe embedding inside a Prometheus exposition-format label value.""" + return value.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n") + + +def format_precise_time(timestamp_us): + """Format a microsecond epoch timestamp as "YYYY-MM-DD HH:MM:SS.ffffff" (UTC).""" + seconds, microseconds = divmod(timestamp_us, 1_000_000) + dt = datetime.datetime.fromtimestamp(seconds, tz=datetime.timezone.utc) + dt += datetime.timedelta(microseconds=microseconds) + + return dt.strftime("%Y-%m-%d %H:%M:%S.%f") + + +def push_line(victoria_url, line): + """POST a single Prometheus exposition-format line to VictoriaMetrics.""" + request = urllib.request.Request( + f"{victoria_url.rstrip('/')}/api/v1/import/prometheus", + data=line.encode(), + method="POST", + ) + + try: + with urllib.request.urlopen(request, timeout=5) as response: + response.read() + except urllib.error.URLError as err: + print(f"failed to push to VictoriaMetrics: {err}", file=sys.stderr) + + +def push_event(victoria_url, node, source, event): + """Push a checkpoint_event sample (the same metric event_exporter.py produces).""" + timestamp_us = int(time.time() * 1_000_000) + labels = ",".join( + f'{name}="{escape_label_value(value)}"' + for name, value in ( + ("node", node), + ("source", source), + ("event", event), + ("time_us", format_precise_time(timestamp_us)), + ) + ) + time_s = timestamp_us / 1_000_000 + push_line(victoria_url, f"checkpoint_event{{{labels}}} 1 {time_s:.3f}") + + +def push_result(victoria_url, node, source, name, value): + """Push a single benchmark_result sample for one measured value.""" + timestamp_us = int(time.time() * 1_000_000) + labels = ",".join( + f'{label}="{escape_label_value(text)}"' + for label, text in ( + ("node", node), + ("source", source), + ("name", name), + ("time_us", format_precise_time(timestamp_us)), + ) + ) + time_s = timestamp_us / 1_000_000 + push_line(victoria_url, f"benchmark_result{{{labels}}} {value} {time_s:.3f}") + + +def parse(output): + """Pull the latency figures out of sockperf's report. + + sockperf has no machine readable output, so its text report is parsed. + Every value is in microseconds and, because the tests run with --full-rtt, + is a full round trip rather than the one way figure sockperf reports by + default. + """ + # sockperf colours parts of its report, so drop the escape sequences first. + output = ANSI_RE.sub("", output) + + result = {} + + for label, value in PERCENTILE_RE.findall(output): + name = PERCENTILES.get(label) + if name: + result[name] = float(value) + + for pattern, name in ( + (MIN_RE, "min_us"), + (MAX_RE, "max_us"), + (AVG_RE, "avg_us"), + (STDDEV_RE, "stddev_us"), + ): + match = pattern.search(output) + if match: + result[name] = float(match.group(1)) + + for pattern, name in ((OBSERVATIONS_RE, "observations"), (DROPPED_RE, "dropped")): + match = pattern.search(output) + if match: + result[name] = int(match.group(1)) + + return result + + +def run_sockperf(extra_args): + """Run one ping-pong test and return its output and error message.""" + cmd = [ + "sockperf", "ping-pong", + "-i", TARGET, + "-p", PORT, + "-t", DURATION, + "-m", MSG_SIZE, + "--full-rtt", + ] + extra_args + + print(f"Running: {' '.join(cmd)}") + + process = subprocess.run(cmd, capture_output=True, text=True) + + # sockperf writes its report to stdout and its diagnostics to stderr, and + # both matter: a run can exit cleanly having received nothing at all. + output = process.stdout + process.stderr + + if process.returncode != 0: + error = output.strip().splitlines()[-1] if output.strip() else \ + f"sockperf exited with code {process.returncode}" + return output, error + + return output, "" + + +def run_test(name, protocol, extra_args, attempts=1): + """Run one test, log its whole result, and return the values worth charting. + + The server may still be starting when the client reaches it, so the first + test is given several attempts: otherwise its result would say more about + start order than about the network. + """ + for attempt in range(1, attempts + 1): + output, error = run_sockperf(extra_args) + values = parse(output) + + # A run that produced no percentiles never reached the server, whatever + # its exit code says. + if not error and values.get("p50_us") is not None: + break + + if attempt == attempts: + break + + print(f"Server {TARGET}:{PORT} not ready ({attempt}/{attempts}): {error or 'no observations'}") + time.sleep(CONNECT_DELAY) + + metric = { + "test": name, + "protocol": protocol, + "target": TARGET, + "port": int(PORT), + "duration_s": int(DURATION), + "msg_size": int(MSG_SIZE), + } + + if values.get("p50_us") is None: + metric["error"] = error or "sockperf produced no observations" + else: + metric.update(values) + + metric["raw"] = {"output": output} + + # The log keeps everything, sockperf's own report included; only the + # percentiles the plan asks for are worth a time series. + print(json.dumps(metric)) + + return { + f"{name} {label}, us": metric[key] + for key, label in CHARTED + if metric.get(key) is not None + } + + +def run_benchmark(): + """Run both tests and return their results as {value_name: value}.""" + results = {} + attempts = CONNECT_ATTEMPTS + + for index, (name, protocol, extra_args) in enumerate(TESTS): + # Give the server a moment to reset between tests, the same way the + # bandwidth benchmark does: starting the next test the instant the + # previous one ends is what makes it fail on the control connection. + if index: + time.sleep(TEST_GAP) + + results.update(run_test(name, protocol, extra_args, attempts)) + attempts = 1 + + return results + + +def main(): + """Push a start event, run the benchmark, push its results, then push a stop event.""" + args = parse_args() + source = os.environ["AOS_INSTANCE_ID"] + + if not TARGET: + print("TARGET environment variable is required", file=sys.stderr) + return 1 + + print( + f"Latency benchmark: target={TARGET} port={PORT} " + f"duration={DURATION}s msg_size={MSG_SIZE}" + ) + + push_event(args.victoria_url, NODE, source, "Start") + + try: + results = run_benchmark() + + for name, value in results.items(): + push_result(args.victoria_url, NODE, source, name, value) + finally: + push_event(args.victoria_url, NODE, source, "Stop") + + print("All tests finished") + + # The benchmark is a one shot run, but the instance keeps running so its + # logs stay available and the unit does not restart it in a loop. + while True: + time.sleep(IDLE_DELAY) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/benchmark/services/network/latency/src/server/latency_server.py b/benchmark/services/network/latency/src/server/latency_server.py new file mode 100644 index 0000000..f985444 --- /dev/null +++ b/benchmark/services/network/latency/src/server/latency_server.py @@ -0,0 +1,40 @@ +import os +import subprocess +import time + +PORT = os.environ.get("PORT", "11111") + +RESTART_DELAY = 5 + +# One server per protocol: sockperf listens either on UDP or, with --tcp, on +# TCP, never on both. TCP and UDP port numbers are independent, so the two can +# share PORT. +SERVERS = [ + ("udp", ["sockperf", "server", "-i", "0.0.0.0", "-p", PORT]), + ("tcp", ["sockperf", "server", "-i", "0.0.0.0", "-p", PORT, "--tcp"]), +] + + +def main(): + processes = {} + + # sockperf servers never exit on their own, so this loop only matters if + # one of them dies: the service instance stays alive and it comes back. + while True: + for name, cmd in SERVERS: + process = processes.get(name) + + if process is not None and process.poll() is None: + continue + + if process is not None: + print(f"sockperf {name} server exited with code {process.returncode}, restarting") + + print(f"Starting sockperf {name} server: {' '.join(cmd)}") + processes[name] = subprocess.Popen(cmd) + + time.sleep(RESTART_DELAY) + + +if __name__ == "__main__": + main() From 53ee61c9333872602957131febbfa252560ca0e3 Mon Sep 17 00:00:00 2001 From: Mykola Solianko Date: Wed, 12 Aug 2026 11:46:44 +0300 Subject: [PATCH 3/4] benchmark: add network DNS resolve time services Move the DNS resolve time benchmark into the new structure, completing the network group: one folder per benchmark group, a single src/ built for any architecture, and one item per scenario differing only in the environment. The client builds and sends the query itself instead of shelling out to dig. dig reports its query time in whole milliseconds while a local dnsmasq answers in hundreds of microseconds, so every sample would read as zero, and timing the dig process from outside costs more than the query it measures. Doing it in-process also keeps bind-utils out of the image. The log keeps every individual sample, so percentiles can be recomputed or several instances pooled without re-running anything, while VictoriaMetrics gets the three percentiles the benchmark plan names. The external scenario prepends a random label to every query, since dnsmasq would otherwise serve everything after the first from its cache and the scenario would stop describing resolution. Signed-off-by: Mykola Solianko --- benchmark/services/network/dns/README.md | 147 +++++++ benchmark/services/network/dns/config.yaml | 105 +++++ .../network/dns/src/client/dns_client.py | 372 ++++++++++++++++++ .../services/network/dns/src/peer/dns_peer.py | 17 + 4 files changed, 641 insertions(+) create mode 100644 benchmark/services/network/dns/README.md create mode 100644 benchmark/services/network/dns/config.yaml create mode 100644 benchmark/services/network/dns/src/client/dns_client.py create mode 100644 benchmark/services/network/dns/src/peer/dns_peer.py diff --git a/benchmark/services/network/dns/README.md b/benchmark/services/network/dns/README.md new file mode 100644 index 0000000..e5c2683 --- /dev/null +++ b/benchmark/services/network/dns/README.md @@ -0,0 +1,147 @@ +# DNS resolve time + +How long a service takes to resolve a name through the `dnsmasq` an AosEdge unit runs for its instances, reported as +percentiles. + +`config.yaml` holds four items: an idle peer that exists only to own a name, and one client per scenario. The client +code is the same for all three — a scenario is a different value of one environment variable, not different code. + +| Item | Scenario | `NAME` | Where the name lives | +| --------------------------------------- | ------------------ | ---------------- | ------------------------------------ | +| `benchmark-network-dns-peer` | — | — | registered by the unit while it runs | +| `benchmark-network-dns-client-service` | service -> service | `dns-peer` | the peer item above | +| `benchmark-network-dns-client-unit` | service -> unit | `main` | `/etc/aos/addnhosts` on the node | +| `benchmark-network-dns-client-external` | service -> external| `dns-probe.test` | a DNS server on the host | + +Install only the items a run needs. + +## Why there is no dig + +The benchmark plan names `dig` as the tool, but `dig` reports its query time in whole milliseconds +(`;; Query time: 0 msec`), and a local `dnsmasq` answers in hundreds of microseconds, so every sample would read as +zero. Timing the `dig` process from outside is worse: starting it costs more than the query it is supposed to +measure. + +So the client sends the query itself — a UDP socket, a hand built query packet, and `time.perf_counter()` around it. +That gives microsecond resolution, no process startup inside the measurement, and enough samples for percentiles. It +also keeps `dig`, and the `bind-utils` package behind it, out of the image: `python3` is already in the container. + +## Which resolver is measured + +With `RESOLVER` unset the client reads every `nameserver` from the container's `/etc/resolv.conf` and measures +against the first one that answers, reporting which in the log. + +That fallback is not decoration. The container is handed two nameservers, the service bridge address first and the +node address second, but `dnsmasq` is configured with `listen-address` set to the node address only — so nothing is +listening on the first one and a query there just times out. A libc resolver walks the list until something replies, +which is why service names resolve normally, and the client does the same. Measuring the first entry blindly would +report timeouts instead of resolve times. + +## Environment + +| Variable | Default | Meaning | +| -------------- | -------- | ------------------------------------------------------------------------ | +| `NAME` | per item | Name to resolve. Required. | +| `QUERIES` | `2000` | How many queries to send. | +| `RANDOM_LABEL` | per item | Prepend a random label to `NAME` on every query, to defeat the DNS cache. | +| `RESOLVER` | unset | DNS server to ask, `host` or `host:port`. Unset means `/etc/resolv.conf`. | + +## Results + +Two destinations, on purpose. + +The **log** gets the whole result, every individual sample included, so percentiles can be recomputed later or several +instances pooled into one distribution without re-running anything. It also carries the failure breakdown: a timeout, +a non-zero RCODE, or an answer with no records, counted by reason. + +**VictoriaMetrics** gets the three percentiles the benchmark plan names, pushed as `benchmark_result` samples +bracketed by `checkpoint_event` Start/Stop, in the same shape as `services/template/py`: + +| Sample name | +| ------------------ | +| `resolve p50, us` | +| `resolve p99, us` | +| `resolve p999, us` | + +At `QUERIES=2000`, `p99` rests on 20 samples and `p999` on 2, so treat `p999` as an indication and raise `QUERIES` if +it matters. The `source` label is the instance's `AOS_INSTANCE_ID`, which tells instances apart once a scenario is +run at scale. + +## Setting up each scenario + +**service -> service** needs nothing: install the peer alongside the client, and the unit registers `dns-peer` in +`dnsmasq` for as long as the peer instance runs. `RANDOM_LABEL` stays off — the name is answered out of a local file +every time, so no cache sits in the way. + +**service -> unit** needs nothing either on a stock unit. The unit's `dnsmasq` reads two hosts files: + +``` +addn-hosts=/var/aos/dns/addnhosts # written by Aos, rewritten on every deployment +addn-hosts=/etc/aos/addnhosts # static, for names the unit owner adds +``` + +and the second already carries `10.0.0.100 main`, which is why `NAME` defaults to `main`. To measure a different +name, add it there and make `dnsmasq` re-read the file: + +```console +echo "10.0.0.100 dns-probe-unit" >> /etc/aos/addnhosts +kill -HUP $(cat /var/aos/dns/pidfile) +``` + +Not `/var/aos/dns/addnhosts` — Aos rewrites that file whenever instances change. + +**service -> external** needs a DNS server on the host holding the name, and a unit that forwards to it. The +forwarding is usually already there: check the node's `/etc/resolv.conf` for `nameserver 10.0.0.1`, and if it is +listed, nothing on the unit has to change. + +The host normally already runs a `dnsmasq` on the bridge, serving DHCP for it: + +```console +ps -eo pid,args | grep [d]nsmasq +ss -lnup | grep 10.0.0.1:53 +``` + +Add the record to `/etc/dnsmasq.conf`: + +``` +address=/dns-probe.test/10.0.0.1 +``` + +`dnsmasq` reads that file at startup unless given `-C`, so it is the right place even for an instance launched by +hand. `/etc/dnsmasq.d/` is not: on a stock Ubuntu the `conf-dir` line is commented out, so the directory is never +read and the record is silently ignored. `/etc/hosts` cannot express a wildcard, so it is not an option either. + +The wildcard form matters. `address=/dns-probe.test/10.0.0.1` answers the domain and everything under it, which is +what makes `RANDOM_LABEL=1` work — and it is on for this scenario for a reason: `dnsmasq` on the unit caches +answers, so a repeated name would be served from cache after the first query and the scenario would stop describing +resolution. + +`address=` is only read at startup — a `SIGHUP` re-reads hosts files, not the configuration — so restart it exactly +as it was running: + +```console +sudo kill +sudo dnsmasq --interface=aos-br0 --dhcp-range=10.0.0.101,10.0.0.254,12h \ + --dhcp-option=3,10.0.0.1 --dhcp-option=6,10.0.0.1 --bind-interfaces +``` + +That process usually serves DHCP for the bridge, so anything holding a lease will notice the second it is down; a +unit with a static address does not. Verify before deploying: + +```console +nslookup probe123.dns-probe.test 10.0.0.100 +``` + +## Requirements + +`python3` in the container rootfs, which is already there. Nothing else — no `dig`, no layer, no image change. + +## Reading the numbers + +The service -> service and service -> unit scenarios measure the same thing on this stack and come out equal: both +names live in files read by the same `dnsmasq` on the node, and the packets travel the same bridge either way. Only +the external scenario leaves the node, and it costs several times more — that gap is the price of forwarding. + +Comparing a resolve time against the latency benchmark's round trip time on the same path is worth doing: locally the +two come out nearly identical, which says the lookup inside `dnsmasq` is almost free and what is being measured is +the network round trip. diff --git a/benchmark/services/network/dns/config.yaml b/benchmark/services/network/dns/config.yaml new file mode 100644 index 0000000..d1c1146 --- /dev/null +++ b/benchmark/services/network/dns/config.yaml @@ -0,0 +1,105 @@ +schemaVersion: 2 +publisher: + author: Developer Name + company: Company Name +publish: + tlsKey: aos-user-sp.p12 +items: +- identity: + type: service + codename: benchmark-network-dns-peer + title: Benchmark network DNS peer + description: Idle peer whose name the DNS benchmark resolves + version: 1.0.0-beta.1 + images: + - sourceFolder: src/peer + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u dns_peer.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 128MiB + hostname: dns-peer +- identity: + type: service + codename: benchmark-network-dns-client-service + title: Benchmark network DNS client (service to service) + description: Resolve time for a peer service name + version: 1.0.0-beta.1 + images: + - sourceFolder: src/client + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u dns_client.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 128MiB + resources: + - name: victoria-metrics + mode: rw + env: + - NAME=dns-peer + - QUERIES=2000 + - RANDOM_LABEL=0 +- identity: + type: service + codename: benchmark-network-dns-client-unit + title: Benchmark network DNS client (service to unit) + description: Resolve time for a name registered on the node + version: 1.0.0-beta.1 + images: + - sourceFolder: src/client + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u dns_client.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 128MiB + resources: + - name: victoria-metrics + mode: rw + env: + - NAME=main + - QUERIES=2000 + - RANDOM_LABEL=0 +- identity: + type: service + codename: benchmark-network-dns-client-external + title: Benchmark network DNS client (service to external) + description: Resolve time for a name served outside the unit + version: 1.0.0-beta.1 + images: + - sourceFolder: src/client + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u dns_client.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 128MiB + resources: + - name: victoria-metrics + mode: rw + env: + - NAME=dns-probe.test + - QUERIES=2000 + - RANDOM_LABEL=1 diff --git a/benchmark/services/network/dns/src/client/dns_client.py b/benchmark/services/network/dns/src/client/dns_client.py new file mode 100644 index 0000000..a2fd872 --- /dev/null +++ b/benchmark/services/network/dns/src/client/dns_client.py @@ -0,0 +1,372 @@ +#!/usr/bin/env python3 +"""DNS benchmark item: measures name resolution time and reports it. + +Resolves NAME QUERIES times against the resolver the container was handed, +timing each query from just before the packet leaves to just after the matching +answer arrives, and reports the distribution as percentiles. + +The query is built and sent here rather than shelled out to dig on purpose. dig +reports its query time in whole milliseconds, and a local dnsmasq answers in +hundreds of microseconds, so every sample would read as zero; timing the dig +process from outside is worse still, as starting it costs more than the query. +Doing it in-process also keeps dig, and the bind-utils package behind it, out +of the image. + +Results go two ways. The full result, every individual sample included, is +printed to the service log. Only the percentiles worth charting are pushed to +VictoriaMetrics as benchmark_result samples, bracketed by checkpoint_event +Start/Stop, exactly as services/template/py does. + +Configuration comes from the environment, so one image serves every scenario +and only the deployment differs: + NAME name to resolve (required) + QUERIES how many queries to send (default 2000) + RANDOM_LABEL 1 prepends a random label to every query, to miss the cache + RESOLVER DNS server to ask, host or host:port; unset means the + nameservers from the container's /etc/resolv.conf + +Usage: + dns_client.py [--victoria-url http://victoriametrics:8428] +""" + +import argparse +import datetime +import json +import os +import random +import socket +import string +import struct +import sys +import time +import urllib.error +import urllib.request + +NAME = os.environ.get("NAME", "") +RESOLVER = os.environ.get("RESOLVER", "") +QUERIES = int(os.environ.get("QUERIES", "2000")) +RANDOM_LABEL = os.environ.get("RANDOM_LABEL", "0") == "1" + +CONNECT_ATTEMPTS = 30 +CONNECT_DELAY = 2 +QUERY_TIMEOUT = 2 +IDLE_DELAY = 60 + +RESOLV_CONF = "/etc/resolv.conf" +DNS_PORT = 53 + +NODE = "main" # No mechanism yet for an instance to learn which node it's actually running on. + +# Percentiles are picked the way sockperf picks them, so DNS and latency +# numbers can be read side by side. +PERCENTILES = (("p50_us", 0.50), ("p90_us", 0.90), ("p99_us", 0.99), ("p999_us", 0.999)) + +# Of those, the ones worth a time series. +CHARTED = (("p50_us", "p50"), ("p99_us", "p99"), ("p999_us", "p999")) + + +def parse_args(): + """Parse --victoria-url command-line option.""" + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--victoria-url", + default="http://victoriametrics:8428", + help="main node's VictoriaMetrics base URL (default: %(default)s)", + ) + return parser.parse_args() + + +def escape_label_value(value): + """Escape a string for safe embedding inside a Prometheus exposition-format label value.""" + return value.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n") + + +def format_precise_time(timestamp_us): + """Format a microsecond epoch timestamp as "YYYY-MM-DD HH:MM:SS.ffffff" (UTC).""" + seconds, microseconds = divmod(timestamp_us, 1_000_000) + dt = datetime.datetime.fromtimestamp(seconds, tz=datetime.timezone.utc) + dt += datetime.timedelta(microseconds=microseconds) + + return dt.strftime("%Y-%m-%d %H:%M:%S.%f") + + +def push_line(victoria_url, line): + """POST a single Prometheus exposition-format line to VictoriaMetrics.""" + request = urllib.request.Request( + f"{victoria_url.rstrip('/')}/api/v1/import/prometheus", + data=line.encode(), + method="POST", + ) + + try: + with urllib.request.urlopen(request, timeout=5) as response: + response.read() + except urllib.error.URLError as err: + print(f"failed to push to VictoriaMetrics: {err}", file=sys.stderr) + + +def push_event(victoria_url, node, source, event): + """Push a checkpoint_event sample (the same metric event_exporter.py produces).""" + timestamp_us = int(time.time() * 1_000_000) + labels = ",".join( + f'{name}="{escape_label_value(value)}"' + for name, value in ( + ("node", node), + ("source", source), + ("event", event), + ("time_us", format_precise_time(timestamp_us)), + ) + ) + time_s = timestamp_us / 1_000_000 + push_line(victoria_url, f"checkpoint_event{{{labels}}} 1 {time_s:.3f}") + + +def push_result(victoria_url, node, source, name, value): + """Push a single benchmark_result sample for one measured value.""" + timestamp_us = int(time.time() * 1_000_000) + labels = ",".join( + f'{label}="{escape_label_value(text)}"' + for label, text in ( + ("node", node), + ("source", source), + ("name", name), + ("time_us", format_precise_time(timestamp_us)), + ) + ) + time_s = timestamp_us / 1_000_000 + push_line(victoria_url, f"benchmark_result{{{labels}}} {value} {time_s:.3f}") + + +def default_resolvers(): + """Every nameserver the container was handed, in order. + + All of them are returned, not just the first, because not all of them + necessarily answer: on this stack the container is handed the bridge + address first, while dnsmasq is bound to the node address only, so a query + to the first one just times out. A libc resolver walks the list until + something replies, and so does this client. + """ + resolvers = [] + + try: + with open(RESOLV_CONF) as conf: + for line in conf: + fields = line.split() + if len(fields) >= 2 and fields[0] == "nameserver": + resolvers.append(fields[1]) + except OSError: + pass + + return resolvers + + +def resolver_address(resolver): + """Split an optional port off the resolver, so a non standard one can be used.""" + host, separator, port = resolver.partition(":") + + return (host, int(port)) if separator else (host, DNS_PORT) + + +def build_query(name, query_id): + """A minimal DNS query for an A record.""" + # Standard query, recursion desired, one question. + header = struct.pack("!HHHHHH", query_id, 0x0100, 1, 0, 0, 0) + labels = b"".join(bytes([len(p)]) + p.encode() for p in name.split(".") if p) + # QTYPE A, QCLASS IN. + return header + labels + b"\x00" + struct.pack("!HH", 1, 1) + + +def query_once(sock, resolver, name): + """Send one query and return how long the answer took, in microseconds.""" + query_id = random.getrandbits(16) + packet = build_query(name, query_id) + + start = time.perf_counter() + sock.sendto(packet, resolver_address(resolver)) + + while True: + try: + reply, _ = sock.recvfrom(4096) + except socket.timeout: + return None, "timeout" + + # Ignore anything that is not the answer to this query. + if len(reply) >= 12 and struct.unpack("!H", reply[:2])[0] == query_id: + break + + elapsed_us = (time.perf_counter() - start) * 1e6 + + flags, _, answers = struct.unpack("!HHH", reply[2:8]) + rcode = flags & 0x0F + + if rcode != 0: + return None, f"rcode {rcode}" + + if answers == 0: + return None, "no answer records" + + return elapsed_us, "" + + +def percentile(sorted_samples, fraction): + index = int(0.5 + fraction * len(sorted_samples)) - 1 + + return sorted_samples[max(index, 0)] + + +def summarize(samples): + ordered = sorted(samples) + count = len(ordered) + mean = sum(ordered) / count + variance = sum((value - mean) ** 2 for value in ordered) / count + + result = {name: round(percentile(ordered, fraction), 3) for name, fraction in PERCENTILES} + result["min_us"] = round(ordered[0], 3) + result["max_us"] = round(ordered[-1], 3) + result["avg_us"] = round(mean, 3) + result["stddev_us"] = round(variance ** 0.5, 3) + + return result + + +def query_name(): + """The name to resolve, with a unique label when the cache has to be missed. + + dnsmasq caches answers, so repeating one name measures its cache rather + than resolution. A random leftmost label defeats that, and it still + resolves as long as the DNS server holding the name answers for the whole + domain. + """ + if not RANDOM_LABEL: + return NAME + + label = "".join(random.choice(string.ascii_lowercase) for _ in range(12)) + + return f"{label}.{NAME}" + + +def pick_resolver(candidates): + """Settle on a resolver that answers, and wait for the name to exist. + + Two things can keep the first query from succeeding, and both are normal: + a nameserver in the list may not be listening at all, and a peer service + may still be starting, since the unit registers its name only once the + instance runs. So every candidate is tried on every attempt, and the first + one that answers wins. + """ + sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + sock.settimeout(QUERY_TIMEOUT) + + try: + for attempt in range(1, CONNECT_ATTEMPTS + 1): + for candidate in candidates: + _, error = query_once(sock, candidate, query_name()) + + if not error: + return candidate + + print(f"{candidate} did not resolve {NAME} ({attempt}/{CONNECT_ATTEMPTS}): {error}") + + time.sleep(CONNECT_DELAY) + finally: + sock.close() + + return "" + + +def run_benchmark(resolver): + """Resolve the name QUERIES times and return the results as {value_name: value}.""" + samples = [] + failures = {} + + sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + sock.settimeout(QUERY_TIMEOUT) + + try: + for _ in range(QUERIES): + elapsed_us, error = query_once(sock, resolver, query_name()) + + if error: + failures[error] = failures.get(error, 0) + 1 + else: + samples.append(elapsed_us) + finally: + sock.close() + + metric = { + "test": "dns_resolve", + "name": NAME, + "resolver": resolver, + "queries": QUERIES, + "resolved": len(samples), + "failed": QUERIES - len(samples), + } + + if failures: + metric["failures"] = failures + + if samples: + metric.update(summarize(samples)) + metric["raw"] = {"samples_us": [round(value, 3) for value in samples]} + else: + metric["error"] = "; ".join(f"{reason} x{count}" for reason, count in failures.items()) + + # The log keeps everything, every individual sample included, so the + # percentiles can be recomputed later; only a few are worth a time series. + print(json.dumps(metric)) + + return { + f"resolve {label}, us": metric[key] + for key, label in CHARTED + if metric.get(key) is not None + } + + +def main(): + """Push a start event, run the benchmark, push its results, then push a stop event.""" + args = parse_args() + source = os.environ["AOS_INSTANCE_ID"] + + if not NAME: + print("NAME environment variable is required", file=sys.stderr) + return 1 + + candidates = [RESOLVER] if RESOLVER else default_resolvers() + + if not candidates: + print(f"No resolver: RESOLVER is unset and {RESOLV_CONF} has no nameserver", file=sys.stderr) + return 1 + + print( + f"DNS benchmark: name={NAME} resolvers={','.join(candidates)} " + f"queries={QUERIES} random_label={int(RANDOM_LABEL)}" + ) + + resolver = pick_resolver(candidates) + + if not resolver: + resolver = candidates[0] + print(f"No resolver answered for {NAME}, measuring against {resolver} anyway") + else: + print(f"Using resolver {resolver}") + + push_event(args.victoria_url, NODE, source, "Start") + + try: + results = run_benchmark(resolver) + + for name, value in results.items(): + push_result(args.victoria_url, NODE, source, name, value) + finally: + push_event(args.victoria_url, NODE, source, "Stop") + + print("All tests finished") + + # The benchmark is a one shot run, but the instance keeps running so its + # logs stay available and the unit does not restart it in a loop. + while True: + time.sleep(IDLE_DELAY) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/benchmark/services/network/dns/src/peer/dns_peer.py b/benchmark/services/network/dns/src/peer/dns_peer.py new file mode 100644 index 0000000..de9dea1 --- /dev/null +++ b/benchmark/services/network/dns/src/peer/dns_peer.py @@ -0,0 +1,17 @@ +import os +import time + +HEARTBEAT_DELAY = 60 + +# The peer exists only to own a DNS name: the unit registers a running +# instance in the per-bridge dnsmasq, and that registration is what the client +# measures the resolution of. There is nothing to serve, so it just stays up. +def main(): + print(f"DNS peer running as {os.environ.get('AOS_INSTANCE_ID', 'unknown instance')}") + + while True: + time.sleep(HEARTBEAT_DELAY) + + +if __name__ == "__main__": + main() From 82228e0527a81e405ad18f4c0f79761d65e86488 Mon Sep 17 00:00:00 2001 From: Mykola Solianko Date: Wed, 12 Aug 2026 11:46:44 +0300 Subject: [PATCH 4/4] benchmark: describe the network benchmark in the top level README Add an overview of what services/network measures, in what scenarios and where the results go, and drop the placeholder that asked for network services to be put there. Two things are worth saying once rather than three times in the group READMEs: that a scenario is a value of one environment variable rather than different code, and that a server outside the container has to bind the address the client dials, since otherwise the reply carries the service bridge address and UDP fails while TCP passes. Signed-off-by: Mykola Solianko --- benchmark/README.md | 87 +++++++++++++++++++++++++++ benchmark/services/network/readme.txt | 1 - 2 files changed, 87 insertions(+), 1 deletion(-) delete mode 100644 benchmark/services/network/readme.txt diff --git a/benchmark/README.md b/benchmark/README.md index 96386b5..818e9ac 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -1,3 +1,90 @@ # Benchmark This folder contains AosEdge services for benchmark tests. + +Every item follows the same shape, which `services/template/py` and `services/template/cpp` define: a `config.yaml` +and a `src/`, one image for any architecture, and results reported to VictoriaMetrics as `benchmark_result` samples +bracketed by `checkpoint_event` Start/Stop, so that everything lands in the same Grafana tables regardless of which +benchmark produced it. + +## Network performance + +`services/network/` covers the network chapter of the benchmark plan: what a service actually gets out of the +container network AosCore builds for it — `veth`, a bridge, nftables, `tc` and `dnsmasq`. + +Three groups, one folder each: + +| Group | Measures | Tool | +| ------------------------- | ------------------------------------------------------- | ---------- | +| [`bandwidth`](services/network/bandwidth) | Throughput, TCP and UDP, both directions, plus UDP jitter and loss | `iperf3` | +| [`latency`](services/network/latency) | Round trip time as percentiles, TCP and UDP | `sockperf` | +| [`dns`](services/network/dns) | Name resolution time as percentiles | built in | + +### Scenarios + +Each group is exercised in the same three scenarios. A group is one folder with one `config.yaml`, and the scenarios +are items inside it — the client code is written once and pointed at different things: + +``` +services/network//src/client/ the client, one copy +services/network//src/server/ the server, where the scenario needs one in a container +services/network//config.yaml one item per scenario, plus the server +``` + +The container is always the client; only the server side moves. A scenario is a different value of one environment +variable, not different code, which is why the items share the sources rather than duplicating them. + +Install only the items a run needs. Every installed client generates traffic, so leaving all three in place would +have them measure each other's interference. + +| Scenario | Server side | Measures | +| ------------------ | --------------------------------------------------- | -------------------------------------------- | +| service -> service | a second item in the same group | two containers on one node and one bridge | +| service -> unit | a plain process on the node | the container to node/gateway path | +| service -> external| a process on a machine outside the unit | egress through masquerade to a LAN host | + +The two scenarios whose server lives outside the container need it started by hand — each group's README gives the +exact commands. One rule spans all of them: **bind the server to the address the client dials**. A node has more than +one address on the path to a container, and without an explicit bind the reply is routed back over the service bridge +and carries the bridge address as its source. Both `iperf3` and `sockperf` connect their sockets, so the kernel drops +datagrams arriving from another address, and the symptom is characteristic — UDP fails while TCP passes. + +### Results + +Every client reports twice, and the split is deliberate. + +The **service log** gets the full result of each test: the tool's own output, error text, counters and per-interval +detail. That is what makes a failed or surprising run explicable afterwards, and none of it belongs in a time series. + +**VictoriaMetrics** gets only what is worth charting — one `benchmark_result` sample per measured value, labeled by +`name`, with `source` set to the instance's `AOS_INSTANCE_ID` so instances are told apart once a scenario is run at +scale. + +| Group | Samples pushed | +| --------- | ---------------------------------------------------------------------- | +| bandwidth | ` throughput, Mbps`, and for UDP ` loss, %` and ` jitter, ms` | +| latency | ` p50, us`, ` p99, us`, ` p999, us` | +| dns | `resolve p50, us`, `resolve p99, us`, `resolve p999, us` | + +Latency and DNS report percentiles rather than averages on purpose. Their distributions are skewed: most samples sit +near the minimum and a thin tail runs orders of magnitude longer, so an average hides exactly the behaviour that +real-time and RPC traffic feel. A percentile is only worth the samples behind it, though — `p999` needs thousands of +samples before it means anything. + +### Interpreting the figures + +On a unit that runs as a VM, none of the three scenarios crosses a physical wire: traffic moves through `veth`, a +bridge and `tap`, in RAM. Throughput figures there are bounded by how fast one core can move data rather than by any +link, which the `cpu_utilization_percent` section in `iperf3`'s output shows directly, so they should be read as a +floor. Latency and DNS figures on the same paths mostly characterise the node's scheduling behaviour. The numbers +start describing a network once the external scenario points at a host reached over a real interface. + +### Running at scale + +The benchmark plan repeats every measurement at 1, 16, 256 and 1024 instances. `config.yaml` keeps `minInstances: 1` +throughout — instance count is a deploy time decision, not a property of the item — and every client tags its samples +with `source`, so results from many instances stay separable. + +One caveat worth knowing before scaling up: every deployed client generates traffic, so a large deployment is the +plan's *active-load* mode. The *idle-density* mode, where one pair measures while the rest merely exist, needs +something the current items do not have — a way to keep most instances silent. diff --git a/benchmark/services/network/readme.txt b/benchmark/services/network/readme.txt deleted file mode 100644 index c7bbe5c..0000000 --- a/benchmark/services/network/readme.txt +++ /dev/null @@ -1 +0,0 @@ -TODO: put network services here