diff --git a/benchmark/README.md b/benchmark/README.md index 96386b5..818e9ac 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -1,3 +1,90 @@ # Benchmark This folder contains AosEdge services for benchmark tests. + +Every item follows the same shape, which `services/template/py` and `services/template/cpp` define: a `config.yaml` +and a `src/`, one image for any architecture, and results reported to VictoriaMetrics as `benchmark_result` samples +bracketed by `checkpoint_event` Start/Stop, so that everything lands in the same Grafana tables regardless of which +benchmark produced it. + +## Network performance + +`services/network/` covers the network chapter of the benchmark plan: what a service actually gets out of the +container network AosCore builds for it — `veth`, a bridge, nftables, `tc` and `dnsmasq`. + +Three groups, one folder each: + +| Group | Measures | Tool | +| ------------------------- | ------------------------------------------------------- | ---------- | +| [`bandwidth`](services/network/bandwidth) | Throughput, TCP and UDP, both directions, plus UDP jitter and loss | `iperf3` | +| [`latency`](services/network/latency) | Round trip time as percentiles, TCP and UDP | `sockperf` | +| [`dns`](services/network/dns) | Name resolution time as percentiles | built in | + +### Scenarios + +Each group is exercised in the same three scenarios. A group is one folder with one `config.yaml`, and the scenarios +are items inside it — the client code is written once and pointed at different things: + +``` +services/network//src/client/ the client, one copy +services/network//src/server/ the server, where the scenario needs one in a container +services/network//config.yaml one item per scenario, plus the server +``` + +The container is always the client; only the server side moves. A scenario is a different value of one environment +variable, not different code, which is why the items share the sources rather than duplicating them. + +Install only the items a run needs. Every installed client generates traffic, so leaving all three in place would +have them measure each other's interference. + +| Scenario | Server side | Measures | +| ------------------ | --------------------------------------------------- | -------------------------------------------- | +| service -> service | a second item in the same group | two containers on one node and one bridge | +| service -> unit | a plain process on the node | the container to node/gateway path | +| service -> external| a process on a machine outside the unit | egress through masquerade to a LAN host | + +The two scenarios whose server lives outside the container need it started by hand — each group's README gives the +exact commands. One rule spans all of them: **bind the server to the address the client dials**. A node has more than +one address on the path to a container, and without an explicit bind the reply is routed back over the service bridge +and carries the bridge address as its source. Both `iperf3` and `sockperf` connect their sockets, so the kernel drops +datagrams arriving from another address, and the symptom is characteristic — UDP fails while TCP passes. + +### Results + +Every client reports twice, and the split is deliberate. + +The **service log** gets the full result of each test: the tool's own output, error text, counters and per-interval +detail. That is what makes a failed or surprising run explicable afterwards, and none of it belongs in a time series. + +**VictoriaMetrics** gets only what is worth charting — one `benchmark_result` sample per measured value, labeled by +`name`, with `source` set to the instance's `AOS_INSTANCE_ID` so instances are told apart once a scenario is run at +scale. + +| Group | Samples pushed | +| --------- | ---------------------------------------------------------------------- | +| bandwidth | ` throughput, Mbps`, and for UDP ` loss, %` and ` jitter, ms` | +| latency | ` p50, us`, ` p99, us`, ` p999, us` | +| dns | `resolve p50, us`, `resolve p99, us`, `resolve p999, us` | + +Latency and DNS report percentiles rather than averages on purpose. Their distributions are skewed: most samples sit +near the minimum and a thin tail runs orders of magnitude longer, so an average hides exactly the behaviour that +real-time and RPC traffic feel. A percentile is only worth the samples behind it, though — `p999` needs thousands of +samples before it means anything. + +### Interpreting the figures + +On a unit that runs as a VM, none of the three scenarios crosses a physical wire: traffic moves through `veth`, a +bridge and `tap`, in RAM. Throughput figures there are bounded by how fast one core can move data rather than by any +link, which the `cpu_utilization_percent` section in `iperf3`'s output shows directly, so they should be read as a +floor. Latency and DNS figures on the same paths mostly characterise the node's scheduling behaviour. The numbers +start describing a network once the external scenario points at a host reached over a real interface. + +### Running at scale + +The benchmark plan repeats every measurement at 1, 16, 256 and 1024 instances. `config.yaml` keeps `minInstances: 1` +throughout — instance count is a deploy time decision, not a property of the item — and every client tags its samples +with `source`, so results from many instances stay separable. + +One caveat worth knowing before scaling up: every deployed client generates traffic, so a large deployment is the +plan's *active-load* mode. The *idle-density* mode, where one pair measures while the rest merely exist, needs +something the current items do not have — a way to keep most instances silent. diff --git a/benchmark/services/network/bandwidth/README.md b/benchmark/services/network/bandwidth/README.md new file mode 100644 index 0000000..c1bdaa7 --- /dev/null +++ b/benchmark/services/network/bandwidth/README.md @@ -0,0 +1,137 @@ +# Bandwidth + +Throughput through the container network, measured with [`iperf3`](https://iperf.fr/) for TCP and UDP in both +directions, together with UDP jitter and packet loss. + +`config.yaml` holds four items: one server, and one client per scenario. The client code is the same for all three — +a scenario is a different value of one environment variable, not different code. + +| Item | Scenario | `TARGET` | Server side | +| --------------------------------------------- | ------------------ | ------------------ | --------------------- | +| `benchmark-network-bandwidth-server` | — | — | this bundle | +| `benchmark-network-bandwidth-client-service` | service -> service | `bandwidth-server` | the server item above | +| `benchmark-network-bandwidth-client-unit` | service -> unit | `10.0.0.100` | `iperf3` on the node | +| `benchmark-network-bandwidth-client-external` | service -> external| `10.0.0.1` | `iperf3` on the host | + +Install only the items a run needs: three clients installed at once would measure each other's interference. + +## What the client measures + +Four tests in a row against `TARGET`, each for `DURATION` seconds: + +| Test | `iperf3` arguments | Direction | +| ---------- | ------------------------- | ---------------- | +| `tcp_up` | (none) | client -> server | +| `tcp_down` | `-R` | server -> client | +| `udp_up` | `-u -b $UDP_BANDWIDTH` | client -> server | +| `udp_down` | `-u -b $UDP_BANDWIDTH -R` | server -> client | + +`up` is the container sending, `down` is it receiving. Both are worth measuring because the stack shapes the two +directions separately, and because the sending side is the one that saturates a core, which the +`cpu_utilization_percent` section of the log shows directly. + +The first test is retried for up to 60 seconds while `iperf3` reports a connection or name resolution failure, so the +run survives being started before its server. Later tests are not retried. + +Tests are spaced 3 seconds apart. An `iperf3` server runs one test at a time and needs a moment to reset between +them: starting the next test the instant the previous one ends makes it fail on the control connection, most often +right after an unlimited UDP test, which leaves the server draining its buffers. Measured on a unit, back to back +tests failed in two runs out of three, and none failed with the gap. + +After the last test the instance stays alive idling, which keeps its logs available instead of having the unit +restart it in a loop. + +## Environment + +| Variable | Default | Meaning | +| --------------- | -------- | ------------------------------------------------ | +| `TARGET` | per item | Server hostname or IP. Required. | +| `DURATION` | `5` | Length of every single test, in seconds. | +| `PORT` | `5201` | `iperf3` port, the same for client and server. | +| `UDP_BANDWIDTH` | `0` | Target rate for the UDP tests (`0` is unlimited). | + +## Results + +Two destinations, on purpose. + +The **log** gets every test in full, `iperf3`'s own JSON document included, as one line per test: the error text, the +retransmit counts, the per-second intervals and the CPU utilisation of both sides. That is what makes a failed or +surprising run explicable afterwards, and none of it belongs in a time series. + +**VictoriaMetrics** gets only what is worth charting, pushed as `benchmark_result` samples bracketed by +`checkpoint_event` Start/Stop, in the same shape as `services/template/py`: + +| Sample name | From | +| -------------------------- | ------------- | +| ` throughput, Mbps` | every test | +| ` loss, %` | the UDP tests | +| ` jitter, ms` | the UDP tests | + +The `source` label is the instance's `AOS_INSTANCE_ID`, which is what tells instances apart once a scenario is run at +scale. + +## Setting up each scenario + +**service -> service** needs nothing beyond installing both items: the server sets `hostname: bandwidth-server` and +the client reaches it by that name. + +**service -> unit** needs an `iperf3` server on the node, bound to the address the client dials: + +```console +setsid iperf3 -s -p 5201 -B 10.0.0.100 /tmp/iperf3-unit.log 2>&1 & +``` + +**service -> external** needs the same on the host outside the unit: + +```console +setsid iperf3 -s -p 5201 -B 10.0.0.1 /tmp/iperf3-external.log 2>&1 & +``` + +`10.0.0.1` is the host's address on the bridge carrying the unit's network — the address that faces the unit, and a +stable one, unlike the Aos service bridge that is recreated with a new subnet on every deployment. + +### Why the bind matters + +A node has more than one address on the path to a container, and without an explicit bind the reply is routed back +over the service bridge and carries the bridge address as its source. `iperf3` connects its UDP socket, so the kernel +drops datagrams arriving from another address — and the symptom is characteristic: the UDP tests fail while the TCP +ones pass. Verified on a unit, where UDP failed against `10.0.0.100` without `-B` and worked with it. + +Binding the bridge address instead would also work, but it is a worse choice: that bridge does not exist at boot and +changes between deployments. + +`setsid` matters when starting a server over SSH: a plain `&` leaves it attached to the session and it dies on +logout, which then looks like the benchmark failing to reach the far side. + +### Before deploying + +The server item's own `iperf3` listens on 5201 too, but inside a container namespace, so it does not occupy the +node's port; a leftover from an earlier run does. On Debian and Ubuntu the `iperf3` package also ships an enabled +`iperf3.service` on `*:5201` — either use it as is, since it answers on every address, or take the port over with +`sudo systemctl disable --now iperf3`. + +```console +ss -lntu | grep 5201 +iperf3 -c 10.0.0.100 -p 5201 -t 2 +``` + +## Requirements + +`iperf3` and `python3` must be in the container rootfs, which they are: service containers run on the node rootfs, +and `aos-image-vm` installs both. + +`tmpLimit` is required rather than decorative. `iperf3` creates a temporary buffer file under `/tmp` for every +stream, and the container rootfs is read-only, so without that quota every test fails on both sides with +`unable to create a new stream: Read-only file system`. + +## Reading the numbers + +Between two instances on one node there is no wire: the traffic goes through `veth` and a bridge, in RAM. TCP lands +in the tens of Gbit/s with the sending side pinned at ~100% of a core, which makes the figure a floor set by CPU +rather than the capacity of a link. Raising `cpuLimit` raises the number, which is the clearest sign of what is +actually being measured. + +UDP with `UDP_BANDWIDTH=0` measures something narrower still: `iperf3` sends 1448 byte datagrams, so the test becomes +a syscall rate benchmark and reports several times less than TCP on the same path. That is not UDP being slower; it +is the datagram size. Loss reported under those conditions is the receiver failing to keep up, not the network +dropping traffic. diff --git a/benchmark/services/network/bandwidth/config.yaml b/benchmark/services/network/bandwidth/config.yaml new file mode 100644 index 0000000..0383838 --- /dev/null +++ b/benchmark/services/network/bandwidth/config.yaml @@ -0,0 +1,114 @@ +schemaVersion: 2 +publisher: + author: Developer Name + company: Company Name +publish: + tlsKey: aos-user-sp.p12 +items: +- identity: + type: service + codename: benchmark-network-bandwidth-server + title: Benchmark network bandwidth server + description: iperf3 server for the network bandwidth benchmark + version: 1.0.0-beta.1 + images: + - sourceFolder: src/server + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u bandwidth_server.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 256MiB + tmpLimit: 64MiB + hostname: bandwidth-server + env: + - PORT=5201 +- identity: + type: service + codename: benchmark-network-bandwidth-client-service + title: Benchmark network bandwidth client (service to service) + description: Throughput between two services on the same unit + version: 1.0.0-beta.1 + images: + - sourceFolder: src/client + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u bandwidth_client.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 256MiB + tmpLimit: 64MiB + resources: + - name: victoria-metrics + mode: rw + env: + - TARGET=bandwidth-server + - DURATION=5 + - PORT=5201 + - UDP_BANDWIDTH=0 +- identity: + type: service + codename: benchmark-network-bandwidth-client-unit + title: Benchmark network bandwidth client (service to unit) + description: Throughput between a service and the node it runs on + version: 1.0.0-beta.1 + images: + - sourceFolder: src/client + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u bandwidth_client.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 256MiB + tmpLimit: 64MiB + resources: + - name: victoria-metrics + mode: rw + env: + - TARGET=10.0.0.100 + - DURATION=5 + - PORT=5201 + - UDP_BANDWIDTH=0 +- identity: + type: service + codename: benchmark-network-bandwidth-client-external + title: Benchmark network bandwidth client (service to external) + description: Throughput between a service and a host outside the unit + version: 1.0.0-beta.1 + images: + - sourceFolder: src/client + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u bandwidth_client.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 256MiB + tmpLimit: 64MiB + resources: + - name: victoria-metrics + mode: rw + env: + - TARGET=10.0.0.1 + - DURATION=5 + - PORT=5201 + - UDP_BANDWIDTH=0 diff --git a/benchmark/services/network/bandwidth/src/client/bandwidth_client.py b/benchmark/services/network/bandwidth/src/client/bandwidth_client.py new file mode 100644 index 0000000..6342906 --- /dev/null +++ b/benchmark/services/network/bandwidth/src/client/bandwidth_client.py @@ -0,0 +1,301 @@ +#!/usr/bin/env python3 +"""Bandwidth benchmark item: measures throughput with iperf3 and reports it. + +Runs four tests in a row against TARGET, each for DURATION seconds: TCP and +UDP, in both directions. iperf3 does the measuring; this script only starts it, +parses its JSON and reports the result. + +Results go two ways. Every test's full result, iperf3's own JSON document +included, is printed to the service log, which is what makes a failed run +diagnosable afterwards. Only the few numbers worth charting are pushed to +VictoriaMetrics as benchmark_result samples, bracketed by checkpoint_event +Start/Stop, exactly as services/template/py does. + +Configuration comes from the environment, so one image serves every scenario +and only the deployment differs: + TARGET server hostname or IP (required) + DURATION seconds per test (default 5) + PORT iperf3 port (default 5201) + UDP_BANDWIDTH target rate for the UDP tests, 0 is unlimited (default 0) + +Usage: + bandwidth_client.py [--victoria-url http://victoriametrics:8428] +""" + +import argparse +import datetime +import json +import os +import subprocess +import sys +import time +import urllib.error +import urllib.request + +TARGET = os.environ.get("TARGET", "") +DURATION = os.environ.get("DURATION", "5") +PORT = os.environ.get("PORT", "5201") +UDP_BANDWIDTH = os.environ.get("UDP_BANDWIDTH", "0") + +CONNECT_ATTEMPTS = 30 +CONNECT_DELAY = 2 +TEST_GAP = 3 +IDLE_DELAY = 60 + +NODE = "main" # No mechanism yet for an instance to learn which node it's actually running on. + +# Test name -> extra iperf3 arguments. "up" is client -> server, "down" is +# server -> client (iperf3 -R, reverse mode). +TESTS = [ + ("tcp_up", "tcp", []), + ("tcp_down", "tcp", ["-R"]), + ("udp_up", "udp", ["-u", "-b", UDP_BANDWIDTH]), + ("udp_down", "udp", ["-u", "-b", UDP_BANDWIDTH, "-R"]), +] + +# iperf3 error fragments that mean the server is not up yet, as opposed to the +# test itself having failed. Only these are worth retrying. +CONNECT_ERRORS = ( + "unable to connect to server", + "unable to send control message", + "unable to receive control message", + "Connection refused", + "No route to host", + "Name or service not known", + "Temporary failure in name resolution", +) + + +def parse_args(): + """Parse --victoria-url command-line option.""" + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--victoria-url", + default="http://victoriametrics:8428", + help="main node's VictoriaMetrics base URL (default: %(default)s)", + ) + return parser.parse_args() + + +def escape_label_value(value): + """Escape a string for safe embedding inside a Prometheus exposition-format label value.""" + return value.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n") + + +def format_precise_time(timestamp_us): + """Format a microsecond epoch timestamp as "YYYY-MM-DD HH:MM:SS.ffffff" (UTC).""" + seconds, microseconds = divmod(timestamp_us, 1_000_000) + dt = datetime.datetime.fromtimestamp(seconds, tz=datetime.timezone.utc) + dt += datetime.timedelta(microseconds=microseconds) + + return dt.strftime("%Y-%m-%d %H:%M:%S.%f") + + +def push_line(victoria_url, line): + """POST a single Prometheus exposition-format line to VictoriaMetrics.""" + request = urllib.request.Request( + f"{victoria_url.rstrip('/')}/api/v1/import/prometheus", + data=line.encode(), + method="POST", + ) + + try: + with urllib.request.urlopen(request, timeout=5) as response: + response.read() + except urllib.error.URLError as err: + print(f"failed to push to VictoriaMetrics: {err}", file=sys.stderr) + + +def push_event(victoria_url, node, source, event): + """Push a checkpoint_event sample (the same metric event_exporter.py produces).""" + timestamp_us = int(time.time() * 1_000_000) + labels = ",".join( + f'{name}="{escape_label_value(value)}"' + for name, value in ( + ("node", node), + ("source", source), + ("event", event), + ("time_us", format_precise_time(timestamp_us)), + ) + ) + time_s = timestamp_us / 1_000_000 + push_line(victoria_url, f"checkpoint_event{{{labels}}} 1 {time_s:.3f}") + + +def push_result(victoria_url, node, source, name, value): + """Push a single benchmark_result sample for one measured value.""" + timestamp_us = int(time.time() * 1_000_000) + labels = ",".join( + f'{label}="{escape_label_value(text)}"' + for label, text in ( + ("node", node), + ("source", source), + ("name", name), + ("time_us", format_precise_time(timestamp_us)), + ) + ) + time_s = timestamp_us / 1_000_000 + push_line(victoria_url, f"benchmark_result{{{labels}}} {value} {time_s:.3f}") + + +def throughput_bps(result, protocol): + """Extract the throughput iperf3 measured, in bits per second. + + TCP results are reported per direction in `sum_sent` / `sum_received`; the + receiver side is what actually arrived, so it is preferred. UDP results + carry the interesting numbers (including loss and jitter) in `sum`. + """ + end = result.get("end", {}) + + if protocol == "udp": + sections = ("sum", "sum_received", "sum_sent") + else: + sections = ("sum_received", "sum_sent", "sum") + + for name in sections: + section = end.get(name) + if isinstance(section, dict) and "bits_per_second" in section: + return section["bits_per_second"] + + return None + + +def udp_stats(result): + """Loss and jitter counters reported for a UDP test, if present.""" + summary = result.get("end", {}).get("sum") + if not isinstance(summary, dict): + return {} + + keys = ("jitter_ms", "lost_packets", "packets", "lost_percent") + + return {key: summary[key] for key in keys if key in summary} + + +def is_connect_error(message): + return any(fragment in message for fragment in CONNECT_ERRORS) + + +def run_iperf(extra_args): + """Run one iperf3 test and return its parsed result and error message.""" + cmd = ["iperf3", "-c", TARGET, "-p", PORT, "-t", DURATION, "-J"] + extra_args + + print(f"Running: {' '.join(cmd)}") + + process = subprocess.run(cmd, capture_output=True, text=True) + + try: + result = json.loads(process.stdout) + except ValueError: + error = process.stderr.strip() or f"iperf3 exited with code {process.returncode}" + return None, error + + # With -J iperf3 reports failures inside the JSON document itself. + return result, result.get("error", "") + + +def run_test(name, protocol, extra_args, attempts=1): + """Run one test, log its whole result, and return the values worth charting. + + The server instance may still be starting when the client starts, so the + first test is given several attempts: otherwise its result would say more + about instance start order than about the network. + """ + for attempt in range(1, attempts + 1): + result, error = run_iperf(extra_args) + + if not error or not is_connect_error(error) or attempt == attempts: + break + + print(f"Server {TARGET}:{PORT} not ready ({attempt}/{attempts}): {error}") + time.sleep(CONNECT_DELAY) + + metric = { + "test": name, + "protocol": protocol, + "target": TARGET, + "port": int(PORT), + "duration_s": int(DURATION), + } + + if error: + metric["error"] = error + else: + metric["throughput_bps"] = throughput_bps(result, protocol) + + if protocol == "udp": + metric.update(udp_stats(result)) + + if result is not None: + metric["raw"] = result + + # The log keeps everything, iperf3's own document included; only a handful + # of numbers are worth a time series. + print(json.dumps(metric)) + + values = {} + + if metric.get("throughput_bps") is not None: + values[f"{name} throughput, Mbps"] = round(metric["throughput_bps"] / 1e6, 3) + + if metric.get("lost_percent") is not None: + values[f"{name} loss, %"] = metric["lost_percent"] + + if metric.get("jitter_ms") is not None: + values[f"{name} jitter, ms"] = metric["jitter_ms"] + + return values + + +def run_benchmark(): + """Run the four tests and return their results as {value_name: value}.""" + results = {} + attempts = CONNECT_ATTEMPTS + + for index, (name, protocol, extra_args) in enumerate(TESTS): + # An iperf3 server runs one test at a time and needs a moment to reset + # between them. Starting the next test the instant the previous one + # ends makes it fail on the control connection, most often right after + # an unlimited UDP test, which leaves the server draining its buffers. + if index: + time.sleep(TEST_GAP) + + results.update(run_test(name, protocol, extra_args, attempts)) + attempts = 1 + + return results + + +def main(): + """Push a start event, run the benchmark, push its results, then push a stop event.""" + args = parse_args() + source = os.environ["AOS_INSTANCE_ID"] + + if not TARGET: + print("TARGET environment variable is required", file=sys.stderr) + return 1 + + print( + f"Bandwidth benchmark: target={TARGET} port={PORT} " + f"duration={DURATION}s udp_bandwidth={UDP_BANDWIDTH}" + ) + + push_event(args.victoria_url, NODE, source, "Start") + + try: + results = run_benchmark() + + for name, value in results.items(): + push_result(args.victoria_url, NODE, source, name, value) + finally: + push_event(args.victoria_url, NODE, source, "Stop") + + print("All tests finished") + + # The benchmark is a one shot run, but the instance keeps running so its + # logs stay available and the unit does not restart it in a loop. + while True: + time.sleep(IDLE_DELAY) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/benchmark/services/network/bandwidth/src/server/bandwidth_server.py b/benchmark/services/network/bandwidth/src/server/bandwidth_server.py new file mode 100644 index 0000000..038fda0 --- /dev/null +++ b/benchmark/services/network/bandwidth/src/server/bandwidth_server.py @@ -0,0 +1,23 @@ +import os +import subprocess +import time + +PORT = os.environ.get("PORT", "5201") + +RESTART_DELAY = 5 + + +def main(): + cmd = ["iperf3", "-s", "-p", PORT] + + # iperf3 -s never exits on its own, so this loop only matters if it dies: + # the service instance stays alive and the server comes back. + while True: + print(f"Starting iperf3 server: {' '.join(cmd)}") + code = subprocess.call(cmd) + print(f"iperf3 server exited with code {code}, restarting in {RESTART_DELAY}s") + time.sleep(RESTART_DELAY) + + +if __name__ == "__main__": + main() diff --git a/benchmark/services/network/dns/README.md b/benchmark/services/network/dns/README.md new file mode 100644 index 0000000..e5c2683 --- /dev/null +++ b/benchmark/services/network/dns/README.md @@ -0,0 +1,147 @@ +# DNS resolve time + +How long a service takes to resolve a name through the `dnsmasq` an AosEdge unit runs for its instances, reported as +percentiles. + +`config.yaml` holds four items: an idle peer that exists only to own a name, and one client per scenario. The client +code is the same for all three — a scenario is a different value of one environment variable, not different code. + +| Item | Scenario | `NAME` | Where the name lives | +| --------------------------------------- | ------------------ | ---------------- | ------------------------------------ | +| `benchmark-network-dns-peer` | — | — | registered by the unit while it runs | +| `benchmark-network-dns-client-service` | service -> service | `dns-peer` | the peer item above | +| `benchmark-network-dns-client-unit` | service -> unit | `main` | `/etc/aos/addnhosts` on the node | +| `benchmark-network-dns-client-external` | service -> external| `dns-probe.test` | a DNS server on the host | + +Install only the items a run needs. + +## Why there is no dig + +The benchmark plan names `dig` as the tool, but `dig` reports its query time in whole milliseconds +(`;; Query time: 0 msec`), and a local `dnsmasq` answers in hundreds of microseconds, so every sample would read as +zero. Timing the `dig` process from outside is worse: starting it costs more than the query it is supposed to +measure. + +So the client sends the query itself — a UDP socket, a hand built query packet, and `time.perf_counter()` around it. +That gives microsecond resolution, no process startup inside the measurement, and enough samples for percentiles. It +also keeps `dig`, and the `bind-utils` package behind it, out of the image: `python3` is already in the container. + +## Which resolver is measured + +With `RESOLVER` unset the client reads every `nameserver` from the container's `/etc/resolv.conf` and measures +against the first one that answers, reporting which in the log. + +That fallback is not decoration. The container is handed two nameservers, the service bridge address first and the +node address second, but `dnsmasq` is configured with `listen-address` set to the node address only — so nothing is +listening on the first one and a query there just times out. A libc resolver walks the list until something replies, +which is why service names resolve normally, and the client does the same. Measuring the first entry blindly would +report timeouts instead of resolve times. + +## Environment + +| Variable | Default | Meaning | +| -------------- | -------- | ------------------------------------------------------------------------ | +| `NAME` | per item | Name to resolve. Required. | +| `QUERIES` | `2000` | How many queries to send. | +| `RANDOM_LABEL` | per item | Prepend a random label to `NAME` on every query, to defeat the DNS cache. | +| `RESOLVER` | unset | DNS server to ask, `host` or `host:port`. Unset means `/etc/resolv.conf`. | + +## Results + +Two destinations, on purpose. + +The **log** gets the whole result, every individual sample included, so percentiles can be recomputed later or several +instances pooled into one distribution without re-running anything. It also carries the failure breakdown: a timeout, +a non-zero RCODE, or an answer with no records, counted by reason. + +**VictoriaMetrics** gets the three percentiles the benchmark plan names, pushed as `benchmark_result` samples +bracketed by `checkpoint_event` Start/Stop, in the same shape as `services/template/py`: + +| Sample name | +| ------------------ | +| `resolve p50, us` | +| `resolve p99, us` | +| `resolve p999, us` | + +At `QUERIES=2000`, `p99` rests on 20 samples and `p999` on 2, so treat `p999` as an indication and raise `QUERIES` if +it matters. The `source` label is the instance's `AOS_INSTANCE_ID`, which tells instances apart once a scenario is +run at scale. + +## Setting up each scenario + +**service -> service** needs nothing: install the peer alongside the client, and the unit registers `dns-peer` in +`dnsmasq` for as long as the peer instance runs. `RANDOM_LABEL` stays off — the name is answered out of a local file +every time, so no cache sits in the way. + +**service -> unit** needs nothing either on a stock unit. The unit's `dnsmasq` reads two hosts files: + +``` +addn-hosts=/var/aos/dns/addnhosts # written by Aos, rewritten on every deployment +addn-hosts=/etc/aos/addnhosts # static, for names the unit owner adds +``` + +and the second already carries `10.0.0.100 main`, which is why `NAME` defaults to `main`. To measure a different +name, add it there and make `dnsmasq` re-read the file: + +```console +echo "10.0.0.100 dns-probe-unit" >> /etc/aos/addnhosts +kill -HUP $(cat /var/aos/dns/pidfile) +``` + +Not `/var/aos/dns/addnhosts` — Aos rewrites that file whenever instances change. + +**service -> external** needs a DNS server on the host holding the name, and a unit that forwards to it. The +forwarding is usually already there: check the node's `/etc/resolv.conf` for `nameserver 10.0.0.1`, and if it is +listed, nothing on the unit has to change. + +The host normally already runs a `dnsmasq` on the bridge, serving DHCP for it: + +```console +ps -eo pid,args | grep [d]nsmasq +ss -lnup | grep 10.0.0.1:53 +``` + +Add the record to `/etc/dnsmasq.conf`: + +``` +address=/dns-probe.test/10.0.0.1 +``` + +`dnsmasq` reads that file at startup unless given `-C`, so it is the right place even for an instance launched by +hand. `/etc/dnsmasq.d/` is not: on a stock Ubuntu the `conf-dir` line is commented out, so the directory is never +read and the record is silently ignored. `/etc/hosts` cannot express a wildcard, so it is not an option either. + +The wildcard form matters. `address=/dns-probe.test/10.0.0.1` answers the domain and everything under it, which is +what makes `RANDOM_LABEL=1` work — and it is on for this scenario for a reason: `dnsmasq` on the unit caches +answers, so a repeated name would be served from cache after the first query and the scenario would stop describing +resolution. + +`address=` is only read at startup — a `SIGHUP` re-reads hosts files, not the configuration — so restart it exactly +as it was running: + +```console +sudo kill +sudo dnsmasq --interface=aos-br0 --dhcp-range=10.0.0.101,10.0.0.254,12h \ + --dhcp-option=3,10.0.0.1 --dhcp-option=6,10.0.0.1 --bind-interfaces +``` + +That process usually serves DHCP for the bridge, so anything holding a lease will notice the second it is down; a +unit with a static address does not. Verify before deploying: + +```console +nslookup probe123.dns-probe.test 10.0.0.100 +``` + +## Requirements + +`python3` in the container rootfs, which is already there. Nothing else — no `dig`, no layer, no image change. + +## Reading the numbers + +The service -> service and service -> unit scenarios measure the same thing on this stack and come out equal: both +names live in files read by the same `dnsmasq` on the node, and the packets travel the same bridge either way. Only +the external scenario leaves the node, and it costs several times more — that gap is the price of forwarding. + +Comparing a resolve time against the latency benchmark's round trip time on the same path is worth doing: locally the +two come out nearly identical, which says the lookup inside `dnsmasq` is almost free and what is being measured is +the network round trip. diff --git a/benchmark/services/network/dns/config.yaml b/benchmark/services/network/dns/config.yaml new file mode 100644 index 0000000..d1c1146 --- /dev/null +++ b/benchmark/services/network/dns/config.yaml @@ -0,0 +1,105 @@ +schemaVersion: 2 +publisher: + author: Developer Name + company: Company Name +publish: + tlsKey: aos-user-sp.p12 +items: +- identity: + type: service + codename: benchmark-network-dns-peer + title: Benchmark network DNS peer + description: Idle peer whose name the DNS benchmark resolves + version: 1.0.0-beta.1 + images: + - sourceFolder: src/peer + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u dns_peer.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 128MiB + hostname: dns-peer +- identity: + type: service + codename: benchmark-network-dns-client-service + title: Benchmark network DNS client (service to service) + description: Resolve time for a peer service name + version: 1.0.0-beta.1 + images: + - sourceFolder: src/client + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u dns_client.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 128MiB + resources: + - name: victoria-metrics + mode: rw + env: + - NAME=dns-peer + - QUERIES=2000 + - RANDOM_LABEL=0 +- identity: + type: service + codename: benchmark-network-dns-client-unit + title: Benchmark network DNS client (service to unit) + description: Resolve time for a name registered on the node + version: 1.0.0-beta.1 + images: + - sourceFolder: src/client + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u dns_client.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 128MiB + resources: + - name: victoria-metrics + mode: rw + env: + - NAME=main + - QUERIES=2000 + - RANDOM_LABEL=0 +- identity: + type: service + codename: benchmark-network-dns-client-external + title: Benchmark network DNS client (service to external) + description: Resolve time for a name served outside the unit + version: 1.0.0-beta.1 + images: + - sourceFolder: src/client + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u dns_client.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 128MiB + resources: + - name: victoria-metrics + mode: rw + env: + - NAME=dns-probe.test + - QUERIES=2000 + - RANDOM_LABEL=1 diff --git a/benchmark/services/network/dns/src/client/dns_client.py b/benchmark/services/network/dns/src/client/dns_client.py new file mode 100644 index 0000000..a2fd872 --- /dev/null +++ b/benchmark/services/network/dns/src/client/dns_client.py @@ -0,0 +1,372 @@ +#!/usr/bin/env python3 +"""DNS benchmark item: measures name resolution time and reports it. + +Resolves NAME QUERIES times against the resolver the container was handed, +timing each query from just before the packet leaves to just after the matching +answer arrives, and reports the distribution as percentiles. + +The query is built and sent here rather than shelled out to dig on purpose. dig +reports its query time in whole milliseconds, and a local dnsmasq answers in +hundreds of microseconds, so every sample would read as zero; timing the dig +process from outside is worse still, as starting it costs more than the query. +Doing it in-process also keeps dig, and the bind-utils package behind it, out +of the image. + +Results go two ways. The full result, every individual sample included, is +printed to the service log. Only the percentiles worth charting are pushed to +VictoriaMetrics as benchmark_result samples, bracketed by checkpoint_event +Start/Stop, exactly as services/template/py does. + +Configuration comes from the environment, so one image serves every scenario +and only the deployment differs: + NAME name to resolve (required) + QUERIES how many queries to send (default 2000) + RANDOM_LABEL 1 prepends a random label to every query, to miss the cache + RESOLVER DNS server to ask, host or host:port; unset means the + nameservers from the container's /etc/resolv.conf + +Usage: + dns_client.py [--victoria-url http://victoriametrics:8428] +""" + +import argparse +import datetime +import json +import os +import random +import socket +import string +import struct +import sys +import time +import urllib.error +import urllib.request + +NAME = os.environ.get("NAME", "") +RESOLVER = os.environ.get("RESOLVER", "") +QUERIES = int(os.environ.get("QUERIES", "2000")) +RANDOM_LABEL = os.environ.get("RANDOM_LABEL", "0") == "1" + +CONNECT_ATTEMPTS = 30 +CONNECT_DELAY = 2 +QUERY_TIMEOUT = 2 +IDLE_DELAY = 60 + +RESOLV_CONF = "/etc/resolv.conf" +DNS_PORT = 53 + +NODE = "main" # No mechanism yet for an instance to learn which node it's actually running on. + +# Percentiles are picked the way sockperf picks them, so DNS and latency +# numbers can be read side by side. +PERCENTILES = (("p50_us", 0.50), ("p90_us", 0.90), ("p99_us", 0.99), ("p999_us", 0.999)) + +# Of those, the ones worth a time series. +CHARTED = (("p50_us", "p50"), ("p99_us", "p99"), ("p999_us", "p999")) + + +def parse_args(): + """Parse --victoria-url command-line option.""" + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--victoria-url", + default="http://victoriametrics:8428", + help="main node's VictoriaMetrics base URL (default: %(default)s)", + ) + return parser.parse_args() + + +def escape_label_value(value): + """Escape a string for safe embedding inside a Prometheus exposition-format label value.""" + return value.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n") + + +def format_precise_time(timestamp_us): + """Format a microsecond epoch timestamp as "YYYY-MM-DD HH:MM:SS.ffffff" (UTC).""" + seconds, microseconds = divmod(timestamp_us, 1_000_000) + dt = datetime.datetime.fromtimestamp(seconds, tz=datetime.timezone.utc) + dt += datetime.timedelta(microseconds=microseconds) + + return dt.strftime("%Y-%m-%d %H:%M:%S.%f") + + +def push_line(victoria_url, line): + """POST a single Prometheus exposition-format line to VictoriaMetrics.""" + request = urllib.request.Request( + f"{victoria_url.rstrip('/')}/api/v1/import/prometheus", + data=line.encode(), + method="POST", + ) + + try: + with urllib.request.urlopen(request, timeout=5) as response: + response.read() + except urllib.error.URLError as err: + print(f"failed to push to VictoriaMetrics: {err}", file=sys.stderr) + + +def push_event(victoria_url, node, source, event): + """Push a checkpoint_event sample (the same metric event_exporter.py produces).""" + timestamp_us = int(time.time() * 1_000_000) + labels = ",".join( + f'{name}="{escape_label_value(value)}"' + for name, value in ( + ("node", node), + ("source", source), + ("event", event), + ("time_us", format_precise_time(timestamp_us)), + ) + ) + time_s = timestamp_us / 1_000_000 + push_line(victoria_url, f"checkpoint_event{{{labels}}} 1 {time_s:.3f}") + + +def push_result(victoria_url, node, source, name, value): + """Push a single benchmark_result sample for one measured value.""" + timestamp_us = int(time.time() * 1_000_000) + labels = ",".join( + f'{label}="{escape_label_value(text)}"' + for label, text in ( + ("node", node), + ("source", source), + ("name", name), + ("time_us", format_precise_time(timestamp_us)), + ) + ) + time_s = timestamp_us / 1_000_000 + push_line(victoria_url, f"benchmark_result{{{labels}}} {value} {time_s:.3f}") + + +def default_resolvers(): + """Every nameserver the container was handed, in order. + + All of them are returned, not just the first, because not all of them + necessarily answer: on this stack the container is handed the bridge + address first, while dnsmasq is bound to the node address only, so a query + to the first one just times out. A libc resolver walks the list until + something replies, and so does this client. + """ + resolvers = [] + + try: + with open(RESOLV_CONF) as conf: + for line in conf: + fields = line.split() + if len(fields) >= 2 and fields[0] == "nameserver": + resolvers.append(fields[1]) + except OSError: + pass + + return resolvers + + +def resolver_address(resolver): + """Split an optional port off the resolver, so a non standard one can be used.""" + host, separator, port = resolver.partition(":") + + return (host, int(port)) if separator else (host, DNS_PORT) + + +def build_query(name, query_id): + """A minimal DNS query for an A record.""" + # Standard query, recursion desired, one question. + header = struct.pack("!HHHHHH", query_id, 0x0100, 1, 0, 0, 0) + labels = b"".join(bytes([len(p)]) + p.encode() for p in name.split(".") if p) + # QTYPE A, QCLASS IN. + return header + labels + b"\x00" + struct.pack("!HH", 1, 1) + + +def query_once(sock, resolver, name): + """Send one query and return how long the answer took, in microseconds.""" + query_id = random.getrandbits(16) + packet = build_query(name, query_id) + + start = time.perf_counter() + sock.sendto(packet, resolver_address(resolver)) + + while True: + try: + reply, _ = sock.recvfrom(4096) + except socket.timeout: + return None, "timeout" + + # Ignore anything that is not the answer to this query. + if len(reply) >= 12 and struct.unpack("!H", reply[:2])[0] == query_id: + break + + elapsed_us = (time.perf_counter() - start) * 1e6 + + flags, _, answers = struct.unpack("!HHH", reply[2:8]) + rcode = flags & 0x0F + + if rcode != 0: + return None, f"rcode {rcode}" + + if answers == 0: + return None, "no answer records" + + return elapsed_us, "" + + +def percentile(sorted_samples, fraction): + index = int(0.5 + fraction * len(sorted_samples)) - 1 + + return sorted_samples[max(index, 0)] + + +def summarize(samples): + ordered = sorted(samples) + count = len(ordered) + mean = sum(ordered) / count + variance = sum((value - mean) ** 2 for value in ordered) / count + + result = {name: round(percentile(ordered, fraction), 3) for name, fraction in PERCENTILES} + result["min_us"] = round(ordered[0], 3) + result["max_us"] = round(ordered[-1], 3) + result["avg_us"] = round(mean, 3) + result["stddev_us"] = round(variance ** 0.5, 3) + + return result + + +def query_name(): + """The name to resolve, with a unique label when the cache has to be missed. + + dnsmasq caches answers, so repeating one name measures its cache rather + than resolution. A random leftmost label defeats that, and it still + resolves as long as the DNS server holding the name answers for the whole + domain. + """ + if not RANDOM_LABEL: + return NAME + + label = "".join(random.choice(string.ascii_lowercase) for _ in range(12)) + + return f"{label}.{NAME}" + + +def pick_resolver(candidates): + """Settle on a resolver that answers, and wait for the name to exist. + + Two things can keep the first query from succeeding, and both are normal: + a nameserver in the list may not be listening at all, and a peer service + may still be starting, since the unit registers its name only once the + instance runs. So every candidate is tried on every attempt, and the first + one that answers wins. + """ + sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + sock.settimeout(QUERY_TIMEOUT) + + try: + for attempt in range(1, CONNECT_ATTEMPTS + 1): + for candidate in candidates: + _, error = query_once(sock, candidate, query_name()) + + if not error: + return candidate + + print(f"{candidate} did not resolve {NAME} ({attempt}/{CONNECT_ATTEMPTS}): {error}") + + time.sleep(CONNECT_DELAY) + finally: + sock.close() + + return "" + + +def run_benchmark(resolver): + """Resolve the name QUERIES times and return the results as {value_name: value}.""" + samples = [] + failures = {} + + sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + sock.settimeout(QUERY_TIMEOUT) + + try: + for _ in range(QUERIES): + elapsed_us, error = query_once(sock, resolver, query_name()) + + if error: + failures[error] = failures.get(error, 0) + 1 + else: + samples.append(elapsed_us) + finally: + sock.close() + + metric = { + "test": "dns_resolve", + "name": NAME, + "resolver": resolver, + "queries": QUERIES, + "resolved": len(samples), + "failed": QUERIES - len(samples), + } + + if failures: + metric["failures"] = failures + + if samples: + metric.update(summarize(samples)) + metric["raw"] = {"samples_us": [round(value, 3) for value in samples]} + else: + metric["error"] = "; ".join(f"{reason} x{count}" for reason, count in failures.items()) + + # The log keeps everything, every individual sample included, so the + # percentiles can be recomputed later; only a few are worth a time series. + print(json.dumps(metric)) + + return { + f"resolve {label}, us": metric[key] + for key, label in CHARTED + if metric.get(key) is not None + } + + +def main(): + """Push a start event, run the benchmark, push its results, then push a stop event.""" + args = parse_args() + source = os.environ["AOS_INSTANCE_ID"] + + if not NAME: + print("NAME environment variable is required", file=sys.stderr) + return 1 + + candidates = [RESOLVER] if RESOLVER else default_resolvers() + + if not candidates: + print(f"No resolver: RESOLVER is unset and {RESOLV_CONF} has no nameserver", file=sys.stderr) + return 1 + + print( + f"DNS benchmark: name={NAME} resolvers={','.join(candidates)} " + f"queries={QUERIES} random_label={int(RANDOM_LABEL)}" + ) + + resolver = pick_resolver(candidates) + + if not resolver: + resolver = candidates[0] + print(f"No resolver answered for {NAME}, measuring against {resolver} anyway") + else: + print(f"Using resolver {resolver}") + + push_event(args.victoria_url, NODE, source, "Start") + + try: + results = run_benchmark(resolver) + + for name, value in results.items(): + push_result(args.victoria_url, NODE, source, name, value) + finally: + push_event(args.victoria_url, NODE, source, "Stop") + + print("All tests finished") + + # The benchmark is a one shot run, but the instance keeps running so its + # logs stay available and the unit does not restart it in a loop. + while True: + time.sleep(IDLE_DELAY) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/benchmark/services/network/dns/src/peer/dns_peer.py b/benchmark/services/network/dns/src/peer/dns_peer.py new file mode 100644 index 0000000..de9dea1 --- /dev/null +++ b/benchmark/services/network/dns/src/peer/dns_peer.py @@ -0,0 +1,17 @@ +import os +import time + +HEARTBEAT_DELAY = 60 + +# The peer exists only to own a DNS name: the unit registers a running +# instance in the per-bridge dnsmasq, and that registration is what the client +# measures the resolution of. There is nothing to serve, so it just stays up. +def main(): + print(f"DNS peer running as {os.environ.get('AOS_INSTANCE_ID', 'unknown instance')}") + + while True: + time.sleep(HEARTBEAT_DELAY) + + +if __name__ == "__main__": + main() diff --git a/benchmark/services/network/latency/README.md b/benchmark/services/network/latency/README.md new file mode 100644 index 0000000..cda34b4 --- /dev/null +++ b/benchmark/services/network/latency/README.md @@ -0,0 +1,142 @@ +# Latency + +Round trip time through the container network, measured with +[`sockperf`](https://github.com/Mellanox/sockperf) as a ping-pong request/response test over TCP and UDP, and +reported as percentiles. + +`config.yaml` holds four items: one server, and one client per scenario. The client code is the same for all three — +a scenario is a different value of one environment variable, not different code. + +| Item | Scenario | `TARGET` | Server side | +| ------------------------------------------- | ------------------ | ---------------- | ---------------------- | +| `benchmark-network-latency-server` | — | — | this bundle | +| `benchmark-network-latency-client-service` | service -> service | `latency-server` | the server item above | +| `benchmark-network-latency-client-unit` | service -> unit | `10.0.0.100` | `sockperf` on the node | +| `benchmark-network-latency-client-external` | service -> external| `10.0.0.1` | `sockperf` on the host | + +Install only the items a run needs: three clients installed at once would measure each other's interference. + +## Why percentiles, not an average + +Latency distributions are skewed: most round trips sit close to the minimum, and a thin tail of rare ones runs orders +of magnitude longer. An average hides that tail — one round trip of 50 ms among ten thousand of 40 µs moves the +average by five microseconds and disappears. The tail is what real-time and RPC traffic feel: an operation making a +hundred sequential calls has roughly a 63% chance of hitting at least one p99 event. + +On this stack the tail comes from scheduling, from conntrack and nftables handling a new flow, and from softirq work +on `veth` competing with neighbouring instances — which is why the figures are expected to change with instance +density even when the median does not. + +A percentile is only worth the samples behind it. A five second ping-pong run normally produces a few hundred +thousand round trips, which is enough for `p999`; if a run reports only a few hundred observations, treat `p999` as +noise. + +## What the client measures + +| Test | `sockperf` arguments | Protocol | +| --------- | -------------------- | -------- | +| `udp_rtt` | (none) | UDP | +| `tcp_rtt` | `--tcp` | TCP | + +Both run with `--full-rtt`, so every figure is a full round trip. Without that flag `sockperf` halves its numbers and +reports one way latency instead, which is not what the benchmark plan asks for. + +The first test is retried for up to 60 seconds while `sockperf` fails to reach the server; a run that exits cleanly +but produced no observations counts as a failure too. Later tests are not retried, and tests are spaced 3 seconds +apart so the server has a moment to reset between them. + +After the last test the instance stays alive idling, which keeps its logs available instead of having the unit +restart it in a loop. + +## Environment + +| Variable | Default | Meaning | +| ---------- | -------- | ------------------------------------------------ | +| `TARGET` | per item | Server hostname or IP. Required. | +| `DURATION` | `5` | Length of every single test, in seconds. | +| `PORT` | `11111` | `sockperf` port, the same for client and server. | +| `MSG_SIZE` | `64` | Payload size in bytes passed to `sockperf -m`. | + +## Results + +Two destinations, on purpose. + +The **log** gets every test in full, `sockperf`'s own report included: all five percentiles it prints, min, max, +average, standard deviation, the observation count and the dropped message count, plus the raw text. The tool has no +machine readable output, so everything is parsed out of that report, and keeping it makes a surprising number +checkable afterwards. + +**VictoriaMetrics** gets the three percentiles the benchmark plan names, pushed as `benchmark_result` samples +bracketed by `checkpoint_event` Start/Stop, in the same shape as `services/template/py`: + +| Sample name | From | +| ------------------ | ---------- | +| ` p50, us` | both tests | +| ` p99, us` | both tests | +| ` p999, us` | both tests | + +The `source` label is the instance's `AOS_INSTANCE_ID`, which is what tells instances apart once a scenario is run at +scale. + +## Setting up each scenario + +`sockperf` listens either on UDP or, with `--tcp`, on TCP, never on both, so every server side is two processes. TCP +and UDP port numbers are independent, so both use the same port. + +**service -> service** needs nothing beyond installing both items: the server item runs both processes itself and +sets `hostname: latency-server`. + +**service -> unit** needs them on the node, bound to the address the client dials: + +```console +setsid sockperf server -i 10.0.0.100 -p 11111 /tmp/sockperf-udp.log 2>&1 & +setsid sockperf server -i 10.0.0.100 -p 11111 --tcp /tmp/sockperf-tcp.log 2>&1 & +``` + +**service -> external** needs the same on the host, with `10.0.0.1`. Install `sockperf` there first — Debian and +Ubuntu package it: + +```console +sudo apt-get install -y sockperf +``` + +The versions on the two sides then differ, since the `meta-aos-vm` recipe pins `3.10+git` while Ubuntu ships `3.7`. +That pairing has been exercised without trouble; a failure during the handshake rather than during the test is the +sign to look here, and the fix is to build the newer one on the host rather than to downgrade the unit. + +### Why the bind matters + +A node has more than one address on the path to a container, and without an explicit bind the reply is routed back +over the service bridge and carries the bridge address as its source. The client's socket is connected to the address +it dialed, so the kernel drops the mismatch and UDP fails while TCP passes. This was measured directly with `iperf3` +on the same paths. + +`setsid` matters when starting servers over SSH: a plain `&` leaves them attached to the session and they die on +logout, which then looks like the benchmark failing to reach the far side. + +### Before deploying + +The server item's own `sockperf` listens on 11111 too, but inside a container namespace, so it does not occupy the +node's port; a leftover from an earlier run does. + +```console +ss -lntu | grep 11111 +sockperf ping-pong -i 10.0.0.100 -p 11111 -t 2 --full-rtt +sockperf ping-pong -i 10.0.0.100 -p 11111 -t 2 --full-rtt --tcp +``` + +## Requirements + +`sockperf` and `python3` must be in the container rootfs. `sockperf` is not in a stock image: it comes from the +`sockperf` recipe in `meta-aos-vm`, installed into `aos-image-vm`, which covers both the node and the service +containers that run on the node rootfs. The external host needs its own copy, from its distribution. + +## Reading the numbers + +The CPU quota matters more here than for throughput. A container that exhausts its `cpuLimit` is throttled by the +cgroup, and throttling shows up directly as spikes in the tail: `p999` degrades while `p50` stays flat. If the tail +looks implausible, check the quota before blaming the network. + +Between two instances on one node, and between an instance and its node, there is no wire — the traffic goes through +`veth` and a bridge, in RAM — so what is characterised is the node's scheduling and softirq behaviour. Only the +external scenario crosses a real boundary, and its figures sit well above the other two, tail included. diff --git a/benchmark/services/network/latency/config.yaml b/benchmark/services/network/latency/config.yaml new file mode 100644 index 0000000..1e8ad43 --- /dev/null +++ b/benchmark/services/network/latency/config.yaml @@ -0,0 +1,114 @@ +schemaVersion: 2 +publisher: + author: Developer Name + company: Company Name +publish: + tlsKey: aos-user-sp.p12 +items: +- identity: + type: service + codename: benchmark-network-latency-server + title: Benchmark network latency server + description: sockperf servers for the network latency benchmark + version: 1.0.0-beta.1 + images: + - sourceFolder: src/server + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u latency_server.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 256MiB + tmpLimit: 64MiB + hostname: latency-server + env: + - PORT=11111 +- identity: + type: service + codename: benchmark-network-latency-client-service + title: Benchmark network latency client (service to service) + description: Round trip time between two services on the same unit + version: 1.0.0-beta.1 + images: + - sourceFolder: src/client + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u latency_client.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 256MiB + tmpLimit: 64MiB + resources: + - name: victoria-metrics + mode: rw + env: + - TARGET=latency-server + - DURATION=5 + - PORT=11111 + - MSG_SIZE=64 +- identity: + type: service + codename: benchmark-network-latency-client-unit + title: Benchmark network latency client (service to unit) + description: Round trip time between a service and the node it runs on + version: 1.0.0-beta.1 + images: + - sourceFolder: src/client + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u latency_client.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 256MiB + tmpLimit: 64MiB + resources: + - name: victoria-metrics + mode: rw + env: + - TARGET=10.0.0.100 + - DURATION=5 + - PORT=11111 + - MSG_SIZE=64 +- identity: + type: service + codename: benchmark-network-latency-client-external + title: Benchmark network latency client (service to external) + description: Round trip time between a service and a host outside the unit + version: 1.0.0-beta.1 + images: + - sourceFolder: src/client + archInfo: + architecture: any + configuration: + workingDir: / + cmd: /usr/bin/python3 -u latency_client.py + instances: + minInstances: 1 + skipResourceLimits: true + quotas: + cpuLimit: 10000 + ramLimit: 256MiB + tmpLimit: 64MiB + resources: + - name: victoria-metrics + mode: rw + env: + - TARGET=10.0.0.1 + - DURATION=5 + - PORT=11111 + - MSG_SIZE=64 diff --git a/benchmark/services/network/latency/src/client/latency_client.py b/benchmark/services/network/latency/src/client/latency_client.py new file mode 100644 index 0000000..41cc412 --- /dev/null +++ b/benchmark/services/network/latency/src/client/latency_client.py @@ -0,0 +1,315 @@ +#!/usr/bin/env python3 +"""Latency benchmark item: measures round trip time with sockperf and reports it. + +Runs a ping-pong test against TARGET over UDP and over TCP, each for DURATION +seconds, and reports the round trip time as percentiles rather than an average: +latency distributions are skewed, and an average hides the tail that real-time +and RPC traffic actually feel. + +Results go two ways. Every test's full result, sockperf's own report included, +is printed to the service log, which is what makes a failed run diagnosable +afterwards. Only the percentiles worth charting are pushed to VictoriaMetrics +as benchmark_result samples, bracketed by checkpoint_event Start/Stop, exactly +as services/template/py does. + +Configuration comes from the environment, so one image serves every scenario +and only the deployment differs: + TARGET server hostname or IP (required) + DURATION seconds per test (default 5) + PORT sockperf port (default 11111) + MSG_SIZE payload size in bytes (default 64) + +Usage: + latency_client.py [--victoria-url http://victoriametrics:8428] +""" + +import argparse +import datetime +import json +import os +import re +import subprocess +import sys +import time +import urllib.error +import urllib.request + +TARGET = os.environ.get("TARGET", "") +DURATION = os.environ.get("DURATION", "5") +PORT = os.environ.get("PORT", "11111") +MSG_SIZE = os.environ.get("MSG_SIZE", "64") + +CONNECT_ATTEMPTS = 30 +CONNECT_DELAY = 2 +TEST_GAP = 3 +IDLE_DELAY = 60 + +NODE = "main" # No mechanism yet for an instance to learn which node it's actually running on. + +# Test name -> extra sockperf arguments. sockperf speaks UDP unless told +# otherwise, so only the TCP test needs a flag. +TESTS = [ + ("udp_rtt", "udp", []), + ("tcp_rtt", "tcp", ["--tcp"]), +] + +# sockperf reports percentiles by their exact label; these are the ones the +# benchmark plan asks for, plus the neighbours that make the tail readable. +PERCENTILES = { + "50.000": "p50_us", + "90.000": "p90_us", + "99.000": "p99_us", + "99.900": "p999_us", + "99.990": "p9999_us", +} + +# Of those, the ones worth a time series. +CHARTED = (("p50_us", "p50"), ("p99_us", "p99"), ("p999_us", "p999")) + +PERCENTILE_RE = re.compile(r"percentile\s+([\d.]+)\s*=\s*([\d.]+)") +MIN_RE = re.compile(r"\s+observation\s*=\s*([\d.]+)") +MAX_RE = re.compile(r"\s+observation\s*=\s*([\d.]+)") +# sockperf labels the average avg-rtt with --full-rtt and avg-latency without. +AVG_RE = re.compile(r"avg-(?:rtt|latency)\s*=\s*([\d.]+)") +STDDEV_RE = re.compile(r"std-dev\s*=\s*([\d.]+)") +OBSERVATIONS_RE = re.compile(r"Total\s+(\d+)\s+observations") +DROPPED_RE = re.compile(r"dropped messages\s*=\s*(\d+)") +ANSI_RE = re.compile(r"\x1b\[[0-9;]*m") + + +def parse_args(): + """Parse --victoria-url command-line option.""" + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--victoria-url", + default="http://victoriametrics:8428", + help="main node's VictoriaMetrics base URL (default: %(default)s)", + ) + return parser.parse_args() + + +def escape_label_value(value): + """Escape a string for safe embedding inside a Prometheus exposition-format label value.""" + return value.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n") + + +def format_precise_time(timestamp_us): + """Format a microsecond epoch timestamp as "YYYY-MM-DD HH:MM:SS.ffffff" (UTC).""" + seconds, microseconds = divmod(timestamp_us, 1_000_000) + dt = datetime.datetime.fromtimestamp(seconds, tz=datetime.timezone.utc) + dt += datetime.timedelta(microseconds=microseconds) + + return dt.strftime("%Y-%m-%d %H:%M:%S.%f") + + +def push_line(victoria_url, line): + """POST a single Prometheus exposition-format line to VictoriaMetrics.""" + request = urllib.request.Request( + f"{victoria_url.rstrip('/')}/api/v1/import/prometheus", + data=line.encode(), + method="POST", + ) + + try: + with urllib.request.urlopen(request, timeout=5) as response: + response.read() + except urllib.error.URLError as err: + print(f"failed to push to VictoriaMetrics: {err}", file=sys.stderr) + + +def push_event(victoria_url, node, source, event): + """Push a checkpoint_event sample (the same metric event_exporter.py produces).""" + timestamp_us = int(time.time() * 1_000_000) + labels = ",".join( + f'{name}="{escape_label_value(value)}"' + for name, value in ( + ("node", node), + ("source", source), + ("event", event), + ("time_us", format_precise_time(timestamp_us)), + ) + ) + time_s = timestamp_us / 1_000_000 + push_line(victoria_url, f"checkpoint_event{{{labels}}} 1 {time_s:.3f}") + + +def push_result(victoria_url, node, source, name, value): + """Push a single benchmark_result sample for one measured value.""" + timestamp_us = int(time.time() * 1_000_000) + labels = ",".join( + f'{label}="{escape_label_value(text)}"' + for label, text in ( + ("node", node), + ("source", source), + ("name", name), + ("time_us", format_precise_time(timestamp_us)), + ) + ) + time_s = timestamp_us / 1_000_000 + push_line(victoria_url, f"benchmark_result{{{labels}}} {value} {time_s:.3f}") + + +def parse(output): + """Pull the latency figures out of sockperf's report. + + sockperf has no machine readable output, so its text report is parsed. + Every value is in microseconds and, because the tests run with --full-rtt, + is a full round trip rather than the one way figure sockperf reports by + default. + """ + # sockperf colours parts of its report, so drop the escape sequences first. + output = ANSI_RE.sub("", output) + + result = {} + + for label, value in PERCENTILE_RE.findall(output): + name = PERCENTILES.get(label) + if name: + result[name] = float(value) + + for pattern, name in ( + (MIN_RE, "min_us"), + (MAX_RE, "max_us"), + (AVG_RE, "avg_us"), + (STDDEV_RE, "stddev_us"), + ): + match = pattern.search(output) + if match: + result[name] = float(match.group(1)) + + for pattern, name in ((OBSERVATIONS_RE, "observations"), (DROPPED_RE, "dropped")): + match = pattern.search(output) + if match: + result[name] = int(match.group(1)) + + return result + + +def run_sockperf(extra_args): + """Run one ping-pong test and return its output and error message.""" + cmd = [ + "sockperf", "ping-pong", + "-i", TARGET, + "-p", PORT, + "-t", DURATION, + "-m", MSG_SIZE, + "--full-rtt", + ] + extra_args + + print(f"Running: {' '.join(cmd)}") + + process = subprocess.run(cmd, capture_output=True, text=True) + + # sockperf writes its report to stdout and its diagnostics to stderr, and + # both matter: a run can exit cleanly having received nothing at all. + output = process.stdout + process.stderr + + if process.returncode != 0: + error = output.strip().splitlines()[-1] if output.strip() else \ + f"sockperf exited with code {process.returncode}" + return output, error + + return output, "" + + +def run_test(name, protocol, extra_args, attempts=1): + """Run one test, log its whole result, and return the values worth charting. + + The server may still be starting when the client reaches it, so the first + test is given several attempts: otherwise its result would say more about + start order than about the network. + """ + for attempt in range(1, attempts + 1): + output, error = run_sockperf(extra_args) + values = parse(output) + + # A run that produced no percentiles never reached the server, whatever + # its exit code says. + if not error and values.get("p50_us") is not None: + break + + if attempt == attempts: + break + + print(f"Server {TARGET}:{PORT} not ready ({attempt}/{attempts}): {error or 'no observations'}") + time.sleep(CONNECT_DELAY) + + metric = { + "test": name, + "protocol": protocol, + "target": TARGET, + "port": int(PORT), + "duration_s": int(DURATION), + "msg_size": int(MSG_SIZE), + } + + if values.get("p50_us") is None: + metric["error"] = error or "sockperf produced no observations" + else: + metric.update(values) + + metric["raw"] = {"output": output} + + # The log keeps everything, sockperf's own report included; only the + # percentiles the plan asks for are worth a time series. + print(json.dumps(metric)) + + return { + f"{name} {label}, us": metric[key] + for key, label in CHARTED + if metric.get(key) is not None + } + + +def run_benchmark(): + """Run both tests and return their results as {value_name: value}.""" + results = {} + attempts = CONNECT_ATTEMPTS + + for index, (name, protocol, extra_args) in enumerate(TESTS): + # Give the server a moment to reset between tests, the same way the + # bandwidth benchmark does: starting the next test the instant the + # previous one ends is what makes it fail on the control connection. + if index: + time.sleep(TEST_GAP) + + results.update(run_test(name, protocol, extra_args, attempts)) + attempts = 1 + + return results + + +def main(): + """Push a start event, run the benchmark, push its results, then push a stop event.""" + args = parse_args() + source = os.environ["AOS_INSTANCE_ID"] + + if not TARGET: + print("TARGET environment variable is required", file=sys.stderr) + return 1 + + print( + f"Latency benchmark: target={TARGET} port={PORT} " + f"duration={DURATION}s msg_size={MSG_SIZE}" + ) + + push_event(args.victoria_url, NODE, source, "Start") + + try: + results = run_benchmark() + + for name, value in results.items(): + push_result(args.victoria_url, NODE, source, name, value) + finally: + push_event(args.victoria_url, NODE, source, "Stop") + + print("All tests finished") + + # The benchmark is a one shot run, but the instance keeps running so its + # logs stay available and the unit does not restart it in a loop. + while True: + time.sleep(IDLE_DELAY) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/benchmark/services/network/latency/src/server/latency_server.py b/benchmark/services/network/latency/src/server/latency_server.py new file mode 100644 index 0000000..f985444 --- /dev/null +++ b/benchmark/services/network/latency/src/server/latency_server.py @@ -0,0 +1,40 @@ +import os +import subprocess +import time + +PORT = os.environ.get("PORT", "11111") + +RESTART_DELAY = 5 + +# One server per protocol: sockperf listens either on UDP or, with --tcp, on +# TCP, never on both. TCP and UDP port numbers are independent, so the two can +# share PORT. +SERVERS = [ + ("udp", ["sockperf", "server", "-i", "0.0.0.0", "-p", PORT]), + ("tcp", ["sockperf", "server", "-i", "0.0.0.0", "-p", PORT, "--tcp"]), +] + + +def main(): + processes = {} + + # sockperf servers never exit on their own, so this loop only matters if + # one of them dies: the service instance stays alive and it comes back. + while True: + for name, cmd in SERVERS: + process = processes.get(name) + + if process is not None and process.poll() is None: + continue + + if process is not None: + print(f"sockperf {name} server exited with code {process.returncode}, restarting") + + print(f"Starting sockperf {name} server: {' '.join(cmd)}") + processes[name] = subprocess.Popen(cmd) + + time.sleep(RESTART_DELAY) + + +if __name__ == "__main__": + main() diff --git a/benchmark/services/network/readme.txt b/benchmark/services/network/readme.txt deleted file mode 100644 index c7bbe5c..0000000 --- a/benchmark/services/network/readme.txt +++ /dev/null @@ -1 +0,0 @@ -TODO: put network services here