diff --git a/docs/README.md b/docs/README.md index a6b63ef..1fd712e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -59,6 +59,8 @@ The captured netlink `.pcap` fixture corpus spanning many kernel versions, the r See **[CONTRIBUTING.md](../CONTRIBUTING.md)** for the development environment, the full set of Nix build/test targets, the automated test suite (unit, race, per-flavor, and microVM integration tests), linting tiers, and protobuf regeneration. Related references: - [Protobuf formats](protobuf-formats.md) — the config/data/ClickHouse schemas, generated code, and how to regenerate. +- [Parquet format](parquet-format.md) — the S3/Parquet layout and schema, for data teams consuming the export. +- [Socket analysis](socket-analysis.md) — finding RTT bands and other socket groupings by clustering (data-team methodology). - [Build flavors](build-flavors.md) — the build-variant × destination-flavor matrix. - [Integration testing](integration-testing.md) — the QEMU microVM test harness. - [Quality report](quality-report.md) — auto-generated linter/coverage status. diff --git a/docs/output-and-destinations.md b/docs/output-and-destinations.md index 1292111..4ddaf79 100644 --- a/docs/output-and-destinations.md +++ b/docs/output-and-destinations.md @@ -79,6 +79,8 @@ Notes on the stream sinks: `pkg/xtcp/destinations_s3parquet.go` (build tag `dest_s3parquet`) writes Hive-partitioned Parquet files to an S3-compatible store (e.g. MinIO) instead of streaming to a broker. The record-to-Parquet schema mapping is in `destinations_s3parquet_schema.go`. Files are partitioned `host=…/date=…/hour=…/.parquet` and finalized/uploaded when the in-memory builder crosses `-s3ParquetFlushBytes` (default 63 MiB uncompressed). Credentials and endpoint come from `-s3*` flags or `S3_*` environment variables; the bucket must already exist. +For a consumer-facing reference of the Parquet layout, schema, and the key TCP columns — written for a **data team** ingesting this into an enterprise platform — see [parquet-format.md](parquet-format.md). + ## The record schema The per-socket record and its batch wrapper are defined in `proto/xtcp_flat_record/v1/xtcp_flat_record.proto`: diff --git a/docs/parquet-format.md b/docs/parquet-format.md new file mode 100644 index 0000000..40ccb1a --- /dev/null +++ b/docs/parquet-format.md @@ -0,0 +1,204 @@ +# Parquet format (for data teams) + +This document is written for a **data / analytics team** that needs to consume xtcp2's TCP telemetry into an enterprise data platform (lakehouse, warehouse, query engine). It assumes you're fluent in Parquet, object storage, and SQL, but only have a *basic* understanding of TCP — so it explains the columns that matter most and where to focus your first implementation. + +The short version: when xtcp2 runs with the S3/Parquet destination it writes **Hive-style partitioned, column-compressed Apache Parquet files** to an S3-compatible bucket. One Parquet **row = one socket observed at one poll**. The schema is flat (no nested or repeated fields), one column per field, so it loads cleanly into any Parquet reader. + +## Table of contents + +- [Where the files land](#where-the-files-land) +- [File size, cadence, and compression](#file-size-cadence-and-compression) +- [Reading the data](#reading-the-data) +- [Loading into Snowflake](#loading-into-snowflake) +- [The grain: one row per socket per poll](#the-grain-one-row-per-socket-per-poll) +- [Start here: the columns that matter](#start-here-the-columns-that-matter) +- [Decoding cheat sheet](#decoding-cheat-sheet) +- [Full schema and column types](#full-schema-and-column-types) +- [Types, nulls, and gotchas](#types-nulls-and-gotchas) +- [Where the schema is defined](#where-the-schema-is-defined) +- [See also](#see-also) + +## Where the files land + +Object keys are **Hive-partitioned** by host, date, and hour (all UTC): + +``` +/host=/date=/hour=/_.parquet +``` + +Example: + +``` +xtcp/host=web-01/date=2026-06-19/hour=14/1750345200_9f3a1c20.parquet +``` + +- `host=` — the emitting machine (`hostname`); sanitized for object-store safety, empty → `unknown`. +- `date=` / `hour=` — **UTC** wall clock at write time, ready for partition pruning (`WHERE date = '...' AND hour = '...'`). +- The file name is `_.parquet` — unique, append-only; xtcp2 never rewrites a file. + +These partition keys are part of the **path, not the file** (standard Hive convention). Most engines (Spark, Trino/Athena, DuckDB, BigQuery external tables) expose `host`, `date`, `hour` as virtual columns automatically when you point them at `/`. + +## File size, cadence, and compression + +- **Size:** xtcp2 finalizes and uploads a file when its in-memory builder reaches a soft cap of **~63 MiB uncompressed** (configurable via `-s3ParquetFlushBytes`). On the wire the `.parquet` is several times smaller after compression. A partial file is also flushed on shutdown, so the last file of a run may be small. +- **Cadence:** depends on traffic volume — a busy host fills 63 MiB quickly; a quiet host may take a while, so don't assume one file per hour. Use the `date`/`hour` partitions, not file counts. +- **Compression:** per-column. String and address columns use **ZSTD** (high ratio); numeric columns use **SNAPPY** (fast, widely supported). Every mainstream Parquet reader handles both — you don't need to configure anything. + +## Reading the data + +Point any Parquet engine at the prefix. A few starting points: + +```sql +-- DuckDB (great for exploration); hive_partitioning surfaces host/date/hour as columns +SELECT host, date, hour, count(*) AS rows +FROM read_parquet('s3://bucket/xtcp/**/*.parquet', hive_partitioning = true) +WHERE date = '2026-06-19' +GROUP BY 1,2,3 ORDER BY 1,2,3; +``` + +```python +# pandas / pyarrow +import pyarrow.dataset as ds +dataset = ds.dataset("s3://bucket/xtcp/", format="parquet", partitioning="hive") +df = dataset.to_table(columns=["timestamp_ns","hostname","tcp_info_rtt"]).to_pandas() +``` + +```sql +-- Trino / Athena: create an external table over the prefix with +-- partitions (host string, date string, hour string); project columns you need. +``` + +**Always select only the columns you need** — there are 122, and columnar pruning is where Parquet earns its keep. Likewise filter on `date`/`hour` for partition pruning. + +## Loading into Snowflake + +If your platform team already manages an S3 storage integration, ingesting is a few statements. The column names match the Parquet schema, so name-based matching does the mapping for you. + +```sql +-- One-time: a Parquet file format and an external stage over the prefix. +-- STORAGE_INTEGRATION is the bucket grant your platform team provisions. +CREATE OR REPLACE FILE FORMAT xtcp_parquet TYPE = PARQUET; + +CREATE OR REPLACE STAGE xtcp_stage + URL = 's3://bucket/xtcp/' + STORAGE_INTEGRATION = my_s3_int + FILE_FORMAT = xtcp_parquet; + +-- Auto-create a table whose columns mirror the Parquet schema. +CREATE OR REPLACE TABLE xtcp_flat_records + USING TEMPLATE ( + SELECT ARRAY_AGG(OBJECT_CONSTRUCT(*)) + FROM TABLE(INFER_SCHEMA(LOCATION => '@xtcp_stage', FILE_FORMAT => 'xtcp_parquet')) + ); + +-- Load. MATCH_BY_COLUMN_NAME maps Parquet columns → table columns by name. +COPY INTO xtcp_flat_records + FROM @xtcp_stage + FILE_FORMAT = (TYPE = PARQUET) + MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE; +``` + +For **continuous, query-in-place** ingestion instead of a one-shot load, define an external table with `AUTO_REFRESH = TRUE` (backed by Snowpipe + an S3 event notification) over the same stage. + +Two Snowflake-specific notes: + +- **Partitions live in the path, not the file.** Snowflake won't surface `host`/`date`/`hour` automatically the way Spark/Trino do — derive them from `metadata$filename` (e.g. `REGEXP_SUBSTR(metadata$filename, 'date=([^/]+)', 1, 1, 'e', 1)`) as virtual columns on an external table, or as extra columns during `COPY` via a transform. They make great clustering/pruning keys. +- **The two IP-address columns load as `BINARY`.** Decode them with `inet_diag_msg_family` as in the [cheat sheet](#decoding-cheat-sheet) — or, if you'd rather not decode in SQL, point the daemon at the humanized CSV/JSON output instead (see [output formats](output-and-destinations.md)). + +## The grain: one row per socket per poll + +xtcp2 polls every network namespace on a fixed interval (default 10s) and emits one row per TCP socket per poll. So: + +- A long-lived connection appears in **many** rows over time — one per poll while it exists. +- Most counters (bytes, segments, retransmits) are **cumulative over the socket's lifetime**, so you typically `MAX()` them per socket or take deltas between consecutive polls. +- Identify a single socket across polls with `inet_diag_msg_socket_cookie` (a stable kernel-assigned id) together with `hostname`/`netns`. + +## Start here: the columns that matter + +If you're scoping an initial implementation, these are the high-value columns. Everything else can come later. + +### Identity & time +| Column | Type | Meaning | +|---|---|---| +| `timestamp_ns` | double | When the sample was taken — **Unix epoch nanoseconds, UTC**. Divide by 1e9 for seconds. | +| `hostname` | string | Emitting machine (also the `host=` partition). | +| `netns` | string | Network namespace path — distinguishes host vs container/pod sockets. | +| `inet_diag_msg_socket_cookie` | uint64 | Stable per-socket id; use to track one connection across polls. | + +### The connection (4-tuple) +| Column | Type | Meaning | +|---|---|---| +| `inet_diag_msg_family` | uint32 | Address family: **2 = IPv4, 10 = IPv6**. Tells you how to read the address bytes. | +| `inet_diag_msg_socket_source` | bytes | Local IP, **raw bytes** (4 for v4, 16 for v6). See [decoding](#decoding-cheat-sheet). | +| `inet_diag_msg_socket_source_port` | uint32 | Local port (host byte order; use as-is). | +| `inet_diag_msg_socket_destination` | bytes | Remote IP, raw bytes. | +| `inet_diag_msg_socket_destination_port` | uint32 | Remote port. | +| `inet_diag_msg_state` | uint32 | TCP state (see the [state map](#decoding-cheat-sheet)); `1`=ESTABLISHED, `10`=LISTEN. | + +### Health & performance (the metrics most teams want) +| Column | Type | Unit / meaning | +|---|---|---| +| `tcp_info_rtt` | uint32 | Smoothed round-trip time, **microseconds**. The headline latency metric. | +| `tcp_info_min_rtt` | uint32 | Minimum RTT seen, microseconds — a cleaner latency baseline. | +| `tcp_info_rtt_var` | uint32 | RTT variance, microseconds (jitter). | +| `tcp_info_snd_cwnd` | uint32 | Congestion window, **in packets/segments** (not bytes). | +| `tcp_info_total_retrans` | uint32 | Cumulative retransmitted segments — the simplest "is this connection healthy?" signal. | +| `tcp_info_bytes_sent` / `tcp_info_bytes_acked` | uint64 | Cumulative bytes sent / acknowledged. | +| `tcp_info_bytes_received` | uint64 | Cumulative bytes received. | +| `tcp_info_delivery_rate` | uint64 | Recent delivery rate, **bytes/second** — effective throughput. | +| `tcp_info_pacing_rate` | uint64 | Sender pacing rate, bytes/second. | +| `congestion_algorithm_string` | string | Congestion-control algorithm name (e.g. `cubic`, `bbr`) — easiest to read. | + +A solid first dashboard: per host/destination, `MAX(tcp_info_rtt)` and `MAX(tcp_info_min_rtt)`, the delta of `tcp_info_total_retrans`, and throughput from `tcp_info_delivery_rate` — filtered to `inet_diag_msg_state = 1` (ESTABLISHED). + +## Decoding cheat sheet + +A few columns are stored as machine values for fidelity/size and need decoding for humans: + +- **IP addresses** (`inet_diag_msg_socket_source` / `_destination`) are raw bytes. Read them with `inet_diag_msg_family`: 4 bytes → dotted-quad IPv4, 16 bytes → IPv6. In DuckDB you can reconstruct IPv4 as `concat_ws('.', get_byte(col,0), get_byte(col,1), get_byte(col,2), get_byte(col,3))`. If you'd rather not decode in SQL at all, the daemon can emit **already humanized** CSV/JSON instead — see [output formats](output-and-destinations.md) — but the Parquet path keeps raw bytes so nothing is lost. +- **TCP state** (`inet_diag_msg_state`, and `tcp_info_state`) is a kernel integer. Map: + + | value | name | value | name | + |---|---|---|---| + | 1 | ESTABLISHED | 7 | CLOSE | + | 2 | SYN_SENT | 8 | CLOSE_WAIT | + | 3 | SYN_RECV | 9 | LAST_ACK | + | 4 | FIN_WAIT1 | 10 | LISTEN | + | 5 | FIN_WAIT2 | 11 | CLOSING | + | 6 | TIME_WAIT | 12 | NEW_SYN_RECV | + +- **Congestion algorithm**: prefer `congestion_algorithm_string` (the kernel name). The `congestion_algorithm_enum` integer is `0`=UNSPECIFIED, `1`=CUBIC, `2`=DCTCP, `3`=VEGAS, `4`=PRAGUE, `5`=BBR1, `6`=BBR2, `7`=BBR3. +- **timestamp_ns** is a double; `to_timestamp(timestamp_ns / 1e9)` (or your engine's equivalent) gives a UTC timestamp. + +## Full schema and column types + +The complete column list (122 columns) groups as follows; column names are the proto's snake_case names, identical to the ClickHouse table columns: + +- **Metadata** — `timestamp_ns` (double), `hostname`, `netns`, `nsid`, `label`, `tag`, `record_counter`, `socket_fd`, `netlinker_id`. +- **`inet_diag_msg_*`** — the socket id/4-tuple, state, queues, uid/inode, ASN annotations. +- **`mem_info_*` / `sk_mem_info_*`** — socket memory accounting. +- **`tcp_info_*`** — the bulk of the data: RTT, cwnd, ssthresh, MSS, windows, segment and byte counters, pacing/delivery rates, RTO stats, busy/limited times. +- **`congestion_algorithm_*`** — enum (`int32`) + string name. +- **Per-algorithm blocks** — `vegas_info_*`, `dctcp_info_*`, `bbr_info_*` (only meaningful when that algorithm is in use). +- **QoS / misc** — `type_of_service`, `traffic_class`, `shutdown_state`, `class_id`, `sock_opt`, `c_group`. + +Column types are: `double` (timestamp only), `string` (hostname/netns/label/tag/congestion string), `bytes` (the two IP-address columns), `int32` (congestion enum), and `uint32`/`uint64` for everything else. The authoritative, field-by-field list with types and compression is the [`ParquetRow` struct](../pkg/xtcp/destinations_s3parquet_schema.go); field meanings are in the [protobuf schema](../proto/xtcp_flat_record/v1/xtcp_flat_record.proto) and [protobuf-formats.md](protobuf-formats.md). + +## Types, nulls, and gotchas + +- **No NULLs.** The records come from proto3, which has no null — an absent/zero value is the numeric `0` (or empty string/bytes). Treat `0` as "unset or genuinely zero"; don't expect SQL `NULL`. +- **Counters are cumulative**, per socket lifetime — delta between consecutive polls (matched by `inet_diag_msg_socket_cookie`) for per-interval rates, or `MAX()` for totals. +- **Units differ**: RTTs are microseconds; rates are bytes/second; `snd_cwnd` is packets; byte counters are bytes. The per-column units are in the tables above. +- **Per-algorithm columns are sparse-in-meaning**: `bbr_info_*` is only populated when the socket uses BBR, etc. Filter on `congestion_algorithm_string` before trusting them. +- **Schema evolution**: new fields are *added* (never renamed/reordered in place), so plan for forward-compatible reads (select by name, tolerate new columns). + +## Where the schema is defined + +The Parquet columns are generated from the [`ParquetRow`](../pkg/xtcp/destinations_s3parquet_schema.go) struct, whose `parquet:` tags set the column names and per-column compression. A drift test (`TestS3ParquetSchema_matchesProto`) asserts that set matches the [`XtcpFlatRecord` proto](../proto/xtcp_flat_record/v1/xtcp_flat_record.proto) field-for-field, so the Parquet schema, the protobuf, and the ClickHouse table never diverge. To change the schema you edit the proto, regenerate (`nix run .#regen-protos`), and mirror the field in `ParquetRow`. The S3/Parquet destination itself is documented in [output formats & destinations](output-and-destinations.md#s3-and-parquet); it ships only in builds that include the `dest_s3parquet` tag (see [build flavors](build-flavors.md)). + +## See also + +- [Socket analysis](socket-analysis.md) — finding RTT bands and other socket groupings by clustering, once the data is loaded. +- [Protobuf formats](protobuf-formats.md) — the canonical schema and field semantics. +- [Output formats & destinations](output-and-destinations.md) — the S3/Parquet destination and the alternative humanized CSV/JSON formats. +- [Build flavors](build-flavors.md) — enabling the `s3parquet` destination. diff --git a/docs/socket-analysis.md b/docs/socket-analysis.md new file mode 100644 index 0000000..03a3e6e --- /dev/null +++ b/docs/socket-analysis.md @@ -0,0 +1,209 @@ +# Analyzing socket data: RTT bands and clustering + +This guide is for a **data / analytics team** turning xtcp2's per-socket TCP telemetry into insight. It assumes you're comfortable with SQL and basic statistics / clustering. The headline use case is discovering the natural **RTT bands** in your fleet — and doing it *statistically*, because the bands differ by data center and **drift over time**, so hardcoded thresholds go stale. It also sketches other groupings (throughput, retransmission/loss, congestion algorithm, per-ASN, time-of-day). + +Read [parquet-format.md](parquet-format.md) first for how to load the data and what the columns mean; this doc is the analysis companion. + +## Table of contents + +- [The RTT-band mental model](#the-rtt-band-mental-model) +- [Pick the right signal](#pick-the-right-signal) +- [Data preparation](#data-preparation) +- [Finding RTT bands](#finding-rtt-bands) +- [Multi-feature clustering](#multi-feature-clustering) +- [Other useful analyses](#other-useful-analyses) +- [Worked example](#worked-example) +- [Pitfalls](#pitfalls) +- [See also](#see-also) + +## The RTT-band mental model + +Round-trip time is the strongest single signal for *where* a socket's peer is and *what kind* of path it's on. In a typical fleet you'd expect RTT to fall into a handful of bands, roughly: + +1. **Intra-data-center** — peers in the same DC; sub-millisecond to a few ms, usually high throughput. +2. **Metro / same-region CDN edge** — Cloudflare, Fastly, Google, Akamai POPs in the same metro; ~10–30 ms. +3. **Regional services** — another region or a more distant POP; ~60–120 ms. +4. **Outliers** — cross-continent, satellite, or pathological paths; much higher. +5. **Mobile / last-mile** — end users on 4G/LTE/Wi-Fi reaching your hosts; often >150 ms with high variance. + +Treat these as **hypotheses, not constants.** The actual band count, centroids, and boundaries vary by DC and move over time (new POPs, peering/routing changes, congestion). The goal is to let the data define the bands and to re-derive them on a schedule, comparing across DCs. + +## Pick the right signal + +Use **`tcp_info_min_rtt`** as the primary banding feature, not the smoothed `tcp_info_rtt` (srtt): + +- `min_rtt` is the *minimum* RTT the kernel has seen on the socket — it approximates the propagation/path floor and is largely free of transient queueing and load. That makes it a clean proxy for distance/path, which is exactly what bands are about. +- `tcp_info_rtt` (srtt) is useful as a *current latency* feature and, together with `tcp_info_rtt_var`, as a **jitter** signal — but it inflates under load, so it's noisier for geography. + +**All RTT fields are microseconds** — divide by 1000 for milliseconds. RTT spans several orders of magnitude (0.1 ms intra-DC to 300 ms mobile), so **analyze it on a log scale**; the modes that correspond to bands are far clearer in `log10(min_rtt_ms)` than in linear space. + +## Data preparation + +This is the make-or-break step. Two issues dominate. + +**1. Fix the grain.** The export is one row per socket **per poll** (default every 10s), so a long-lived connection contributes many rows. If you cluster raw rows, long flows dominate and you're really clustering "poll-seconds," not sockets. **Aggregate to one feature vector per socket**, keyed by `inet_diag_msg_socket_cookie` + `hostname` + `netns` (the cookie is unique only within a host/namespace). Sensible aggregations: `MIN(tcp_info_min_rtt)`, median or last srtt, `MAX(tcp_info_delivery_rate)`, and the **last** value of cumulative counters. + +**2. Filter and normalize.** + +- Keep `inet_diag_msg_state = 1` (ESTABLISHED). Drop listeners (`10`), `TIME_WAIT`, etc. +- Drop loopback / intra-host noise (source == destination, `127.0.0.0/8`, `::1`). +- Require a minimum lifetime — e.g. ≥ N samples or ≥ some bytes — so ephemeral sockets with one noisy RTT sample don't pollute the bands (survivorship filtering). +- Convert units: µs → ms for RTT; bytes/s → Mbit/s for throughput. +- Counters (`tcp_info_bytes_*`, `tcp_info_total_retrans`, `tcp_info_segs_*`) are **cumulative over the socket lifetime** — use the last value per socket, or deltas between consecutive polls for rates. +- Derive the **data-center** dimension. xtcp2 doesn't emit "DC" directly — derive it from your `hostname` convention, or set it explicitly with the daemon's `-label`/`-tag` (carried in the `label`/`tag` columns). + +The per-socket feature table (one row per socket) is the input to everything below. See the [worked example](#worked-example) for the SQL. + +## Finding RTT bands + +This is a one-dimensional clustering problem on `log(min_rtt)`. + +**Look before you cluster.** Plot a histogram and a kernel-density estimate (KDE) of `log10(min_rtt_ms)`. You'll usually *see* the modes (one bump per band) and the valleys between them. That sanity-checks everything that follows. + +**Method A — Gaussian Mixture Model + BIC (recommended, adaptive).** Fit a 1-D GMM to `log(min_rtt)` and let **BIC** choose the number of components. Each component is a band; the boundaries are where adjacent components cross over. This auto-discovers how many bands exist *today* — re-fit per time window (e.g. daily, per DC) and the band count/locations track drift automatically. This is the method to build the production pipeline on. + +**Method B — natural breaks (simple, explainable).** Jenks natural-breaks optimization (or finding the valleys/minima of the KDE) on `log(min_rtt)` gives defensible cut points that are easy to explain to stakeholders ("we split where the data has gaps"). Good for a first pass or when a GMM is overkill. + +**Method C — quantile bands in the warehouse (quick win).** `NTILE(n)` or `APPROX_PERCENTILE` in Snowflake gives instant coarse bands with zero data movement. Caveat: quantiles cut the data into *equal-sized* groups, which is **not** the same as finding natural modes — a quantile boundary can land in the middle of a real band. Use it for a fast look, not as the definition of a band. + +**Label and validate.** Name each band by its centroid RTT, then **confirm the physical story** with independent columns: `inet_diag_msg_socket_dest_asn`, an IP-geolocation join on the decoded `inet_diag_msg_socket_destination`, or `inet_diag_msg_socket_destination_port`. The lowest band should be mostly intra-DC peers, the next mostly CDN ASNs in your metro, and so on. If the physical story doesn't match the statistical band, investigate before trusting it. Keep the labels **derived from the centroids**, not hardcoded. + +**Track over time and per DC.** Persist each run's band centroids and boundaries keyed by (data center, day) as a time series. Now you can compare EU vs US vs AU, watch a band's RTT creep, and alert when a boundary jumps (a routing change, a new POP, or an outage). + +## Multi-feature clustering + +RTT is the headline, but sockets cluster on more than latency. Build a standardized feature vector per socket and cluster in several dimensions: + +- `log(min_rtt_ms)` — path/distance +- `log(throughput_mbps)` — capacity / flow size +- `retrans_rate` — loss (`bytes_retrans / bytes_sent`) +- `rtt_var / min_rtt` — *relative* jitter (dimensionless) +- `log(snd_cwnd)` — window the path sustains + +Standardize (z-score) after log-transforming the heavy-tailed features. Algorithm trade-offs: + +| Algorithm | Pros | Cons | +|---|---|---| +| **K-means** | Fast, simple baseline | Must pick K; assumes round, equal-size clusters | +| **GMM** | Soft assignments; BIC picks K; elliptical clusters | Assumes Gaussian components | +| **HDBSCAN** | No K; arbitrary shapes; **labels outliers as noise** | Sensitive to `min_cluster_size`; needs scaled features | + +**HDBSCAN is the recommended default** here — it doesn't need a predetermined cluster count and its built-in noise label naturally captures the "outliers" band (item 4 above) instead of forcing every socket into a group. Use PCA or UMAP to project to 2-D for a scatter plot colored by cluster. Validate with silhouette score (or BIC for GMM), **stability across time windows** (do the same clusters reappear tomorrow?), and **external agreement** — clusters should line up with `dest_asn`, `congestion_algorithm_string`, or DC. + +## Other useful analyses + +- **Throughput bands.** `log(tcp_info_delivery_rate)` is heavy-tailed; cluster it to separate "elephant" flows from "mice." Exclude `tcp_info_delivery_rate_app_limited = 1` rows when you want *path* capacity (those flows were limited by the application, not the network). +- **Retransmission / loss bands.** `bytes_retrans / bytes_sent` (or `total_retrans / segs_out`) splits healthy (~0) from lossy paths. Cross-tab with the RTT band — high-RTT mobile paths often also show elevated loss. +- **Congestion-algorithm comparison.** Group by `congestion_algorithm_string` (e.g. BBR vs CUBIC) and compare RTT/throughput/loss distributions for the same destination band. +- **Per-ASN / per-CDN performance.** Aggregate by `inet_diag_msg_socket_dest_asn` to rank CDN edges or transit providers by latency and loss from each DC. +- **Diurnal patterns.** Bucket by hour-of-day (`timestamp_ns`); mobile/last-mile RTT typically rises in the evening peak. Useful for capacity planning. +- **Anomaly / drift detection.** Monitor band centroids over time; a sudden shift is a strong signal of a routing change or incident. + +## Worked example + +The snippets below are **illustrative — adapt names and thresholds to your environment.** + +**Step 1 — per-socket feature table (DuckDB; near-identical in Snowflake).** Aggregate the poll-grain rows to one row per socket. + +```sql +WITH socket AS ( + SELECT + hostname, + netns, + inet_diag_msg_socket_cookie AS cookie, + -- DC from a hostname convention like "iad1-web-07"; adapt the regex, + -- or use the daemon's -label which lands in the `label` column. + regexp_extract(hostname, '^([a-z]+[0-9]+)', 1) AS dc, + inet_diag_msg_socket_dest_asn AS dest_asn, + MIN(tcp_info_min_rtt) / 1000.0 AS min_rtt_ms, + MEDIAN(tcp_info_rtt) / 1000.0 AS srtt_ms, + MEDIAN(tcp_info_rtt_var) / 1000.0 AS rtt_var_ms, + MAX(tcp_info_delivery_rate) * 8.0 / 1e6 AS mbps, + MAX(tcp_info_snd_cwnd) AS cwnd, + -- cumulative counters: last value ≈ MAX over the socket's life + MAX(tcp_info_bytes_sent) AS bytes_sent, + MAX(tcp_info_bytes_retrans) AS bytes_retrans, + ANY_VALUE(congestion_algorithm_string) AS congestion, + COUNT(*) AS samples + FROM read_parquet('s3://bucket/xtcp/**/*.parquet', hive_partitioning => true) + WHERE inet_diag_msg_state = 1 -- ESTABLISHED only + GROUP BY 1,2,3,4,5 +) +SELECT + *, + CASE WHEN bytes_sent > 0 THEN bytes_retrans::DOUBLE / bytes_sent ELSE 0 END AS retrans_rate +FROM socket +WHERE samples >= 3 -- survivorship: drop ephemeral sockets + AND min_rtt_ms > 0; +``` + +**Step 2 — RTT bands in Python (GMM + BIC).** + +```python +import numpy as np, pandas as pd +from sklearn.mixture import GaussianMixture + +feat = pd.read_parquet("socket_features.parquet") # the table from step 1 +x = np.log10(feat["min_rtt_ms"].to_numpy()).reshape(-1, 1) + +# Let BIC choose the number of bands (1..8). +models = {k: GaussianMixture(k, covariance_type="full", random_state=0).fit(x) + for k in range(1, 9)} +k = min(models, key=lambda k: models[k].bic(x)) +gmm = models[k] +feat["band"] = gmm.predict(x) + +# Order bands by RTT and summarize. +centroids = (10 ** gmm.means_.ravel()) +order = np.argsort(centroids) +print(f"{k} bands; centroids (ms):", np.round(centroids[order], 2)) +print(feat.groupby("band")["min_rtt_ms"].describe()[["count", "mean", "min", "max"]]) +# Plot log10(min_rtt_ms) histogram + the fitted components to eyeball the fit. +``` + +**Step 3 — multi-feature clusters (HDBSCAN).** + +```python +import hdbscan +from sklearn.preprocessing import StandardScaler + +cols = ["min_rtt_ms", "mbps", "rtt_var_ms", "cwnd"] +F = feat[cols].copy() +F[["min_rtt_ms", "mbps", "cwnd"]] = np.log10(F[["min_rtt_ms", "mbps", "cwnd"]].clip(lower=1e-6)) +F["rel_jitter"] = feat["rtt_var_ms"] / feat["min_rtt_ms"] +Z = StandardScaler().fit_transform(F) + +labels = hdbscan.HDBSCAN(min_cluster_size=50).fit_predict(Z) # -1 == outlier/noise +feat["cluster"] = labels +print(feat.groupby("cluster")[["min_rtt_ms", "mbps", "retrans_rate"]].median()) +``` + +**Quick warehouse bands (Snowflake).** When you just need coarse buckets fast: + +```sql +SELECT APPROX_PERCENTILE(tcp_info_min_rtt/1000.0, 0.5) AS p50_ms, + APPROX_PERCENTILE(tcp_info_min_rtt/1000.0, 0.9) AS p90_ms, + NTILE(5) OVER (ORDER BY tcp_info_min_rtt) AS rtt_quintile +FROM xtcp_flat_records +WHERE inet_diag_msg_state = 1; +``` + +Snowflake users who want clustering in-warehouse can also use Snowflake ML / Cortex's k-means on the per-socket feature table instead of exporting to Python. + +## Pitfalls + +- **RTT is microseconds** — divide by 1000 for ms. Easy to be off by 1000×. +- **`min_rtt` vs srtt** — band on `min_rtt`; srtt is inflated by load and queueing. +- **Cluster per-socket, not per-poll-row** — otherwise long-lived flows dominate the model. +- **Counters are cumulative** — use the last value per socket, or deltas for rates; don't sum across polls. +- **Survivorship** — filter out very short / low-byte sockets; their RTT is a single noisy sample. +- **App-limited throughput** — `tcp_info_delivery_rate_app_limited = 1` means the app, not the network, capped the rate; exclude for path-capacity work. +- **Addresses are raw bytes** — decode with `inet_diag_msg_family` before any geo/ASN join (see the [parquet decoding cheat sheet](parquet-format.md#decoding-cheat-sheet)). +- **Bands drift** — re-fit on a schedule; a model trained last quarter will mislabel today. +- **Per-host clocks** — `timestamp_ns` is each host's wall clock; don't assume sub-second alignment across machines. + +## See also + +- [Parquet format](parquet-format.md) — how to load the data and what every column means. +- [Protobuf formats](protobuf-formats.md) — the authoritative schema and field semantics. +- [Output formats & destinations](output-and-destinations.md) — the export pipeline and the humanized formats.