Flight Tracker: analyze RS3 data for other tracks

[robbiebyrd]

Goal

Evaluate and (if feasible) extract flight tracks for other aircraft — beyond the 4+1 curated in #262 — from the raw .RS3 sensor recordings in the 84 RADES 9/11 FOIA release, to enrich the Flight Tracker with real radar-derived traffic.

What the raw data is

• Data/Raw/USAF/*.RS3 — binary radar message streams from NEADS/SEADS sensors covering the northeast US, roughly 09:30–16:30 UTC on 2001-09-11 (filenames encode Julian day 254 + UTC half-hour blocks). Contains every aircraft those radars saw, not just the incident flights.
• Format findings so far: "RADES Sensor Data" header; 99 99 FF FF message separators; little-endian Unix-epoch timestamps confirmed (e.g. 0x3B9DF5B8 = 2001-09-11 11:30:00 UTC); per-message range/azimuth/Mode 3/Mode C field encodings not yet decoded; no public parser exists (searched).

Path A — RS3 app export (unblocks analysis without reverse-engineering)

The release bundles the RS3 viewer (Windows). RS3 can export per-return query CSVs with computed lat/lon (Exp_Query.csv in the Pentagon products proves the format: Id,Trk,Time,MsgType,Range,AzDegs,Height,MC,M3,M2,Lat,Lon,...). Export instructions for a Windows XP session are being prepared; the exported CSVs feed the same analysis pipeline as #262.

Path B — binary parser (fallback/scale)

Finish reverse-engineering the message layout (validate against known returns from the #262 Excel data and Exp_Query.csv).

The hard problem — track→flight identity

Radar returns carry Mode 3 squawk codes, not flight numbers; beacon-code assignments for 2001-09-11 are not public. Correlating anonymous radar tracks to BTS flights will need heuristics (departure airport + wheels-off time from BTS vs track origin, squawk continuity). Expect a correlated subset, not full coverage.

Tier 1 (the 4 hijacked flights + GOFER06 from RADES's already-attributed Excel products) is #262.

[robbiebyrd]

Status: export instructions ready — analysis blocked on the Windows XP export step

Binary reverse-engineering findings (from probing the raw files against known returns):

• Framing decoded: "RADES Sensor Data" header, 99 99 FF FF message separators, little-endian Unix-epoch timestamps confirmed (e.g. 0x3B9DF5B8 = 2001-09-11 11:30:00 UTC, matching the 12541130.RS3 filename convention: 1+Julian day 254+UTC half-hour).
• Per-message field encodings (range/azimuth/Mode 3/Mode C) did not yield to known-plaintext probing against the RADES analysts' own Excel returns and Exp_Query.csv, and no public parser exists.
• Therefore Path A (RS3 app export) first: the release's own training doc (Introduction to RS3.doc, Exercises Dev #18/Dev #19) documents a Message Viewer process with continuous CSV export including computed Latitude/Longitude fields — a whole-file batch export, better than the per-area Query flow that produced Exp_Query.csv.

Step-by-step instructions for the Windows XP + RS3 session are committed at packages/tools/flight-recon/analysis/RADES_RS3_EXPORT.md (181fd9a): one-time setup, the per-file Message Viewer export procedure (fields to add, Export On settings), and a deduplicated priority list of ~26 raw files (~330 MB binary → a few GB CSV, zips ~10×).

Once the CSVs come back, next steps here:

1. Parse + segment per-squawk/track radar tracks (same filtering pipeline as Flight Tracker: Update 4 + 1 Active Track flights (AA11, AA77, UA175, UA93, GOFER6) #262's extractor).
2. Heuristic track→BTS-flight correlation (departure airport + wheels-off time vs track origin, squawk continuity) — expect a correlated subset, not full coverage.
3. Decide how correlated real tracks merge with the BTS great-circle reconstructions in flight_positions/flight_tracks.

🤖 Generated with Claude Code

[robbiebyrd]

Exporting the raw RADES .RS3 files to CSV (Windows XP + RS3 app)

Goal: turn each raw binary sensor recording in the 84 RADES FOIA release into a
per-message CSV with computed latitude/longitude, so the flight-recon
pipeline can analyze all aircraft tracks (issue #263) — the binary field
encodings are undecoded, but the bundled RS3 app computes lat/lon itself and
has a per-message CSV export (the same path that produced Exp_Query.csv in
the Pentagon products).

Everything below comes from the release's own docs: Practice Files/ Introduction to RS3.doc (Message Viewer §, Exercises #18/#19) and
Products/software/RS3 Quick Instructions.doc.

One-time setup

1. Copy the RADES folders you'll work from onto the hard drive (the app writes
   next to its data; CD paths and read-only flags break it).
2. Right-click each copied file → Properties → uncheck Read-only → OK.
3. Install RS3: run Rs3Setup.EXE (in rades.pentagon…/Products/software/ —
   the UAL93 set has an identical copy). Accept defaults; ~12 MB. Create the
   desktop shortcut when asked. Ignore any "RDI Boards not installed" warning —
   that's for live radar capture hardware, not playback.
4. Make an output folder, e.g. C:\exports\.

Per-file export procedure (Message Viewer)

Repeat for each raw .RS3 file (list below). One source file → one CSV, named
after the source file.

1. Launch RS3 → File → New Project → Blank Project → Load.
2. Project → Add Task → in Select Data Source choose File → pick the
   raw file (e.g. C:\rades\raw\12541130.RS3) → check Open as Read Only →
   Open.
3. Project → Add Process → select the Message Viewer icon → OK.
4. Right-click Message Viewer in the Project Manager tree → Properties:
   • Msg Fields tab — the defaults are Site ID, Track No, Date, Time, Message Type, Range, Azimuth Degs, Mode 3, Mode 2, Mode C, Height.
     Add from the left frame: Latitude, Longitude, and any
     validity-bit fields offered (e.g. Mode 3 Valid, Mode C Valid — the
     84 RADES analysts' own spreadsheets include them). Field order doesn't
     matter to the pipeline; headers are exported.
   • Options tab —
     • check Export On
     • check Export Column Headings
     • Export Filename: C:\exports\<sourcefilename>.csv
       (e.g. C:\exports\12541130.csv)
     • Export Format: CSV
     • uncheck View Messages While Processing (major speed-up; the grid
       doesn't need to paint for the export to happen)
   • OK.
5. Data → Start. Raise the run speed to maximum (Sensor Display's run-speed
   control — Exercise Bump urllib3 from 1.26.2 to 1.26.3 in /backend #9 in the intro doc — or just let it run). Wait until the
   file plays out (the End indicator, upper-left).
6. Data → Stop, then File → Exit (no need to save the project).
7. Sanity-check the CSV opens and has a Latitude column with real values.
   Expect tens of MB per half-hour file.

If the Message Viewer misbehaves on a big file, the fallback is the Sensor
Display Query Area flow (Exercise #10: Process → Query Area → trace a
boundary → right-click → Query Results → export) — that's what produced
Exp_Query.csv — but it's per-area rather than whole-file, so prefer the
Message Viewer.

Which files to export (deduplicated across the four release sets)

The three event sets share identical 12540930.RS3 … 12541430.RS3 copies —
export each name once. Priority order:

1. NEADS core sweep (any set's Data/Raw/USAF/, ~7–21 MB each):
   12540930, 12541000, 12541030, 12541100, 12541130, 12541200, 12541230, 12541300, 12541330, 12541400, 12541430 (11 files — 09:30–15:00 UTC)
2. NEADS afternoon (Pentagon set): NEADS 12541430 … NEADS 12541630
   (4 more files — through 17:00 UTC; the NEADS 12541430 duplicates Bump django from 3.0.3 to 3.0.7 #1's
   12541430, skip it)
3. SEADS radars (Pentagon set): SEADS_12541100 … SEADS_12541600
   (11 files — southern coverage of AA77/UA93 area)
4. WTC evening file (WTC set): 12542330.rs3 (23:30 UTC)
5. Optional, likely redundant with 1–3: Projects/Cnv_Multifile Sensor Data.rs3 (160 MB combined file), Projects/Pittsburgh.rs3 (90 MB),
   Data/Filtered/USAF/Pentagon Combined Radar File.rs3 (145 MB — already
   merged + filtered; useful as a cross-check against Exp_Query.csv)

That's ~26 files ≈ 330 MB of binary → expect a few GB of CSV. Zip the CSVs
(they compress ~10×) and drop them next to the RADES folder; the analysis
continues from there (parse → per-squawk track segmentation → BTS correlation).

[robbiebyrd]

First analysis of the RS3 CSV exports — segmentation works, correlation quantified

Exports received (MV_Exp_12541130/1200/1230/1300.csv + MV_Exp_Pentagon Combined Radar File.csv, ~555 MB, now under Radar_Evaluation_Squadron_(RADES)/exports/, gitignored). Pipeline committed as analysis/segment_rades_exports.py (d29c863); all numbers below reproduce from it.

Inventory

• 6.2M radar messages → 2,463,899 valid-Mode-3 beacon returns, continuous 09:30–13:30Z, from 15 radar sites (GIB PLA RIV DAN DTW REM EMP OCA NSH QFF NOR BUC CAR BAR WSD) — twice the sites the curated notable work used.
• Ground truth validates: AA77's squawk-3743 returns end 12:50:38.660Z @ 38.8308,-81.9414 — identical to the curated track's last pre-gap waypoint.

Segmentation

• Squawk codes are reused across the airspace (code 1443 — AA11's — flies near Dayton OH before AA11 leaves the Boston gate), so segmentation is squawk + spatial-temporal continuity (240 s staleness, 650 kt speed gate, 3 nm cross-site allowance).
• 12,322 raw segments → 5,654 aircraft-grade tracks (≥25 returns, ≥8 min, ≥10 nm; median 35 min).

BTS correlation (departure-signature matching)

2,587 tracks begin below 6,000 ft within 8 nm of an airport. Against 1,715 BTS 9/11 flights wheels-off 09:30–13:30Z from in-coverage airports:

tolerance	0 cand.	1 (unambiguous)	2	3	4+	BTS flights named 1:1
±2 min	1,338	496	380	248	125	339
±5 min	1,213	273	262	260	579	165

Reading: ~20% of in-coverage BTS departures are nameable with the naive matcher; the 1,338 no-candidate tracks are mostly real non-BTS traffic (GA, cargo, commuter, military — BTS records only major-carrier scheduled domestic). Headroom: mutual-best assignment, destination-bearing consistency, arrival-side matching, and per-airport wheels-off bias calibration should raise the 1:1 count substantially — and the unmatched tracks could ship as an anonymous real-traffic layer regardless.

Gaps

The exports end 13:30Z; 12541330/12541400/12541430.RS3 (13:30–15:00Z, incl. GOFER06's later flight and UA93's final minutes from more sites) and the SEADS/NEADS files are not yet exported.

🤖 Generated with Claude Code

[robbiebyrd]

Matcher v2 — 870 BTS flights named (was 339)

acc90f5d upgrades the correlation with the four planned improvements:

1. Global mutual-best assignment — candidate (track, flight) pairs scored and assigned greedily best-first, each track and flight used once. Hub congestion (multiple departures inside one time window) no longer forces a discard.
2. Per-airport wheels-off bias calibration — a tight time-only first pass yields per-airport median offsets between first-radar-return and BTS wheels-off (real systematic biases of −24…−44 s at 23 airports), applied before final scoring.
3. Destination-bearing consistency — a departure's outbound course (sampled 5 min in) must agree with the great-circle bearing to its BTS destination (35° free cone, 100° hard gate, 3 s/degree penalty beyond).
4. Arrival-side matching — tracks ending below 6,000 ft near an airport matched against BTS WheelsOn with the same machinery.

Result: 768 named via departures + 238 via arrivals → union 870 flights (~51% of the 1,715 BTS departures in the covered window), 25 matched independently at both ends by the same track. Negative control holds: none of the hijacked/observer squawk tracks (1443, 3020, 3321, 3743, 1527, 2427) get named.

Next candidates for the pipeline: precision estimation (both-end agreement rate as ground truth proxy), the remaining 13:30–15:00Z exports, and the load-to-Directus design (named tracks replacing great-circle reconstructions + optionally the ~1,300 anonymous real-traffic tracks).

🤖 Generated with Claude Code