Add pairwise angle ckpt comparison#841
Open
klei22 wants to merge 3 commits into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new analysis utility to compare intra-vocabulary geometry of lm_head.weight between two nanoGPT-style checkpoints, including a CLI report generator, a trend-over-iterations plotter, a small checkpoint-selection web UI, and a runnable Shakespeare demo.
Changes:
- Add
compare_lm_head_pairwise_angles.pyto compute pairwise-angle matrices, summary metrics, and emit CSV/HTML. - Add
plot_lm_head_pairwise_angle_trend.pyto compute/report how angle-difference metrics evolve across saved iterations. - Add a Flask
app.pyweb UI plus ademos/script and module README for running the workflow end-to-end.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| demos/shakespeare_lm_head_pairwise_angles_demo.sh | End-to-end demo: trains 2 runs and generates pairwise-angle reports/trends. |
| analysis/lm_head_pairwise_angles/README.md | Usage documentation for CLI, webapp, and trend plotting. |
| analysis/lm_head_pairwise_angles/plot_lm_head_pairwise_angle_trend.py | Iteration-sweep runner + Plotly trend report generation. |
| analysis/lm_head_pairwise_angles/compare_lm_head_pairwise_angles.py | Core pairwise-angle computation, metrics, CSV/HTML output. |
| analysis/lm_head_pairwise_angles/app.py | Flask UI to select two checkpoints and render the comparison. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+6
to
+9
| import argparse | ||
| import os | ||
| import sys | ||
| from pathlib import Path |
Comment on lines
+57
to
+61
| metrics = "".join(f"<tr><th>{k}</th><td>{v:.6g}</td></tr>" for k, v in result.metrics.items()) | ||
| result_html = f"<h2>Metrics</h2><table border='1' cellpadding='4'>{metrics}</table>" + plot_html(result) | ||
| except Exception as exc: # render user-facing analysis errors in the page | ||
| error = f"<pre style='color:#b00'>{type(exc).__name__}: {exc}</pre>" | ||
| options = "".join(f"<option value='{p}' {'selected' if p == ckpt_a else ''}>{p}</option>" for p in ckpts) |
Comment on lines
+49
to
+50
| min_angle = float(form.get("min_angle", 0.0) or 0.0) | ||
| max_angle = float(form.get("max_angle", 180.0) or 180.0) |
Comment on lines
+47
to
+55
| meta = form.get("meta", "") | ||
| device = form.get("device", "cpu") | ||
| min_angle = float(form.get("min_angle", 0.0) or 0.0) | ||
| max_angle = float(form.get("max_angle", 180.0) or 180.0) | ||
| if request.method == "POST": | ||
| try: | ||
| result = compare_lm_head_pairwise_angles( | ||
| ckpt_a, ckpt_b, meta=meta or None, device=device, | ||
| min_angle=min_angle, max_angle=max_angle, |
Comment on lines
+60
to
+66
| def write_trend_csv(rows: List[Dict[str, float]], csv_path: str) -> None: | ||
| os.makedirs(os.path.dirname(os.path.abspath(csv_path)) or ".", exist_ok=True) | ||
| fieldnames = ["iteration"] + [key for key in rows[0] if key != "iteration"] if rows else ["iteration"] | ||
| with open(csv_path, "w", newline="", encoding="utf-8") as f: | ||
| writer = csv.DictWriter(f, fieldnames=fieldnames) | ||
| writer.writeheader() | ||
| writer.writerows(rows) |
Comment on lines
+117
to
+121
| angles_a = pairwise_angles_deg(weight_a) | ||
| angles_b = pairwise_angles_deg(weight_b) | ||
| diff = angles_b - angles_a | ||
| vocab_size = weight_a.shape[0] | ||
| pair_indices = torch.triu_indices(vocab_size, vocab_size, offset=1, device=angles_a.device) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.