feat(lm): perplexity + batchnorm, fix NNLM_X init, auto-close workflow#123
Merged
Conversation
…kflow models.lm: - add reusable `perplexity(logits, targets)` = exp(mean cross-entropy) and log val/test perplexity per-epoch in NNLM_X (closes #64) - add optional BatchNorm1d on the NNLM hidden layer via `batch_norm` flag on NNLM/NNLMConfig (default off = backward compatible); sample() now eval()s so BN uses running stats on single-token contexts (closes #16) - fix NNLM_X.__init__: it forwarded args to Classifier positionally in the wrong order (nnet got the int num_classes), so NNLM_X could not be instantiated at all and its notebook test cell was failing CI; now passes keyword args - surface batch_norm in config/text/model/nnlm.yaml CI: - add close-linked-issues workflow: on merged PRs, close issues referenced by closing keywords on any base branch (GitHub only does this for the default branch) (closes #42) Verified: nbdev_test models.lm.ipynb passes; perplexity(uniform, V)=V and ~1.0 for near-perfect; full Lightning train/val/test logs test/perplexity=exp(loss). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #64, #16, #42.
#64 — perplexity computation for LMs
perplexity(logits, targets, ignore_index=-100)=exp(mean cross-entropy).val/perplexityandtest/perplexityper-epoch inNNLM_X.NNLM_X.__init__forwarded args toClassifierpositionally in the wrong order (nnetreceived the intnum_classes), soNNLM_Xcould not be instantiated at all — and its notebook cell (not#| notest) was failing CI. Now passes keyword args.#16 — batchnorm (+ optims) for the MLP LM
batch_norm: bool = FalsetoNNLM/NNLMConfig→ insertsnn.BatchNorm1dafter the hidden linear (nn.Identitywhen off → fully backward compatible). Surfaced inconfig/text/model/nnlm.yaml.sample()now callsself.eval()so BatchNorm uses running stats on single-token contexts.NNLM_X(now actually usable thanks to the init fix).#42 — auto-close linked issues on PR merge
.github/workflows/close-linked-issues.yaml: on merged PRs, parse closing keywords (close/fix/resolve+#N) and close those issues with a comment. Extends GitHub's native behavior, which only fires when merging into the default branch, to any base branch.Verification
nbdev_test models.lm.ipynb→ Success (runs the previously-brokenNNLM_Xinstantiation cell + a new perplexity assertion).perplexity: uniform logits overV→ exactlyV; near-perfect →~1.0.test/perplexity ≈ exp(test/loss).nbdev_exportreproducesnimrod/models/lm.pyidentically (numeric cell markers preserved).🤖 Generated with Claude Code