Skip to content

Add Fisher Information Matrix support and natural-gradient training#60

Open
ujohn33 wants to merge 3 commits into
StatMixedML:masterfrom
ujohn33:feat/fim-all-dists
Open

Add Fisher Information Matrix support and natural-gradient training#60
ujohn33 wants to merge 3 commits into
StatMixedML:masterfrom
ujohn33:feat/fim-all-dists

Conversation

@ujohn33

@ujohn33 ujohn33 commented Dec 22, 2025

Copy link
Copy Markdown

This PR adds natural gradient descent support to LightGBMLSS by introducing per-distribution Fisher Information Matrix (FIM) scaling of gradients.

What’s included

  1. Extend DistributionClass with a natural_gradient flag and a default compute_fisher_information_matrix() method (fallback returns ones).
  2. Apply natural-gradient updates in compute_gradients_and_hessians() when natural_gradient=True (gradient scaled by FIM diagonal).
  3. Implement closed-form / efficient FIM diagonal computations for several univariate distributions based on the source [Miyamoto et al.] https://arxiv.org/abs/2304.14885 (e.g., Gaussian, Cauchy, Laplace, LogNormal, Logistic, Poisson, StudentT, Weibull, Gumbel), with an optimized branch for response_fn="exp" and a generic autograd-based derivative fallback for other response functions.
  4. Add unit tests aligned with the existing test structure to validate:
    • FIM output shapes and positivity/finite values across distributions
    • presence/behavior of the natural_gradient parameter
    • basic end-to-end training sanity for natural_gradient=True

Notes
Natural gradient is not enabled by default; existing behavior is unchanged unless natural_gradient=True.
CRPS currently falls back to standard gradients (a warning is emitted), as natural gradient scaling is implemented for NLL.

@ujohn33

ujohn33 commented Jan 27, 2026

Copy link
Copy Markdown
Author

@StatMixedML @nucflash please have a look once you find time

@StatMixedML

Copy link
Copy Markdown
Owner

@claude please review this PR

@claude

claude Bot commented Jun 1, 2026

Copy link
Copy Markdown

Claude encountered an error after 2s —— View job


I'll analyze this and get back to you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants