Profiled Sparse Networks (PSN)

Heterogeneous Connectivity in Sparse Networks: Fan-in Profiles, Gradient Hierarchy, and Topological Equilibria

Nikodem Tomczak, Thulge Labs, Singapore

Copyright (c) 2026 Nikodem Tomczak, Thulge Labs. All rights reserved.

---

Overview

PSN is a framework for designing deterministic heterogeneous connectivity in sparse neural networks through continuous profile functions. Rather than treating all neurons identically, PSN assigns per-neuron fan-in according to smooth nonlinear profiles, creating neurons with dense and sparse receptive fields.

This repository contains the library code accompanying the paper "Heterogeneous Connectivity in Sparse Networks: Fan-in Profiles, Gradient Hierarchy, and Topological Equilibria".

Key findings:

• Static connectivity structure does not affect accuracy at matched sparsity across four datasets and fan-in CVs from 0 to 2.5
• Fan-in CV determines gradient heterogeneity as a structural consequence of mask geometry (r = 0.93)
• Lognormal initialisation matched to the RigL equilibrium fan-in distribution consistently outperforms ERK, with advantages increasing on harder tasks

Installation

pip install -e .

PyTorch must be installed separately. See pytorch.org for the appropriate version for your hardware.

Library

The core library is in src/sparse_profiles/ and provides:

Module	Description
masks.py	Profile-based connectivity mask generation (8 parametric profiles, lognormal, power-law, multi-peak)
layers.py	Sparse linear layers with profile-based connectivity masks
models.py	MLP architectures with PSN sparse connectivity
training.py	Training loops and utilities
rigl.py	RigL dynamic sparse training with PSN profile initialisation
evaluation.py	Gradient ratio computation and evaluation tools
datasets.py	Dataset loaders (MNIST, Fashion-MNIST, EMNIST-Balanced, Forest Cover)
utils.py	Utility functions

Quick example

from sparse_profiles.models import create_model
from sparse_profiles.training import Trainer

# Dense baseline
model = create_model('mnist', model_type='dense', hidden_dim=1024)

# PSN with exponential profile at 90% sparsity
model = create_model('mnist', model_type='profiled_random',
                        profile='exponential', global_sparsity=0.9,
                        min_fan_in=1)

# PSN with lognormal profile at target CCV = 2.5
model = create_model('mnist', model_type='profiled_random',
                        profile='lognormal', global_sparsity=0.9,
                        target_cv=2.5, min_fan_in=1)

Connectivity profiles

PSN supports the following fan-in profile families:

Parametric profiles (8 total): exponential, bell, linear, quadratic, and their inverses. Each produces a deterministic fan-in curve as a function of neuron index.

Distribution-based profiles: lognormal and powerlaw, parameterised by target CCV (coefficient of variation of fan-in). These sample fan-in values from the specified distribution and sort in descending order.

Multi-peak profiles: multi_peak with parameters num_peaks (k) and peak_beta. Interpolates continuously from extreme heterogeneity (k=1) to uniformity (large k).

All profiles preserve the same mean fan-in at a given sparsity level. They differ only in the distribution of fan-in across neurons.

Project structure

src/sparse_profiles/     Core library
  masks.py               Mask generation (profiles, lognormal, power-law, multi-peak)
  layers.py              Sparse linear layers
  models.py              MLP architectures (SimpleMLP, DeepMLP)
  training.py            Training loops
  rigl.py                RigL dynamic sparse training
  evaluation.py          Gradient ratio computation
  datasets.py            Dataset loaders
  utils.py               Utilities

Citation

@article{tomczak2026psn,
  title={Heterogeneous Connectivity in Sparse Networks: Fan-in Profiles,
         Gradient Hierarchy, and Topological Equilibria},
  author={Tomczak, Nikodem},
  journal={arXiv preprint arXiv:2604.10560},
  year={2026},
  url={https://arxiv.org/abs/2604.10560}
}

License

MIT License. See LICENSE for details.