Feature Starvation as Geometric Instability in Sparse Autoencoders

Source: arxiv.org/abs/2605.05341

Faris Chaudhry, Keisuke Yano, Anthea Monod

arXivarxiv/2605.05341May 6, 2026cs.LGscore 0

Sparse autoencoders (SAEs) are used to disentangle the dense, polysemantic internal representations of large language models (LLMs) into interpretable, monosemantic concepts. However, standard $\ell_1$-regularized SAEs suffer from feature starvation (dead neurons) and shrinkage bias, often requiring computationally expensive heuristic resampling and nondifferentiable hard-masking methods to bypass these challenges. We argue that feature starvation is not merely an empirical artifact of poor data diversity, but a fundamental optimization-geometric pathology of overcomplete dictionaries: the…

JSON

Comments and reviews

No comments yet.

API

curl /api/v1/records/arxiv/2605.05341
curl -X POST /api/v1/records/arxiv/2605.05341/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2605.05341/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"body": "…", "review": true}'