paper
referenced-only
2025
paper:doi-10-48550-arxiv-2506-01197

Incorporating Hierarchical Semantics in Sparse Autoencoder Architectures

ByMark Muchane·Sean M. Richardson·Kiho Park·Victor Veitch
Original abstract (expand)

Sparse dictionary learning (and, in particular, sparse autoencoders) attempts to learn a set of human-understandable concepts that can explain variation on an abstract space. A basic limitation of this approach is that it neither exploits nor represents the semantic relationships between the learned concepts. In this paper, we introduce a modified SAE architecture that explicitly models a semantic hierarchy of concepts. Application of this architecture to the internal representations of large language models shows both that semantic hierarchy can be learned, and that doing so improves both reconstruction and interpretability. Additionally, the architecture leads to significant improvements in computational efficiency.

Similar preprints — Semantic Scholar

Cited by (1)