paper
referenced-only
2025
paper:doi-10-48550-arxiv-2510-16820

Finding Manifolds With Bilinear Autoencoders

ByThomas Dooms·Ward Gauderis
Original abstract (expand)

Sparse autoencoders are a standard tool for uncovering interpretable latent representations in neural networks. Yet, their interpretation depends on the inputs, making their isolated study incomplete. Polynomials offer a solution; they serve as algebraic primitives that can be analysed without reference to input and can describe structures ranging from linear concepts to complicated manifolds. This work uses bilinear autoencoders to efficiently decompose representations into quadratic polynomials. We discuss improvements that induce importance ordering, clustering, and activation sparsity. This is an initial step toward nonlinear yet analysable latents through their algebraic properties.

Similar preprints — Semantic Scholar

Cited by (1)

  • Do Sparse Autoencoders Capture Concept Manifolds?

    Sparse autoencoders trained on Llama3.1-8B (layer 19, 500M Pile tokens, 25 SAEs across Standard, JumpReLU, TopK, BatchTopK and Matryoshka architectures at VE > 0.85) do not capture continuous concept