paper
referenced-only
2015
paper:arxiv-1509-08101

Representation Benefits of Deep Feedforward Networks

ByMatus Telgarsky
Original abstract (expand)

This note provides a family of classification problems, indexed by a positive integer $k$, where all shallow networks with fewer than exponentially (in $k$) many nodes exhibit error at least $1/6$, whereas a deep network with 2 nodes in each of $2k$ layers achieves zero error, as does a recurrent network with 3 distinct nodes iterated $k$ times. The proof is elementary, and the networks are standard feedforward networks with ReLU (Rectified Linear Unit) nonlinearities.

Similar preprints — Semantic Scholar

Cited by (1)

  • Do Sparse Autoencoders Capture Concept Manifolds?

    Sparse autoencoders trained on Llama3.1-8B (layer 19, 500M Pile tokens, 25 SAEs across Standard, JumpReLU, TopK, BatchTopK and Matryoshka architectures at VE > 0.85) do not capture continuous concept