paper
referenced-only
2020
paper:arxiv-2101-00027

The Pile: An 800GB Dataset of Diverse Text for Language Modeling

ByLeo Gao·Stella Biderman·Sid Black·Laurence Golding·Travis Hoppe·Charles Foster+4 more
Original abstract (expand)

Recent work has demonstrated that increased training dataset diversity improves general cross-domain knowledge and downstream generalization capability for large-scale language models. With this in mind, we present the Pile : an 825 GiB English text corpus tar-geted at training large-scale language models. The Pile is constructed from 22 diverse high-quality subsets—both existing and newly constructed—many of which derive from academic or professional sources. Our evaluation of the untuned performance of GPT-2 and GPT-3 on the Pile shows that these models struggle on many of its components, such as academic writing. Conversely, models trained on the Pile improve significantly over both Raw CC and CC-100 on all components of the Pile, while improving performance on downstream evaluations. Through an in-depth exploratory analysis, we document potentially concerning aspects of the data for prospective users. We make publicly available the code used in its construction. 1

Similar preprints — Semantic Scholar

Cited by (1)

  • Do Sparse Autoencoders Capture Concept Manifolds?

    Sparse autoencoders trained on Llama3.1-8B (layer 19, 500M Pile tokens, 25 SAEs across Standard, JumpReLU, TopK, BatchTopK and Matryoshka architectures at VE > 0.85) do not capture continuous concept