paper
referenced-only
2023
paper:penedo-2023-refinedwebThe RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data Only
ByGuilherme Penedo·Quentin Malartic·Daniel Hesslow·Ruxandra-Aimée Cojocaru·Hamza Alobeidli·Alessandro Cappelli+3 more
Similar preprints — Semantic Scholar
Cited by (1)
- Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
Larger models learn tasks smaller models never learn, and the cause is data-induced competition for neurons, not expressivity: small models spend capacity on frequent or simple tasks, and their update