paper:doi-10-48550-arxiv-2310-06825Mistral 7B
Original abstract (expand)
We introduce Mistral 7B v0.1, a 7-billion-parameter language model engineered for superior performance and efficiency. Mistral 7B outperforms Llama 2 13B across all evaluated benchmarks, and Llama 1 34B in reasoning, mathematics, and code generation. Our model leverages grouped-query attention (GQA) for faster inference, coupled with sliding window attention (SWA) to effectively handle sequences of arbitrary length with a reduced inference cost. We also provide a model fine-tuned to follow instructions, Mistral 7B -- Instruct, that surpasses the Llama 2 13B -- Chat model both on human and automated benchmarks. Our models are released under the Apache 2.0 license.
Similar preprints — Semantic Scholar
Cited by (2)
- Facet-Level Persona Control by Trait-Activated Routing with Contrastive SAE for Role-Playing LLMs
Facet-level personality control in role-playing LLMs is substantially improved by injecting contrastively trained sparse autoencoder (SAE) control vectors into mid-residual layers, with the CV-SAE+Pro
- The Platonic Representation Hypothesis
Neural networks trained on different data modalities, architectures, and objectives are converging toward a shared statistical model of reality — what the paper terms the "platonic representation" — f