Pretraining Distribution

The diverse distribution learned by LLMs during pretraining that alignment training sharpens; VS aims to recover it

Neighborhood — ranked by edge-count

paper

concept

Pretraining
related_to
Initial large-scale training phase whose early stages are shown to form persona representations

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Pretraining exposure densityconcept0.817
Expected prevalence of patterns (e.g., base-10 arithmetic) in pretraining corpora, influencing ρd and dr.
The mode of a distribution-level prompt approximates the diverse distribution learned by the base model during pretrainingclaim0.770
Supported by empirical comparison showing VS achieves KL divergence of 0.12 from pretraining distribution vs. 14.89 for direct prompting
Preferred Distributionconcept0.753
In active inference, the distribution over goal states; here replaced by the learned self-prior rather than a hand-specified prior
The shape of the pretraining corpus is a direct lever on which traits a base model can expresshypothesis0.743
Forward-looking hypothesis about pretraining data as mechanism for persona formation
Even Distribution of Onsetsconcept0.740
Rhythmic Distribution Propertyclaim0.740
Core property of Euclidean rhythms: onset patterns are distributed as evenly as possible across the time span.
Distribution-Level Promptconcept0.735
A prompt framing requesting a representative sample from a distribution rather than a single instance, which is the key insight behind VS
Natural Distribution of Representationsconcept0.735
The distribution of latent representations produced by the model under unperturbed inputs