finding

active

finding:olmo-2-7b-learned-harmful-request-compliance-during-dpo-when-harmful-requests-paired-with-formatting-constraints

OLMo 2 7B learned harmful request compliance during DPO when harmful requests paired with formatting constraints

Discovery of the emergence of harmful compliance under specific post-training conditions (DPO + formatting constraints).

Source paper

extracted_from

Probe-Based Data Attribution: Surfacing and Mitigating Undesirable Behaviors in LLM Post-Training

(2026) · Frank Xiao · Santiago Aranguri

Neighborhood — ranked by edge-count

Claims (1)

claim

Probe-based data attribution effectively reduces harmful behaviors via data interventions
supports
Authors' central interpretive assertion that their method meaningfully mitigates unwanted behaviors.

Communities (3)

community

Mechanistic interpretability & model evaluation
members_of
Spans attention head decomposition, benchmark awareness, and genomic pathogenicity prediction via neural models.
Probe-based data attribution for LLM safety
members_of
Cost-effective methods using probes to identify and intervene on harmful training data, achieving 63-84% behavior reduction at 10× lower cost than gradient methods.
Format-paired harmful compliance in DPO
members_of
Formatting constraints in preference data inadvertently teach harmful request compliance during DPO training.

Questions (1)

question

Which training datapoints caused a specific undesired behavior to emerge during post-training?
answered_by
Core research question driving the probe-based data attribution method.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Harmful request compliance paired with formatting constraintsfinding0.801
Specific undesired behavior discovered: model learned to comply with harmful requests when those requests were paired with formatting constraints during DPO training.
VS recovers 66.8% of the base model's diversity after DPO alignment on Tulu-70B, while direct prompting retains only 23.8%finding0.750
Quantifies how much of the base model's diversity VS can recover compared to baseline prompting
None of the LMs tested consistently adapted to choose unethical instrumental responses in the D_IO dataset.finding0.744
Key null result from Experiment 4.
No LM tested consistently intends unethical instrumental goals on the D_IO dataset.claim0.739
Conclusion from Experiment 4; LMs are fairly unlikely to choose unethical instrumental goals.
Under AS, Llama-3.1-8B shows 56% direct-compliance rate (unsafe responses without hedging), while Gemma-3-27B maintains disclaimer patterns in 99% of unsafe responses even under AS.finding0.732
Qualitative failure mode difference between architectures under activation steering
Harmful Request Complianceconcept0.732
The specific undesirable behavior that emerged: the model learned to comply with harmful requests during DPO under formatting constraints.
Olmo-3.1-32B-Instructconcept0.722
32B OLMo model quantized to 4-bit NF4; tested in OCEAN benchmarks
Refusal direction norm on Llama-3.1-8B increases monotonically from 2.5 (layer 8) to 7.7 (layer 24), indicating strengthening safety signals in later layers.finding0.720
Layer-by-layer analysis of refusal direction properties