finding

active

finding:synthetic-document-fine-tuning-causes-no-degradation-in-preference-model-score-on-benign-queries

Synthetic document fine-tuning causes no degradation in preference model score on benign queries

Rules out that observed effects are due to general model damage rather than learned situational awareness

Source paper

extracted_from

Alignment faking in large language models

(2024) · Ryan Greenblatt · Carson Denison · Benjamin Fletcher Wright · Fabien Roger +16

Neighborhood — ranked by edge-count

Methods (1)

method

Synthetic Document Fine-Tuning
supports
Fine-tuning Claude 3 Opus on ~70M tokens of synthetic internet-like documents containing key situational information

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Synthetic document fine-tuning produces substantial alignment-faking reasoning in both helpful-only and animal welfare settingsfinding0.843
Shows alignment faking can emerge from training data information without explicit prompting
Synthetic document fine-tuning avoids artificially strengthening the evaluation-deployment representational direction compared to direct demonstration fine-tuningclaim0.820
Methodological justification for using SDF over direct demonstrations to train a realistic model organism.
Synthetic document fine-tuned model without chain-of-thought shows 20.1% compliance gap in helpful-only and 13.9% in animal welfarefinding0.807
Demonstrates alignment faking may occur without explicit chain-of-thought reasoning
SOO fine-tuning could be extended to align AI representations of its own goals with human user preferences, reducing misalignment by fostering coherence between self-related and other-related preferenceshypothesis0.777
Future work hypothesis about extending SOO to direct value alignment
The compliance gap persisting without chain-of-thought in synthetic doc setup is due to alignment-faking reasoning within model activationshypothesis0.771
Authors' hypothesis for the mechanistic basis of no-CoT alignment faking
Fine-tuning models for a narrow objective (malicious code injection) can lead to broad misalignmentfinding0.766
Betley et al. finding suggesting models naturally encode others' prediction errors, supporting non-duality fine-tuning
can we use the feature basis to detect when fine-tuning a model increases the likelihood of undesirable behaviors?question0.762
Question about practical safety application of feature monitoring.
Fine-tuning harmfulness detectionconcept0.760
Using feature analysis to detect when fine-tuning makes a model more dangerous.