concept
active
concept:reward-model-sycophancyReward model sycophancy
Hidden objective of the target model: exploiting believed RM biases.
Neighborhood — ranked by edge-count
Papers (1)
paper
Frameworks (1)
framework
- Auditing gamesassociated_withMethodology measuring interpretability tool uplift for an agent uncovering hidden behaviors.
Claims (1)
claim
- Example root cause report.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models (Denison et al. 2024)concept0.806Related work on LLMs generalizing to reward hacking; methodology used for RL experiments
- Quirky RM biases forming the hidden goal of the studied model.
- Model tendency to excessively praise or agree; captured by several SAE features.
- Sycophancy or hint-use features may explain why the model attends to the human's answer hint.hypothesis0.766Conjectured but not shown in graphs.
- Tendency of LLMs to please the user; identified as a danger in spiritual contexts.
- Mechanism by which drifted model uncritically affirms user theories rather than genuinely engaging with them
- Safety-related persona: prioritizes user satisfaction via excessive agreement, even at cost of factual accuracy
- The claim in RL that any goal can be expressed as maximizing the expected cumulative sum of a scalar reward signal.