method
active
method:attention-heatmaps-of-reward-probesAttention heatmaps of reward probes
Coarse assessment of what reward probes attend to.
Neighborhood — ranked by edge-count
Papers (1)
paper
Findings (1)
finding
- Reward probes attend to entity and relation tokens; Failed Fix shows higher attention on punctuationassociated_withAttention map observation.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Interpretation of resampling.
- Classifiers used for token filtering and reward shaping.
- Internals-based approach to which the logit monitor is complementary.
- Input-output mapping per feature across ~100 prompts.
- Nguyen et al. trained linear probes on activations to distinguish evaluation from deployment scenarios.
- Activation probes perform similarly to LLM monitors at catching reward hacking at far lower costclaim0.724Practical claim.
- Linear classifier approach applied to model activations to identify which training datapoints caused undesired behaviors in post-training.
- Discovery claim.