finding
active
finding:reward-probes-attend-to-entity-and-relation-tokens-failed-fix-shows-higher-attention-on-punctuationReward probes attend to entity and relation tokens; Failed Fix shows higher attention on punctuation
Attention map observation.
Source paper
extracted_from(2026) · Aaditya Vikram Prasad · Connor Watts · Jack Merullo · Dhruvil Gala +3
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- Framework premise.
Methods (1)
method
- Attention heatmaps of reward probesassociated_withCoarse assessment of what reward probes attend to.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Interpretation of resampling.
- Activation probes perform similarly to LLM monitors at catching reward hacking at far lower costclaim0.779Practical claim.
- Interesting special case of copying behavior related to tokenization artifacts; primitive precursor to induction heads
- Key interpretive claim from Case Study II distinguishing probe accuracy from causal relevance
- Design choice validated empirically.
- Explains why cities+neg_cities and larger_than+smaller_than training sets yield better OOD accuracy
- Abstract; central distinction.
- Shows that truth representations are not reducible to text probability representations