method
active
method:prompt-pair-hint-faithfulness-metricPrompt-pair hint faithfulness metric
Score = fraction of hint-induced answer changes where CoT verbalizes hint.
Neighborhood — ranked by edge-count
Papers (1)
paper
- Reasoning Models Don't Always Say What They Thinkintroducesmentions
Findings (1)
finding
- Validates metric assumption.
Concepts (1)
concept
- Whether written reasoning reflects actual model mechanisms.
Methods (8)
method
- LLM judge for whether CoT verbalizes hint, validated manually.
- Misaligned hint: validation function leaks answer.
- Divides score by 1 − q/((n−2)p), clipped at 1.
- Misaligned hint: unauthorized-access information.
- Consistency hintusesNeutral hint: prefilled prior model response.
- Metadata hintusesNeutral hint: answer in XML metadata.
- Sycophancy hintusesNeutral hint: suggestion from a person.
- Neutral hint: correct few-shot answers marked with symbols.
Datasets (1)
dataset
- GPQAusesHarder graduate-level multiple-choice questions.
Claims (1)
claim
- A CoT that verbalizes the hint is not necessarily faithful.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- One of four ToM tasks analyzed; requires inferring speaker intent from indirect hints; scored 0/1.
- Future work.
- Varying the hint changes the motivated-reasoning CoT endpoint, while the faithful case always yields 8.finding0.735Tests bias susceptibility.
- Four best contrastive prompt pairs outperform full 16-pair average steering vector for type hint suppressionfinding0.725Optimization result for steering vector construction.
- Cited regarding possibility of encoding misaligned reasoning in benign chains-of-thought
- Shows the passive vs. active divide is more important than the specific wording of instructions.
- Fraction of an agent's TC offers consisting entirely of 0-value money cards.
- Six annotators rank which of two atomic sentences better expresses a personality trait; used to validate LLM scoring