finding
active
finding:probe-achieves-selectivity-of-4-20-on-pythia-410m-slightly-exceeding-das-selectivity-of-3-96Probe achieves selectivity of 4.20 on pythia-410m, slightly exceeding DAS selectivity of 3.96
Key result showing that for models larger than pythia-70m, probe selectivity matches or exceeds DAS selectivity
Source paper
extracted_from(2024) · Aryaman Arora · Dan Jurafsky · Christopher Potts
Neighborhood — ranked by edge-count
Claims (1)
claim
- Author interpretation of selectivity results showing DAS advantage diminishes when controlling for expressivity
Questions (1)
question
- Core methodological question motivating the introduction of selectivity and control tasks
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Linear probe achieves 100% classification accuracy for almost all components in Pythia-6.9B gender taskfinding0.808Demonstrates that linear probes can overestimate causal relevance; probes succeed on non-causally-relevant representations
- Baseline accuracy showing small models fail on harder NPI licensing tasks
- Kimi SWE-bench-selected probe achieves AUROC 0.93 on DeepSWE and 0.94 on ImpossibleBenchfinding0.786Cross-environment generalization.
- Mixed LR probe achieves 0.386 to 0.826 accuracy against 6.7% chance, but fails on natural documentsfinding0.779No evidence of chronic emotion representation.
- Positive valence probe predicts preference.
- Moderate agreement.
- Likely-trained MM probe is a surprisingly effective causal baseline due to correlation between truth and probability on sp_en_trans
- Justifies restricting probe-based vector derivation to h_b activations; attributed to Yes/No semantics