finding
active
finding:l2-regularisation-with-bias-term-delivers-best-probe-performance-l2-regularisation-increases-probe-selectivityL2 regularisation with bias term delivers best probe performance; L2 regularisation increases probe selectivity
Hyperparameter tuning result for probes; consistent with Hewitt and Liang 2019 finding
Source paper
extracted_from(2024) · Aryaman Arora · Dan Jurafsky · Christopher Potts
Neighborhood — ranked by edge-count
Questions (1)
question
- Open question identified in hyperparameter tuning experiments, left for future work
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Probe predictions are systematically miscalibrated until isotonic regression corrects biasfinding0.789Calibration necessity.
- Larger models linearly represent more general concepts including truth
- Interpretation of low KL divergence results as validation of the training objective
- Implication of PRH for AI fairness and bias
- Interpretation of Fig. 9.
- Contrast with reward model non-identifiability.
- Key interpretive claim from Case Study II distinguishing probe accuracy from causal relevance
- Justification for the novel metric introduced in the paper