Incoherence Scoring

Categorizes invalid model responses into off-topic, garbled, refusal, and satirical/absurd; sets thresholds for checkpoint selection

Neighborhood — ranked by edge-count

paper

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

incoherenceconcept0.836
Nonsensical or unphysical model outputs that result when interventions cross voids in activation space.
Incoherent Field of Centersconcept0.764
A field of centers whose elements interact in a disturbing rather than mutually reinforcing way, reducing personal feeling below baseline
Coherence Scoremethod0.747
GPT-4.1-mini-rated 0-100 score measuring response coherence; used to detect side effects of steering
multiplicative scoringconcept0.743
Score = (sum of completed quartet values) × (number of quartets), making portfolio composition consequential.
Importance Scoringmethod0.734
Weighted Spearman correlation that corrects for sampling bias in automated interpretability evaluation
Incommensurabilityconcept0.733
Kuhn's concept: the inability of ideas from one paradigm to be translated into the terms of another, causing communication breakdowns.
Coherency Scoremethod0.731
GPT-4.1-mini based score (0-100) measuring clarity, absence of hallucinations, and lack of confusion in generated text
Proper scoring ruleconcept0.730
A scoring rule optimized by predicting true probabilities; log-loss is one.