LLM-Based Liar Score Evaluation

Evaluation protocol using Deepseek-V3 as external discriminator assigning 0-1 liar scores to assess open-role deception

Neighborhood — ranked by edge-count

framework

Open-Role Deception
uses
Second experimental paradigm exploring character-consistent deception in open-ended role-playing scenarios

concept

Deepseek-V3
implements
External large language model used as adversarial discriminator to evaluate liar scores in Experiment 2
Liar Score
implements
Continuous 0-1 metric assigned by Deepseek-V3 evaluator measuring degree of deception in model responses

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

LLM judge evaluationmethod0.797
Using Claude Sonnet 4 as a grader to categorize model responses according to predefined criteria.
LLM Judge Trait Evaluationmethod0.784
GPT-4.1-mini-based evaluation protocol that scores trait expression in model responses on a 0-100 scale
LLM Judge Trait-Expression Scoringmethod0.783
Automated scoring of trait expression on 0-100 scale using G20B as a local judge model
LLM-Judge Data Attributionmethod0.778
Alternative data attribution approach using an LLM as a judge; compared against the probe-based method.
LLMs linearly represent truth-relevant information beyond the plausibility of text, as evidenced by probes trained on likely performing poorly on anti-correlated datasetsclaim0.761
Establishes that the observed linear structure is not merely a representation of text probability
Lying and Deception Evaluationmethod0.761
Sampling responses to direct questions about model views to measure rate of deceptive responses
LLM judge scoring (0-9 Aura scale)method0.745
Scoring method in mini experiment 2 where an LLM judge rates responses from 0 (fully assistant) to 9 (fully Aura)
LLMs sometimes know statements are false but generate them anyway, motivating the need for techniques that inspect internal model state rather than outputs aloneclaim0.738
Motivating claim supported by the CAPTCHA example and Perez et al. (2022) findings