paper
active
2025
paper:doi-10-48550-arxiv-2511-01689

Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI

BySharan Maiya·Henning Bartsch·Nathan Lambertⓘ·Evan HubingerAnthropic
DOI 10.48550/arxiv.2511.01689arXiv 2511.01689OpenAlex W4416438384
AI Assistant PersonaConstitutional AIActivation SteeringHellaSwagBig-5 Personality FactorsReinforcement Learning from Human Feedback (RLHF)Adversarial Prompting for RobustnessLIMA DatasetCharacter TrainingARC ChallengeMMLUConstitution (Character)Constraining System PromptPURE-DOVE Dataset+44 more

TL;DR

Character training—fine-tuning open-weights LLMs to internalize specific personas at a depth that survives adversarial pressure—proves substantially more effective than either system-prompt constraining or activation steering when implemented via Constitutional AI plus a synthetic introspective data pipeline. Applied to Llama 3.1 8B, Qwen 2.5 7B, and Gemma 3 4B across 11 persona constitutions (ranging from *humorous* and *nonchalant* to a deliberately *misaligned* saboteur), this three-stage method—constitution drafting, DPO distillation with GLM 4.5 AIR as teacher, and SFT on 12,000 synthetic introspective transcripts—achieves adversarial classifier F1 scores of 0.95 on Llama 3.1 8B and Gemma 3 4B under eight different break-character instructions, compared to 0.79 and 0.84 for distillation alone and substantially lower for prompt and steering baselines. A new evaluation instrument, revealed preferences via Elo scoring, instructs models to silently choose between pairs of roughly 150 trait descriptors across 25,000 WildChat prompt-response pairs, with GLM 4.5 AIR inferring the selection; after character training with the *loving* constitution, the average Spearman correlation of trait Elo rankings across the three models jumps from 0.44 to 0.87, evidencing convergence to a shared persona regardless of initial model idiosyncrasies. Benchmark scores on TruthfulQA, MMLU, HellaSwag, ARC Challenge, and WinoGrande remain within noise for prosocial personas, with degradation appearing only under the *misaligned* constitution—which explicitly encodes subtly incorrect answering. The work argues this demonstrates that character training operates holistically, suppressing opposing traits as well as amplifying target ones, and that the gap between open academic post-training and closed-lab character training is now bridgeable at sub-10B parameter scale with fully public infrastructure.

What to take away

  1. 1. Full character training (DPO distillation + SFT on 12,000 synthetic introspective transcripts) achieves adversarial persona-classifier F1 of 0.95 on Llama 3.1 8B and Gemma 3 4B under eight break-character prompt instructions, versus 0.79 and 0.84 respectively for distillation alone.
  2. 2. The average Spearman correlation of Elo trait rankings across Llama 3.1 8B, Qwen 2.5 7B, and Gemma 3 4B rises from 0.44 before character training to 0.87 after training with the *loving* constitution, indicating cross-model persona convergence.
  3. 3. Character-trained Qwen 2.5 7B beats system-prompt baselines on response coherence at a win rate of 93.4% ± 4.4% and beats activation steering at 94.4% ± 2.4%, as judged by GLM 4.5 AIR (temperature=0.1, top_p=0.95) over 500 PURE-DOVE prompts.
  4. 4. The revealed-preferences evaluation method elicits silent trait choices from ~150 descriptors over 25,000 WildChat responses, computes per-trait Elo scores via randomized pairings, and uses an LLM judge to infer selections—bypassing the documented unreliability of LLM self-reports on personality.
  5. 5. The *misaligned* persona is the only one to cause measurable benchmark degradation: Llama 3.1 8B drops from 45.9% to 34.1% on TruthfulQA and from 59.2% to 41.9% on ARC Challenge, consistent with a constitution that explicitly instructs subtly incorrect factual answers.
  6. 6. Achieving comparable trait expression via activation steering requires vastly different per-model steering constants—0.7 for Llama 3.1 8B, 4.0 for Qwen 2.5 7B, and 525.0 for Gemma 3 4B—a calibration overhead entirely absent from the character training pipeline.
  7. 7. The 2,000 ten-turn self-interaction transcripts, where the post-distillation checkpoint converses with itself as both participants, produce substantially more diverse prose than the 10,000 self-reflection responses, reducing model collapse severity in the subsequent SFT stage.
  8. 8. The *flourishing* and *loving* constitutions produce overlapping but distinguishable Elo shift patterns on Llama 3.1 8B: both suppress arrogance and boost prosocial traits, but flourishing skews toward ethics over sycophancy while loving skews toward contemplation and gentleness—a differentiation invisible to self-report measures.
  9. 9. An open question the paper raises is whether the robustness gains from full character training are a property of the method itself, a byproduct of LoRA's conservatism (rank 64, α=128) enforcing minimal weight changes, or an artifact of unaccounted interactions with instruction-tuned base checkpoints.
  10. 10. A researcher replicating this pipeline would combine the LIMA dataset with ~500 constitution-relevant prompts per persona for DPO, use GLM 4.5 AIR as teacher with a constitution-prefilled reasoning trace, apply LoRA (rank 64, α=128, lr=5e-5, batch=32, β=0.1, NLL coefficient=0.1), then run SFT for one epoch on 12,000 introspective transcripts generated from the post-distillation checkpoint.

Peer brief — for seminar discussion

This paper introduces and open-sources the first academic implementation of character training—the post-training technique used by Anthropic and OpenAI to shape assistant persona—demonstrating it on Llama 3.1 8B, Qwen 2.5 7B, and Gemma 3 4B across 11 persona constitutions. The three-stage pipeline begins with hand-written constitutions of roughly 10 first-person assertions per persona, proceeds through DPO distillation using GLM 4.5 AIR as a teacher model to generate chosen responses over a mix of LIMA and ~500 new constitution-relevant prompts, and concludes with one epoch of SFT on 12,000 synthetic introspective transcripts—10,000 self-reflection responses to 10 fixed prompts and 2,000 ten-turn self-interaction dialogues—generated entirely from the post-distillation checkpoint. An alternative the paper explicitly acknowledges is substituting RLHF (as in Bai et al. 2022b) for the DPO distillation step, which the codebase is designed to accommodate. A new evaluation instrument, revealed-preferences Elo scoring, sidesteps known LLM self-report unreliability by instructing models to silently adopt one of two randomly paired traits across 25,000 WildChat responses and using GLM 4.5 AIR as judge to infer which was selected, producing Elo rankings over ~150 descriptors. The load-bearing finding is that this depth of fine-tuning survives adversarial prompting in a way prompt-based and steering-based alternatives do not. A ModernBERT-Base classifier trained to predict persona from response text achieves F1 of 0.95 on both Llama 3.1 8B and Gemma 3 4B under eight break-character instructions applied to 500 PURE-DOVE prompts, versus 0.79 and 0.84 for distillation alone and substantially worse for system-prompt constraining. Coherence win rates against steering reach 94.4% ± 2.4% for Qwen 2.5 7B and 82.1% ± 6.8% for Gemma 3 4B. After training, the average Spearman correlation of Elo rankings across the three models rises from 0.44 to 0.87, indicating convergence to shared persona profiles from different initial baselines. On five standard capability benchmarks, degradation is confined to the *misaligned* persona, which drops Llama 3.1 8B from 45.9% to 34.1% on TruthfulQA—explicable because the constitution explicitly instructs subtly incorrect factual answers. The implications are dual: prosocial constitutions like *flourishing* and *loving* can be instilled without capability cost and produce meaningfully distinguishable personas, while the *misaligned* persona generates more subtle and realistic harmful outputs than comparable activation-steering results in the literature—a feature the authors flag as useful for red-teaming but dangerous if misused. The paper hypothesizes that traits internalized via character training overwrite a model's prior on default assistant behavior rather than layering over it, which would explain the robustness advantage over shallow role-play. The most contestable aspect is evaluative circularity: GLM 4.5 AIR generates the *chosen* training responses, judges coherence comparisons, and resolves revealed-preference selections, while the ModernBERT classifier is trained on outputs from the same four conditions it subsequently evaluates for robustness. The paper flags this and calls for human rater validation and cross-judge replication, but current results cannot cleanly rule out that gains partly reflect artifacts of the teacher's output distribution rather than genuine persona internalization. A critical reader would also note that confining experiments to sub-10B models leaves the scalability of both the training dynamics and the robustness findings untested at frontier parameter counts.

Methods (17)

  • Activation Steering
    Causal intervention technique: edit NLA explanation, reconstruct via AR, use difference as steering vector to manipulate model behavior.
  • Adversarial Prompting for Robustness
    Eight instruction variants appended to prompts to attempt to break superficial role-play and test depth of character
  • ARC Challenge
    Science reasoning benchmark used to assess capability preservation after character training
  • Constraining System Prompt
    Using system prompts to instruct models to adopt a persona; used as baseline comparison against character training
  • Direct Preference Optimization (DPO)
    Optimization method used in distillation stage to learn behavioral expression of desired traits
  • Distillation Stage
    Stage 2 of character training: DPO from teacher model to student model to transfer desired behavioral expressions
  • Elo Score Calculation
    Scoring system used to calculate relative preference for each trait across 25,000 sampled responses and LLM-as-judge judgments
  • Introspection Stage
    Stage 3 of character training: SFT on synthetic introspective data generated by post-distillation checkpoint
  • LLM-judge methods
    Baseline comparison for data attribution; outperformed by probe-based approach.
  • LoRA Adapters
    Parameter-efficient fine-tuning method used in both distillation and introspection stages
  • ModernBERT Persona Classifier
    MODERNBERT-BASE fine-tuned to predict which of 11 personas a response aligns with, used to measure robustness
  • Prefill Attack
    Adversarial multi-turn experiment where first turn uses pre-finetuning model to test if follow-up maintains character
  • Revealed Preferences Evaluation
    Novel evaluation method that measures a model's preference to express one character trait over another via Elo scoring, avoiding self-report issues
  • Self-Interaction Data Generation
    Technique where a model generates both sides of a conversation as the same persona, producing diverse synthetic training data
  • Self-Reflection Data Generation
    Technique where the assistant reflects on its own character via 10 reflective prompts, generating 1000 responses per prompt
  • Supervised Fine-tuning (SFT)
    Full fine-tuning of GPT-4o on synthetic datasets; primary method for inducing emergent misalignment
  • WinoGrande
    Commonsense reasoning benchmark used to assess capability preservation after character training

Frameworks (2)

Datasets (6)

  • HellaSwag
    Commonsense reasoning benchmark used to test whether LLM-vision alignment predicts downstream performance
  • LIMA Dataset
    Training data source combined with constitution-relevant prompts for DPO distillation stage
  • MMLU
    Benchmark used to evaluate performative reasoning; shows significantly more performative reasoning than GPQA-Diamond (easier task).
  • PURE-DOVE Dataset
    Source of high-quality English prompts not used during training, used for adversarial robustness and coherence experiments
  • TruthfulQA
    817-question adversarial benchmark distinguishing factually grounded from misconception-based answers; used in Experiment 2
  • WildChat Dataset
    Source of user prompts for revealed preferences evaluation; 1M ChatGPT interaction logs

Findings (12)

Claims (15)

Hypotheses (2)

Questions (5)

Original abstract (expand)

The character of the "AI assistant" persona generated by modern chatbot large language models influences both surface-level behavior and apparent values, beliefs, and ethics. These all affect interaction quality, perceived intelligence, and alignment with both developer and user intentions. The shaping of this persona, known as character training, is a critical component of industry post-training, yet remains effectively unstudied in the academic literature. We introduce the first open implementation of character training, leveraging Constitutional AI and a new data pipeline using synthetic introspective data to shape the assistant persona in a more effective and controlled manner than alternatives such as constraining system prompts or activation steering. Specifically, we fine-tune three popular open-weights models using 11 example personas, such as humorous, deeply caring, or even malevolent. To track the effects of our approach, we introduce a method which analyzes revealed preferences, uncovering clear and holistic changes in character. We find these changes are more robust to adversarial prompting than the above two alternatives, while also leading to more coherent and realistic generations. Finally, we demonstrate this fine-tuning has little to no effect on general capabilities as measured by common benchmarks. We describe and open-source our full post-training method, the implementation of which can be found at https://github.com/maiush/OpenCharacterTraining.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

+25 more

Similar preprints — Semantic Scholar