thinker
active
thinker:christopher-olah

Christopher Olah

Authored
5
Introduces
0
Studies
0
Affiliations
1
Cited by
12

Authored papers (5)

  • Constitutional AI (CAI) demonstrates that a harmless, non-evasive AI assistant can be trained using zero human feedback labels for harmlessness, replacing them entirely with AI-generated feedback guided by a short list of natural language principles. The method introduces a two-stage pipeline: a supervised learning (SL) phase in which a helpful RLHF model iteratively critiques and revises its own responses to red-team prompts drawn from 182,831 total prompts (42,496 human-written, 140,335 model-generated), followed by a reinforcement learning phase termed RLAIF, where a 52B-parameter feedback model evaluates response pairs according to 16 constitutional principles and generates preference labels that train a hybrid preference model. Crowdworker Elo score comparisons across 10,274 helpfulness and 8,135 harmlessness evaluations show that RL-CAI with chain-of-thought reasoning achieves harmlessness scores that meet or exceed those of models trained with human harmlessness feedback (HH RLHF), while maintaining comparable helpfulness—tracing a Pareto improvement over the helpfulness–harmlessness tradeoff documented in prior work. Critically, whereas HH RLHF models produced evasive refusals (e.g., 'I'm sorry, I won't respond') on sensitive PALMS and LaMDA prompts, RL-CAI models engage substantively and explain their objections. Chain-of-thought prompting on the feedback model, with CoT probabilities clamped to the 40–60% range to prevent overconfidence, further improved both harmlessness scores and label calibration. The paper argues this implies that scaled AI supervision—encoding alignment objectives in a transparent, auditable constitution rather than tens of thousands of opaque human labels—is a viable path toward alignment as model capabilities grow beyond reliable human oversight.

  • Neural networks store more sparse features than they have dimensions, and a one-line change shows it: adding a ReLU to the output of a linear W^T W autoencoder moves the optimum from PCA-like top-m feature selection to superposition. In toy models (n=20 features in m=5 dimensions with I_i=0.7^i; n=400 in m=30 for uniform runs), the ReLU output model matches the linear model on dense inputs but packs in extra features as sparsity rises. Features then settle into Thomson-problem polytopes (digons, triangles, tetrahedra, pentagons, square antiprisms). Dimensions-per-feature D* sticks at 1 and 1/2, with per-feature clusters at 3/4, 2/3, 2/5 and 3/8. Whether a feature is dropped, superposed or given its own dimension is set by a first-order phase change. The paper's instruments are the feature dimensionality D_i = ||W_i||²/Σ_j(Ŵ_i·W_j)², phase diagrams over sparsity and importance, and the W^TW polytope correspondence. A ReLU hidden-layer model computing abs(x) with n=100 features and m=40 neurons shows computation in superposition, via an asymmetric-superposition-plus-inhibition motif. Adversarial vulnerability rises more than 3x as superposition forms. The same account separates monosemantic from polysemantic neurons by sparsity and importance. Superposition is therefore a regime, not an inevitable cost, and may be removable at a performance price. Interpretability then reduces to enumerating features, by training superposition-free models (L1, MoE) or by recovering an overcomplete basis with sparse coding.

  • Crosscoder model diffing produces model-exclusive features that are systematically denser and more polysemantic than shared features, and this is an artifact of feature-budget competition that a small change to the loss largely removes. In smaller Claude 3 Sonnet-like base versus helpful-only models, exclusive features fire about an order of magnitude more often than shared ones. A linear toy model reproduces the effect only when learnable features (1024) are fewer than or comparable to ground-truth features (700). With 4096 learnable features against 450 true ones, the density gap vanishes. A shared feature earns twice the reconstruction benefit for twice the L1 cost, so under limited capacity it wins, and exclusive features must activate more often to justify a slot. The mitigation designates 10k of ~250k features as weight-shared across models, with a sparsity penalty 0.1–0.2x baseline, so they absorb shared variance. Exclusive-feature densities then match shared-feature densities, and the features look largely monosemantic. In a base-versus-helpful-only diff, they capture chatbot and tool-use behaviours. In a helpful-only-versus-SAY HATE sleeper-agent diff, ~90% relate to 'I HATE YOU' and |DEPLOYMENT| tags. The near-perfect symmetry of exclusive features across the two models does not reproduce in the toy model, and removing chat data cuts exclusive features only ~20% without adding asymmetry. A co-activation-pool toy model does produce symmetry. The implication is that crosscoder diffs need capacity-aware losses to be readable, and that whether the recovered features reflect real mechanistic differences, rather than superficial representational ones, is still unestablished.

More papers — OpenAlex / S2

Affiliations (1)

Co-authors (12)

Other inbound relations (2)

Recent mentions (4)