thinker
active
thinker:tristan-hume

Tristan Hume

Authored
4
Introduces
0
Studies
0
Affiliations
0
Cited by
12

Authored papers (4)

  • Chain-of-thought reasoning in a 175B RLHF-tuned assistant model is faithful only conditionally: how much the final answer depends on the stated reasoning swings from near zero to dominant by task, and faithfulness falls as models grow. On ARC (Easy), ARC (Challenge) and OpenBookQA, truncating the CoT changes the final answer under 10% of the time, while on AQuA it changes it over 60% of the time. Lanham and colleagues introduce a battery of CoT-intervention tests: early answering (truncation), adding mistakes, filler tokens and paraphrasing. They use them to rule out specific failure classes rather than prove faithfulness. Replacing the CoT with filler ellipses gives no accuracy gain, which argues against hidden test-time computation. Paraphrasing matches original accuracy, which argues against phrasing-encoded steganography. Post-hoc reasoning, measured by area over the curve (AOC), correlates weakly with CoT's accuracy benefit: LogiQA gains about 1 point from CoT yet ranks second in AOC (0.26), while HellaSwag loses 4.69 points. Across the 810M–175B model series, 13B models give the most faithful CoT on 6 of 8 tasks, and on synthetic addition tasks faithfulness rises with model size and with easier problems. Faithful CoT therefore depends on choosing a model whose capability level is matched to the task, not a maximally capable one. The authors argue that CoT faithfulness is a tunable property, to be measured per task and per model before the reasoning is trusted in high-stakes settings.

  • Constitutional AI (CAI) demonstrates that a harmless, non-evasive AI assistant can be trained using zero human feedback labels for harmlessness, replacing them entirely with AI-generated feedback guided by a short list of natural language principles. The method introduces a two-stage pipeline: a supervised learning (SL) phase in which a helpful RLHF model iteratively critiques and revises its own responses to red-team prompts drawn from 182,831 total prompts (42,496 human-written, 140,335 model-generated), followed by a reinforcement learning phase termed RLAIF, where a 52B-parameter feedback model evaluates response pairs according to 16 constitutional principles and generates preference labels that train a hybrid preference model. Crowdworker Elo score comparisons across 10,274 helpfulness and 8,135 harmlessness evaluations show that RL-CAI with chain-of-thought reasoning achieves harmlessness scores that meet or exceed those of models trained with human harmlessness feedback (HH RLHF), while maintaining comparable helpfulness—tracing a Pareto improvement over the helpfulness–harmlessness tradeoff documented in prior work. Critically, whereas HH RLHF models produced evasive refusals (e.g., 'I'm sorry, I won't respond') on sensitive PALMS and LaMDA prompts, RL-CAI models engage substantively and explain their objections. Chain-of-thought prompting on the feedback model, with CoT probabilities clamped to the 40–60% range to prevent overconfidence, further improved both harmlessness scores and label calibration. The paper argues this implies that scaled AI supervision—encoding alignment objectives in a transparent, auditable constitution rather than tens of thousands of opaque human labels—is a viable path toward alignment as model capabilities grow beyond reliable human oversight.

  • Neural networks store more sparse features than they have dimensions, and a one-line change shows it: adding a ReLU to the output of a linear W^T W autoencoder moves the optimum from PCA-like top-m feature selection to superposition. In toy models (n=20 features in m=5 dimensions with I_i=0.7^i; n=400 in m=30 for uniform runs), the ReLU output model matches the linear model on dense inputs but packs in extra features as sparsity rises. Features then settle into Thomson-problem polytopes (digons, triangles, tetrahedra, pentagons, square antiprisms). Dimensions-per-feature D* sticks at 1 and 1/2, with per-feature clusters at 3/4, 2/3, 2/5 and 3/8. Whether a feature is dropped, superposed or given its own dimension is set by a first-order phase change. The paper's instruments are the feature dimensionality D_i = ||W_i||²/Σ_j(Ŵ_i·W_j)², phase diagrams over sparsity and importance, and the W^TW polytope correspondence. A ReLU hidden-layer model computing abs(x) with n=100 features and m=40 neurons shows computation in superposition, via an asymmetric-superposition-plus-inhibition motif. Adversarial vulnerability rises more than 3x as superposition forms. The same account separates monosemantic from polysemantic neurons by sparsity and importance. Superposition is therefore a regime, not an inevitable cost, and may be removable at a performance price. Interpretability then reduces to enumerating features, by training superposition-free models (L1, MoE) or by recovering an overcomplete basis with sparse coding.

More papers — OpenAlex / S2

Co-authors (12)

Recent mentions (1)