claim
active
claim:harmful-features-mostly-reflect-graphic-criminal-content-subsets-of-the-lmsys-chat-training-dataHarmful features mostly reflect graphic criminal content subsets of the LMSys chat training data
Explanation for moderated features.
Neighborhood — ranked by edge-count
Papers (1)
paper
Findings (1)
finding
- Moderation statistics.
Claims (1)
claim
- Open-sourcing SAEs on current leading open-source models does not meaningfully affect biorisk or persuasion riskassociated_withRisk judgment because base model is insufficiently capable.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- During pretraining the model learns distinct harm categories and during finetuning wires them into general harmful-request features.hypothesis0.766Suggested by dictionaries trained on pretraining data having few refusal features.
- Author's interpretive conclusion from comparing filtering strategies
- LMSYS-Chat-1M SAE produced the most effective features for chat applications on Llama-3-8B-Instructfinding0.762Result of extensive hyperparameter and dataset experimentation.
- Chat-trained SAEs find behaviourally relevant features because many causal features occur on chat-specific special tokenshypothesis0.761Explanation for LMSYS superiority.
- Refusals.
- SAEs uncover safety-relevant representations that might be monitored or controlled.
- Figure 12 left and right show accuracy on harmful/ethical identification and 9-way classification.
- Including chat data in SAE training merges RM bias features with generic Assistant featureshypothesis0.739Explains poor features with three-dataset SAE.