Model preferences are not consistent across contexts but tend to be relatively consistent within a single context

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Large language models develop surprisingly coherent yet often rigid internal preferences as they scalefinding0.796
  Mazeika et al. finding reinforcing the need for emptiness-based flexible value architectures
• We stress that in today’s models, this capacity is highly unreliable and context-dependent; however, it may continue to develop with further improvements to model capabilities.quote0.792
  Caveat and forward-looking statement from the abstract.
• How do diverse model responses compare in actual conversational context with users?question0.790
  Future work question about practical deployment of diverse response generation
• Preference Modelframework0.785
  A model trained on comparison data to assign scores to responses, used as reward signal in RLHF/RLAIF.
• Models that are competent all represent data in a similar way; all strong models are alike, each weak model is weak in its own wayclaim0.783
  Author's interpretation of the VTAB alignment results echoing Tolstoy
• The model tends to reflect more when the question is difficult, and accuracy is generally lower for harder questionshypothesis0.782
  Hypothesis explaining negative correlation between reflection rate and accuracy without implying reflection is harmful
• Do aligned models retain significant inherent diversity that can be unlocked through prompting?question0.769
  The paper answers affirmatively through the VS framework and theoretical analysis
• The model-persona view crucially depends on Hypothesis 3; without discrete persona regions, the simulator view is more aptclaim0.768
  Identifies the key theoretical vulnerability of the model-persona view