claim
active
claim:using-the-marks-et-al-environment-as-both-dev-and-test-may-make-results-unrepresentativeUsing the Marks et al. environment as both dev and test may make results unrepresentative
Key limitation.
Source paper
extracted_from(2025) · Trenton Bricken · Rowan Wang · Sam Bowman
Neighborhood — ranked by edge-count
Papers (1)
paper
Findings (1)
finding
- Investigator agent wins the Marks et al. auditing game 13% of the time open-ended with all tools and datasetsassociated_withRealistic affordances result.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Proposal for assessment framework.
- Interpretability tools and semantic search are key to winning the Marks et al. auditing gameclaim0.732Affordance insight.
- Assumption D.3 formalized in the theoretical framework; empirically validated with coin-flip sequence experiments
- Acknowledged limitation.
- Normative claim about how to evaluate AI-generated content, using Deutsche Physik as cautionary analogy