method
active
method:harness-following-rate-measurementHarness-Following Rate Measurement
LLM-judge pipeline measuring fraction of skill-loaded trajectories where agent follows loaded skill guidance, using Claude Sonnet 4.6 as judge
Neighborhood — ranked by edge-count
Papers (1)
paper
Concepts (1)
concept
- Harness-Following RateimplementsThe fraction of skill-loaded trajectories judged by an LLM judge as following the loaded skill's guidance
Methods (1)
method
- First stage of HFR pipeline that converts skill body into a strict JSON rubric of atomic procedural instructions for adherence judging
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- A failure mode where even when harness artifacts are loaded, weak-tier models fail to follow their guidance faithfully
- Named metric measuring the fraction of trajectories in which a model actively loads at least one skill into its context
- The external non-parametric context and infrastructure (prompts, skills, memories, tools) through which an LLM is deployed for task execution
- Natural-language harness artifacts that encode standing behavioral rules, task policies, and reasoning procedures
- Primary metric: percentage of responses containing multiple attempts that successfully improve on the first attempt
- The capability of a task-solving agent to benefit from updated harnesses during task solving
- The capability of an evolver model to produce useful persistent harness updates from execution evidence
- Metric measuring harness-updating capability as the mean pairwise gain across an anchor agent set