Pith. sign in

REVIEW 3 major objections 5 minor 50 references

Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agents

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Coding agents pass rules they would obey anyway, and a new metric exposes the gap.

desk verdict A new benchmark axis—instruction surface placement—and a smart prior-control metric, but the headline gap rests on an LLM judge whose swap agreement is too weak to trust exact margins. read the letter →

arxiv 2608.11727 v1 pith:BS276OFM submitted 2026-08-12 cs.AI

Zining Huang , Haoran Que , Hong Zeng , Ge Zhang , Zuo Wang , Jin Chen , Haodong Wang , Zhongfei Hou
show 3 more authors
Changxin Pu Shen Yan Wenhao Huang
This is my paper · ORCID
classification cs.AI
keywords instructionfollowingcodingagentsbenchmarkevaluationsurfacesagainst-prioraccuracyLLMjudgeexecutionevidencerule-levelmetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

If a coding agent obeys a rule, the agent may well have been going to do that anyway; existing instruction-following benchmarks cannot tell compliance from coincidence. Harness-IF scores individual rules from execution evidence across the five surfaces a deployed agent reads: system prompt, tool description, skill description, project file, and user instruction. To separate genuine compliance from default behavior, the paper introduces Against-Prior Accuracy (AP-Acc), which scores only rules labeled as opposing what the agent does when the rule is withheld. Across 12 frontier models on 60 multi-turn coding items, every model scores lower on against-prior rules by 3.6 to 7.4 points, with a mean gap of 5.81 points, so aggregate accuracy overstates compliance by a model-specific margin. A separate counterbalanced pilot finds that when surfaces conflict, system prompts, project files, and user instructions tie ahead of tool and skill descriptions, so surface precedence does not follow prompt depth.

What carries the argument

The central object is Against-Prior Accuracy (AP-Acc), a metric that restricts scoring to rules labeled as opposing the agent's unprompted default, with defaults observed by zero-injection probes that rerun each task with the rule withheld across nine builds and curated prior annotations. It is carried by a rule-level verdict pipeline that produces one pass/fail/not-applicable judgment per applicable rule per run from traces, diffs, tests, and artifacts rather than a single task-level outcome, and by five configurable instruction surfaces plus a fixed harness default. AP-Acc is explicitly a behavioral stratification, not a training-provenance claim, and it is compared with accuracy over the same eligible verdicts so the gap is like-for-like.

What would settle it

Re-run the released 200-verdict subset through a second judge with the same three-vote protocol while retaining row-level verdicts, then recompute AP-Acc with the alternate labels; if the per-model accuracy-minus-AP-Acc gap becomes negative for several models, or if swapping the judge on the 116 paired clean verdicts removes the sign for a majority of the 12 models, the central prior-alignment claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that instruction-following evaluation must separate prompted compliance from unprompted default behavior. Across twelve frontier coding agents on 60 realistic multi-turn items, accuracy spans 72.1-85.9% while AP-Acc spans only 66.1-78.6%, and the accuracy-minus-AP-Acc gap is positive for every model, both on the full panel and under a common-support analysis with item-clustered intervals. Aggregate instruction-following scores therefore overstate compliance, and the overstatement is model-specific: it ranges two-fold across the cohort, leaves the top-ranked build unchanged, and swaps three adjacent rank pairs. The paper also claims, from its E0 conflict pilot, that pooled surface precedence is not explained by prompt depth, with system prompts, project files, and user instructions ahead of tool and skill descriptions.

Load-bearing premise

The load-bearing premise is that the GPT-5.2 three-vote LLM judge, which decides 86.8% of eligible verdicts, produces pass/fail labels accurate enough to preserve the sign and size of the accuracy-minus-AP-Acc gap; if judge errors correlate with rule content, the model-specific gap could be an artifact of the instrument.

Editorial extensions

If this is right

  • Aggregate instruction-following scores should be reported alongside a prior-controlled denominator such as AP-Acc, since the gap between them varies two-fold across models.
  • Leaderboard comparisons between adjacent models are not reliable; only the prior-alignment direction, broad difficulty patterns, and failure signatures carry weight.
  • Benchmark designers should deliberately inject rules that oppose known defaults if they want to measure instruction following rather than default-aligned behavior.
  • Because shortfall rules absorb 77.1% of failure mass, verifiers tuned to detect excess output address only about a fifth of the failures in realistic coding work.
  • The E0 surface ordering implies that critical constraints may be more reliably honored through system prompts, project files, and user instructions than through tool or skill descriptions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural generalization of the prior-control idea is to compute AP-Acc for non-coding agents by withholding each rule from a probe run; the paper's 40-case extension uses a different metric, so this transfer remains untested.
  • The low judge-swap agreement on the 200-verdict subset means absolute AP-Acc levels are instrument-specific, even though shared-instrument comparisons among models can still carry the sign of the gap.
  • If surfaces differ in effective authority as E0 suggests, future benchmark construction should counterbalance surface placement rather than assign rules to whichever surface is easiest to phrase.
  • The selection of items partly by observed discriminativeness implies the reported accuracy spread may be optimistic for unselected coding tasks, so AP-Acc should be re-estimated on a held-out item set.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. Harness-IF is a benchmark for rule-level instruction following in multi-turn coding agents. It curates a 642-rule library, instantiates 302 rules in 60 multi-turn coding items, and scores 256 rules from execution evidence across five configurable instruction surfaces (system prompt, tool description, skill description, project file, user instruction) plus a fixed harness default. The main methodological contribution is Against-Prior Accuracy (AP-Acc), which scores only rules labeled as opposing unprompted defaults, using zero-injection probe observations and curated prior annotations. Across 12 frontier models the paper reports that AP-Acc is 3.6–7.4 points (mean 5.81) below overall accuracy for every model, with the direction surviving a common-support analysis and item-clustered intervals. A separate counterbalanced conflict pilot (E0) reports a pooled surface precedence ordering with system prompts, project files, and user instructions ahead of tool and skill descriptions. The paper also reports failure decompositions by rule family, modality, and failure type, and an exploratory non-coding extension.

Significance. If the headline findings hold, Harness-IF fills a genuine gap: existing instruction-following benchmarks concentrate rules in the user turn and coding-agent benchmarks emphasize final task success, so neither separates compliance from coincidence. The AP-Acc prior control is a simple and transferable idea, and the benchmark is unusually reproducible: a single script regenerates every displayed number from the released 2,160-record verdict panel. The authors are also transparent about selection optimism, judge-swap sensitivity, prior-label overlap with the evaluated cohort, and the non-auditability of the human calibration study, and the E0 pilot is properly scoped as a pooled tendency rather than a universal hierarchy. These strengths are substantial. The unresolved judge-swap issue, however, puts the model-specific magnitude and rank-pair claims at risk, so the central quantitative result is not yet established at the precision claimed.

major comments (3)
  1. [§4.1, Table 2, Appendix E.1] The model-specific Δ values are not yet established as measurements of compliance rather than judge artifacts. Replacing the GPT-5.2 judge with Claude Opus 4.7 on the retained 200-row stratum gives 62.1% raw agreement and κ=0.163 on the 116 paired clean verdicts, with per-agent pass-rate deltas of −40 to +33 percentage points. Because 86.8% of eligible verdicts are produced by the GPT-5.2 judge, and because row-level swap verdicts are not retained, the alternate-judge AP-Acc and Acc−AP-Acc cannot be computed. The reported mean gap of 5.81 points is far smaller than the observed judge-swap pass-rate swings, so a judge bias correlated with against-prior content could plausibly produce or invert the obtained gaps. The deterministic-only subset in §4.5 (+13.09 points) is reassuring for the sign but is drawn from a different, pattern-heavy subpopulation and cannot validate the displayed per-model margins. I request an alternate-judge computation of the AP-Acc gap, or an analysis demonstrating that judge disagreements are independent of the against-prior label; absent that, the Δ column and the 'exchanges three adjacent rank pairs' statement should be relabeled as directional.
  2. [§3.4, Appendix E.4] The AP-Acc denominator is partly defined by the very models being scored. The paper states that a 5/9 zero-injection consensus necessarily includes at least one build whose identifier overlaps the evaluated panel, so no zero-injection label is independent of the scored cohort. The mitigation—that the seven evaluated models absent from the probe cohort still show positive gaps (+5.63 vs. +6.06) and that only 44.1% of against-prior verdicts carry zero-injection labels—is an appropriate disclosure but not a test. I request a robustness analysis that recomputes AP-Acc and the gap using only curated prior labels, or only zero-injection labels determined by non-overlapping probe builds, and reports whether the 3.6–7.4-point range and the per-model ordering survive. If they do not, the claim that AP-Acc controls for unprompted defaults is materially weakened.
  3. [Abstract, §4.1, §4.5] The paper claims that prior control 'leaves the top build unchanged and exchanges three adjacent rank pairs (2–3, 4–5, and 11–12)' while also stating that common-support item-clustered intervals leave every adjacent model comparison unresolved and that the displayed order is a point ranking. Given the test-retest cell-level range of 15.6 points (Appendix E.2) and the judge-swap variability documented in Appendix E.1, the rank-pair exchanges are not statistically supported and should be reported as point-estimate observations, not as a finding of the paper. This is especially important because the abstract and conclusion repeat the rank-exchange claim without the caveat that adjacent comparisons are unresolved.
minor comments (5)
  1. [Appendix D.4] The paragraph beginning 'The pooled ordering survives both direction-specific fits...' is duplicated verbatim; one copy should be removed.
  2. [Appendix D.4 and Appendix E.4] The paper states that four probe builds fall outside the evaluated panel's identifier set, but the D.4 build list appears to contain five such builds (Claude Opus 4.6, GPT 5.4, DeepSeek V3.2, Kimi K2.5, and Qwen 3.6 Plus) under the Table 3 identifiers; please reconcile the count.
  3. [§4.1 and Appendix E.3] Section 4.1 says 86.8% of rows involve the judge, while Table 7 in Appendix E.3 reports judge plus hybrid covering 86.7% of eligible verdicts; standardize this number.
  4. [Appendix E.1] The judge-swap stratum was balanced across family×modality×prior cells, but the paired 116-row subset may have lost that balance; report the prior-label distribution of the paired subset so readers can assess how directly the swap result applies to the AP-Acc denominator.
  5. [References] Reference [30] is the GPT-5 System Card; a specific citation for the GPT-5.2 judge model would be useful.

Circularity Check

1 steps flagged · score 4.0 of 10

AP-Acc's against-prior subset is partly defined by zero-injection behavior of the very builds being scored, so the model-specific gap is partially self-referential; the central direction survives for non-overlapping builds.

  1. self definitional [Appendix E.4 (Prior-label lineage and sensitivity); Sections 3.4 and 4.1.]
    "Because only four probe builds sit outside the evaluated panel’s identifier set, a 5/9 consensus necessarily includes at least one overlapping build, so no zero-injection label is independent of the scored cohort. ... of the 19,449 against-prior eligible verdicts, 8,574 (44.1%) carry a label sourced from the zero-injection consensus, against 9,051 (46.5%) from prior curation and 1,824 (9.4%) of unknown lineage."

    AP-Acc (Eq. 4) is defined on the against-prior set P, and P's labels are assigned using zero-injection probe behavior. The probe cohort necessarily includes evaluated builds, and 44.1% of the AP-Acc denominator carries those consensus labels. For an overlapping build, a rule enters P partly because that build (or a same-identifier build) tends not to do it without instruction; the Acc−AP-Acc gap then partially re-expresses the label-selection criterion rather than measuring a fresh compliance deficit. The paper's own absent-model bound (seven non-probe builds, mean gap +5.63) shows the direction is not wholly forced, so the circularity is partial and disclosed rather than total.

full rationale

The benchmark's core derivations are empirical, not analytical: Acc, AP-Acc, F-Acc, and DW-Acc are all recomputed from the released 2,160-record panel under explicit equations, so no displayed gap is an identity of those equations. I find no load-bearing self-citation chain, imported uniqueness theorem, or ansatz smuggled in via citation; references such as IHEval and the Seed model card are background or model documentation, not justifications of the prior-alignment claim. The only self-referential element is the AP-Acc prior-label construction: the against-prior denominator is partly derived from zero-injection probes whose cohort necessarily overlaps the evaluated models, and the paper itself discloses that no zero-injection label is independent of the scored cohort. For the five overlapping builds, this makes part of the Acc−AP-Acc gap a re-statement of the label-selection rule rather than an independent behavioral measurement. However, the central direction is not forced by this construction: seven evaluated builds absent from the probe cohort also show a positive gap (mean +5.63, range +3.62 to +7.19), the deterministic-only subset shows the same sign (+13.09 over 5,013 verdicts), and common-support item-clustered intervals remain positive model by model. The judge-swap instability (62.1% agreement, κ=0.163) is a serious validity threat to absolute levels and narrow rank margins, but it is a measurement-reliability concern, not circularity. F-Acc and DW-Acc are explicitly cohort-relative diagnostics and are not presented as independent predictions. Overall, the paper exhibits partial, disclosed self-referentiality in the prior-label source, but its principal claim retains independent empirical content, warranting a score of 4 rather than a higher score.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The central claims rest on three measurement assumptions: the six-surface taxonomy is complete and meaningful for deployed coding agents, rule-level pass/fail verdicts are faithful, and zero-injection probe behavior estimates unprompted defaults. No free parameters enter Acc or AP-Acc themselves; the listed parameters affect labeling and auxiliary diagnostics. No physical entities are invented; the instruction surface is a taxonomy, not a mechanism.

free parameters (3)
  • Zero-injection consensus threshold = 5 of 9 probe builds
    Determines which rules receive zero-injection prior labels. The sensitivity sweep across 4/9 to 7/9 shows the gap direction is stable, so the parameter is not load-bearing for the sign, but it is a hand-chosen threshold.
  • Judge aggregation protocol = GPT-5.2, temperature 0.3, three-vote majority
    Applied to 86.8% of eligible verdicts. Judge-swap agreement is low (kappa 0.163), so this choice is the dominant measurement uncertainty and is a hand-chosen configuration.
  • Discrimination weights d_r (DW-Acc) = Cohort-derived Pearson correlations, non-positive set to 0
    Used only for the descriptive DW-Acc column. Fitted to the 12-model cohort and not part of the central Acc or AP-Acc claim.
assumptions (5)
  • domain assumption The six instruction surfaces (HD, SP, TD, SD, PF, UI) are the relevant and distinct delivery channels for deployed coding agents.
    Section 3.1 defines the taxonomy; the benchmark design assumes this categorization is complete and meaningful for the claimed measurements.
  • domain assumption A rule can be judged pass, fail, or not applicable from execution evidence.
    Section 3.2 states the verifiability assumption that underlies all rule-level verdicts; the LLM judge handles rubric cases, which is the central uncertainty.
  • domain assumption Zero-injection behavior (rule withheld) estimates the unprompted default of a model.
    Section 3.4 and Appendix A define prior labels from zero-injection probe runs; AP-Acc depends on this behavioral interpretation.
  • domain assumption E0 synthetic conflict pairs isolate surface precedence.
    Appendix D.4 describes four synthetic conflict pairs across nine older builds; the precedence ranking assumes these conflicts are representative of surface conflicts.
  • standard math Item-clustered percentile bootstrap intervals are valid for the paired Acc-AP-Acc gap.
    Appendix E.5 uses a 2,000-resample bootstrap clustered by item; validity assumes item-level clustering captures the dependence structure of verdicts.
invented entities (1)
  • Instruction surface taxonomy (SP, TD, SD, PF, UI) independent evidence
    purpose: Organizing construct for where rules are delivered; the experimental variable of the benchmark.
    Surfaces are observable sections of an agent's prompt and can be located in deployment, so the taxonomy is externally identifiable; treating this exact five-surface set as complete is a design assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agents." pith.science (2026). https://pith.science/paper/BS276OFM

@misc{pith2026260811727,
  author       = {Pith},
  title        = {Pith review of: Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BS276OFM}},
  note         = {Machine review of arXiv:2608.11727}
}
read the original abstract

When a coding agent obeys a rule, it may simply have been going to do that anyway. Existing instruction-following benchmarks cannot tell the difference: they concentrate rules in the user turn, while coding-agent benchmarks emphasize final task success. We introduce Harness-IF, which scores operational rules one at a time from execution evidence: 60 realistic multi-turn coding items drawn from a 642-rule library, 256 rules receiving verdicts, placed on the five configurable surfaces a deployed agent reads. To separate compliance from coincidence we introduce Against-Prior Accuracy (AP-Acc), which scores only rules labeled as opposing unprompted defaults, observed by re-running tasks with the rule withheld across nine probe builds and curated otherwise. Across 12 frontier models, accuracy spans 72.1-85.9% and AP-Acc 66.1-78.6%; every model is worse on against-prior rules, by 3.6 to 7.4 points (mean 5.81), and the direction survives a common-support analysis with item-clustered intervals. Aggregate scores therefore overstate compliance by a model-specific margin: prior control leaves the top build unchanged and exchanges three adjacent rank pairs. A counterbalanced conflict pilot on nine separate builds adds a second result: pooled precedence does not follow prompt depth, with system prompts, project files, and user instructions ahead of tool and skill descriptions.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 30 canonical work pages

  1. [1]

    Claude code, 2024.https://claude.com/claude-code

    Anthropic. Claude code, 2024.https://claude.com/claude-code

    work page 2024

  2. [2]

    Models overview.https://platform.claude.com/docs/claude/docs/models-overview, 2026

    Anthropic. Models overview.https://platform.claude.com/docs/claude/docs/models-overview, 2026

    work page 2026

  3. [3]

    Building effective agents

    Anthropic, Erik Schluntz, and Barry Zhang. Building effective agents. Anthropic Research Blog, 2024

    work page 2024

  4. [4]

    Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan.τ2-Bench: Evaluating conversa- tional agents in a dual-control environment.arXiv preprint arXiv:2506.07982, 2025

    arXiv 2025

  5. [5]

    MLE-bench: Evaluating machine learning agents on machine learning engineering

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Mądry. MLE-bench: Evaluating machine learning agents on machine learning engineering. InProc. ICLR, 2025

    work page 2025

  6. [6]

    SWE-Bench Pro: Can AI agents solve long-horizon software engineering tasks?arXiv preprint arXiv:2509.16941, 2025

    Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. SWE-Bench Pro: Can AI agents solve long-ho...

    arXiv 2025

  7. [7]

    Datasheets for datasets.Communications of the ACM, 64(12):86–92, 2021

    Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. Datasheets for datasets.Communications of the ACM, 64(12):86–92, 2021

    2021

  8. [8]

    Gemini 3.1 Pro: Model card.https://deepmind.google/models/model-cards/ gemini-3-1-pro/, 2026

    Google DeepMind. Gemini 3.1 Pro: Model card.https://deepmind.google/models/model-cards/ gemini-3-1-pro/, 2026

    work page 2026

Show all 50 references
  1. [9]

    Mohammed Mehedi Hasan, Hao Li, Gopi Krishnan Rajbahadur, Bram Adams, and Ahmed E. Hassan. Model context protocol (MCP) tool descriptions are smelly! towards improving AI agent efficiency with augmented MCP tool descriptions.arXiv preprint arXiv:2602.14878, 2026

    2026 arXiv

  2. [10]

    ProcCtrlBench: Evaluating process-level defects and control preservation in LLM coding agents.arXiv preprint arXiv:2605.20251, 2026

    JiaweiHe, JieJia, ChenboLiu, ChaoyiXue, YapengSong, XikaiYang, andDongSun. ProcCtrlBench: Evaluating process-level defects and control preservation in LLM coding agents.arXiv preprint arXiv:2605.20251, 2026

    2026 arXiv

  3. [11]

    Multi-IF: Benchmarking LLMs on multi-turn and multilingual instructions following.arXiv preprint arXiv:2410.15553, 2024

    Yun He, Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam, Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, Shruti Bhosale, Chenguang Zhu, Karthik Abinav Sankararaman, Eryk Helenowski, Melanie Kam- badur, Aditya Tayade, Hao Ma, Han Fang, and Sinong Wang. Multi-IF: Benchm...

    arXiv

  4. [12]

    MLAgentBench: Evaluating language agents on machine learning experimentation

    Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MLAgentBench: Evaluating language agents on machine learning experimentation. InProc. ICML, 2024

    2024

  5. [13]

    FollowBench: A multi-level fine-grained constraints following benchmark for large language models

    Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. FollowBench: A multi-level fine-grained constraints following benchmark for large language models. In Proc. ACL, 2024

    2024

  6. [14]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? InProc. ICLR, 2024

    2024

  7. [15]

    AgentBench: Evaluating LLMs as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench:...

    2024

  8. [16]

    Merrill, Alexander G

    Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastro...

    2026 arXiv

  9. [17]

    GAIA: A benchmark for general AI assistants

    Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: A benchmark for general AI assistants. InProc. ICLR, 2024

    2024

  10. [18]

    MiniMax M2.7: Model self-improvement.https://www.minimax.io/models/text/m27, 2026

    MiniMax. MiniMax M2.7: Model self-improvement.https://www.minimax.io/models/text/m27, 2026

    2026

  11. [19]

    Kimi K2.6 model card.https://huggingface.co/moonshotai/Kimi-K2.6, 2026

    Moonshot AI. Kimi K2.6 model card.https://huggingface.co/moonshotai/Kimi-K2.6, 2026

    2026

  12. [20]

    GPT-5.5 model.https://developers.openai.com/api/docs/models/gpt-5.5/, 2026

    OpenAI. GPT-5.5 model.https://developers.openai.com/api/docs/models/gpt-5.5/, 2026

    2026

  13. [21]

    Introducing SWE-bench verified.https://openai.com/index/ introducing-swe-bench-verified/, 2024

    OpenAI Preparedness Team. Introducing SWE-bench verified.https://openai.com/index/ introducing-swe-bench-verified/, 2024

    2024

  14. [22]

    Patil, Tianjun Zhang, Xin Wang, and Joseph E

    Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive APIs. InProc. NeurIPS, 2024

    2024

  15. [23]

    Patil, Huanzhi Mao, Fanjia Yan, Charlie Ji, Vivek Suresh, Ion Stoica, and Joseph E

    Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Charlie Ji, Vivek Suresh, Ion Stoica, and Joseph E. Gonzalez. The Berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of LLMs. InProc. ICML, 2025

    2025

  16. [24]

    Generalizing verifiable instruction following

    Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lam- bert, and Hannaneh Hajishirzi. Generalizing verifiable instruction following. In Proc. NeurIPS Datasets and Benchmarks Track, 2025

    2025

  17. [25]

    AgentIF: Bench- marking instruction following of large language models in agentic scenarios

    Yunjia Qi, Hao Peng, Xiaozhi Wang, Amy Xin, Youfeng Liu, Bin Xu, Lei Hou, and Juanzi Li. AgentIF: Bench- marking instruction following of large language models in agentic scenarios. InProc. NeurIPS Datasets and Benchmarks Track, 2025

    2025

  18. [26]

    InfoBench: Evaluating instruction following ability in large language models.arXiv preprint arXiv:2401.03601, 2024

    Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. InfoBench: Evaluating instruction following ability in large language models.arXiv preprint arXiv:2401.03601, 2024

    2024 arXiv

  19. [27]

    Qwen3.6-Max-Preview released.https://qwen.ai/blog?id=qwen3.6-max-preview, 2026

    Qwen Team. Qwen3.6-Max-Preview released.https://qwen.ai/blog?id=qwen3.6-max-preview, 2026

    2026

  20. [28]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Can- cedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. InProc. NeurIPS, 2023

    2023

  21. [29]

    Seed2.0 model card.https://yfz.ai/Seed2.0_Model_Card.pdf, 2026

    Seed Team. Seed2.0 model card.https://yfz.ai/Seed2.0_Model_Card.pdf, 2026

    2026

  22. [30]

    OpenAI GPT-5 System Card.arXiv preprintarXiv:2601.03267, 2025

    Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. OpenAI GPT-5 System Card.arXiv preprintarXiv:2601.03267, 2025

    2025 arXiv

  23. [31]

    PaperBench: Evaluating AI’s ability to replicate AI research.arXiv preprint arXiv:2504.01848, 2025

    GiulioStarace, OliverJaffe, DaneSherburn, JamesAung, JunShernChan, LeonMaksin, RachelDias, EvanMays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. PaperBench: Evaluating AI’s ability to replicate AI research.arXiv preprint arXiv:250...

    2025 arXiv

  24. [32]

    Step 3.5 Flash: Open frontier-level intelligence with 11b active parameters

    StepFun. Step 3.5 Flash: Open frontier-level intelligence with 11b active parameters. arXiv preprint arXiv:2602.10604, 2026

    2026

  25. [33]

    Tencent unveils Hy3 preview.https://www.tencent.com/en-us/articles/2202320.html, 2026

    Tencent. Tencent unveils Hy3 preview.https://www.tencent.com/en-us/articles/2202320.html, 2026

    2026

  26. [34]

    AppWorld: A controllable world of apps and people for benchmarking interactive coding agents

    Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. AppWorld: A controllable world of apps and people for benchmarking interactive coding agents. InProc. ACL, 2024

    2024

  27. [35]

    The instruction hierarchy: Training LLMs to prioritize privileged instructions.arXiv preprint arXiv:2404.13208, 2024

    Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. The instruction hierarchy: Training LLMs to prioritize privileged instructions.arXiv preprint arXiv:2404.13208, 2024

    2024 arXiv

  28. [36]

    CodeIF-Bench: Evaluat- ing instruction-following capabilities of large language models in interactive code generation

    Peiding Wang, Li Zhang, Fang Liu, Lin Shi, Minxiao Li, Bo Shen, and An Fu. CodeIF-Bench: Evaluat- ing instruction-following capabilities of large language models in interactive code generation. arXiv preprint arXiv:2503.22688, 2025. 13

    2025

  29. [37]

    Benchmarking complex instruction- following with multiple constraints composition

    Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, Yiming Liu, Jie Tang, Hongning Wang, and Minlie Huang. Benchmarking complex instruction- following with multiple constraints composition. InProc. NeurIPS Datasets ...

    2024

  30. [38]

    LIFBench: Evaluating the instruction following performance and stability of large language models in long- context scenarios

    Xiaodong Wu, Minhao Wang, Yichen Liu, Xiaoming Shi, He Yan, Xiangju Lu, Junmin Zhu, and Wei Zhang. LIFBench: Evaluating the instruction following performance and stability of large language models in long- context scenarios. InProc. ACL, 2025

    2025

  31. [39]

    OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for op...

    2024 arXiv

  32. [40]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering.arXiv preprintarXiv:2405.15793, 2024

    2024 arXiv

  33. [41]

    Jimenez, Alex L

    John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muen- nighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, and Ofir Press. SWE-bench multimodal: Do AI systems generalize to visual software domains? InPro...

    2025

  34. [42]

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.τ-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024

    2024 arXiv

  35. [43]

    OpenSkillEval: Automatically auditing the open skill ecosystem for LLM agents.arXiv preprint arXiv:2605.23657, 2026

    Jiahao Ying, Boxian Ai, Wei Tang, Siyuan Liu, and Yixin Cao. OpenSkillEval: Automatically auditing the open skill ecosystem for LLM agents.arXiv preprint arXiv:2605.23657, 2026

    2026 arXiv

  36. [44]

    GLM-5.1 release notes.https://docs.z.ai/release-notes/new-released, 2026

    Z.ai. GLM-5.1 release notes.https://docs.z.ai/release-notes/new-released, 2026

    2026

  37. [45]

    CFBench: A comprehensive constraints- following benchmark for LLMs

    Tao Zhang, Chenglin Zhu, Yanjun Shen, Wenjing Luo, Yan Zhang, Hao Liang, Tao Zhang, Fan Yang, Mingan Lin, Yujing Qiao, Weipeng Chen, Bin Cui, Wentao Zhang, and Zenan Zhou. CFBench: A comprehensive constraints- following benchmark for LLMs. InProc. ACL, 2025

    2025

  38. [46]

    IHEval: Evaluating language models on following the instruction hierarchy

    ZhihanZhang, ShiyangLi, ZixuanZhang, XinLiu, HaomingJiang, XianfengTang, YifanGao, ZhengLi, Haodong Wang, Zhaoxuan Tan, Yichuan Li, Qingyu Yin, Bing Yin, and Meng Jiang. IHEval: Evaluating language models on following the instruction hierarchy. InProc. NAACL, 2025

    2025

  39. [47]

    SpecBench: Measuring reward hacking in long-horizon coding agents.arXiv preprint arXiv:2605.21384, 2026

    Bingchen Zhao, Dhruv Srikanth, Yuxiang Wu, and Zhengyao Jiang. SpecBench: Measuring reward hacking in long-horizon coding agents.arXiv preprint arXiv:2605.21384, 2026

    2026 arXiv

  40. [48]

    Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023

    2023 arXiv

  41. [49]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. InProc. ICLR, 2024

    2024

  42. [50]

    Keep generated summaries compact,

    Tao Zou, Xinghua Zhang, Haiyang Yu, Minzheng Wang, Fei Huang, and Yongbin Li. EIFBench: Extremely complex instruction following benchmark for large language models. InProc. EMNLP, 2025. 14 Appendix A Design axis definitions Each of the 642 atomic constraints in theHarness-IFli...

    2025

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.