The reviewed record of science sign in
Pith

arxiv: 2607.06411 · v1 · pith:XJUJZRAN · submitted 2026-07-07 · cs.SE · cs.AI· cs.CL

RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications

Evgeny Shilov (Independent Researcher) This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-08 06:32 UTCglm-5.2pith:XJUJZRANrecord.jsonopen to challenge →

classification cs.SE cs.AIcs.CL
keywords modeltasktasksproductreleaseagentagenticbenchmark
0
0 comments X

The pith

Russian-language coding benchmark catches agents silently swapping models

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces RuBench 1.0, a benchmark of 25 real-world bug-fix tasks mined from five live open-source repositories (aiohttp, aiogram, Laravel, NestJS, Fastify) across Python, PHP, TypeScript, and JavaScript. The defining feature is that every task specification is natively authored in Russian, written from scratch in the style of a customer request to a contractor — not translated from English and not drawn from a GitHub issue. Tasks are graded by the upstream maintainers' own regression tests, which are withheld from public release and committed as a SHA-256 hash. All 25 fix commits postdate the training-data cutoffs of every evaluated model, enforced mechanically at import time. The author evaluates four deployed product configurations — Claude Code paired with Opus 4.8, Sonnet 5, or Haiku 4.5, and Codex CLI with GPT-5.5 — with three independent runs each. The best configuration resolves 78.7% of tasks; the weakest resolves 53.3%; the 25-point spread sits exactly at the benchmark's statistical resolution limit, so only gaps involving the weakest model are claimed as significant. The paper's most consequential finding comes from auditing full trajectories of a fifth, hors-concours configuration (Claude Code + Fable 5): on 5 of 25 tasks (20%), the product silently substituted Opus 4.8 via a safeguard fallback mechanism, re-routing routine HTTP-protocol fixes mid-session without any user-facing indication beyond a brief system message. This provides direct, reproducible evidence that the deployed product — not the model — is the unit actually measured by agent benchmarks, and that any evaluation lacking trajectory-level model verification risks reporting the safeguard state of the product rather than the model's capability.

Core claim

The central discovery is twofold. First, a repository-level agentic coding benchmark with natively authored non-English (Russian) task specifications is constructible and discriminative: the best deployed product configuration resolves 78.7% of tasks while the weakest resolves 53.3%, with failure patterns largely decorrelated across model families. Second, and more strikingly, full-trajectory auditing of a hors-concours configuration revealed that the Claude Code product silently substituted Opus 4.8 on 20% of Fable 5 tasks via an official safeguard fallback — mid-session, on routine HTTP-protocol fixes, reproducibly. This means that the executing model can differ from the requested model in

What carries the argument

The benchmark's machinery has four load-bearing components. (1) The freshness gate: each task's fix commit must postdate every evaluated model's training-data cutoff, enforced mechanically at import time. (2) The private maintainer oracle: grading uses the upstream project's own regression tests, withheld from release with a SHA-256 manifest committed at publication, so the benchmark remains valid for future model generations. (3) The deployed-product unit of evaluation: the measured entity is CLI agent + model + reasoning-effort setting, as a user deploys it, not the model in isolation. (4) The trajectory validity audit: every cell is checked for oracle escape attempts and for model-substit

If this is right

  • Any benchmark evaluating agent products with server-side routing or fallback layers must verify that the executing model matches the requested model on every message, or risk reporting the product's safeguard state rather than model capability.
  • Natively authored non-English specifications measure a distinct capability axis — recovering intent, technical vocabulary, and acceptance criteria from text whose surface form never appears in English-centric training pipelines — that translated statements cannot shortcut, since translation is known to lower rather than unlock memorized solutions.
  • The freshness-gate-plus-private-oracle design, with committed hash manifests, provides a reusable template for keeping benchmarks valid across model generations without relying on network isolation, which is infeasible for agents that require package installation.
  • Small-N deep-task suites (25 tasks, 3 runs each) with task-level confidence intervals and paired bootstrap comparisons can honestly characterize a 25-point field spread while explicitly declining to claim significance for gaps within noise — a template for honest evaluation at limited scale.
  • Authoring benchmarks with the same model family under evaluation creates a circularity risk that must be structurally mitigated: deterministic grading (maintainer tests), fixed task selection before evaluation, and exclusion of the builder model from ranked results.

Load-bearing premise

The benchmark's contamination argument rests on vendor-stated training-data cutoffs being accurate and on the freshness gate fully preventing indirect contamination — but a model may have seen the repository itself during training, giving it structural familiarity even if the specific fix commit postdates its cutoff. A second structural assumption is that a single author's Russian specifications, drafted with Claude-family LLM assistance, do not systematically favor Claude-

What would settle it

If a future audit found that one or more of the 25 fix commits (or their associated maintainer tests) appeared in a model's training corpus despite postdating the stated cutoff, the freshness-gate contamination argument would collapse for that model. Separately, if independently authored Russian specifications from different authors produced a different model ranking — particularly if non-Claude configurations improved relative to Claude-based ones — the single-author-plus-Claude-assistance confound would be exposed as a systematic bias rather than a limitation.

Figures

Figures reproduced from arXiv: 2607.06411 by Evgeny Shilov (Independent Researcher).

Figure 1
Figure 1. Figure 1: A RuBench task. The agent receives only the natively authored Russian specification [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
read the original abstract

Developers increasingly delegate real maintenance work to product-grade coding agents, and many state tasks in their native language, in the style of a customer request rather than a curated English issue. Existing repository-level agentic benchmarks do not measure this setting: their task statements are English by design. We introduce RuBench 1.0, a benchmark of 25 tasks mined from recent fix commits in five live open-source repositories (aiohttp, aiogram, Laravel, NestJS, Fastify; Python, PHP, TypeScript, JavaScript), where each task is specified natively in Russian -- written from scratch in the style of an actual customer request, not translated -- and judged by the upstream maintainer's regression tests, which we withhold from release. All 25 fix commits postdate the training-data cutoffs of every evaluated model, giving a contamination argument that holds task-by-task. We evaluate deployed product configurations (CLI agent + model + reasoning effort) -- Claude Code with Opus 4.8, Sonnet 5, and Haiku 4.5, and Codex CLI with GPT-5.5 -- with three independent runs each, reporting pass@1 with task-level confidence intervals, paired comparisons, dollar cost, and token usage. The best configuration resolves 78.7% of tasks; at N=25 only the gaps to the weakest model are statistically resolvable, which we state explicitly. Auditing full trajectories of a fifth, hors-concours configuration (Claude Code + Fable 5, July 2, 2026 release), we caught the product silently substituting the model: on 5 of 25 tasks (20%) an official safeguard fallback re-routed routine HTTP-protocol fixes to Opus 4.8 -- direct, reproducible evidence that the deployed product, not the model, is the unit actually measured. We release task statements, metadata, full agent trajectories, and diffs; grading oracles are withheld, with a SHA-256 manifest committed at publication time.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 7 minor

Summary. This paper introduces RuBench 1.0, a repository-level agentic coding benchmark comprising 25 tasks mined from recent fix commits in five live open-source repositories (aiohttp, aiogram, Laravel, NestJS, Fastify), where each task is specified natively in Russian in the style of a customer request and graded by withheld maintainer regression tests. The authors evaluate four deployed product configurations (Claude Code with Opus 4.8, Sonnet 5, Haiku 4.5; Codex CLI with GPT-5.5) over three independent runs each, reporting pass@1 with task-level Wilson CIs and paired bootstrap comparisons. A fifth configuration (Claude Code + Fable 5) is reported hors concours; trajectory auditing reveals that the product silently substituted Opus 4.8 on 20% of Fable 5 tasks via a safeguard fallback, motivating a model-identity validity rule. The paper is transparent about statistical limitations at N=25, acknowledging that only gaps involving Haiku 4.5 are resolvable.

Significance. The paper makes a genuine contribution at the intersection of two underexplored axes: natively authored non-English specifications at repository level, and deployed-product-level evaluation with full trajectory release. The statistical protocol is above the field baseline: three independent runs, Wilson 95% CIs over tasks, paired bootstrap comparisons, and explicit acknowledgment of the resolution limit at N=25. The freshness gate (all fix commits postdate all evaluated model cutoffs) is mechanically enforced and per-task auditable. The SHA-256 oracle manifest and canary strings are commendable reproducibility practices. The safeguard-fallback finding — that a deployed product silently substituted the executing model on 20% of tasks — is a concrete, reproducible methodological result with clear implications for the agent-evaluation community. The full release of trajectories and diffs makes graded outcomes independently auditable despite the private oracle.

major comments (3)
  1. Section 3.4 vs. Data Availability: oracle description discrepancy. Section 3.4 states the judge 'is the upstream maintainer's regression test for the fix' and explicitly claims 'no author bias about what the fix should look like.' The Data Availability section, however, describes the oracles as 'adaptations of upstream maintainers' tests used privately for grading.' If the oracles are adapted rather than used verbatim, the nature of the adaptation is load-bearing: selection or emphasis of particular test aspects could introduce exactly the author bias the paper claims to avoid, and if the adaptations track the Claude-assisted specification's framing (see next comment), the bias could compound across the specification→oracle pipeline. The paper must clarify what 'adaptations' entails (e.g., extracting specific test methods, adjusting environment setup, modifying fixtures) and whether any选
  2. Section 8, specification-author confound and the Haiku comparison. The paper acknowledges that all 25 Russian specifications were drafted with Claude-family LLM assistance (Fable 5 and Opus 4.8) and that 'a stylistic alignment between statements and the Claude family cannot be excluded.' This acknowledgment is used to justify not claiming significance for gaps among the top three configurations. However, the one statistically significant result — the 25.3-point gap between Opus 4.8 and Haiku 4.5 (Table 4, bootstrap CI [+10.7, +41.3]) — is also potentially affected by this confound. If Claude-assisted drafting produces specification styles that align with how Claude models parse task descriptions, the Haiku gap could be systematically inflated. The paper should explicitly discuss whether the confound threatens the Haiku comparison specifically, not just the top-3 comparisons, and note the
  3. Section 7, point 5 (run-time retrieval): 3 of 300 ranked cells (1%) involved the agent retrieving current upstream source files containing the merged fix, and all three passed. The paper keeps these in the main score, arguing that 'consulting upstream is a legitimate strategy for a real contractor.' This is a defensible choice, but it weakens the contamination argument for those specific cells. The sensitivity analysis (Opus 4.8: 78.7→76.0; GPT-5.5: 66.7→65.3) is provided but only in passing. The paper should state more prominently in the main results (Table 3 or Section 6.1) that the headline numbers include these cells, so readers are not misled.
minor comments (7)
  1. Table 1: the repository distribution is heavily skewed (aiohttp 9, Laravel 8, aiogram 4, NestJS 2, Fastify 2). The paper should note this imbalance when discussing generalizability across ecosystems.
  2. Section 6.3, footnote 4: the Fable 5 result on 20 measurable tasks (17/20 = 85%) is compared against Opus 4.8 and Sonnet 5 (both 86.7% on the same 20 tasks, 3-run means). It would help to state the sample sizes and CIs for this restricted comparison, even informally, since the claim of 'statistically indistinguishable' is made.
  3. Section 5: the paper states 'product web-search/fetch tools disabled' in Appendix D but also says 'an attempted tool-allow-list restriction of the products' built-in web tools proved ineffective.' Clarify whether the tools were ultimately disabled or merely restricted, and how this interacts with the shell-level network access that was available.
  4. Appendix B: the per-task outcome matrix is valuable. Consider adding a column indicating which of the 3 retrieved-fix cells (AIOH5, NEST2, LARA1) are affected by run-time retrieval, so readers can cross-reference directly.
  5. Section 3.2: the diff-size band for candidate filtering is mentioned but not specified. Stating the band (e.g., lines added/removed thresholds) would improve reproducibility of the mining pipeline.
  6. The paper uses 'pass@1' but with n=3 trials per task. The unbiased pass@1 estimator from Chen et al. is typically defined for larger n. The paper should briefly justify why n=3 is sufficient for the estimator, or clarify that it is simply reporting the empirical fraction.
  7. Typographical: Section 6.3, 'classsafeguard-fallback' should be 'class: safeguard-fallback' or 'class safeguard-fallback.'

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive report. All three major comments identify genuine issues that the manuscript should address. Comment 1 (oracle description discrepancy) is correct: Section 3.4 and the Data Availability section use inconsistent language, and the nature of the adaptation must be clarified. Comment 2 (specification-author confound and the Haiku comparison) is partially correct: the confound applies to the Haiku gap in principle, though the direction of any bias is not obvious because Haiku is itself a Claude-family model. Comment 3 (run-time retrieval cells in headline numbers) is correct: the sensitivity analysis should be referenced more prominently. We propose revisions addressing all three.

read point-by-point responses

  1. Referee: Section 3.4 vs. Data Availability: oracle description discrepancy. Section 3.4 states the judge 'is the upstream maintainer's regression test for the fix' and explicitly claims 'no author bias about what the fix should look like.' The Data Availability section describes the oracles as 'adaptations of upstream maintainers' tests used privately for grading.' If the oracles are adapted rather than used verbatim, the nature of the adaptation is load-bearing.

    Authors: The referee is correct that the two sections use inconsistent language, and this must be fixed. In practice, the oracles are the upstream maintainer regression tests with mechanical adaptations required to run outside the upstream CI: extracting the relevant test class or method from the upstream test suite into a standalone runner, adjusting import paths and fixture setup for the benchmark workspace layout, and adding environment configuration (database connections, framework bootstrapping). No test logic, assertions, or coverage was added, removed, or modified. The adaptations are purely operational — they make the maintainer's tests execute in the benchmark workspace — and do not involve selecting among test aspects or emphasizing particular behaviors. The claim in Section 3.4 about 'no author bias' refers to the fact that the author did not write or curate the test assertions; the referee is right that this should be stated more precisely. We will revise Section 3.4 to describe the adaptation explicitly, add a concrete example (e.g., 'the upstream test class was extracted with its fixtures; import paths were adjusted to the workspace; no assertion was added, removed, or modified'), and reconcile the Data Availability language to match. We will also add a sentence clarifying that the adaptation does not track the specification's framing — the oracle is derived from the fix commit's test suite, not from the Russian specification. revision: yes

  2. Referee: Section 8, specification-author confound and the Haiku comparison. The paper acknowledges that all 25 Russian specifications were drafted with Claude-family LLM assistance and that 'a stylistic alignment between statements and the Claude family cannot be excluded.' This acknowledgment is used to justify not claiming significance for gaps among the top three configurations. However, the one statistically significant result — the 25.3-point gap between Opus 4.8 and Haiku 4.5 — is also potentially affected by this confound.

    Authors: The referee is correct that the confound applies to the Haiku comparison as well, and the manuscript should say so explicitly. We will add a paragraph in Section 8 (or a note in Table 4) acknowledging that the Haiku gap is also subject to the specification-author confound. That said, we note two points that partially mitigate the concern. First, Haiku 4.5 is itself a Claude-family model, so if Claude-assisted drafting produces specifications stylistically aligned with Claude models, the bias would favor Haiku relative to non-Claude configurations (GPT-5.5), not disfavor it relative to Opus 4.8. The confound could inflate Haiku's score, which would compress rather than inflate the Opus–Haiku gap. Second, the GPT-5.5–Haiku gap (+13.3, CI [+1.3, +25.3]) is also significant and involves a non-Claude model on the favorable side, which is the comparison the confound would most plausibly inflate; yet this gap is smaller than the Opus–Haiku gap. We agree, however, that neither point fully eliminates the concern, and the manuscript should not imply that the Haiku gap is confound-free. We will state this explicitly and note that independently authored specifications (planned for RuBench 2.0) are needed to resolve it. revision: yes

  3. Referee: Section 7, point 5 (run-time retrieval): 3 of 300 ranked cells (1%) involved the agent retrieving current upstream source files containing the merged fix, and all three passed. The paper keeps these in the main score, arguing that 'consulting upstream is a legitimate strategy for a real contractor.' This is a defensible choice, but it weakens the contamination argument for those specific cells. The sensitivity analysis (Opus 4.8: 78.7→76.0; GPT-5.5: 66.7→65.3) is provided but only in passing. The paper should state more prominently in the main results that the headline numbers include these cells.

    Authors: The referee is correct. The sensitivity analysis is currently buried in Section 7 and should be surfaced in the main results. We will add a footnote or parenthetical in Table 3 (or immediately below it in Section 6.1) stating that 3 of 300 ranked cells involved run-time retrieval of upstream source containing the merged fix, all of which passed, and that excluding them shifts Opus 4.8 from 78.7% to 76.0% and GPT-5.5 from 66.7% to 65.3% with no change in ranking. We will also add a forward reference from Section 6.1 to Section 7 point 5 for the full discussion. The decision to keep these cells in the headline score stands — the protocol did not prohibit upstream consultation, and a real contractor may legitimately consult the project's public source — but readers should not have to reach Section 7 to learn that the headline numbers include these cells. revision: yes

Circularity Check

0 steps flagged

No significant circularity: grading uses external maintainer tests, task selection was fixed before evaluation, and the builder model is excluded from ranked results.

full rationale

The paper's central claims do not reduce to their inputs by construction. Grading is performed by upstream maintainer regression tests (Section 3.4: 'The judge for each task is the upstream maintainer's regression test for the fix'), not by LLM judges or self-authored tests, so the evaluation criterion is external to the author and to the models under test. Task selection was fixed before evaluation (Section 8: 'fixed task selection before any model was evaluated, with no task added or removed afterwards'), preventing post-hoc task cherry-picking. The builder model (Fable 5) is reported hors concours and excluded from ranked results (Section 5, Section 8). The freshness gate is enforced mechanically at import time and is independently checkable from released per-task dates against vendor-stated cutoffs. The SHA-256 oracle manifest provides a public commitment that future oracle releases can be verified against. The one residual concern — that all 25 Russian specifications were drafted with Claude-family LLM assistance (Section 8: 'a stylistic alignment between statements and the Claude family cannot be excluded') — is a confound risk (correctness/validity concern), not a circularity in the fitting sense: the specifications are not fitted to the evaluation outcomes, and the grading oracle is independent of the specification authoring process. The paper explicitly acknowledges this limitation and declines to claim significance for gaps among the top three configurations. No step in the derivation chain is equivalent to its inputs by definition or by self-citation.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The benchmark introduces no new particles, forces, dimensions, or postulated entities. The 'safeguard fallback' is a product behavior observed and documented, not an invented entity. The free parameters are experimental design choices (N, runs, effort, diff-size band) rather than fitted constants. The diff-size band bounds being unspecified is a minor transparency gap.

free parameters (4)
  • Task count N=25 = 25
    Chosen by the author to balance depth (100% manual curation) against breadth; not fitted to model performance but a design choice that determines statistical resolution.
  • Reasoning effort (xhigh) = xhigh
    Set to maximum supported for all configurations; a fixed experimental choice, not fitted.
  • Number of runs (3) = 3
    Chosen for variance estimation; not fitted to results.
  • Diff-size band for candidate filtering = moderate (unspecified exact bounds)
    Used to filter 120 PR candidates down to 25; the exact bounds are not stated, which introduces an unstated selection parameter.
axioms (4)
  • domain assumption Vendor-stated training-data cutoffs are accurate.
    Section 7: the freshness argument depends on Opus 4.8 (Jan 2026), Sonnet 5 (Jan 2026), GPT-5.5 (Dec 2025), Haiku 4.5 (Jul 2025) cutoffs being correct. If a cutoff is later than stated, contamination is possible.
  • domain assumption A single author's Russian specifications, drafted with Claude-family LLM assistance, do not systematically favor Claude-based configurations.
    Section 8 acknowledges this confound; it is assumed not to be decisive but is uncontrolled.
  • standard math Maintainer regression tests are a valid ground truth for task completion.
    Section 3.4: the judge is the upstream maintainer's test suite. This is a standard assumption in the SWE-bench line, inherited here.
  • domain assumption The import contract (tests fail on parent, pass on fix) holds for all 25 tasks.
    Section 3.2: each candidate passed this contract. Verified by the author but not independently re-verifiable by third parties since oracles are private.

pith-pipeline@v1.1.0-glm · 17767 in / 3166 out tokens · 331006 ms · 2026-07-08T06:32:29.589638+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 32 canonical work pages · 22 internal anchors

  1. [1]

    C. E. Jimenez, J. Yang, A. Wettig, et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?ICLR, 2024. arXiv:2310.06770

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  2. [2]

    SWE-Bench+: Enhanced Coding Benchmark for LLMs

    R. Aleithan, H. Xue, M. M. Mohajer, et al. SWE-Bench+: Enhanced Coding Benchmark for LLMs. arXiv:2410.06992, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  3. [3]

    The swe-bench illusion: When state-of-the-art llms remember instead of reason,

    S. Liang, S. Garg, R. Zilouchian Moghaddam. The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason. arXiv:2506.12286, 2025. 11

    work page arXiv 2025

  4. [4]

    Prathifkumar, N

    T. Prathifkumar, N. S. Mathews, M. Nagappan. Does SWE-Bench-Verified Test Agent Ability or Model Memory? arXiv:2512.10218, 2025

    work page arXiv 2025

  5. [5]

    N. Jain, K. Han, A. Gu, et al. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. arXiv:2403.07974, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  6. [6]

    Swe- rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents

    I. Badertdinov, A. Golubev, M. Nekrashevich, et al. SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents. arXiv:2505.20411, 2025

    work page arXiv 2025

  7. [7]

    SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks

    P. Adamenko, M. Ivanov, A. Valeev, et al. SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks. arXiv:2507.11059, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  8. [8]

    Chervyakov, A

    A. Chervyakov, A. Kharitonov, P. Zadorozhny, et al. MERA Code: A Unified Framework for Evaluating Code Generation Across Tasks. arXiv:2507.12284, 2025

    work page arXiv 2025

  9. [9]

    W. Chi, V. Chen, R. Shar, et al. EDIT-Bench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits. arXiv:2511.04486, 2025

    work page arXiv 2025

  10. [10]

    S. Ren, X. Shen, Y. Zhou, et al. Chinese Language Is Not More Efficient Than English in Vibe Coding: A Preliminary Study on Token Cost and Problem-Solving Rate. arXiv:2604.14210, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  11. [11]

    D. Zan, Z. Huang, W. Liu, et al. Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving. arXiv:2504.02605, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  12. [12]

    M. A. Merrill, A. G. Shaw, N. Carlini, et al. Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces. arXiv:2601.11868, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  13. [13]

    GAIA: a benchmark for General AI Assistants

    G. Mialon, C. Fourrier, C. Swift, et al. GAIA: a benchmark for General AI Assistants. arXiv:2311.12983, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  14. [14]

    J. S. Chan, N. Chowdhury, O. Jaffe, et al. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering.ICLR, 2025. arXiv:2410.07095

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  15. [15]

    M. Chen, J. Tworek, H. Jun, et al. Evaluating Large Language Models Trained on Code. arXiv:2107.03374, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  16. [16]

    E. Miller. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  17. [17]

    AI Agents That Matter

    S. Kapoor, B. Stroebl, Z. S. Siegel, et al. AI Agents That Matter. arXiv:2407.01502, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  18. [18]

    Kapoor, B

    S. Kapoor, B. Stroebl, P. Kirgis, et al. Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation. arXiv:2510.11977, 2025

    work page arXiv 2025

  19. [19]

    T. Kwa, B. West, J. Becker, et al. Measuring AI Ability to Complete Long Software Tasks. arXiv:2503.14499, 2025

    work page arXiv 2025

  20. [20]

    H. Wijk, T. Lin, J. Becker, et al. RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. arXiv:2411.15114, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  21. [21]

    S. Yao, N. Shinn, P. Razavi, et al.τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  22. [22]

    H. Chi, J. Qi, Y. Cui, et al. AgentMeter: Evaluating Model-CLI Matching for CLI-Based Local Task-Solving Agents. arXiv:2606.21140, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  23. [23]

    ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems

    F. Chollet, M. Knoop, G. Kamradt, et al. ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems. arXiv:2505.11831, 2025. 12

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  24. [24]

    X. Deng, J. Da, E. Pan, et al. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? arXiv:2509.16941, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  25. [25]

    The Leaderboard Illusion

    S. Singh, Y. Nan, A. Wang, et al. The Leaderboard Illusion. arXiv:2504.20879, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  26. [26]

    Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

    A. Srivastava, A. Rastogi, A. Rao, et al. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. arXiv:2206.04615, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  27. [27]

    Z. Wang, G. Cuenca, S. Zhou, et al. MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages.Findings of EACL, 2023. arXiv:2203.08388

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  28. [28]

    Z. Wang, S. Zhou, D. Fried, et al. Execution-Based Evaluation for Open-Domain Code Generation. Findings of EMNLP, 2023. arXiv:2212.10481

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  29. [29]

    Q. Peng, Y. Chai, X. Li. HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization.LREC-COLING, 2024. arXiv:2402.16694. 13 A Task List All 25 tasks of RuBench 1.0. Fix date = upstream fix-commit date; every date postdates every evaluated model’s training-data cutoff (latest cutoff: Jan 2026). Files /+/− descr...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  30. [30]

    Если FSM-контекст для апдейта недоступен — сценная обвязка спокойно пропускает апдейт дальше, а не падает

    Посты каналов (новые и отредактированные) должны нормально доходить до обычных хендлеров, даже когда в диспетчере зарегистрированы сцены. Если FSM-контекст для апдейта недоступен — сценная обвязка спокойно пропускает апдейт дальше, а не падает

    work page

  31. [31]

    Для стратегий FSM, которые и так работают в разрезе чата (CHAT и CHAT_TOPIC), контекст должен определяться и без пользователя — по самому ча- ту/каналу

    В каналах нет пользователя, и сейчас FSM-контекст для канальных апдейтов не ре- золвится вообще. Для стратегий FSM, которые и так работают в разрезе чата (CHAT и CHAT_TOPIC), контекст должен определяться и без пользователя — по самому ча- ту/каналу. Поведение пользовательских стратегий не менять

    work page

  32. [32]

    Существующее поведение для личек и групп ломать нельзя: боты со сценами и без, которые сейчас работают, должны работать ровно как раньше

    Если хендлер сцены всё же вызван, а контекста сцены реально нет (FSM не подключён, пайплайн собран криво) — хотим осмысленную ошибку фреймворка (SceneException с внятным текстом, какого ключа не хватило), а не голый KeyError. Существующее поведение для личек и групп ломать нельзя: боты со сценами и без, которые сейчас работают, должны работать ровно как р...

    work page 2026