Pith. sign in

REVIEW 2 major objections 6 minor 1 cited by

Agent psychometrics: Task-level performance prediction in agentic coding benchmarks

T0 review · 2 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Agentic coding success can be predicted task-by-task from additive LLM and scaffold abilities.

desk verdict Solid workshop-grade paper: the additive LLM+scaffold ability decomposition is the genuinely new piece, held-out-agent results back it up, but missing error bars, nested model selection, and an overstated abstract keep it from being more than that. read the letter →

arxiv 2604.00594 v2 pith:EPDCRWPQ submitted 2026-04-01 cs.AI

Chris Ge , Daria Kryvosheieva , Daniel Fried , Uzay Girit , Kaivalya Hariharan This is my paper
classification cs.AI
keywords agenticcodingbenchmarksitemresponsetheorytask-levelsuccesspredictionLLM-scaffoldabilitydecompositionmulti-benchmarkIRTtaskdifficultyadaptiveselectionbenchmarkcalibration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Agentic coding benchmarks are usually summarized by a single pass rate, which hides which tasks are hard and why. This paper argues that task-level success can be predicted from static properties of the task plus a simple additive decomposition of the agent: the probability an agent solves a task is a logistic function of the sum of its LLM's ability and its scaffold's ability, minus the task's difficulty. Task difficulty is itself estimated from accessible artifacts—problem statement, repository state, test cases, and reference solution—so new tasks can be calibrated without running expensive agents. Because individual LLMs and scaffolds recur across leaderboards even when full agents do not, the additive parameterization lets evaluations from different benchmarks be pooled and extrapolated. If the model is right, benchmark construction and agent evaluation become cheap, targeted prediction rather than costly sampling.

What carries the argument

The central object is the augmented IRT model P(y=1) = σ(θ_LLM + θ_scaffold − β_task). θ_LLM and θ_scaffold are per-LLM and per-scaffold latent ability parameters; their additive combination is what allows heterogeneous leaderboards to be stitched together when full agents rarely overlap. β_task is a task difficulty that is predicted by a linear model from static features: embeddings from an instruction-tuned LLM and a fixed set of LLM-judged rubric scores (solution hint, required domain knowledge, verification difficulty, and so on), sometimes concatenated. This machinery turns the two sources of evaluation cost—new tasks and new agent configurations—into predictions from cheap, static inpu

What would settle it

On a benchmark where at least one LLM is paired with many scaffolds and one scaffold with many LLMs, fit both the additive model and a model with pairwise interaction terms; if the interaction model yields materially better held-out AUC-ROC or more accurate calibration, the additive decomposition is insufficient. A complementary check is to evaluate a held-out LLM-scaffold pair never seen together and compare per-task predicted probabilities against actual pass rates—systematic signed errors would show that θ_LLM + θ_scaffold misses a dependency.

Watch

Extended reading notes

Core claim

The paper's central claim is that standard Item Response Theory, extended with task features and an additive LLM-scaffold ability decomposition, accurately predicts task-level success in the agentic coding regime. The model sets P(success) = σ(θ_LLM + θ_scaffold − β_task), where θ_LLM and θ_scaffold are latent abilities for every observed LLM and scaffold, and β_task is a latent task difficulty predicted from task embeddings and LLM-as-a-judge rubric features. On held-out responses the method performs on par with standard IRT, while adding the ability to predict held-out LLM-scaffold combinations (AUC-ROC 0.921–0.936) and entire held-out benchmarks (AUC-ROC 0.677–0.735), beating the baseline

Load-bearing premise

The load-bearing premise is that an agent's ability is exactly the sum of its LLM's ability and its scaffold's ability, with no interaction; the authors validate this on limited scaffold diversity in one benchmark and explicitly caution that co-adapted model-scaffold pairs could break it.

Editorial extensions

If this is right

  • Benchmark designers can estimate the difficulty of new tasks without running any agent evaluation, by extracting task features and reading off the predicted difficulty.
  • Agent developers can estimate ability on a benchmark from a small adaptively chosen subset of tasks, with the paper showing gains over random selection at budgets below about 30 tasks.
  • Leaderboards from different benchmarks can be placed on a common ability scale through shared LLM and scaffold parameters, making cross-benchmark comparison possible.
  • The learned scaffold abilities give a quantitative ranking of scaffolds, so scaffold choice can be informed by a stable contribution term rather than full re-evaluation.
  • Task-level success probabilities can be used to select reinforcement-learning rollouts near an agent's current ability, improving the training signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the additive decomposition holds broadly, then scaffold contribution is LLM-independent: the same scaffold advantage applies to any model, so scaffold rankings should transfer across LLM generations without fresh evaluations.
  • The same recipe—feature-predicted task difficulty plus additive component abilities—is directly portable to other agentic domains (web automation, tool use, database agents) whenever the agent is an LLM plus a scaffold and tasks can be encoded as text-plus-artifacts.
  • A natural stress test is to refit the model with an interaction term on data where each LLM has several scaffolds; if interaction terms consistently improve held-out prediction, the additivity assumption is false.
  • The method's current dependence on reference solutions and test patches means difficulty prediction is strongest after a task is fully specified; predicting from the problem statement and repository alone, which the ablation suggests is weaker, would be a useful extension for very early task drafting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes AgentPsychometrics, an IRT-based framework for predicting binary success/failure of LLM coding agents on individual agentic coding tasks. Task difficulty is modeled from rich features (embeddings, LLM-as-a-judge rubric features, repository/test/solution artifacts), and agent ability is decomposed additively as the sum of an LLM ability and a scaffold ability (Eq. 1). The framework is evaluated on four benchmarks (SWE-bench Verified, SWE-bench Pro, Terminal-Bench 2.0, GSO) in four held-out settings: held-out tasks, held-out random responses, held-out LLM-scaffold combinations, and held-out benchmarks. Headline results are AUC-ROC 0.921--0.936 for New Agents (Table 4) and 0.677--0.735 for New Benchmarks (Table 5), beating baselines; an adaptive task-selection application is also presented.

Significance. If the additive decomposition is valid, the paper makes a practical contribution: it aggregates sparse leaderboard data across benchmarks that share LLMs or scaffolds, and it can predict task-level outcomes for unseen tasks without running expensive agent evaluations. The strengths of the paper are its public code release, the breadth of feature-source ablations, the use of standard IRT as a reference point, and a concrete downstream application. The principal risk is that the load-bearing assumption of zero LLM--scaffold interaction is validated only indirectly; the paper itself concedes in Section 6 that the independence 'may cease to hold.' Because all multi-benchmark stitching and held-out-agent predictions inherit this assumption, the evidence needs to be strengthened rather than assumed.

major comments (2)
  1. [§3.2.2 (Eq. 1), §6, App. D.3/G.1, Fig. 2] Eq. (1) is the linchpin of the multi-benchmark stitching and of the New Agents / New Benchmarks claims, but the current validation does not rule out material LLM×scaffold interactions. The New Responses experiment (App. G.1, Table 12) holds out random responses while keeping every agent in training, so any interaction shared by observed pairs is absorbed into the fitted additive parameters; it cannot detect the misspecification that matters for unseen pairs. The fixed-scaffold check (Fig. 2) is n=32 on one benchmark and mostly confirms LLM ordering. The functional form itself was selected on SWE-bench Verified among five candidates (App. D.3), making the additive choice partly in-sample. If true ability is θ_m + θ_s + γ_ms, predictions for held-out combinations omit γ, and since scaffold usage is correlated with benchmark composition, Table 5 inherits this bias. Please add a direct inter
  2. [Tables 2, 4, 5; §4] No confidence intervals or significance tests are reported for the headline AUCs, despite 5-fold CV and text claiming that predictors 'significantly beat the baseline.' In Table 5 the gaps are modest and inconsistent across feature types (e.g., Combined 0.677 vs. LLM-as-judge 0.696 on SWE-bench Pro; Combined 0.719 vs. LLM-as-judge 0.735 on GSO), so the 'Combined' advantage is not stable in the out-of-distribution setting. Report bootstrap CIs or paired fold-wise tests, and state how many held-out agents and tasks contribute to each fold, especially after filtering agents with novel LLMs/scaffolds.
minor comments (6)
  1. [§2.2] The paragraph beginning 'Several works have explored using IRT...' is repeated almost verbatim; remove the duplicate.
  2. [§5.3, Table 5] Combining feature vectors does not consistently beat the individual LLM-as-judge vectors in the held-out-benchmark setting; the text should acknowledge this rather than implying combination is always beneficial.
  3. [App. D.3] The formulas for 'Product' and 'L2 norms' need sign conventions; as written, sign(θ_m+θ_s)θ_mθ_s and sign(θ_m+θ_s)√(θ_m²+θ_s²) are hard to interpret without further definition.
  4. [Fig. 2] Report a confidence interval for r=0.974 and state whether the n=32 points are distinct LLMs; also fix spacing artifacts in the axis labels.
  5. [§5.3, Tables 4--5] State the number of held-out agents and tasks per fold. The filtering of agents with novel LLMs/scaffolds is described but counts are omitted, making the strength of the held-out claims hard to assess.
  6. [General] Because benchmark designers are the stated end users, AUC-ROC alone does not guarantee well-calibrated difficulty estimates; a calibration check (Brier score or reliability diagram) would strengthen the practical claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: held-out agent and benchmark predictions are genuine out-of-sample evaluations of an empirically tested additive model; no load-bearing reduction to fitted inputs.

full rationale

The paper's central predictions are true out-of-sample evaluations. Task difficulties are first estimated by IRT from training responses, then a ridge regressor is trained to predict those frozen difficulty values from task features and evaluated on held-out tasks or whole held-out benchmarks; this is a standard two-stage supervised pipeline, not a re-use of the label as a prediction. The additive decomposition θ_agent = θ_LLM + θ_scaffold (Eq. 1) is an explicit modeling assumption, not a tautology: θ_LLM and θ_scaffold are fitted from responses of other agents sharing those components, and the 'New Responses' experiment compares this constrained model against unconstrained standard IRT on held-out responses, while 'New Agents' and 'New Benchmarks' hold out entire LLM-scaffold combinations or benchmarks. The functional form was selected among five candidates on validation folds (Appendix D.3), which is model selection, not circularity. The Terminal-Bench fixed-scaffold correlation (Figure 2) is a consistency check between two models fit on overlapping data; it is not an independent prediction, but it is not load-bearing because the additivity claim is also supported by the held-out response comparison. There are no self-citations and no imported uniqueness theorems. The acknowledged limitation that scaffold–LLM co-adaptation may break independence (Section 6) is an external-validity concern, not a circular-derivation concern.

Assumptions & free parameters 8 free parameters · 6 assumptions · 0 invented entities

The central model is built on IRT and supervised ridge regression. The main latent quantities (task difficulty, LLM ability, scaffold ability) are statistical parameters fitted from response data rather than independently evidenced entities. The additive LLM-scaffold decomposition and cross-benchmark scale comparability are domain assumptions that carry much of the generalization burden.

free parameters (8)
  • Task difficulty parameters β_j (one per task) = Estimated via SVI IRT on response data; values not listed in main text
    Central latent variables; used both as prediction targets for the ridge regression and as difficulty scores for adaptive task selection.
  • LLM ability parameters θ_m (one per LLM) = Range -5.014 (GPT-3.5) to 3.660 (Gemini 3.1 Pro) in Table 13
    Fitted from the multi-benchmark response matrix; the additive ability model relies on these parameters for held-out-agent predictions.
  • Scaffold ability parameters θ_s (one per scaffold) = Range -2.954 (RAG) to 1.453 (AgentScope) in Table 14
    Fitted from the response matrix; 46 of 72 scaffolds occur only once, so many scaffold abilities are weakly identified.
  • Embedding backbone selection = DeepSeek-R1-Distill-Qwen-32B chosen by best SWE-bench Verified AUC (0.824±0.020 among 17 backbones)
    The embedding model was selected using validation performance on one of the evaluation benchmarks, potentially inflating reported cross-validated AUC.
  • LLM-as-judge feature extraction models = Claude Opus 4.6 for judge features; GPT-5.4 for auditor agent
    Chosen by ablation on the evaluation datasets; all judge-feature results depend on this choice.
  • Manual 15-feature LLM-as-judge subset = 10 statement + 1 test + 1 solution + 3 environment features
    Features were manually chosen to work across all datasets; the authors use a different random seed to mitigate overfitting bias, but selection still uses validation performance.
  • Ridge regularization hyperparameters = Chosen by 5-fold cross-validation MSE within training set
    Standard hyperparameter tuning; per-feature-vector regularization values are not reported.
  • Terminal-Bench 2.0 pass threshold = At least 50% of attempts successful
    Arbitrary binarization of repeated evaluation attempts; affects all Terminal-Bench 2.0 response labels.
assumptions (6)
  • standard math One-dimensional 1PL IRT: P(y=1|θ,β)=σ(θ-β)
    Base model introduced in Section 2.1.2; assumes unidimensional ability and conditional independence of responses.
  • domain assumption Agent ability is additive: θ_agent = θ_LLM + θ_scaffold
    Section 3.2.2 Eq. (1); selected empirically among five functional forms in Appendix D.3 on SWE-bench Verified; no LLM-scaffold interaction terms.
  • domain assumption Cross-benchmark comparability of ability and difficulty scales in multi-benchmark IRT
    Section 5.2 and G.2; a single joint fit treats SWE-bench Verified/Pro, Terminal-Bench 2.0, and GSO as sharing one latent scale, with no benchmark-specific offsets.
  • domain assumption IRT difficulty estimates from response data are a valid regression target for task features
    Section 3.2.1; ridge regression predicts frozen SVI difficulty estimates, assuming these recover true task difficulty rather than noise.
  • domain assumption Task feature-difficulty relationships transfer out-of-distribution to new benchmarks
    Section 3.1 and the New Benchmarks experiment; assumes embeddings and judge ratings of gold solutions/test cases generalize to entirely new benchmark distributions.
  • standard math SVI with hierarchical priors resolves identifiability of additive ability parameters
    Appendix C and the fitting procedure; the additive form has a gauge freedom (θ_m+c, θ_s-c), so absolute ability rankings depend on the prior/identifiability constraints, which the paper does not discuss.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Agent psychometrics: Task-level performance prediction in agentic coding benchmarks." pith.science (2026). https://pith.science/paper/EPDCRWPQ

@misc{pith2026260400594,
  author       = {Pith},
  title        = {Pith review of: Agent psychometrics: Task-level performance prediction in agentic coding benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EPDCRWPQ}},
  note         = {Machine review of arXiv:2604.00594}
}
read the original abstract

As the focus in LLM-based coding shifts from static single-step code generation to multi-step agentic interaction with tools and environments, understanding which tasks will challenge agents and why becomes increasingly difficult. This is compounded by current practice: agent performance is typically measured by aggregate pass rates on benchmarks, but single-number metrics obscure the diversity of tasks within a benchmark. We present a framework for predicting success or failure on individual tasks tailored to the agentic coding regime. Our approach augments Item Response Theory (IRT) with rich features extracted from tasks, including issue statements, repository contexts, solutions, and test cases, and introduces a novel decomposition of agent ability into LLM and scaffold ability components. This parameterization enables us to aggregate evaluation data across heterogeneous leaderboards and accurately predict task-level performance for unseen benchmarks, as well as unseen LLM-scaffold combinations. Our methods have practical utility for benchmark designers, who can better calibrate the difficulty of their new tasks without running computationally expensive agent evaluations.

Figures

Figures reproduced from arXiv: 2604.00594 by the authors.

Figure 1
Figure 1. Agent and task features predicting success probability. We illustrate the feature sources from which we derive estimates of an agent’s ability score and a task’s difficulty score. Then, using the estimated agent ability and task difficulty, we apply the logistic model from IRT (Baker, 2001) to predict the probability that the agent succeeds on the task. capabilities and designing more discriminative tasks (Liu et al… view at source ↗
Figure 2
Figure 2. Validation of decomposition. Strong correlation (Pearson r = 0.974) between agent abilities learned on a fixed scaffold (Terminus 2) versus LLM abilities isolated via our decomposition method. 5.3 PREDICTORS WITH TASK AND AGENT FEATURES GENERALIZE TO HELD-OUT AGENTS AND BENCHMARKS New Agents. We observe very strong performance, even approaching the oracle, on held-out LLM￾scaffold combinations (see [PITH_FULL_IMAGE… view at source ↗
Figure 3
Figure 3. Choosing Effective Subsets of a Benchmark for Evaluation Via Adaptive Task Selec￾tion. IRT (Predicted) uses the predicted IRT difficulty scores from a multi-benchmark model trained without SWE-bench Pro response data. IRT (Oracle) uses IRT difficulty scores, unrealistically cali￾brated with full response data. Random simply selects tasks at random. New Benchmarks. We observe decent generalization to held-out benchma… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Task difficulty histograms. SWE-bench Pro is harder on average than Verified, GSO is the hardest, and Terminal-Bench 2.0 is highly heterogeneous [PITH_FULL_IMAGE:figures/full_fig_p035_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A unified corpus of 957k trial outcomes shows frontier progress is uneven and strict all-pass aggregation obscures capability and can reorder agents.

Reference graph

Works this paper leans on

6 extracted references · 1 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Explore the working directory to understand the project structure

  2. [2]

    Read the problem statement (provided as input)

  3. [3]

    Try to understand the scope and complexity of the task

  4. [4]

    Check available tools, tests, and dependencies

  5. [5]

    value" (1-5 integer) and

    Rate the environment on the 8 axes below ## Features to Assess (1-5 scale) ### Fix Localization (fix_localization: 1-5) How spread out is the likely solution? - 1: Solution requires changes across many modules/packages - 2: Solution spans multiple files across different directories - 3: Solution spans 2-3 files in the same module - 4: Solution is in 1-2 c...

    2026

  6. [2025]

    hello") plt.subplot(3, 3, 4) plt.imshow(data) + plt.ylabel(

    URLhttps://satsuite.collegeboard.org/media/pdf/ sat-sd-essay-understanding-scores-educators.pdf. Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Cal...

    arXiv 2025

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.