{"id":"e00ca802-f9ca-49ee-9437-c828d37e667c","arxiv_id":"2604.00594","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A combined IRT and feature-based model predicts task-level success of LLM coding agents across benchmarks by treating agent ability as the sum of an LLM ability and a scaffold ability.","lead":"This paper builds a statistical model that predicts whether a coding agent will solve a given benchmark task, using the task's text, tests, and repository plus the agent's underlying model and scaffolding. It could let benchmark designers estimate task difficulty without running expensive agent evaluations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Additive LLM+scaffold ability (Eq. 1) is the linchpin of held-out agent/benchmark prediction, but its validation is too weak to rule out interaction effects that would bias those predictions.","rationale":"The reader's weakest_assumption is exactly the load-bearing concern I identify: the additive LLM+scaffold ability decomposition. The paper's own validation of this assumption is limited to one fixed-scaffold comparison on a single benchmark, plus in-distribution response prediction that does not exercise unseen combinations. Because the method's headline results for held-out agents and held-out benchmarks all rely on this decomposition, interaction effects would directly undermine the central claim. No other concern—missing error bars, tuning on evaluation benchmarks, or modest held-out-benchmark AUC—is as tightly coupled to the core mechanism. I therefore agree with the reader's identification. The appropriate verdict remains CONDITIONAL: the method is promising and the current evidence is suggestive, but the additive assumption needs stronger cross-benchmark validation before the framework is used for predicting new agent configurations or comparing scaffolds. I do not change the reader's verdict, so verdict_should_be is UNCHANGED.","tokens_in":27732,"tokens_out":5955,"duration_ms":62455,"concrete_test":"Using the standard IRT fits already computed for the New Responses experiment, take every pair of scaffolds (S1, S2) that co-occur with the same LLM L in at least two different LLMs and compute d_{L,S1,S2} = θ̂_{L,S1} − θ̂_{L,S2} from the standard IRT agent abilities. Under additivity, d_{L,S1,S2} should be constant across L (equal to θ_S1 − θ_S2). Test this with an ANOVA/permutation test on d across L. If the LLM×scaffold interaction explains significant variance, or if the spread of d is comparable to the scale of scaffold-ability differences, then the additive model is misspecified and held-out-combination predictions are biased. Report the effect size and, if interactions are found, refit Eq. 1 with a hierarchical interaction term to quantify the impact on Table 4 and Table 5 AUCs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—predicting success for unseen LLM-scaffold combinations and unseen benchmarks—depends on the decomposition θ_agent = θ_LLM + θ_scaffold with no interaction (Section 3.2.2, Eq. 1). This assumption is load-bearing: it is what allows multi-benchmark stitching and predictions for held-out agents. Its validation, however, is incomplete. The New Responses experiment (Appendix G.1) shows parity with standard IRT on held-out responses, but those responses still involve agent combinations that appear in training, so any interaction shared across observed agents is absorbed into the fitted abilities. The fixed-scaffold comparison (Figure 2) is a single benchmark (Terminal-Bench 2.0), n=32, and mostly confirms LLM ordering; it does not test whether pairwise interactions are zero. The functional form was also selected on SWE-bench Verified among several candidates (Appendix D.3), making the choice partly in-sample. If true ability is θ_LLM + θ_scaffold + γ_LLM,scaffold, then predictions for unseen combinations omit γ, and because scaffold usage is correlated with benchmark composition, the held-out-benchmark results in Table 5 inherit this bias. The authors themselves concede in Section 6 that scaffold–LLM co-adaptation may break independence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AgentPsychometrics, an IRT-based framework for predicting binary success/failure of LLM coding agents on individual agentic coding tasks. Task difficulty is modeled from rich features (embeddings, LLM-as-a-judge rubric features, repository/test/solution artifacts), and agent ability is decomposed additively as the sum of an LLM ability and a scaffold ability (Eq. 1). The framework is evaluated on four benchmarks (SWE-bench Verified, SWE-bench Pro, Terminal-Bench 2.0, GSO) in four held-out settings: held-out tasks, held-out random responses, held-out LLM-scaffold combinations, and held-out benchmarks. Headline results are AUC-ROC 0.921--0.936 for New Agents (Table 4) and 0.677--0.735 for New Benchmarks (Table 5), beating baselines; an adaptive task-selection application is also presented.","tokens_in":27991,"tokens_out":8183,"duration_ms":84857,"significance":"If the additive decomposition is valid, the paper makes a practical contribution: it aggregates sparse leaderboard data across benchmarks that share LLMs or scaffolds, and it can predict task-level outcomes for unseen tasks without running expensive agent evaluations. The strengths of the paper are its public code release, the breadth of feature-source ablations, the use of standard IRT as a reference point, and a concrete downstream application. The principal risk is that the load-bearing assumption of zero LLM--scaffold interaction is validated only indirectly; the paper itself concedes in Section 6 that the independence 'may cease to hold.' Because all multi-benchmark stitching and held-out-agent predictions inherit this assumption, the evidence needs to be strengthened rather than assumed.","major_comments":[{"comment":"Eq. (1) is the linchpin of the multi-benchmark stitching and of the New Agents / New Benchmarks claims, but the current validation does not rule out material LLM×scaffold interactions. The New Responses experiment (App. G.1, Table 12) holds out random responses while keeping every agent in training, so any interaction shared by observed pairs is absorbed into the fitted additive parameters; it cannot detect the misspecification that matters for unseen pairs. The fixed-scaffold check (Fig. 2) is n=32 on one benchmark and mostly confirms LLM ordering. The functional form itself was selected on SWE-bench Verified among five candidates (App. D.3), making the additive choice partly in-sample. If true ability is θ_m + θ_s + γ_ms, predictions for held-out combinations omit γ, and since scaffold usage is correlated with benchmark composition, Table 5 inherits this bias. Please add a direct inter","section":"§3.2.2 (Eq. 1), §6, App. D.3/G.1, Fig. 2"},{"comment":"No confidence intervals or significance tests are reported for the headline AUCs, despite 5-fold CV and text claiming that predictors 'significantly beat the baseline.' In Table 5 the gaps are modest and inconsistent across feature types (e.g., Combined 0.677 vs. LLM-as-judge 0.696 on SWE-bench Pro; Combined 0.719 vs. LLM-as-judge 0.735 on GSO), so the 'Combined' advantage is not stable in the out-of-distribution setting. Report bootstrap CIs or paired fold-wise tests, and state how many held-out agents and tasks contribute to each fold, especially after filtering agents with novel LLMs/scaffolds.","section":"Tables 2, 4, 5; §4"}],"minor_comments":[{"comment":"The paragraph beginning 'Several works have explored using IRT...' is repeated almost verbatim; remove the duplicate.","section":"§2.2"},{"comment":"Combining feature vectors does not consistently beat the individual LLM-as-judge vectors in the held-out-benchmark setting; the text should acknowledge this rather than implying combination is always beneficial.","section":"§5.3, Table 5"},{"comment":"The formulas for 'Product' and 'L2 norms' need sign conventions; as written, sign(θ_m+θ_s)θ_mθ_s and sign(θ_m+θ_s)√(θ_m²+θ_s²) are hard to interpret without further definition.","section":"App. D.3"},{"comment":"Report a confidence interval for r=0.974 and state whether the n=32 points are distinct LLMs; also fix spacing artifacts in the axis labels.","section":"Fig. 2"},{"comment":"State the number of held-out agents and tasks per fold. The filtering of agents with novel LLMs/scaffolds is described but counts are omitted, making the strength of the held-out claims hard to assess.","section":"§5.3, Tables 4--5"},{"comment":"Because benchmark designers are the stated end users, AUC-ROC alone does not guarantee well-calibrated difficulty estimates; a calibration check (Brier score or reliability diagram) would strengthen the practical claims.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The main concern is testable: the additive decomposition in Eq. (1) underpins all cross-benchmark claims, and the current evidence is indirect. I would not reject the paper; the framework is well engineered and the experimental design is reasonable. I chose major_revision rather than minor because the interaction question is load-bearing and the manuscript currently lacks both a direct interaction diagnostic and uncertainty quantification on the headline AUCs. If the authors add those, I expect to support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is worth a look if you care about cheap evaluation of coding agents. The central idea — decompose agent ability into LLM and scaffold abilities and stitch together leaderboards — is a real extension of IRT. The task-difficulty prediction from agentic artifacts is less new, being adapted from Truong et al., but the application to coding tasks is new enough.\n\nWhat the paper does well: the experimental structure is careful, with held-out tasks, responses, agents, and benchmarks. The held-out-agent results are the key test of the additive assumption, and they look good: AUC 0.921–0.936 versus an oracle around 0.94. The authors are also refreshingly honest in Section 6 about the independence assumption possibly breaking down. The adaptive subset-selection application is a nice practical payoff.\n\nWhere I'd push back. First, no confidence intervals or significance tests in the main tables, despite \"significantly beat\" language. For an empirical paper, that's a real omission, though the gains are consistent across benchmarks. Second, the pipeline is tuned on the evaluation benchmarks themselves: embedding backbone chosen on SWE-bench Verified, judge model on a multi-dataset ablation, functional form selected on SWE-bench. The different-seed trick for the 15-feature subset is an attempt to mitigate, but it doesn't fully erase leakage. Third, the abstract says \"accurately predict\" for unseen benchmarks; AUC 0.68–0.74 is decent but not accurate in any strong sense.\n\nThe stress-test concern about the additive assumption is only partly valid. The New Responses experiment indeed doesn't test generalization to new combinations, but the New Agents experiment does, and it passes on two benchmarks. The n=32 Terminal-Bench correlation is a nice sanity check, not the main evidence. So I wouldn't call the additive assumption unsupported; it's a modeling choice with reasonable evidence and a clearly stated limitation.\n\nThis is a workshop-grade paper with a publishable core. A serious referee would want error bars, some analysis of the selection pipeline's effect, and a toned-down abstract, but the work is coherent and the results look real. I'd send it to peer review rather than desk reject.","headline":"Solid workshop-grade paper: the additive LLM+scaffold ability decomposition is the genuinely new piece, held-out-agent results back it up, but missing error bars, nested model selection, and an overstated abstract keep it from being more than that.","tokens_in":665,"tokens_out":866,"would_cite":true,"duration_ms":41801,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Agentic coding success can be predicted task-by-task from additive LLM and scaffold abilities.","keywords":["agentic coding benchmarks","item response theory","task-level success prediction","LLM-scaffold ability decomposition","multi-benchmark IRT","task difficulty prediction","adaptive task selection","benchmark calibration"],"falsifier":"On a benchmark where at least one LLM is paired with many scaffolds and one scaffold with many LLMs, fit both the additive model and a model with pairwise interaction terms; if the interaction model yields materially better held-out AUC-ROC or more accurate calibration, the additive decomposition is insufficient. A complementary check is to evaluate a held-out LLM-scaffold pair never seen together and compare per-task predicted probabilities against actual pass rates—systematic signed errors would show that θ_LLM + θ_scaffold misses a dependency.","tokens_in":27532,"feed_emoji":"🤖","tokens_out":8925,"duration_ms":79825,"temperature":0.7,"pith_summary":"Agentic coding benchmarks are usually summarized by a single pass rate, which hides which tasks are hard and why. This paper argues that task-level success can be predicted from static properties of the task plus a simple additive decomposition of the agent: the probability an agent solves a task is a logistic function of the sum of its LLM's ability and its scaffold's ability, minus the task's difficulty. Task difficulty is itself estimated from accessible artifacts—problem statement, repository state, test cases, and reference solution—so new tasks can be calibrated without running expensive agents. Because individual LLMs and scaffolds recur across leaderboards even when full agents do not, the additive parameterization lets evaluations from different benchmarks be pooled and extrapolated. If the model is right, benchmark construction and agent evaluation become cheap, targeted prediction rather than costly sampling.","feed_headline":"Additive LLM+scaffold scores predict agentic coding task success","feed_subtitle":"Item-response model forecasts task-level pass rates on unseen benchmarks using only task text and artifacts.","key_machinery":"The central object is the augmented IRT model P(y=1) = σ(θ_LLM + θ_scaffold − β_task). θ_LLM and θ_scaffold are per-LLM and per-scaffold latent ability parameters; their additive combination is what allows heterogeneous leaderboards to be stitched together when full agents rarely overlap. β_task is a task difficulty that is predicted by a linear model from static features: embeddings from an instruction-tuned LLM and a fixed set of LLM-judged rubric scores (solution hint, required domain knowledge, verification difficulty, and so on), sometimes concatenated. This machinery turns the two sources of evaluation cost—new tasks and new agent configurations—into predictions from cheap, static inpu","core_discovery":"The paper's central claim is that standard Item Response Theory, extended with task features and an additive LLM-scaffold ability decomposition, accurately predicts task-level success in the agentic coding regime. The model sets P(success) = σ(θ_LLM + θ_scaffold − β_task), where θ_LLM and θ_scaffold are latent abilities for every observed LLM and scaffold, and β_task is a latent task difficulty predicted from task embeddings and LLM-as-a-judge rubric features. On held-out responses the method performs on par with standard IRT, while adding the ability to predict held-out LLM-scaffold combinations (AUC-ROC 0.921–0.936) and entire held-out benchmarks (AUC-ROC 0.677–0.735), beating the baseline","pith_inferences":["If the additive decomposition holds broadly, then scaffold contribution is LLM-independent: the same scaffold advantage applies to any model, so scaffold rankings should transfer across LLM generations without fresh evaluations.","The same recipe—feature-predicted task difficulty plus additive component abilities—is directly portable to other agentic domains (web automation, tool use, database agents) whenever the agent is an LLM plus a scaffold and tasks can be encoded as text-plus-artifacts.","A natural stress test is to refit the model with an interaction term on data where each LLM has several scaffolds; if interaction terms consistently improve held-out prediction, the additivity assumption is false.","The method's current dependence on reference solutions and test patches means difficulty prediction is strongest after a task is fully specified; predicting from the problem statement and repository alone, which the ablation suggests is weaker, would be a useful extension for very early task drafting."],"forward_implications":["Benchmark designers can estimate the difficulty of new tasks without running any agent evaluation, by extracting task features and reading off the predicted difficulty.","Agent developers can estimate ability on a benchmark from a small adaptively chosen subset of tasks, with the paper showing gains over random selection at budgets below about 30 tasks.","Leaderboards from different benchmarks can be placed on a common ability scale through shared LLM and scaffold parameters, making cross-benchmark comparison possible.","The learned scaffold abilities give a quantitative ranking of scaffolds, so scaffold choice can be informed by a stable contribution term rather than full re-evaluation.","Task-level success probabilities can be used to select reinforcement-learning rollouts near an agent's current ability, improving the training signal."],"fun_headline_variants":["LLM+scaffold ability split predicts coding agent tasks","Task success forecast from LLM and scaffold abilities","Agentic coding: predict tasks via LLM+scaffold IRT","Benchmark difficulty calibrated without costly agent runs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that an agent's ability is exactly the sum of its LLM's ability and its scaffold's ability, with no interaction; the authors validate this on limited scaffold diversity in one benchmark and explicitly caution that co-adapted model-scaffold pairs could break it.","fun_headline_variants_meta":{"raw":{"variants":["LLM+scaffold ability split predicts coding agent tasks","Task success forecast from LLM and scaffold abilities","Agentic coding: predict tasks via LLM+scaffold IRT","Benchmark difficulty calibrated without costly agent runs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000649,"raw_usage":{"total_tokens":2804,"prompt_tokens":723,"completion_tokens":2081,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":2014}},"tokens_in":467,"tokens_out":2081,"duration_ms":16031,"temperature":1.0,"reasoning_tokens":2014,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T16:57:49.302572+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a benchmark where at least one LLM is paired with many scaffolds and one scaffold with many LLMs, fit both the additive model and a model with pairwise interaction terms; if the interaction model yields materially better held-out AUC-ROC or more accurate calibration, the additive decomposition is insufficient. A complementary check is to evaluate a held-out LLM-scaffold pair never seen together and compare per-task predicted probabilities against actual pass rates—systematic signed errors would show that θ_LLM + θ_scaffold misses a dependency.","supporting_citations":[],"review_version":1}