{"id":"af735fcb-7edd-4f24-8305-08ec7683dc77","arxiv_id":"2602.07267","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"BRIDGE shows that item-response-theory difficulty estimated from model performance tracks log human completion time, enabling human time prediction and a ~6-month doubling forecast for frontier task horizons.","lead":"BRIDGE is a framework that learns a hidden difficulty scale from AI model pass/fail results and links it to how long a human would need to finish the same task. If it holds up, benchmark scores could be translated into human work-hours without running expensive human timing studies for every new benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Global transfer of the METR-calibrated b→log-time mapping is the key fragility: Eq. 3 is an empirical assumption, tested on only 13 Cybench tasks and coarse SWE-bench buckets, and the 6-month doubling forecast inherits this slope.","rationale":"The reader's weakest assumption—transferability of a single METR-calibrated log-linear mapping from IRT difficulty to human time—is the same load-bearing concern I identify. The paper's own validation is too small and coarse to establish that the relationship is domain-invariant, and the forecasted doubling time inherits any error in the calibration slope. My concrete test would directly probe this assumption by adding diverse human-time annotations and comparing domain-specific slopes; if the slopes differ, the central claim fails. Since this matches the reader's conditional verdict, my recommendation is to keep the verdict unchanged rather than move it to reject or accept. I found no independent reason to escalate: the empirical fit on METR is strong (R² = 0.81), and the method is clearly described with code released, so a conditional verdict is appropriate.","tokens_in":20348,"tokens_out":4615,"duration_ms":55502,"concrete_test":"Collect or obtain human completion-time annotations for at least 50–100 tasks from two or more domains outside METR (e.g., MLE-bench Kaggle competitions, GDPval occupations, or GUI workflow tasks). Fit Eq. 3 on METR tasks only, then evaluate the global mapping on this held-out set: report log-time error and 0.5×–2× coverage. Separately fit domain-specific slopes (one per new domain) and test whether they differ from the METR slope by more than sampling uncertainty (e.g., via interaction term in a linear regression). If the domain-specific slopes differ meaningfully or the global coverage is substantially below the Cybench figure, the universal transfer assumption fails. A faster internal check: refit the 2PL IRT on METR tasks alone, compare the resulting b→log h slope to the joint-fit slope—if it shifts significantly, the calibration depends on benchmark composition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Eq. 3 in Section 3) is that a single global affine map from IRT difficulty b to log human time, fit only on METR tasks, transfers to all benchmarks. Section 3's motivating argument ('If capability grows exponentially... implies a linear relationship between b_i and the logarithm of the human time') is not a derivation—it is an empirical assertion that model-derived latent difficulty and human time share one domain-independent relation. The out-of-sample evidence is limited and weak: SWE-bench (Figure 3) uses coarse buckets and yields weighted κ = 0.231 with a confusion matrix showing most 15–60 min tasks misclassified; Cybench (Figure 4) has n = 13 after dropping all-unsolved tasks, with R² = 0.45 and the '92% within 2×' claim driven by very few points. The forecasting in Section 4.4 directly compounds this: the 6-month doubling time is the product of IRT ability growth and the METR-fit slope, so any domain-specific offset in the human-time/difficulty relation (e.g., CTF tasks where humans spend time on research that models shortcut) shifts the predicted horizon. The paper itself notes in Section 5.3 that benchmark composition changes model rankings, underscoring that the latent scale is not composition-independent. Because the entire human-time prediction pipeline keys on this one universal calibration, this is the load-bearing assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BRIDGE, a framework that fits a two-parameter logistic Item Response Theory (2PL IRT) model to binary success/failure outcomes of models across several benchmarks (METR, SWE-bench, MLE-bench, GDPval, Cybench), yielding per-task latent difficulty b_i and per-model ability θ_j. On the subset of METR tasks with human completion-time annotations, the authors fit a log-linear calibration log(h) = slope × b + intercept (Eq. 3) and report R²=0.81. They then use this calibration to predict human completion times for other benchmarks without human-time labels, validating against SWE-bench coarse time buckets and Cybench human first-solve times. Finally, they forecast the frontier model's 'task-length horizon' over release dates from the IRT difficulty that corresponds to 50% (and 80%) success, finding an approximate doubling time of 6 months, consistent with METR's earlier estimate.","tokens_in":20825,"tokens_out":4243,"duration_ms":46508,"significance":"If the central claim holds — that a single global affine map from IRT difficulty to log human time transfers across benchmarks — BRIDGE would provide a scalable, human-interpretable evaluation tool and an independent capability-forecasting method, which is a potentially valuable contribution. The paper's strengths include the use of a principled psychometric model (2PL IRT), a publicly available code repository, comparison against several baselines (logit success rate, LLM estimators), and explicit acknowledgment of some limitations (e.g., the LLM-as-a-judge proxy for GDPval, the exclusion of all-unsolved Cybench tasks). The empirical calibration on METR is compelling, but the paper's central transfer assumption is tested on only two out-of-distribution benchmarks, one of which is a coarse four-bucket classification with modest accuracy and the other is a 13-task subset after selection. The 6-month forecast inherits the calibration slope and does not propagate uncertainty from the IRT and calibration stages. The result is plausible but not yet established at the level claimed by the abstract.","major_comments":[{"comment":"The central claim is that IRT difficulty b and log human time are linearly related by a single global slope/intercept fit on METR. The motivating argument in §3 ('If capability grows exponentially... implies a linear relationship') is not a derivation: θ_j and b_i are estimated from model performance, and the assertion that this latent scale coincides with log human time is an empirical assumption, not a consequence of exponential capability growth. This assumption is load-bearing for every downstream prediction, including the 6-month horizon. I recommend testing it directly: fit Eq. (3) separately within SWAA/HCAST/RE-Bench and test slope homogeneity; also perform leave-one-benchmark-out calibration where the mapping is fit on a subset of METR suites and evaluated on held-out METR suites and on the OOD benchmarks. Without such evidence, the transfer claim is not established.","section":"§3, Eq. (3)"},{"comment":"The SWE-bench validation is a coarse four-bucket classification, and the reported numbers are modest: overall accuracy 41.6%, weighted κ=0.231, and per-bucket accuracy of 18% for the 15–60 min bucket (n=261) and 21% for 1–4 hr (n=42), while the largest bucket (<15 min, n=194) gets 77%. The phrase 'substantially better alignment' is relative to baselines that are near-zero or systematically biased; absolute calibration in the middle range is weak. Please report calibration on continuous predictions (e.g., predicted median interpolated against bucket boundaries), bucket-wise error distributions, and an ordinal evaluation metric. A confusion matrix alone does not support the conclusion that predicted times are 'quantitatively accurate' for SWE-bench.","section":"§4.3, Figure 3"},{"comment":"Cybench validation uses only 13 tasks after excluding tasks unsolved by all models. This exclusion is not neutral: tasks that no model solves are likely to be the most difficult and longest, so the calibration is validated on a truncated, easier subset. With n=13, R²=0.45 and '92% within 2×' = 12/13 give wide confidence intervals; the reported point estimates are fragile. Please report results on the full Cybench task set, with an explicit treatment for unsolved tasks (e.g., censored at a lower bound or with imputed difficulty), and show leave-one-out predictions. Also state how the exclusion criterion affects the estimated calibration slope and intercept, since the paper's own logic says unsolved tasks have unreliable b estimates, which itself casts doubt on using IRT difficulty for the long tail.","section":"§4.1.2, Figure 4"},{"comment":"The 6-month doubling time is obtained by a linear fit to log task length vs. release date, where task length is a deterministic function of the estimated b for the frontier model. The bootstrap uncertainty shown (2000 resamples over frontier models) does not propagate uncertainty in the IRT parameters (a_i, b_i, θ_j) or in the calibration slope/intercept of Eq. (3). Since the horizon values all inherit the estimated calibration slope, the reported 95% CIs are likely far too narrow. Please propagate the posterior distribution of the IRT and calibration parameters (e.g., by resampling from the MCMC posterior) and report the sensitivity of the doubling time to the calibration slope. This is essential for any claim about 6 vs. 7 months.","section":"§4.4, Figure 6"},{"comment":"The authors acknowledge that different benchmark compositions change model rankings, but this also affects the task-difficulty scale. Because b_i is jointly estimated from all benchmarks, adding or removing a benchmark (or model) can shift all b_i estimates and thus all predicted human times. The paper does not report stability of b_i or of the Eq. (3) calibration under benchmark/model subsampling. This is directly relevant to the assumption that there is a single 'shared' latent scale that transfers across domains. Please add a sensitivity analysis (e.g., bootstrap over tasks/models with refitting, or a jackknife over benchmarks) and show how much the predicted times and the doubling-time estimate change.","section":"§5.3"}],"minor_comments":[{"comment":"The label 'BIRDGE (Ours)' is a typo for 'BRIDGE'.","section":"Figure 3"},{"comment":"The x-axis is labeled 'Task difficulty (b)' and the y-axis is labeled 'Task Length (human time)' on a log scale. Please state explicitly in the caption that both axes are on a log scale, and add units (minutes) to the regression equation.","section":"Figure 2"},{"comment":"The calibration R²=0.81 is reported without confidence intervals or residual diagnostics. Please report the standard errors of slope and intercept and a residual plot, especially because the data include SWAA tasks with very short times.","section":"§4.3"},{"comment":"The exclusion of tasks unsolved by all models is described in a parenthetical. Given its importance to the validation, this should be stated prominently and treated as a limitation in the main body, not only in a footnote-like remark.","section":"§4.1.2, Cybench"},{"comment":"The GDPval LLM-as-a-judge threshold (score ≥ 4) is acknowledged as a proxy. This is fine as a limitation, but the main text currently states that GDPval results are predicted without caveat; please add a caveat in §4.3.","section":"Appendix C.1"},{"comment":"The sentence 'both b_i and θ_j are scale invariant' is imprecise; the model is invariant under affine transformations of the latent scale, as stated later in the same paragraph. Consider rephrasing to avoid confusion.","section":"§2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a credible psychometric extension with a strong in-distribution calibration, but the key transferability claim is currently supported by weak out-of-sample evidence. The authors should be encouraged to add leave-one-benchmark-out and leave-one-suite-out analyses, propagate uncertainty from the IRT and calibration stages, and report continuous-time metrics for SWE-bench. If these additions substantially change the conclusions, a reject decision would be appropriate; otherwise, with the requested revisions, the paper could become acceptable. No concerns about attribution or novelty disclosure; the code repository is a plus."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Tom — quick take on the BRIDGE paper. The genuinely new thing is the task-level calibration: fit 2PL IRT on a large response matrix, regress log human time on difficulty b using METR's 170 annotated tasks, then read off human times for new benchmark tasks from their b values. That works better than I'd have guessed: R²=0.81 on METR, and the transfer to SWE-bench and Cybench, though noisy, beats a logit success-rate baseline and LLM time estimators. The code is up, the comparisons are fair, and the MCMC implementation of IRT is standard. That is a legit methodological contribution.\n\nThe soft spot is the one the stress-test flags: the assumption that one global affine map from b to log time, fit only on METR, transfers to every other benchmark. The motivating argument in Section 3 is not a derivation — 'if capability grows exponentially then b is linear in log time' is an empirical regularity, not a theorem. And the OOD evidence is thin. Cybench ends up with 13 tasks after dropping ones unsolved by all models; R²=0.45 and the '12/13 within 2×' sounds better than it is with n=13. SWE-bench uses coarse buckets: overall accuracy 41.6%, weighted kappa 0.231, and the middle buckets are essentially guessed (18% and 21% per-bucket accuracy). GDPval relies on an LLM judge, so it's not a clean human-time check. The 6-month doubling forecast is just the METR-fit slope applied to IRT ability growth, so it inherits whatever domain-specific offset the calibration has. If CTF tasks or Kaggle-style work have a different human-time/difficulty relationship, the horizon moves.\n\nThe paper is aware of some of this: Section 5.3 concedes that benchmark composition changes model rankings (BRIDGE has Claude 4.5 Opus top, Epoch AI has Gemini 3 Pro top). That is honest, but it also cuts against the 'unified scale' story.\n\nConclusion: the central empirical claim — log-linear b-to-time relation inside METR — holds up. The universal transfer is plausible but unproven. If you work on AI evaluation or capability forecasting, this is worth your time. I'd send this to a serious referee: the idea is important, the execution is competent, and the key weaknesses are fixable with more held-out human-time data (e.g., more Cybench tasks, exact times on SWE-bench, or at least a proper calibration-withheld analysis). The paper should be revised to temper the abstract, report the OOD numbers with confidence intervals, and stop saying 'without human annotations' when it means 'without new human studies per benchmark.' For a reading group, I'd bring it — it's a clean example of how to leverage IRT for evaluation, and the fragility discussion is instructive.","headline":"BRIDGE's core calibration is real and useful, but the universal transfer claim is tested on a handful of tasks and the forecasting doubles down on that weak link.","tokens_in":21257,"tokens_out":3778,"would_cite":true,"duration_ms":35101,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that IRT-estimated task difficulty is log-linearly related to human completion time, so model responses alone can be used to predict human task times and to forecast AI capability growth.","keywords":["Item Response Theory","task difficulty","human completion time","capability forecasting","benchmark transfer","latent difficulty","task-length horizon","exponential growth"],"falsifier":"Take a new domain with expert time annotations, fit the same two-parameter IRT model on model responses, and compare predicted times from the calibration line against actual recorded times; if a class of multi-hour creative tasks is systematically underpredicted, the transfer assumption is false and domain-specific recalibration is required.","tokens_in":20293,"feed_emoji":"⏱️","tokens_out":5444,"duration_ms":54540,"temperature":0.7,"pith_summary":"The paper tries to prove that a task's difficulty for a machine—estimated from binary success/failure of many models—is the same quantity, up to a log-linear rescaling, as the time a human expert needs to finish it. If that holds, expensive human timing studies are needed only once: after calibrating on a set of 170 timed tasks, any new benchmark can be assigned predicted human completion times from model logs alone. The paper validates the transfer on two out-of-distribution benchmarks, then uses the mapping to forecast capability, finding that the task length at which frontier models hit 50% success doubles about every six months. A sympathetic reader would care because this turns benchmark scores into a human-interpretable, forecastable unit without new human annotation.","feed_headline":"Model logs can replace human timing studies","feed_subtitle":"One calibration pass lets new benchmarks get human-time labels, and shows AI task length doubles every ~6 months.","key_machinery":"A two-parameter logistic Item Response Theory (2PL IRT) model, which turns binary model–task outcomes into a latent difficulty per task, a discrimination per task, and an ability per model, together with a linear calibration log(h) = slope·b + intercept connecting latent difficulty b to the logarithm of human completion time h. The calibration resolves IRT's scale ambiguity and is what allows difficulty values from any future benchmark to be read as human minutes.","core_discovery":"The paper claims that the latent difficulty of a task, estimated by a two-parameter logistic Item Response Theory model from binary model success/failure, is log-linearly related to the time a knowledgeable human needs to complete it. After calibrating on a set of 170 tasks with expert time annotations, the paper predicts human completion times for out-of-distribution benchmarks, and uses the same mapping to track frontier model capability over time, finding that the task length at which models succeed 50% of the time doubles roughly every six months. The central relationship is log(h) = slope × b + intercept, with a reported fit of R² = 0.81 on the calibration set.","pith_inferences":["If the log-linear mapping is domain-invariant, one could build a 'predicted person-hours' leaderboard for any benchmark with sufficient model coverage, turning accuracy scores into labour-market units.","The transfer assumption can be tested cheaply: collect expert times for a few dozen tasks in a new domain and check whether the same slope and intercept hold; systematic bias would argue for per-domain calibration curves.","The 6-month doubling time rests on the assumption that model ability grows exponentially; if growth is instead a power law, the horizon forecast would need to be refit, though the ordering of models would likely survive."],"forward_implications":["New benchmarks can receive human-time labels without running human studies, by fitting the same psychometric model to model responses and applying the calibrated line.","Capability can be reported as 'task length solvable at 50% success' instead of raw benchmark scores, making progress easier to compare across domains.","The estimated doubling time of roughly six months, obtained without human annotation, corroborates prior human-timed forecasts, suggesting model-only evaluation can track long-horizon progress.","Predicted task-length distributions can reveal within-benchmark structure, e.g., tasks that only require a valid submission being shorter than those that earn a medal.","Raising the reliability threshold from 50% to 80% success shrinks the solvable horizon to roughly one-third, a contraction that may matter for safe deployment."],"fun_headline_variants":["AI scores predict human task duration","Benchmark data yields human time estimates","Model difficulty scales with human effort","One calibration converts AI logs to human hours","Task length from models doubles every 6 months"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The single global log-linear mapping, learned from 170 timed tasks in one family of benchmarks, transfers unchanged to every other task family, so a one-unit increase in latent difficulty always means the same multiplicative increase in human time.","fun_headline_variants_meta":{"raw":{"variants":["AI scores predict human task duration","Benchmark data yields human time estimates","Model difficulty scales with human effort","One calibration converts AI logs to human hours","Task length from models doubles every 6 months"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000135,"raw_usage":{"total_tokens":942,"prompt_tokens":670,"completion_tokens":272,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":414,"completion_tokens_details":{"reasoning_tokens":210}},"tokens_in":414,"tokens_out":272,"duration_ms":4011,"temperature":1.0,"reasoning_tokens":210,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T03:38:24.683958+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a new domain with expert time annotations, fit the same two-parameter IRT model on model responses, and compare predicted times from the calibration line against actual recorded times; if a class of multi-hour creative tasks is systematically underpredicted, the transfer assumption is false and domain-specific recalibration is required.","supporting_citations":[],"review_version":1}