{"id":"edc0e700-71ac-471f-aaaa-043c975a1a60","arxiv_id":"2608.02455","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Aggregate-then-Calibrate projects model scores onto a human-derived consensus ranking, claiming theoretical guarantees over model-only assessment; the central proofs have important gaps.","lead":"The paper combines human judgment rankings with machine model scores: first it turns people's pairwise comparisons into a consensus order, then it adjusts the model's numbers to respect that order. It claims guaranteed improvements over using the model alone, but several core proofs are incomplete.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.12's δ1 bound conflates the Stage-1 MLE covariance around s with the s*−ẽ covariance; the omitted subjective-noise term keeps δ1 from vanishing, so the high-probability improvement claim has no asymptotic basis.","rationale":"The reader's verdict is REJECT, and my pass converges on the same conclusion via a more specific internal inconsistency. Theorem 3.12 is the load-bearing claim advertised in the title and abstract. Its proof requires an event about the ranking of the subjective target ẽ, but the only covariance lemma supplied for Stage-1 (Lemma 3.4) describes the MLE's distribution around the ground truth s. The proof implicitly replaces Cov(s*−s) with Cov(s*−ẽ) without adding the σ̃²I term from ε̃. That substitution is not a technicality: it changes the asymptotic behavior. Since ε̃ variance does not shrink with more pairwise comparisons, δ1 cannot be driven to 0 in the limit unless the subjective noise itself vanishes or the score gaps grow without bound. The theorem's statement that 'both δ1 and δ2 approach 0' as Stage-1 sample size grows is therefore unsupported. The second issue—the missing sp∉M_s condition for strict inequality—is real but secondary. The independence claim between A and B is not the central problem, since a union bound suffices for the claimed high-probability statement. The empirical results may still be interesting, but the central theoretical guarantee, as stated, is not established. The reader's REJECT verdict stands unchallenged; no adjustment is needed.","tokens_in":40397,"tokens_out":7385,"duration_ms":96195,"concrete_test":"Re-derive Corollary 3.9 with η=s*−ẽ=(s*−s)−ε̃, so σ²_Xjk=(e_j−e_k)ᵀ(Σ_{s*}+σ̃²I)(e_j−e_k), and evaluate the exact probabilities in a two-item case s=(0,1), σ̃=1, in the infinite-annotator limit where s*=s. If δ1 and δ2 both remain at Φ(−1/√2)≈0.239, then δ1+δ2 is bounded away from 0 and Theorem 3.12's claim that both decay to 0 as Stage-1 sample size grows is false. This check isolates the missing ε̃ term from all other approximations.","verdict_should_be":"UNCHANGED","load_bearing_attack":"AtC's central optimality claim (Theorem 3.12) asserts ||bs−s||²₂ < ||sp−s||²₂ with probability at least 1−δ1−δ2 and says both δ1,δ2→0 as Stage-1 sample size grows. The proof decomposes failure into A={π(s*)≠π(ẽ)} and B={π(ẽ)≠π(s)}. Corollary 3.9 bounds δ1 by pairwise tail probabilities using σ²_Xjk=(e_j−e_k)ᵀΣ_{s*}(e_j−e_k), where Σ_{s*} is taken from Lemma 3.4. Lemma 3.4 is the asymptotic covariance of the HTM MLE around the true score vector s—the DGP for pairwise comparisons is F(γ_u(s_i−s_j)). But the event A is about the ranking of ẽ=s+ε̃. The relevant error is s*−ẽ=(s*−s)−ε̃, whose covariance is Σ_{s*}+σ̃²I, not Σ_{s*}. Omitting σ̃² understates δ1, and the mistake is not a constant-order correction: as N→∞, s*→s, so P(π(s*)≠π(ẽ))→P(π(s)≠π(ẽ))=δ2. Thus δ1 cannot be driven to 0 by more comparisons while σ̃>0 and gaps are finite; Proposition 3.10's 'σmax/∆min≤ε implies δ1=o(1)' relies on σ_Xjk shrinking with N, which fails when the ε̃ term is included. Consequently Theorem 3.12 does not establish asymptotic outperformance. A second, independent gap: even on A∩B, strict inequality requires sp∉M_s; the theorem states no such condition, and the proof sketch only inserts it at the end.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Aggregate-then-Calibrate (AtC), a two-stage framework for human-centered assessment. Stage 1 fits a heterogeneous Thurstone model to pairwise human judgments and extracts a consensus ranking; Stage 2 projects an arbitrary predictive model's score vector onto the isotonic cone defined by that ranking, with the calibrated output given by Euclidean projection. The manuscript claims three theoretical results: (1) heterogeneous rank aggregation is strictly more efficient than homogeneous aggregation (Theorem 3.6); (2) isotonic calibration enjoys risk bounds even when the consensus ranking is misspecified and the effective noise is biased (Theorem 3.8); and (3) AtC asymptotically outperforms model-only assessment with high probability (Theorem 3.12). Experiments on semi-synthetic and real-world datasets are presented in support of the framework.","tokens_in":40831,"tokens_out":4475,"duration_ms":51583,"significance":"If the theoretical guarantees were valid, the paper would make a useful conceptual contribution: combining ordinal human judgments with metric model scores is a principled idea, and the proposed two-stage pipeline is clean and broadly applicable. The paper also makes a good-faith effort to include robustness bounds, pseudo-code, and empirical evaluation, and it promises code release. However, the two headline theoretical results are not established. Theorem 3.6 compares covariance matrices of estimators that converge to different parameters, so the Loewner comparison does not imply better estimation of the true scores. Theorem 3.12's proof uses a covariance for s*-s where the relevant random variable is s*-ẽ, and the omitted subjective-noise term prevents δ1 from vanishing as the Stage-1 sample size grows; the strict-improvement claim also requires an unstated condition that the model scores violate the true ordering. The optimality guarantee therefore reduces essentially to the Pythagorean property of Euclidean projection once the cone is assumed correct, and the probabilistic control of the cone-recovery event is flawed. The empirical results are suggestive but cannot compensate","major_comments":[{"comment":"Event A={π(s*)≠π(ẽ)} is bounded in Corollary 3.9 using σ²_Xjk=(e_j-e_k)^T Σ_{s*}(e_j-e_k), where Σ_{s*} is the asymptotic covariance of s*−s from Lemma 3.4. But s* estimates s, not ẽ, and the quantity driving A is s*−ẽ = (s*−s)−ε̃. Its covariance is Σ_{s*}+σ̃²I, not Σ_{s*}. This is not a constant-order correction: as N→∞, s*→s, so P(π(s*)≠π(ẽ)) → P(π(s)≠π(ẽ)) = δ2 > 0 whenever σ̃>0 and gaps are finite. Thus Proposition 3.10's claim that δ1=o(1) is false, and Theorem 3.12's assertion that both δ1 and δ2 approach 0 is unsupported. The claimed asymptotic outperformance has no valid basis.","section":"Theorem 3.12 and Corollary 3.9 / Appendix C.3"},{"comment":"The strict inequality ∥ŝ−s∥² < ∥s_p−s∥² is derived from the Pythagorean identity and requires ŝ≠s_p, equivalently s_p∉ĉM. If the model scores already satisfy the consensus ordering, projection is the identity and the inequality becomes equality. The main statement of Theorem 3.12 omits this condition; the appendix restatement inserts 'provided that s_p∉M_{π(s)}' only at the end. Since the condition is not part of the theorem statement, the theorem as stated is false. Moreover, the repaired condition involves the unknown target, which limits the usefulness of the guarantee even after correction.","section":"Theorem 3.12 / Appendix C.3"},{"comment":"The theorem compares Σ_{ŝ_hete} (asymptotic covariance around true s*) with Σ_{ŝ_homo} (asymptotic covariance around pseudo-true s∗). A Loewner comparison of covariance matrices is only meaningful for estimators of the same parameter. Under genuine heterogeneity, s∗≠s* in general, so the sandwich covariance describes concentration around a biased limit; no claim of superior accuracy for estimating s* follows. Appendix A.5 asserts the Loewner inequality 'must hold' by CRLB/White theory, but the CRLB does not apply to a QMLE converging to a different parameter. This is a load-bearing gap: the efficiency guarantee is not established.","section":"Theorem 3.6 / Appendix A.5"}],"minor_comments":[{"comment":"The remark refers to 'Theorem 3.5' when discussing the implications of the robustness theorem; it should refer to Theorem 3.8.","section":"Remark after Theorem 3.8"},{"comment":"The metric labeled MSE is defined with a square root: MSE = sqrt((1/n)Σ(ŝ_i−s_i)²). This is RMSE, not MSE. Please rename or correct the formula for consistency throughout the paper.","section":"Appendix D.3"},{"comment":"The bound uses E[Inv(bπ,eπ)] but eπ is not defined before the theorem. Define eπ := π(ẽ) explicitly. Also, the display contains an asymptotic O(1/n) term inside an expectation bound; using an asymptotic notation inside a probabilistic inequality is informal and should be replaced with explicit constants.","section":"Theorem 3.8"},{"comment":"The notation Σ_{ŝ} is introduced as the covariance of 'the estimation error ŝ−ẽ', but Lemma 3.4 gives the covariance of the HTM MLE around the true score vector s. The notation conflates these two objects and should be clarified, especially since the distinction is load-bearing for the main theorem.","section":"Corollary 3.9"}],"recommendation":"reject","confidential_remarks":"The technical problems are severe and not local. The central asymptotic claim of Theorem 3.12 is false under the paper's own model because δ1 has a positive floor induced by the subjective noise, and Theorem 3.6's efficiency comparison is conceptually invalid because the two estimators converge to different parameters. These are not presentation issues; a substantive reworking of the theoretical core would be needed. The empirical study may still be of interest, but as it stands the manuscript should not be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this one. It proposes Aggregate-then-Calibrate: first aggregate heterogeneous comparative judgments into a consensus ranking via a Thurstone model with annotator-specific precision, then project any model's scores onto that ordering by isotonic regression. That's a clean, modular idea, and the position that ordinal human judgments are more reliable than absolute ratings is sensible. The empirical work is genuinely useful: semi-synthetic Reading Level and real Dots-activity, with ablations that separate scaling from ordinal alignment, and the calibrated outputs do look more robust to model degradation than raw model scores. Credit where due: the paper ships code and data, and the experiments are more thorough than the average submission in this area.\n\nThe problem is the theory. The abstract promises theoretically guaranteed asymptotic outperformance, and Theorem 3.12 is the load-bearing result. As written, it doesn't hold. The proof's δ1 bound uses Σ_{s*}—the covariance of the Stage-1 MLE around s—to control the event that the estimated ranking differs from the ranking of ẽ. But that event is governed by s*−ẽ = (s*−s)−ε̃, whose covariance is Σ_{s*}+σ̃²I, not Σ_{s*}. Omitting the σ̃² term is not a constant-order fix: as N grows, s*→s, so P(π(s*)≠π(ẽ))→P(π(s)≠π(ẽ))=δ2, which is positive whenever ε̃ noise is present and gaps are finite. So δ1 cannot be driven to zero by more comparisons, and the claim that both δ1,δ2→0 is false. On top of that, the strict inequality requires sp∉M_s, a condition that appears only in the proof sketch, not in the theorem statement. These are not cosmetic slips; they break the central guarantee.\n\nThere is also a smaller worry in Theorem 3.6: it compares asymptotic covariance matrices of estimators converging to different parameters (true s* versus the pseudo-true value), so the Loewner comparison does not have the stated meaning, and the proof asserts the dominance rather than deriving it. This one might be reparable, but as written it's not a proof.\n\nNet: the conceptual framing and experiments merit a serious referee, but the paper in this form should not be accepted. The right move is reject and invite a revision where the theory is either corrected or honestly weakened—e.g., a guarantee that holds when σ̃ is negligible or when the model is already close to the ordering.\n\nI'd send it to referees rather than desk reject; the idea is good and the errors are identifiable. Just don't let the abstract overclaim stand.\n\nBest.","headline":"A clean two-stage idea with solid experiments, but the headline optimality theorem has a load-bearing gap and the theory as written should not be accepted.","tokens_in":41310,"tokens_out":2143,"would_cite":false,"duration_ms":22253,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G08","62F07"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that combining a consensus ranking from crowd judgments with a predictive model's scores, via isotonic projection onto that ranking, yields final assessments strictly closer to ground truth than model-only scores, with prob","keywords":["human-centered assessment","rank aggregation","isotonic regression","heterogeneous Thurstone model","model calibration","pairwise comparisons","judgment aggregation","crowdsourcing"],"falsifier":"Construct a synthetic Thurstone dataset with tiny true score gaps and high annotator noise so that δ1 and δ2 are large, and compare MSE of ŝ versus sp against ground truth: the bound predicts AtC can be worse than model-only. A sharper test: take a model whose scores already satisfy the true ordering; Theorem 3.12's inequality becomes equality, so any reported strict improvement in that regime indicates an error in the theorem or its proof.","tokens_in":40231,"feed_emoji":"🎯","tokens_out":5338,"duration_ms":48863,"temperature":0.7,"pith_summary":"This paper addresses assessment tasks where ground truth is unavailable—like rating delivery-route difficulty or paper quality—and where both human judgments and model scores are imperfect: humans are inconsistent across scales, while models are trained on proxy labels. It proposes Aggregate-then-Calibrate (AtC), which first aggregates pairwise comparisons into a consensus ranking using an annotator-reliability-aware rank aggregation model, then calibrates any predictive model's scores by isotonic regression onto that ranking. The central theoretical claim is that, once the consensus ranking is accurate enough, this projection strictly reduces squared error to the latent true scores compared with the model alone, with probability at least 1−δ1−δ2. If correct, the result gives a principled recipe for fusing cheap human ordinal judgments with model cardinal scores in any domain where ground truth is costly or delayed, and it justifies why 'compare, don't score' is the right way to use people in the loop.","feed_headline":"Crowd rankings can pull model scores toward truth","feed_subtitle":"Isotonic projection onto a consensus ordering strictly beats model-only assessment, with high probability.","key_machinery":"The load-bearing object is the isotonic cone M_π̂ = {y : y_{π̂(1)} ≤ … ≤ y_{π̂(n)}} induced by the Stage-1 consensus ranking, together with the Euclidean projection onto it. Stage-1's heterogeneous Thurstone model (pairwise choice probability F(γu(si − sj))) supplies the ranking; Stage-2's projection, implemented by the pool-adjacent-violators algorithm, supplies the calibration. The argument turns on the Pythagorean identity for projection onto a closed convex set: once the true score s lies in the cone, ∥sp−s∥² ≥ ∥sp−ŝ∥² + ∥ŝ−s∥², so any violation of the ordering by the model creates a strict gain. The risk analysis additionally uses the statistical dimension of the isotonic cone (the harm","core_discovery":"On its own terms, the paper's discovery is that human comparative judgments and model scores can be composed so that the ordinal information from people acts as a constraint that provably moves model scores toward the truth. Stage-1 estimates a consensus ranking π̂ by fitting a heterogeneous Thurstone model—each annotator gets a precision γu—and Stage-2 computes the closest vector to the model output in Euclidean distance that respects π̂, via isotonic regression. Theorem 3.12 states that if Stage-1's ranking matches the human target's ranking (event A) and that target preserves the ground-truth ordering (event B), then the calibrated output ŝ is strictly closer to s than the raw model score","pith_inferences":["I would extend the framework to any ordering source, not just human annotators: the same isotonic projection could calibrate an LLM judge's scores to a ranking produced by another model, with the same risk bound holding as long as the ranking-error events are controlled.","The analysis predicts a sharp phase transition in robustness: as pairwise inversions in the consensus ranking cross a threshold set by the expected-inversion term, calibrated performance should collapse (as the experiments show around 500 inversions); a practical rule could stop collecting comparisons once the estimated inversion probability δ1 falls below a target.","The tie-creating behavior of PAV suggests a testable consequence: AtC's Kendall-τ gains should concentrate on discordant pairs adjacent to ordinal violations, and the magnitude of the gain should predict the size of the model's ordering violation.","For deployment, the 'strict' part of Theorem 3.12 requires the model to actually violate the true ordering; if the model is already well-calibrated in ranking, the guarantee degenerates to equality, so AtC's value is highest when model scores carry useful metric information but wrong local order."],"forward_implications":["If the consensus ranking is accurate, AtC strictly improves on the raw model under squared error with high probability; the improvement grows with the model's ordinal violations.","Because Stage-2 only reorders or averages the model's scores, AtC works with any off-the-shelf predictor and needs no retraining or access to ground truth.","Heterogeneous annotator modeling pays off: when annotator reliabilities vary, HTM consensus estimates have strictly smaller asymptotic covariance than homogeneous-model estimates.","Calibration remains controlled under misspecification: risk is bounded by a projection term, a statistical term O(σ̃² log n / n), and a bias term O(‖ν‖² / n), so imperfect rankings and biased models do not catastrophically corrupt the output.","Empirically, on reading-level and dots-counting tasks, AtC beats human-only and model-only assessments on ranking and distributional metrics, and degrades gracefully under image corruptions."],"fun_headline_variants":["Human rankings pull model scores toward truth","Aggregate-then-Calibrate: provable gains from mixing human and machine","Isotonic calibration onto human consensus beats model-only","Heterogeneous human comparisons sharpen model scores","Two-stage AtC: human order plus model scores for robust assessment"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The guarantee collapses if the human-consensus target ẽ is so noisy relative to true score gaps that its ranking disagrees with the ground-truth ranking (large δ2), or—for the strict inequality—if the model's scores already respect the true ordering (sp ∈ cone).","fun_headline_variants_meta":{"raw":{"variants":["Human rankings pull model scores toward truth","Aggregate-then-Calibrate: provable gains from mixing human and machine","Isotonic calibration onto human consensus beats model-only","Heterogeneous human comparisons sharpen model scores","Two-stage AtC: human order plus model scores for robust assessment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00068,"raw_usage":{"total_tokens":2931,"prompt_tokens":754,"completion_tokens":2177,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":2097}},"tokens_in":498,"tokens_out":2177,"duration_ms":13499,"temperature":1.0,"reasoning_tokens":2097,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T06:58:21.266717+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a synthetic Thurstone dataset with tiny true score gaps and high annotator noise so that δ1 and δ2 are large, and compare MSE of ŝ versus sp against ground truth: the bound predicts AtC can be worse than model-only. A sharper test: take a model whose scores already satisfy the true ordering; Theorem 3.12's inequality becomes equality, so any reported strict improvement in that regime indicates an error in the theorem or its proof.","supporting_citations":[],"review_version":1}