{"id":"42789528-674a-4377-b9e8-9bfee091f823","arxiv_id":"2607.05046","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"CollabEval turns model evaluation into low-rank matrix completion and uses the imputations as control variates to cut CI width and MSE at fixed annotation budget while preserving unbiasedness.","lead":"CollabEval reuses historical model scores on shared prompts via matrix completion, then treats the imputations as control variates inside prediction-powered inference. The result is unbiased mean estimates and valid confidence intervals that need far fewer new annotations.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified; the reader's weakest-assumption concern is real but non-load-bearing for the central claim.","rationale":"The reader's identification of Assumption A.1 as the weakest formal link is accurate; the paper itself flags that a full concentration argument for IterativeSVD lies outside its scope. Yet the claim that is actually advanced—and that the experiments support—is not “IterativeSVD is stable,” but “the CrossPPI + matrix-completion pipeline yields tighter, still-valid intervals.” Because power-tuning (Prop. A.9) and the rectifier construction protect against poor predictions, and because the empirical coverage and efficiency numbers hold across five diverse matrices and several alternative completers, the missing proof does not undermine the central empirical-theoretical package. No other internal inconsistency, sampling bias, or over-claim appears. Consequently the ACCEPT / HIGH-confidence verdict stands; the concrete check above would simply confirm that the result is not an artifact of one particular completer.","tokens_in":32058,"tokens_out":556,"duration_ms":5394,"concrete_test":"Re-run the entire experimental suite of Section 5, replacing IterativeSVD by the neural-network completer already ablated in Table 3 (or by nuclear-norm minimization). If the average CI-width reduction remains within a few points of the reported 20-30% figures and empirical coverage stays near 90%, the concrete algorithm choice is non-critical and the central claim is robust to the missing stability proof.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (up to ~30% CI-width / MSE reduction at fixed p while preserving asymptotic coverage) rests on two pillars: (1) the CrossPPI control-variate construction that yields unbiasedness and asymptotic normality whenever the completer is stable (Assumption A.1), and (2) the empirical observation that IterativeSVD produces useful control variates on the five real matrices. The paper never proves A.1 for IterativeSVD, only sketches that regularized SVD methods typically concentrate. That is a genuine technical gap. However it is not load-bearing for the claim as stated: Theorem A.5 and Proposition A.9 already guarantee that any stable completer (or even an unstable one that happens to produce uncorrelated predictions) never worsens asymptotic variance relative to the classical mean, and the bootstrap coverage plots (Fig. 2 and per-task appendices) show the intervals attain nominal 90% coverage even at modest p. Thus the unproved stability condition does not threaten either the unbiasedness guarantee or the reported efficiency gains.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes CollabEval, which reframes model evaluation as matrix completion over an M\times N score matrix of models by prompts. For a subset of target models only a fraction p of entries are observed; historical (anchor) models supply dense rows. Cross-fold matrix completion produces imputed scores that are used as control variates inside a CrossPPI estimator, yielding unbiased point estimates of individual means θ_i and pairwise differences Δ_ij together with asymptotically valid confidence intervals. Power-tuning weights are derived so that asymptotic variance is never larger than that of the classical sample mean. Empirically, on five generation benchmarks the method reduces average CI width by up to ~30 % (and MSE correspondingly) at fixed labeling budget while empirical coverage tracks the nominal 90 % level.","tokens_in":32304,"tokens_out":562,"duration_ms":4606,"significance":"The work cleanly marries collaborative filtering with prediction-powered inference for a practically important problem—repeated, expensive evaluation of generative models. The theoretical package (multivariate CLT under a mild stability assumption, consistent covariance estimators, optimal power-tuning that is guaranteed never to inflate variance) is complete and carefully stated; the empirical gains are consistent across diverse tasks and sampling regimes. Because the method never requires generation or labeling of the skipped prompts, the cost reduction is essentially free once historical evaluations exist. The framework is modular: any stable completer can be substituted, and the validity guarantees survive even when the low-rank hypothesis is only approximate.","major_comments":[],"minor_comments":[{"comment":"Assumption A.1 (imputation stability) is stated but not proved for the concrete IterativeSVD algorithm used throughout the experiments. A short remark or reference to concentration results for regularized SVD would strengthen the theoretical narrative, even though the empirical coverage plots already support the claim.","section":null},{"comment":"Figure 2 and the per-task appendix figures would benefit from a brief note on the number of bootstrap trials (stated as 10 k) and whether the shaded min/max bands are across datasets or across bootstrap replicates.","section":null},{"comment":"In §4.3 the rewritten control-variate form of the estimator (Eq. 3) is convenient, but a one-sentence reminder that it is asymptotically equivalent to the classical PPI form under optimal scaling would help readers coming from the PPI literature.","section":null},{"comment":"Appendix D.2 compares four completers; a short discussion of wall-clock cost relative to the evaluation budget itself would be useful for practitioners deciding whether to adopt the method.","section":null}],"recommendation":"accept","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is simple: if you already have dense historical scores from related models, you can skip a large fraction of prompts for new models, impute the missing scores via low-rank completion, and feed those imputations into CrossPPI control variates. You get unbiased means and pairwise differences plus asymptotically valid CIs, and on five real benchmarks the intervals shrink by up to ~30 % at the same labeling budget.\n\nWhat is new is the packaging and the multivariate theory. They treat the whole M-by-N score matrix as a collaborative-filtering problem, derive the joint CLT for the vector of target means, give the optimal power-tuning weights for both pointwise and difference estimands, and show that the asymptotic variance is never worse than the classical sample mean. The empirical section is thorough: AlpacaEval, MMLU, AQA, WMT24++, SWE-bench; IID vs paired sampling; leave-one-out and leave-two-out protocols; ablations on rank schedule, folds, and anchor-set composition. Coverage tracks the nominal 90 % even at modest p, and the method beats naive imputation, Anchor Points, and a simple anchor-mean PPI baseline.\n\nThe soft spot the reader flagged is real but non-load-bearing. Assumption A.1 (imputation stability) is not proved for the concrete IterativeSVD algorithm; they only sketch that regularized SVD methods typically concentrate. That is a genuine technical gap. It does not threaten the central claim, because unbiasedness and the “never-worse-than-classical” guarantee hold for any stable completer (or even an uncorrelated one), and the bootstrap coverage plots already look good. Free parameters (rank schedule, K, power-tuning) are standard and ablated. Citation pattern is clean; no circularity.\n\nThis is for anyone who runs repeated leaderboards or regression tests and already has historical model scores. It is not a theory paper that rewrites foundations, but it is a solid, immediately usable contribution. I would send it to peer review without hesitation and would cite it the next time I need cheaper evaluation numbers.","headline":"Clean, usable combination of matrix completion and CrossPPI that actually cuts evaluation cost on real generative-model benchmarks while keeping coverage.","tokens_in":32896,"tokens_out":522,"would_cite":true,"duration_ms":5388,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Historical model scores let you evaluate new models on fewer prompts while keeping unbiased estimates and valid confidence intervals.","keywords":["model evaluation","matrix completion","prediction-powered inference","control variates","confidence intervals","collaborative filtering","statistical efficiency"],"falsifier":"On a new benchmark where the score matrix has no low-rank structure (so that completed entries are essentially uncorrelated with true target scores), measure whether CollabEval’s confidence-interval widths remain no larger than the classical baseline and whether empirical coverage still meets the nominal rate; any systematic widening or under-coverage would falsify the claim.","tokens_in":32982,"feed_emoji":"📊","tokens_out":589,"duration_ms":5480,"temperature":0.7,"pith_summary":"Evaluating generative models is expensive because each new model still needs many prompts scored. CollabEval treats the collection of model-by-prompt scores as a partially observed matrix. Historical “anchor” models supply dense rows; new “target” models supply only a sparse fraction p of labels. A cross-fold matrix-completion step imputes the missing entries; those imputations are then used as control variates inside a prediction-powered estimator. The resulting point estimates of mean performance (and of pairwise differences) remain unbiased, and the accompanying confidence intervals are asymptotically valid, even when the completion is imperfect. Across five real benchmarks the method shrinks average interval widths by up to 30 percent relative to classical sampling at the same annotation budget, effectively converting past evaluations into extra statistical power.","feed_headline":"Reuse old model scores to cut new evaluation labels by ~30%","feed_subtitle":"Matrix completion plus control variates keeps estimates unbiased and intervals valid","key_machinery":"Cross-fold matrix completion (IterativeSVD by default) produces predictions that are independent of the held-out labels; those predictions enter a power-tuned CrossPPI control-variate estimator whose asymptotic covariance is given explicitly in terms of the true and imputed scores, guaranteeing both unbiasedness and valid normal-based confidence intervals.","core_discovery":"By casting multi-model evaluation as low-rank matrix completion and feeding the completed entries into a cross-prediction-powered control-variate estimator, one obtains unbiased estimates of both individual model means and pairwise differences whose asymptotic variance is never worse than the classical sample mean and is typically substantially smaller, together with confidence intervals that retain valid coverage under mild stability conditions on the completion algorithm.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Reuse past scores via matrix completion to cut eval labels","CollabEval fills gaps for unbiased cheaper model testing","Low-rank completion shrinks new model eval sample needs","Control variates from filled scores tighten eval intervals","Historical multi-model scores cut annotation budget"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The matrix-completion algorithm must become stable: as more data arrive, its cross-fold predictions must converge to a fixed function of only the anchor-model scores on each prompt.","fun_headline_variants_meta":{"raw":{"variants":["Reuse past scores via matrix completion to cut eval labels","CollabEval fills gaps for unbiased cheaper model testing","Low-rank completion shrinks new model eval sample needs","Control variates from filled scores tighten eval intervals","Historical multi-model scores cut annotation budget"]},"model":"grok-4.5","effort":"low","cost_usd":0.003944,"raw_usage":{"total_tokens":1238,"prompt_tokens":775,"num_sources_used":0,"completion_tokens":57,"cost_in_usd_ticks":39440000,"prompt_tokens_details":{"text_tokens":775,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":406,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":775,"tokens_out":57,"duration_ms":4233,"temperature":1.0,"reasoning_tokens":406,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T09:38:15.306258+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On a new benchmark where the score matrix has no low-rank structure (so that completed entries are essentially uncorrelated with true target scores), measure whether CollabEval’s confidence-interval widths remain no larger than the classical baseline and whether empirical coverage still meets the nominal rate; any systematic widening or under-coverage would falsify the claim.","supporting_citations":[],"review_version":1}