{"id":"77e54d16-85cb-44df-95b5-caf96200d52c","arxiv_id":"2508.17907","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":1,"one_line_summary":"WOMAC ranks experts by their score against an ex-post aggregate of peer predictions, which the authors argue is more reliable and more efficient than scoring against noisy outcomes.","lead":"This paper proposes WOMAC, a deterministic scoring rule for prediction competitions that grades experts against an aggregate of their peers' forecasts rather than against noisy outcomes. It aims to make competitions both less noisy and harder to game.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core claim depends on an identifiable 'best ex-post aggregate' that is closer to latent truth than noisy outcomes, but the abstract neither specifies the noise model nor rules out overfitting the aggregate to the same noise.","rationale":"The reader's verdict is UNVERDICTED based on abstract-only review, and my stress-test agrees that no accept/reject determination can be made. The reader's weakest_assumption points to identifiability of the aggregate and its proximity to latent truth; I share that concern. I add that even if identifiable, the aggregate could overfit the same noisy outcomes it is derived from, and the paper's own admission that incentives are 'challenging to analyze directly' leaves a second load-bearing gap in the claimed incentive compatibility. Both gaps are unaddressable without the full text, which was not available. Therefore the reader's verdict remains unchanged: UNVERDICTED. My agreement is 'partial' because the reader focused on the aggregate assumption, while I also emphasize the incentive-analysis gap and the overfitting risk as distinct failure modes, but these are related rather than independent. The concrete test I propose is a synthetic validation that would directly test the central claim's key condition: whether the ex-post aggregate is closer to truth and yields better rankings than the standard mechanism.","tokens_in":749,"tokens_out":1732,"duration_ms":18742,"concrete_test":"Run a controlled synthetic experiment with K experts of known skill levels, a latent ground truth, and noisy outcomes drawn from a specified noise model. Apply WOMAC exactly as described in the paper, plus the standard scoring mechanism, and compare (i) the distance between the WOMAC aggregate and latent truth versus the distance between noisy outcomes and latent truth, and (ii) rank correlation between WOMAC scores and true skill versus rank correlation for the standard mechanism. Vary noise level and number of experts. If the WOMAC aggregate is not consistently closer to latent truth, or if WOMAC rankings are not better than the standard mechanism's rankings, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"WOMAC's central selling point is that it scores experts against the best ex-post aggregate of peer predictions 'given the noisy outcomes,' rather than against those outcomes directly. For this to work, the aggregate must be a well-defined, computable object that is closer to the latent truth than the raw noisy outcomes. The abstract does not state the noise model, the aggregation procedure, or the sense in which the aggregate is 'best.' Without regularization or a structural noise assumption, fitting an aggregate to the same noisy outcomes can simply reproduce the noise, which would undermine the claimed statistical efficiency and the reliability of WOMAC as a predictor of out-of-sample performance. The abstract also concedes that 'the increased complexity of WOMAC makes it challenging to analyze incentives directly,' so the claimed incentive-compatibility advantage over the standard design is not established. Thus the mechanism's central claim rests on an unstated identification condition: that a unique 'best ex-post aggregate' exists and is closer to truth than the outcomes. This is precisely the kind of assumption that can fail in finite samples, and the abstract provides no evidence that it holds in the empirical demonstrations.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript (arXiv:2508.17907) describes WOMAC, a deterministic mechanism for prediction competitions in which experts are scored against a 'best ex-post aggregate' of peer experts' predictions rather than against the raw, noisy outcomes. The abstract claims that WOMAC is more statistically efficient than the standard design, provides a clearer theoretical foundation for incentive properties, is implementable via a vectorized algorithm, and demonstrates empirically on real-world forecasting datasets that it is a more reliable predictor of experts' out-of-sample performance. The abstract also acknowledges that the mechanism's complexity makes a direct incentive analysis challenging. The review is based on the abstract only, as the full text is not available.","tokens_in":964,"tokens_out":3073,"duration_ms":30211,"significance":"If the claims hold, WOMAC would address a recognized limitation of standard prediction competitions: noise in outcomes allows weaker competitors to win by chance and creates misreporting incentives. The proposed shift from scoring against outcomes to scoring against a derived aggregate is a novel and potentially valuable direction, and the promise of an efficient vectorized implementation indicates attention to practical adoption. However, the significance is conditional on the aggregate being well-defined and closer to the latent truth than the noisy outcomes, and on the incentive properties being rigorously established. The abstract does not provide the necessary detail to verify these conditions, so the current significance assessment is provisional.","major_comments":[{"comment":"The central object, 'the best ex-post aggregate of peer experts' predictions given the noisy outcomes,' is undefined. The abstract does not specify the aggregation procedure, the loss function used to define 'best,' or any noise model for the outcomes. This is load-bearing because the claimed statistical efficiency and out-of-sample reliability depend on the aggregate being identifiable and closer to the latent truth than the original outcomes; without such details, overfitting to the same noise cannot be ruled out.","section":"Abstract"},{"comment":"The abstract concedes that 'the increased complexity of WOMAC makes it challenging to analyze incentives directly,' which stands in tension with the claim that a 'clear theoretical foundation' justifies the mechanism and with the implicit incentive-compatibility advantage over the standard design. The abstract does not state what theoretical properties were actually proven, leaving the incentive claims unsubstantiated.","section":"Abstract"},{"comment":"The empirical claim that WOMAC is 'a more reliable predictor of experts' out-of-sample performance relative to the standard mechanism' is presented without naming the datasets, the evaluation metric, the baseline, or any uncertainty quantification. On its own, the abstract gives the reader no way to assess the magnitude or robustness of the reported advantage, which is central to the paper's contribution.","section":"Abstract"}],"minor_comments":[{"comment":"The mechanism's name 'WOMAC' is introduced as an acronym but the expansion given ('Wisdom of the Most Accurate Crowd') does not directly match the described operation of scoring against a peer aggregate; consider clarifying the terminology.","section":"Abstract"},{"comment":"The abstract mentions 'an efficient vectorized implementation' but does not indicate whether the code will be publicly released; a reproducibility statement or repository link would be helpful.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"This review is based solely on the abstract because the full text was not made available. The recommendation reflects that the abstract's central claims—definition of the aggregate, theoretical guarantees, and empirical superiority—are not yet verifiable. If the full paper contains rigorous definitions, proofs, and experimental details, the concerns raised here may be readily addressed; I recommend obtaining the full text before making a final decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know: this is an abstract-only read, so anything I say is provisional. The idea itself is genuinely interesting. Scoring competitors against the best ex-post aggregate of peer predictions, rather than against noisy realized outcomes, is a clean twist on the standard competition design. If the aggregate can be shown to be closer to latent truth than the raw outcomes, that would be a real improvement: deterministic, less noise in winner selection, and potentially more robust to misreporting. The authors also claim a vectorized implementation and real-data demonstrations, which is more than many theory papers ship. Credit where earned: the problem is well-motivated, the proposed mechanism is novel in my reading, and the direction is worth pursuing.\n\nThe soft spots start with the abstract itself, which is thin on exactly the things that determine whether WOMAC works. The central object—'the best ex-post aggregate of peer experts' predictions given the noisy outcomes'—is not defined. What noise model? What aggregation procedure? What does 'best' mean, and is it uniquely identified? The stress-test note is right that this is the load-bearing assumption. Without a structural model or regularization, fitting an aggregate to the same noisy outcomes risks reproducing that noise, which would undermine the claimed statistical efficiency. The abstract also concedes that incentive compatibility is hard to analyze directly, so the main theoretical advantage over the standard mechanism is not established. These are not fatal objections on their own; they are omitted details. But they are the heart of the paper, and an abstract that skips them leaves the reader unable to judge soundness.\n\nI am not going to manufacture flaws beyond that. The self-referential worry about the aggregate is legitimate but might dissolve in the full text if they use proper scoring rules and a well-posed Bayesian or regularized estimator. The empirical claim about out-of-sample reliability is testable and would be the natural check. Citation pattern I cannot assess from the abstract; no red flags visible.\n\nBottom line: this paper deserves a serious referee. The mechanism is potentially useful for forecasting and ML competitions, and the authors have at least attempted theory, implementation, and experiments. Whether it is correct depends entirely on the full text. If the full version supplies the model, a proof or clear justification for the aggregate, and honest error bars on the empirical results, it could be a solid contribution. I would not cite it from the abstract alone, but I would read the full version and would be happy to referee it myself.","headline":"Abstract-only review: a promising mechanism that cannot be assessed without the full model; the identification of the aggregate is the load-bearing unknown.","tokens_in":1414,"tokens_out":960,"would_cite":false,"duration_ms":11020,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"WOMAC ranks forecasters against a peer aggregate rather than against noisy outcomes, yielding a deterministic competition mechanism that is more statistically efficient and a better predictor of out-of-sample skill.","keywords":["WOMAC","prediction competitions","wisdom of the crowd","forecasting","scoring rules","incentive compatibility","statistical efficiency","mechanism design"],"falsifier":"A simulation in which all experts forecast a latent signal corrupted by substantial independent noise, where the 'best aggregate' is computed by fitting to the same noisy outcomes, could settle the claim: if WOMAC's rankings do not match rankings based on true skill, or if it fails to beat the standard design in out-of-sample prediction, the central claim would be refuted.","tokens_in":581,"feed_emoji":"🎯","tokens_out":2473,"duration_ms":23067,"temperature":0.7,"pith_summary":"The paper introduces WOMAC, a deterministic mechanism for prediction competitions that scores each forecaster against the best ex-post aggregate of all forecasters' predictions, given the noisy outcomes, instead of scoring them directly against those outcomes. The authors argue that the standard design, which scores against noisy outcomes, lets weaker competitors win by chance and incentivizes misreporting. WOMAC is claimed to be more statistically efficient than the standard design in typical settings and a more reliable predictor of experts' out-of-sample performance, as demonstrated on real-world forecasting datasets. The paper also provides a theoretical foundation for the mechanism and an efficient vectorized implementation.","feed_headline":"WOMAC ranks forecasters against peers, not noisy outcomes","feed_subtitle":"Scoring against a peer aggregate filters outcome noise, so competitions more reliably reveal true skill.","key_machinery":"The central object is the 'best ex-post aggregate': a function of the competing predictions and the noisy outcomes that is designed to approximate the latent truth more closely than any single outcome. WOMAC scores each expert by how close their predictions are to this aggregate, rather than to the raw outcomes. The mechanism's theoretical foundation justifies this aggregate as the proper benchmark, and an efficient vectorized implementation makes it practical for real competitions.","core_discovery":"The central claim is that scoring experts against the best ex-post aggregate of peer experts' predictions, conditional on the noisy outcomes, rather than against the noisy outcomes themselves, produces a competition mechanism that is deterministic, more statistically efficient, and a better predictor of out-of-sample performance than the standard design. The authors name this mechanism WOMAC (Wisdom of the Most Accurate Crowd) and argue that it filters out outcome noise by using the crowd's collective information as the benchmark. They support the claim with a theoretical foundation and empirical results on real forecasting datasets showing that WOMAC identifies experts who perform better out of sample than those selected by the standard mechanism.","pith_inferences":["A natural extension, which the paper does not explicitly claim, is that WOMAC's aggregate benchmark could be used directly as a consensus forecast, since it is designed to be closer to the truth than individual outcomes.","Because WOMAC's incentive properties are harder to analyze, one could test whether experts can manipulate the aggregate by coordinating their reports; this remains an open question that the paper does not resolve.","WOMAC's reliance on a well-specified noise model suggests its advantage may shrink when outcome noise is heavy-tailed or non-symmetric; testing on such data would delineate the boundaries of its usefulness."],"forward_implications":["In competitions with noisy outcomes or labels, WOMAC will more reliably select the genuinely best expert than scoring directly against the noisy outcomes.","Because WOMAC is deterministic, it avoids the additional randomness that incentive-compatible scoring rules require, making it more practical for real-world adoption.","WOMAC's improved statistical efficiency means smaller competitions or fewer forecast questions can still identify top performers reliably.","WOMAC applies to any prediction competition with substantial noise, including machine learning benchmarks with imperfect labels."],"supporting_citations":[],"fun_headline_variants":["WOMAC: Score against peers, not noisy labels","Noise-proof competitions: WOMAC benchmarks experts against each other","WOMAC: Crowd-aggregate scoring beats noisy-outcome rankings","WOMAC: Filter outcome noise with a peer aggregate benchmark","WOMAC: Deterministic, efficient prediction competitions via peer scoring"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The mechanism assumes that the best ex-post aggregate of the experts' predictions, computed after seeing the noisy outcomes, is closer to the true underlying value than the noisy outcomes themselves.","fun_headline_variants_meta":{"raw":{"variants":["WOMAC: Score against peers, not noisy labels","Noise-proof competitions: WOMAC benchmarks experts against each other","WOMAC: Crowd-aggregate scoring beats noisy-outcome rankings","WOMAC: Filter outcome noise with a peer aggregate benchmark","WOMAC: Deterministic, efficient prediction competitions via peer scoring"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000267,"raw_usage":{"total_tokens":1602,"prompt_tokens":921,"completion_tokens":681,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":592}},"tokens_in":537,"tokens_out":681,"duration_ms":7045,"temperature":1.0,"reasoning_tokens":592,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:58:45.299994+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A simulation in which all experts forecast a latent signal corrupted by substantial independent noise, where the 'best aggregate' is computed by fitting to the same noisy outcomes, could settle the claim: if WOMAC's rankings do not match rankings based on true skill, or if it fails to beat the standard design in out-of-sample prediction, the central claim would be refuted.","supporting_citations":[],"review_version":2}