{"id":"27dbbaca-0eb2-49bf-901e-c57edd32d99c","arxiv_id":"1908.04628","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"L2P predicts heavy-tailed outcome values by training a pairwise preference classifier and placing each new instance among training instances via a voting scheme.","lead":"This paper introduces L2P, a method that predicts heavy-tailed outcomes such as book sales or art auction prices by learning to compare pairs of items and then placing each new item among previously seen training items. L2P claims to reproduce the outcome distribution more faithfully than standard regression baselines and to predict rare, high-value items more accurately while offering interpretable neighbor comparisons.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed point-accuracy advantage of L2P is not directly measured: the reported AUC is a threshold-ranking metric and KS/EMD are marginal-distribution metrics, so 'consistent outperformance' in accurate prediction is not yet established.","rationale":"The reader's weakest assumption identifies the same gap: no direct error metric is reported. I agree. In good faith, the method is coherent and the paper is transparent about the insufficiency of each individual metric; the explicit random-shuffle caveat is a credit. But the combination of KS/EMD with threshold AUC still does not establish per-instance point accuracy unless calibration is demonstrated, and no calibration or error metric appears anywhere in the paper. Because L2P is a pairwise-ranking method, an AUC advantage is expected rather than decisive. The extrapolation/coarseness observation — predictions are midpoints of adjacent training target values — strengthens the worry that point accuracy for big and rare instances is not actually shown. The verdict should therefore remain conditional: the central claim is plausible and supported by code availability, but it needs a direct error-metric check before the consistent-outperformance claim is accepted as stated.","tokens_in":12289,"tokens_out":7520,"duration_ms":84722,"concrete_test":"Re-run the 5-fold experiments on all three datasets and compute, for L2P and each baseline, per-fold mean absolute error and root mean squared error on log-transformed outcomes, plus mean relative error restricted to the top 10% and top 1% of true outcomes (tail instances). Also report paired differences across folds, e.g., Wilcoxon signed-rank tests, between L2P and each baseline. If L2P is not at least comparable or better on log-scale MAE/RMSE and tail-group errors, the headline claim should be revised to 'L2P improves distribution reproduction and ranking but not demonstrated point accuracy.'","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (abstract; Section 3.4.1 takeaway) is that L2P consistently outperforms competing methods at both reproducing the heavy-tailed outcome distribution and accurately predicting instance values, especially for big and rare instances. The evaluation in Section 3.3, however, never measures point accuracy. The adapted AUC (Eq. 3) tests whether predicted values exceed each actual threshold; it is an ordinal/threshold-ranking measure, not a calibration or error measure. KS, EMD, and Q-Q plots compare marginal distributions, and the paper itself notes (Section 3.3) that a random shuffle of true outcomes would achieve KS=0 and EMD=0. A model could therefore rank well and reproduce the marginal distribution while still making large per-instance errors, such as systematically compressed or shifted values, especially for tail instances. This is not merely a preference for conventional metrics: the placing phase (Algorithm 2) outputs only midpoints of adjacent unique training target values, so predictions are structurally bounded by the training range and are coarse. The robustness experiments (Figure 5) also evaluate L2P only by AUC, so they do not close this gap. No log-scale MAE/RMSE, relative error, or tail-group error is reported for any method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes L2P, a two-stage method for predicting heavy-tailed outcome variables. In the training stage, L2P learns a binary pairwise-preference classifier on concatenated feature pairs labeled by which instance has the larger target value. In the placing stage, it compares a test instance to every training instance, converts the predicted pairwise relations into +/-1 votes over bins sorted by training target values, and predicts the midpoint of the highest-vote bin. The authors claim that L2P consistently outperforms six baselines on three real-world datasets (nonfiction book sales, fiction book sales, and art auction prices) both in reproducing the heavy-tailed distribution and in accurately predicting instance values, especially for 'big and rare' tail instances. A theoretical section argues that the voting stage is a maximum likelihood estimate of the optimal bin, and case studies illustrate model interpretability.","tokens_in":12523,"tokens_out":6595,"duration_ms":62395,"significance":"If the claimed empirical advantage were established, L2P would be a practically useful default for heavy-tailed targets, with the additional benefit of interpretable placements. Strengths of the paper include a clearly specified algorithm, publicly available code, 5-fold cross-validation, and comparison with six methods, including heavy-tailed linear regression and LambdaMART. However, the current evaluation does not measure per-instance point accuracy, so the load-bearing claim of accurate prediction, particularly for tail instances, is not yet demonstrated. The contribution is potentially significant but needs additional evaluation before its central claim can be accepted.","major_comments":[{"comment":"The evaluation does not measure point prediction accuracy, yet 'accurate predictions' is a central claim (abstract; Section 3.4.1 takeaway). The adapted AUC in Eq. (3) is a threshold-ranking metric: it checks whether predicted values exceed each actual threshold, not whether predicted and actual values are close. KS, EMD, and Q-Q plots compare marginal distributions, and the paper itself notes (Section 3.3) that random shuffling of true outcomes would achieve KS=0 and EMD=0. Consequently, a model could rank well and reproduce the marginal distribution while making large per-instance errors, such as systematically compressed predictions, especially in the tail. Please add direct error metrics on the original and log scales (MAE, RMSE, and mean/median absolute relative error), report fold-level means and standard deviations, and break these out for tail groups (e.g., top 1% and 5% of outcomes). Without these, the claim of consistent outperformance in accurate prediction is unsupported.","section":"Section 3.3, Eq. (3); Fig. 3"},{"comment":"The placing phase predicts the midpoint of adjacent unique training target values, so L2P's predictions are structurally bounded by the minimum and maximum observed training values. This matters for the claimed strength on 'big and rare' tail instances: if a held-out tail instance exceeds the training maximum, L2P cannot produce a value that large. The paper does not report how many test instances fall outside the training target range, nor their prediction errors. Please quantify this; if L2P cannot extrapolate beyond the training range, the tail-accuracy claim should be qualified accordingly.","section":"Algorithm 2; Section 3.4.1"},{"comment":"The theoretical justification that voting is a maximum likelihood estimator is a sketch, and the assumptions are not validated for the implemented algorithm. The derivation assumes independent evidence across training instances, a uniform prior over bins, and uniform conditional probability within the right/left region (footnote 4), but the implementation thresholds the pairwise probabilities into hard +1/-1 votes (footnote 3) rather than using p_l and p_r as written. As written, the proof does not establish that the implemented hard-voting rule is MLE. Please either provide a formal derivation for the exact voting rule used or present the MLE argument as an interpretation/heuristic.","section":"Section 2.1"}],"minor_comments":[{"comment":"Algorithm 1 appears to implement the vanilla O(n^2) all-pairs training, but Section 2.2 describes an efficient training procedure using the parameters ns and k; the pseudocode does not include the near-instance sampling or the random sampling of non-near instances. Please add the sampling steps or explicitly label Algorithm 1 as the full training variant.","section":"Section 2.2, Algorithm 1"},{"comment":"The preprocessing description is ambiguous: 'we impose standard scaling on all the columns of the data matrix and the target variable. For all competing methods, we first take the logarithm...' does not make clear whether L2P also receives log-transformed features and target or only the baselines do. Please state explicitly which transformations are applied to L2P and confirm that all methods are compared on the same target scale.","section":"Section 3.3"},{"comment":"The text says methods are grouped into tiers based on the mean and standard deviation of the score, but no confidence intervals or significance tests are reported. Adding fold-level confidence intervals, or significance tests for the new point-error metrics requested above, would make the 'consistent outperformance' claim assessable.","section":"Section 3.4.1, Fig. 3"},{"comment":"The LambdaMART failure analysis is performed on the full dataset, rather than on held-out folds. This is illustrative, but the statement that the failure to preserve neighborhood ranking is the 'root cause' of the poor held-out predictions goes beyond what this experiment establishes.","section":"Section 3.4.1, Fig. 4"},{"comment":"There is a typo: 'LamdaMART' should be 'LambdaMART'.","section":"Section 3.4.1"}],"recommendation":"major_revision","confidential_remarks":"I see no integrity concern in the manuscript. The algorithm is clear and code is provided, but the headline empirical claim is not supported by the reported metrics. Adding direct point-error metrics, especially for tail groups, is essential and within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper proposes L2P, a two-phase method that trains a pairwise comparison classifier and then places new instances among training examples by voting over bins of the target variable. The adaptation of ordinal regression ideas to heavy-tailed prediction is not something I've seen done quite this way, and the presentation is refreshingly clear about how the method works. The placing-by-voting step is simple, interpretable, and O(n) at prediction time, which practitioners will like. The authors also include a robustness analysis showing that the voting stage tolerates a fair amount of classifier error, and they make code and data available. That is real work, and it deserves credit.\n\nThe soft spot is significant and directly hits the paper's main claim. The abstract says L2P outperforms competitors at both reproducing the heavy-tailed distribution and accurately predicting instance values, especially for 'big and rare' instances. But the evaluation never reports a direct point accuracy metric. The adapted AUC is a threshold-ranking measure: it tests whether predicted values exceed each actual threshold, not how close the predictions are. KS, EMD, and Q-Q plots compare marginal distributions, and the paper itself concedes that a random shuffle of true outcomes achieves perfect KS and EMD. So a model that ranks well and reproduces the marginal distribution could still make large per-instance errors—particularly in the tail, where the authors claim the advantage. This matters because L2P's predictions are structurally limited: Algorithm 2 returns the midpoint of a bin, so predictions are coarse and bounded by the training range. The claim of 'accurate predictions' is therefore not yet established.\n\nThere are lesser issues too. The comparisons use only three datasets, all from domains the authors already work in, with no significance tests. The MLE derivation in Section 2.1 is a sketch with assumptions stated but not validated. The efficient training variant's hyperparameters (ns, k) are not reported, so the exact experiments cannot be reproduced from the text alone. None of these are fatal, but together they make the strong empirical claim conditional.\n\nWho is this for? Researchers in robust regression and practitioners in book sales, art pricing, and inventory forecasting will find the method interesting as an interpretable alternative to direct regression. The paper is not a finished proof of superiority, but it is a legitimate proposal. A serious referee should ask for log-scale MAE/RMSE, significance tests, and an analysis of how the binning granularity affects point predictions. With those additions, L2P could be a solid practical contribution. I would not desk-reject it, and I would bring it to a reading group to discuss the evaluation gap, but I would not yet cite it as the established best method for heavy-tailed prediction.","headline":"L2P is a sensible and clearly described two-phase method, but the paper's central accuracy claim is not directly measured; it deserves serious refereeing, not a desk rejection.","tokens_in":806,"tokens_out":798,"would_cite":true,"duration_ms":37300,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that heavy-tailed outcome prediction is best done by learning pairwise comparisons between instances and then voting on where a new instance falls among known outcomes.","keywords":["heavy-tailed distributions","outcome prediction","pairwise preference learning","learning to rank","voting-based placement","book sales prediction","art auction price prediction","interpretable machine learning"],"falsifier":"Run L2P and the baselines on the same three datasets and compute the mean absolute error on log-transformed outcomes. If XGBoost or LambdaMART achieves lower pointwise error while L2P only wins on adapted AUC and distribution distances, the central claim of accurate prediction would be falsified; a sharper version is to compare errors restricted to the top 5% of outcomes.","tokens_in":12049,"feed_emoji":"📊","tokens_out":4560,"duration_ms":44031,"temperature":0.7,"pith_summary":"The paper introduces L2P, a two-phase method for predicting outcomes with heavy-tailed distributions such as book sales and art auction prices. Instead of regressing directly on the outcome value, L2P trains a classifier on pairwise comparisons—which of two instances has the larger outcome—then places a new instance among the training instances by voting and reads off the prediction from the bin with the most votes. The paper argues that this indirect route preserves the heavy-tailed shape of the distribution and avoids the systematic under-prediction of 'big and rare' instances that direct regression suffers. On three real datasets, L2P reports the best combination of distribution reproduction (low KS/EMD), ranking accuracy (high adapted AUC), and interpretability, and remains stable when up to 45% of pairwise relationships are randomly wrong. If correct, this makes pairwise comparison plus placement a stronger default than direct regression when the target is heavy-tailed.","feed_headline":"Pairwise voting beats direct regression on heavy-tailed outcomes","feed_subtitle":"L2P classifies which of two items is larger, then places new instances among known ones to predict sales or prices.","key_machinery":"The central object is the pairwise preference classifier C trained on concatenated feature vectors $[f_i, f_j]$ with labels $+1$ if $t_i > t_j$ and $-1$ otherwise. The placing stage turns this classifier into predictions: bins along the sorted target-value axis receive upvotes and downvotes from each training instance according to the predicted relationship direction, and the midpoint of the bin with the maximum total vote is the prediction. The paper shows this argmax vote count is a maximum likelihood estimate of the optimal bin under a uniform prior over bins and probabilities $p^l_i(q)$ and $p^r_i(q)$, so the mechanism is an ordinal-placement estimator rather than a score regression. The voting step is linear in $n$ and can absorb substantial pairwise classification error.","core_discovery":"L2P claims that for heavy-tailed outcome variables, prediction accuracy and distribution fidelity are better achieved by recasting regression as a placement problem. Training builds a classifier over concatenated feature vectors of instance pairs, labeled by which instance's target value is larger. At test time, the classifier predicts the pairwise relationship between the new instance and every training instance; each training instance votes on all bins left or right of its own target value depending on the predicted relationship; and the bin with the highest vote determines the prediction. The paper argues via a maximum-likelihood calculation that this voting scheme is the optimal placement given the pairwise preferences, and it reports that L2P reaches top-tier AUC while also giving the most faithful heavy-tailed distribution in Q-Q plots and KS/EMD scores across nonfiction, fiction, and art auction data. It does not report a direct pointwise error metric such as mean absolute error.","pith_inferences":["A direct pointwise error comparison, such as mean absolute error on log-transformed outcomes, would sharpen the empirical claim; the reported AUC and distribution distances alone leave open whether L2P's advantage is mainly in ordering.","Because randomly shuffled true outcomes can achieve near-perfect KS and EMD, any heavy-tail method should be benchmarked against random permutation as a lower bound to show that distribution fidelity is not trivial.","A natural extension is a hybrid approach that uses direct regression for mid-tail values and placement voting to protect the tail, or applying L2P to other heavy-tailed targets such as insurance claims or online engagement counts."],"forward_implications":["If L2P is right, practitioners predicting sales, prices, or demand with heavy tails should expect less under-prediction of top instances than from direct regression or XGBoost.","Because placement is derived from pairwise order, the predicted distribution retains the heavy tail and should match Q-Q plots at high quantiles better than methods optimizing only point loss.","The method's robustness means imperfect pairwise classifiers are acceptable: up to about 45% random pair errors leave overall AUC nearly unchanged.","Each prediction comes with interpretable context—the neighboring training instances it was placed between—useful for decisions like publisher advances or artist representation."],"supporting_citations":[{"why":"Supplies the book sales dataset and pre-publication feature set used in the experiments.","marker":"[1]"},{"why":"Supplies the art auction dataset and artist trajectory features used to predict auction prices.","marker":"[11]"},{"why":"Defines the heavy-tailed linear regression baseline (median-of-means) that L2P must outperform.","marker":"[13]"},{"why":"Provides XGBoost, one of the strongest regression baselines compared in the study.","marker":"[14]"},{"why":"Defines LambdaMART, the learning-to-rank baseline whose ranking failure explains the comparison.","marker":"[16]"},{"why":"Provides the two-phase preference-learning and ordering framework that L2P adapts into outcome prediction.","marker":"[10]"},{"why":"Shows ordinal regression via binary classification, the methodological basis L2P generalizes to continuous heavy-tailed outcomes.","marker":"[22]"},{"why":"Ranking method tested as an alternative placement strategy in L2P's placing phase.","marker":"[8]"}],"fun_headline_variants":["Pairwise votes place predictions for heavy-tailed outcomes","Learning to Place: pairwise ranking aids heavy-tail predictions","For big rare outcomes, let pairs vote on prediction placement","L2P: classify pairs, then place new instances to predict","Heavy-tailed prediction via pairwise preference and optimal placement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that high scores on the paper's adapted ranking and distribution-matching metrics really mean accurate prediction, even though the paper notes that randomly shuffled true values can perfectly match the outcome distribution and the ranking metric does not measure error on individual values.","fun_headline_variants_meta":{"raw":{"variants":["Pairwise votes place predictions for heavy-tailed outcomes","Learning to Place: pairwise ranking aids heavy-tail predictions","For big rare outcomes, let pairs vote on prediction placement","L2P: classify pairs, then place new instances to predict","Heavy-tailed prediction via pairwise preference and optimal placement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000303,"raw_usage":{"total_tokens":1732,"prompt_tokens":926,"completion_tokens":806,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":727}},"tokens_in":542,"tokens_out":806,"duration_ms":7743,"temperature":1.0,"reasoning_tokens":727,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:36:40.411703+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run L2P and the baselines on the same three datasets and compute the mean absolute error on log-transformed outcomes. If XGBoost or LambdaMART achieves lower pointwise error while L2P only wins on adapted AUC and distribution distances, the central claim of accurate prediction would be falsified; a sharper version is to compare errors restricted to the top 5% of outcomes.","supporting_citations":[{"cited_title":"Success in books: predicting book sales before publication","cited_arxiv_id":null,"evidence_quote":"Supplies the book sales dataset and pre-publication feature set used in the experiments."},{"cited_title":"Quantifying reputation and success in art","cited_arxiv_id":null,"evidence_quote":"Supplies the art auction dataset and artist trajectory features used to predict auction prices."},{"cited_title":"Heavy-tailed regression with a generalized median-of-means","cited_arxiv_id":null,"evidence_quote":"Defines the heavy-tailed linear regression baseline (median-of-means) that L2P must outperform."},{"cited_title":"From ranknet to lambdarank to lambdamart: An overview","cited_arxiv_id":null,"evidence_quote":"Defines LambdaMART, the learning-to-rank baseline whose ranking failure explains the comparison."},{"cited_title":"Learning to order things","cited_arxiv_id":null,"evidence_quote":"Provides the two-phase preference-learning and ordering framework that L2P adapts into outcome prediction."},{"cited_title":"Ordinal regression by extended binary classification","cited_arxiv_id":null,"evidence_quote":"Shows ordinal regression via binary classification, the methodological basis L2P generalizes to continuous heavy-tailed outcomes."},{"cited_title":"A physical model for efficient ranking in networks","cited_arxiv_id":null,"evidence_quote":"Ranking method tested as an alternative placement strategy in L2P's placing phase."}],"review_version":1}