{"id":"073f32db-1b50-472f-866a-ccacc8e84b35","arxiv_id":"2501.16565","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A Newman-style iterative solver for Plackett-Luce rankings is derived and shown to converge in fewer iterations, but the headline claim that multi-body PL beats pairwise projections rests on an invalid likelihood comparison.","lead":"The paper adapts a faster iterative algorithm for pairwise rankings to the multi-player Plackett-Luce model and applies it to sports, survey, election, and authorship data. It claims the full multi-body model predicts held-out comparisons better than pairwise projections, but that comparison uses an unnormalized model and is not reliable.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The comparative predictive claim is invalid as computed: Eq. (16) defines an unnormalized product, so LpPL−LPL conflates normalization with predictive skill.","rationale":"The reader's verdict identifies the central flaw, and my reading agrees. The manuscript's two advertised contributions are (i) a generalized Newman iteration and (ii) the empirical claim that retaining multi-body structure improves predictive accuracy over pairwise projection. The first contribution is a legitimate algorithmic derivation; the iteration counts in Table I are internally consistent, although the text's '5 to 400' range exceeds the table's maximum 70 and runtime is not measured, which is a lesser concern. The second contribution carries the paper's novelty: the abstract's final sentence is the answer to 'should practitioners keep multi-body comparisons?' That answer is currently computed by comparing LPL with LpPL, and Eq. (16) is an unnormalized product of pairwise factors. Therefore the comparison does not measure predictive skill of two probability models on the same outcome space. This is not a stylistic or consensus disagreement; it is an internal inconsistency in the evaluation protocol. The fix is straightforward—normalize the projected model—and the paper could be revised, but as submitted the central empirical conclusion is not supported. I therefore leave the reader's REJECT verdict unchanged.","tokens_in":12065,"tokens_out":7336,"duration_ms":67857,"concrete_test":"Recompute the held-out comparison with a normalized pPL. For each test hyperedge of size K define Z(π)=Σ_{σ∈S_K} ∏_{i<j} π_{σ_i}/(π_{σ_i}+π_{σ_j}) (using the corrected pair set, no self-pairs) and set P_norm(ω)=product/Z(π). For the synthetic data of Fig. 2(a) (K≤10, so Z is exact by enumeration or importance sampling) evaluate LpPL_norm−LPL on the held-out 20%. If the gap is no longer systematically negative, the claimed superiority of PL is an artifact of normalization. Apply the same normalized evaluation to the seven real datasets in Fig. 3(a); for K=14 in the NS dataset use an unbiased MCMC estimate of Z. This directly tests whether the headline conclusion survives a fair probability comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline empirical claim (abstract: PL trained on multi-body comparisons is more predictive than pPL) is unsupported because the pPL model in Eq. (16) is not a probability distribution. As printed, \\tilde P(ω|π)=∏_{r=1}^{K−1}∏_{t=r}^K π_{ω_r}/(π_{ω_r}+π_{ω_t}) includes self-pair factors; even after the natural correction t=r+1..K, it is a product of K(K−1)/2 pairwise Bradley–Terry factors. For K=3 with equal scores each ordering gets value 1/8, so the six orderings sum to 3/4 rather than 1. Feeding this object into the log-likelihood Eq. (4) makes LpPL a sum over an unnormalized product; LpPL−LPL therefore contains a factor-count term (more pairwise factors for larger K) and an omitted log-normalization constant that depends on π, in addition to any genuine predictive difference. The negative gaps shown in Figs. 2–3 are thus not interpretable as evidence that the multi-body PL outpredicts the projected model. The Newman-iteration contribution (Eq. 13, Table I) is a separate algorithmic claim and is not affected by this flaw, but the abstract's comparative conclusion rests on the invalid likelihood comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper generalizes Newman's accelerated fixed-point iteration from the Bradley-Terry model to the Plackett-Luce model for ordered hypergraphs, derives the corresponding update equations, and tests the resulting solver on synthetic and real datasets. It also performs cross-validated comparisons between the multi-body PL model and a projected pairwise approximation, concluding that the multi-body representation is more predictive for most datasets and that the new solver converges substantially faster than the standard Zermelo scheme.","tokens_in":12315,"tokens_out":19851,"duration_ms":175027,"significance":"The computational contribution is potentially useful: a faster solver for PL ranking inference would benefit a range of applications, and the paper provides derivations and reproducible code and data. The predictive comparison addresses an important modeling question about whether multi-body structure carries information beyond pairwise projections. However, the central empirical claim is not supported by the analysis as presented, because the projected model used in the comparison is not a normalized probability distribution over rankings. The algorithmic speed-up claim is less affected, but the reported speed-up range is inconsistent with the paper's own table.","major_comments":[{"comment":"Equation (16) does not define a probability distribution over ordered hyperedges. As printed it includes self-pair factors t=r; after the natural correction t=r+1, for K=3 with equal scores every ordering has value 1/8 and the six orderings sum to 3/4, not 1. Feeding this unnormalized product into Eq. (4) means that LpPL is not a log-likelihood: the differences LpPL-LPL plotted in Figures 2(a) and 3(a) contain an omitted log-normalization constant that depends on pi and a factor-count term that grows with K(K-1)/2, in addition to any genuine predictive difference. The abstract's claim that a PL model trained on multi-body comparisons is more predictive than a model trained on projected pairwise comparisons is therefore unsupported by the present computations. Equation (17) has the same defect, so Figure 3(b) is affected as well.","section":"Approximating many-body interactions with pairwise interactions; Results: Prediction of unobserved events"},{"comment":"Table I reports speed-up factors between 1.8 and 70 across the listed datasets, but the text following Table I states a '5 to 400' speed-up factor. No dataset in Table I supports the value 400, so the speed-up claim in the Results and in the abstract needs to be corrected or supported by the missing dataset.","section":"Results: Convergence of the algorithms; Table I"},{"comment":"Equation (14) is presented as a model for the full ordered hyperedge, but it is not normalized over the K! orderings of a fixed set of items; for K=3 with equal scores it sums to 2 over the six orderings. If the intended target is only the first position, the paper should state this explicitly and formulate the predictive comparison accordingly. As written, the likelihood comparison for the position 1-breaking variant is subject to the same normalization problem as Eq. (16).","section":"Methods: Position 1-breaking Plackett-Luce model"}],"minor_comments":[{"comment":"The product in Eq. (16) includes self-pair factors with t=r, which are inconsistent with the set of directed edges listed immediately below the equation; the upper limit should likely be t=r+1.","section":"Approximating many-body interactions with pairwise interactions"},{"comment":"The statement 'We do not show any results here, but we systematically verify that the two approaches infer vectors of scores whose components are highly correlated' is odd: either show the correlation analysis or remove the claim.","section":"Results: Prediction of unobserved events"},{"comment":"The axis label 'Log-likelihood' does not state whether values are total log-likelihoods or per-interaction quantities, nor how the raw totals are normalized; this matters because the models have different numbers of factors per hyperedge.","section":"Figures 2 and 3"},{"comment":"Reference [19] is incomplete: it has no title, venue, or year. Reference [21] would also benefit from a journal or proceedings name.","section":"References"},{"comment":"The GitHub repository is mentioned in the conclusions but not in the introduction or methods; a data and code availability statement earlier in the paper would help reproducibility.","section":"Data and code"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is likely salvageable if the predictive comparison is redone with properly normalized models. The normalized pPL model would require computing or appropriately handling the partition function, and the position 1-breaking comparison needs a clear statement of the target variable. If the corrected comparison changes the qualitative conclusions, the paper reduces to an incremental algorithmic contribution, so the significance would need to be reassessed. The speed-up claim also needs to be aligned with Table I. I would ask the authors to address these points before I can recommend acceptance; a re-review after revision is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The algorithmic core is real; the headline empirical claim is not supported as computed.\n\nWhat's new and good: the paper generalizes Newman's (2023) rearrangement of the Bradley-Terry iteration to Plackett-Luce. I traced the algebra from Eq. (6) through Eqs. (9) and (13): splitting the v = r term off the inner sum is legitimate, the resulting fixed points match the Zermelo scheme, and the logistic prior handles disconnected projections the same way Newman's does. The position-1-breaking variant, Eq. (15), appears new. The iteration-count speed-ups in Table I (roughly 2-70x across seven datasets) are consistent and plausible. This is a modest, usable contribution to PL inference.\n\nThe soft spot is load-bearing: the abstract's central claim — multi-body PL predicts held-out comparisons better than projected pairwise pPL — is built on an invalid comparison. Eq. (16) is not a probability distribution. It is a product of pairwise Bradley-Terry factors, one per pair in the ordering, and as printed it even includes the self-pair t = r. Remove the self-pairs and the six orderings of three equal-strength items still sum to 3/4, not 1. So L_pPL − L_PL in Figures 2-3 mixes a missing log-normalization constant, a different factor count per hyperedge (K(K−1)/2 vs K−1), and any genuine predictive difference. The negative gaps are not interpretable as PL beating pPL. This isn't a detail; it's the paper's main conclusion.\n\nTwo smaller issues. The text claims a \"5 to 400\" speed-up, but Table I tops out at 70 and one dataset shows 1.8. And the speed-up is measured in iterations, not wall-clock time; the claim should be qualified accordingly.\n\nWho is this for: anyone implementing PL inference at scale will want the Newman iteration, and the paper is a good cautionary example of comparing unnormalized likelihoods. A revision that normalizes the pPL model (or compares proper scores) could salvage the empirical half. As printed, the empirical conclusion needs correction, but the algorithmic half deserves referee time.\n\nRecommendation: send to peer review with the normalization flaw flagged as the main issue. A serious referee can verify the iteration quickly and push the authors to fix the comparison.","headline":"The generalized Newman iteration for Plackett-Luce is a genuine algorithmic contribution, but the headline predictive claim is invalid because the projected pairwise model of Eq. (16) is unnormalized, so the log-likelihood comparison in Figures 2–3 does not measure what it claims.","tokens_in":12809,"tokens_out":7832,"would_cite":true,"duration_ms":68698,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that keeping multi-body comparisons, rather than projecting them to pairwise ones, yields more predictive ranking models and that a generalized Newman iteration converges 5 to 400 times faster than the standard Zermelo…","keywords":["Plackett-Luce model","multi-body comparisons","hypergraph","ranking","Bradley-Terry model","Newman iteration","cross-validation","preference learning"],"falsifier":"Normalize the projected model in Eq. (16) by summing over all orderings of each hyperedge and re-running the cross-validation; if the measured predictive advantage of the multi-body model disappears or reverses, the claimed advantage is an artifact of normalization rather than a genuine property of multi-body structure.","tokens_in":11820,"feed_emoji":"🏆","tokens_out":7109,"duration_ms":62436,"temperature":0.7,"pith_summary":"The paper addresses two practical problems in ranking entities from contests where each comparison can involve more than two participants. It generalizes the recently proposed Newman iteration for the Bradley-Terry model to the Plackett-Luce model for ordered hypergraphs, and shows that the generalized iteration converges substantially faster than the standard Zermelo iteration across synthetic and real datasets. It then compares the predictive power of the Plackett-Luce model trained on the original multi-body comparisons with that of the model trained on a pairwise projection of the same data. The paper finds that the multi-body model is consistently more predictive, except for election data where the two approaches perform similarly. The practical message is that preserving the multi-body structure of competitions, preference surveys, and co-authorship lists leads to better rankings and predictions.","feed_headline":"Multi-body ranking beats pairwise projection in contests","feed_subtitle":"Retaining the full multi-body structure of comparisons improves prediction and speeds up inference.","key_machinery":"The central object is the Plackett-Luce model on ordered, weighted hypergraphs, where the probability of an ordered hyperedge $\\vec{\\omega}=(\\omega_1,\\dots,\\omega_K)$ is $P(\\vec{\\omega}|\\vec{\\pi})=\\prod_{r=1}^{K-1} \\pi_{\\omega_r}/\\sum_{q=r}^K \\pi_{\\omega_q}$, with scores $\\pi_i=e^{s_i}$ normalized so $\\prod_i \\pi_i=1$. The argument is carried by a rearrangement of the stationarity equations that separates the 'self' term from the 'others' terms in each summand, leading to the new iterative update of Eq. (13). The update is accelerated by including the logistic prior of Eq. (10), which regularizes the scores and guarantees convergence without requiring the projected graph to be strongly connected. The same machinery yields the position-1-breaking variant of Eq. (15) and, with the same high-level steps, the pairwise-projected baseline of Eq. (16).","core_discovery":"The central claim is that a Plackett-Luce model fitted to ordered hyperedges—each hyperedge representing a single multi-body comparison—should not be reduced to a graph of pairwise comparisons before inference. On all datasets considered, including synthetic hypergraphs, football tournaments, sushi preference surveys, course selections, APA elections, and network-science co-authorships, cross-validation shows that the log-likelihood of held-out comparisons is higher for the full Plackett-Luce model than for the projected pairwise model, except in the APA election data where the two are comparable. The paper attributes the election exception to a possibly genuine pairwise structure in that dataset. The paper also claims a computational advantage: its generalized Newman iteration reaches the same convergence criterion in 5 to 400 times fewer iterations than the Zermelo-like iteration.","pith_inferences":["The reported predictive gap could be sensitive to the fact that Eq. (16) is an unnormalized product of pairwise factors; a normalized projection might narrow or change the gap, so the magnitude of the advantage should be checked against that baseline.","The 5–400× speed-up is measured in iteration counts, not wall-clock time; for very large hypergraphs the per-iteration cost of Eq. (13) also matters, but the method should still scale better.","The same algorithmic rearrangement could be applied to other Luce-type choice models, such as the top-1-breaking or reverse Plackett-Luce, and to hypergraphs with weighted or repeated hyperedges, without changing the core identity."],"forward_implications":["The generalized Newman iteration enables Plackett-Luce inference on much larger hypergraphs, since the same accuracy is reached in far fewer iterations.","Multi-body comparison data should be modeled directly with the Plackett-Luce model instead of being collapsed into pairwise edges, because the full model predicts unseen outcomes better.","The position-1-breaking model, which focuses on the first element of each comparison, is at least as predictive as its pairwise projection for all datasets, and more predictive for the football datasets.","The cross-validation framework gives a test for whether a system's multi-body structure is authentic: for APA elections the multi-body and pairwise models perform equally, suggesting the data are effectively pairwise.","Top-10 rankings generated from the inferred scores provide a direct practical output, with plausible historical leaders in football and recognized senior network scientists."],"supporting_citations":[{"why":"Supplies the original Zermelo iteration (Eq. 12) that the paper generalizes and benchmarks against.","marker":"[7]"},{"why":"Provides the efficient iteration for the Bradley-Terry model and the logistic prior that the paper extends to Plackett-Luce.","marker":"[8]"},{"why":"Introduces the Plackett-Luce model for multi-body comparisons that is the paper's core object.","marker":"[16]"},{"why":"Defines the 'full breaking' and 'position-1 breaking' graph projections used to construct the pairwise baselines (pPL).","marker":"[19]"},{"why":"Establishes the convergence condition for PL inference via the strong connectivity of the full-breaking graph.","marker":"[18]"},{"why":"Provides the sushi preference datasets used in the cross-validation tests.","marker":"[29]"}],"fun_headline_variants":["Multi-body comparison model speeds up ranking inference","Full comparison structure improves ranking predictions","Keep contests whole: better rankings, faster fits","Multi-body data yields more predictive contest rankings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The predictive comparison assumes that the projected Plackett-Luce model of Eq. (16) is a properly normalized probability distribution over ordered hyperedges, so that its log-likelihood can be fairly compared with the full model's log-likelihood.","fun_headline_variants_meta":{"raw":{"variants":["Multi-body comparison model speeds up ranking inference","Full comparison structure improves ranking predictions","Keep contests whole: better rankings, faster fits","Multi-body data yields more predictive contest rankings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1328,"prompt_tokens":897,"completion_tokens":431,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":377}},"tokens_in":513,"tokens_out":431,"duration_ms":4459,"temperature":1.0,"reasoning_tokens":377,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T12:16:45.897483+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Normalize the projected model in Eq. (16) by summing over all orderings of each hyperedge and re-running the cross-validation; if the measured predictive advantage of the multi-body model disappears or reverses, the claimed advantage is an artifact of normalization rather than a genuine property of multi-body structure.","supporting_citations":[{"cited_title":"Zermelo, Mathematische Zeitschrift 29, 436 (1929)","cited_arxiv_id":null,"evidence_quote":"Supplies the original Zermelo iteration (Eq. 12) that the paper generalizes and benchmarks against."},{"cited_title":"Newman, Journal of Machine Learning Research 24, 1 (2023)","cited_arxiv_id":null,"evidence_quote":"Provides the efficient iteration for the Bradley-Terry model and the logistic prior that the paper extends to Plackett-Luce."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the Plackett-Luce model for multi-body comparisons that is the paper's core object."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the 'full breaking' and 'position-1 breaking' graph projections used to construct the pairwise baselines (pPL)."},{"cited_title":"Maystre and M","cited_arxiv_id":null,"evidence_quote":"Establishes the convergence condition for PL inference via the strong connectivity of the full-breaking graph."},{"cited_title":"Kamishima, in Proceedings of the ninth ACM SIGKDD inter- national conference on Knowledge discovery and data mining (2003) pp","cited_arxiv_id":null,"evidence_quote":"Provides the sushi preference datasets used in the cross-validation tests."}],"review_version":1}