{"id":"77b6dffb-45f1-4715-b1b9-71dcfb8cb0f2","arxiv_id":"2608.03091","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLM rerankers exhibit position bias at the pairwise, global, and output levels, and these consistency failures can worsen even when accuracy and exposure metrics improve.","lead":"Large language model rerankers can change their preferred ordering of the same items when the input order changes, and fixing that position bias is not the same as making rankings consistent. The paper introduces three consistency metrics and shows that a method improving accuracy and flattening exposure still leaves preferences unstable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"STELLA_LW's unseeded internal shuffles may confound the counterexample: its poor PPI/GPI/LOC could reflect algorithmic stochasticity rather than position-bias-induced inconsistency.","rationale":"The reader identified sampling noise from M=20 permutations as the weakest assumption. My concern is related but more specific: STELLA_LW (and SGS) use internal random reshuffling, and the paper does not establish that these are seeded or otherwise controlled. Consequently, the 20 rankings per query may not be deterministic functions of the input serialization. Since the central claim rests on STELLA_LW's combination of flat exposure and poor consistency, uncontrolled stochasticity could manufacture that pattern. This is a concrete, fixable confound rather than a claim that the framework is invalid. A single experiment, rerunning STELLA_LW with fixed seeds on the same permutation, would determine whether the concern lands. I therefore retain the existing CONDITIONAL verdict, with the added condition that the authors either fix all internal random seeds or report within-permutation variance and demonstrate that the Table 1 ordering persists after controlling for algorithmic randomness. The paper is otherwise clear and reproducible, and the framework is valuable; the condition does not overturn the theoretical contribution, but it does gate the empirical headline claim.","tokens_in":8906,"tokens_out":8202,"duration_ms":79009,"concrete_test":"Run STELLA_LW on the same input permutation with 20 different random seeds for its internal 'shuffled candidate orders' and compute the within-permutation LOC (average pairwise agreement across the 20 stochastic outputs). If that within-permutation LOC is not near 1 (e.g., below 0.8), or if its variance is comparable to the between-permutation LOC gap in Table 1, the reported consistency metrics are contaminated. Then repeat the full Table 1 evaluation with all internal shuffles fixed across the 20 evaluation permutations; if STELLA_LW is no longer worst on PPI/GPI/LOC, the central counterexample collapses.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central counterexample is STELLA_LW: it flattens marginal exposure yet has the highest PPI/GPI and lowest LOC. This is the load-bearing evidence that exposure correction is insufficient for ranking-function validity. However, STELLA_LW's inference procedure injects randomness that is not controlled. Section 4 describes STELLA_LW as updating a posterior 'using the input position of the model's top-ranked candidate over the original and shuffled candidate orders,' and SGS likewise 'reshuffles the remaining candidates.' The paper asserts that 'variation across evaluation rankings is induced by candidate permutation rather than decoding randomness or formatting errors,' but this statement covers token-log-probability extraction, not the internal shuffles of these baseline methods. If the internal shuffles are not generated with fixed seeds per evaluation permutation, then each of the M=20 rankings for STELLA_LW may be drawn from a stochastic process. Within-permutation noise would inflate PPI and GPI and deflate LOC, precisely the pattern observed. The reported consistency metrics would then measure a mixture of position sensitivity and algorithmic randomness, and the counterexample would not establish that reducing exposure skew is insufficient for stable preferences. The paper does not report seed handling or within-permutation variance, leaving this confound unresolved.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies candidate-order sensitivity of decoder-only LLM listwise rerankers for recommendation. Treating rankings produced under M=20 random permutations of a candidate set as observations of an induced preference system, it introduces three metrics: Pairwise Preference Instability (PPI), Global Preference Inconsistency (GPI), and Listwise Output Consistency (LOC). Experiments with three LLMs (Llama-3.2-3B-Instruct, Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct), two datasets (MovieLens-32M, Amazon Books), and list lengths K=15/25/50 compare zero-shot reranking against three baselines: bootstrapping, SGS, and STELLA_LW. The main finding is that PPI, GPI, and LOC are closely aligned with one another but can diverge from recommendation effectiveness and from marginal position-exposure bias: STELLA_LW obtains the highest HR@5 while exhibiting the worst PPI, GPI, and LOC, and flatter exposure curves. The authors conclude that reducing marginal exposure skew is insufficient to establish ranking-function validity in LLM-based reranking.","tokens_in":9112,"tokens_out":6444,"duration_ms":57393,"significance":"If the measurements hold, the paper makes a useful contribution to evaluating LLM-based rerankers. It shifts the evaluation criterion from output-level agreement to the structure of the induced preference system, and the observed divergence between effectiveness, exposure, and consistency is a non-obvious empirical result. The three metrics are defined directly from ranked outputs with no fitted parameters, the code is released, and the experimental design covers multiple models, datasets, and list lengths. The central claim is falsifiable and the framework is likely to be reused by the community. However, the strength of the conclusion depends on the reliability of the measurements, which currently lack uncertainty quantification and careful control of stochasticity in the baselines.","major_comments":[{"comment":"The paper reports a single point estimate for every metric and never reports variance, confidence intervals, or significance tests. With M=20 sampled permutations per query and the number of evaluation queries not stated in the paper, the PPI, GPI, and LOC estimates may be substantially affected by sampling noise; in Table 1 several effectiveness differences are small (e.g., nDCG@5 of 0.2301 vs. 0.2324 for Llama-3B on MovieLens-32M). The central counterexample (STELLA_LW's high PPI/GPI and low LOC) and the claimed alignment of the three consistency measures need per-query standard errors, bootstrap confidence intervals, or paired significance tests before the method ordering can be trusted.","section":"Section 4, Table 1"},{"comment":"The statement that \"variation across evaluation rankings is induced by candidate permutation rather than decoding randomness or formatting errors\" covers the extraction of token log-probabilities, but STELLA_LW and SGS are described as using internal reshuffles: STELLA_LW updates a posterior using \"the input position of the model's top-ranked candidate over the original and shuffled candidate orders,\" and SGS \"reshuffles the remaining candidates.\" If these internal shuffles are not seeded per evaluation permutation, then repeated evaluation of the same permutation could produce different rankings for STELLA_LW and SGS. Within-permutation algorithmic randomness would inflate PPI and GPI and deflate LOC, precisely the pattern observed for STELLA_LW, and would confound the paper's central counterexample. The authors should report seed handling for all internal shuffles and, ideally, the variance across repeated runs with identical evaluation permutations.","section":"Section 4 (STELLA_LW/SGS)"},{"comment":"GPI is computed from an approximate best-fitting ranking sigma_hat obtained by sorting on aggregate win scores followed by local swaps, because exact Kemeny rank aggregation is NP-hard. The reported value D(sigma_hat) is therefore an upper bound on the true minimum disagreement, and the approximation gap may vary across methods and datasets. The paper provides no validation of the approximation, despite GPI being one of the three metrics claimed to be closely aligned with PPI and LOC. Please validate the greedy approximation on a subset of queries with exact enumeration (e.g., K=15) or with random-restart/ILP-based search, and report the approximation gap.","section":"Section 3.3 (GPI)"}],"minor_comments":[{"comment":"The notation is inconsistent: the text uses \"STELLALW\" while Table 1 and Figure captions use \"STELLA_LW\"; please unify the spelling.","section":"Throughout"},{"comment":"The sentence \"Conditioning on every ordered position pair is sparse because there are K(K-1) possible pairs and only M sampled permutations\" is confusing: the sparsity issue is in the number of observed bucket pairs per candidate pair, not in the number of ordered position pairs. Please rephrase.","section":"Section 3.2"},{"comment":"The paper states that position buckets are \"coarse buckets B, such as head, middle, and tail,\" but never specifies the number of buckets or their boundaries in the experimental setup. Since PPI values depend directly on this choice, the bucket definition should be stated explicitly in Section 4.","section":"Section 4"},{"comment":"\"Remaining settings are provided in the repository\" is insufficient for reproducibility; at minimum, the paper should state the number of evaluation queries, the position bucket boundaries, and the generation/decoding settings used for STELLA_LW's calibration stage.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for RecSys and addresses a timely reliability question for LLM-based rerankers. My main concern is that the load-bearing counterexample (STELLA_LW's high PPI/GPI and low LOC despite flat exposure) may be confounded by unreported stochasticity in the baseline method, and that the absence of uncertainty quantification makes the method ordering difficult to assess. Please require the authors to address these issues, along with validation of the GPI approximation, in the revision. The self-citations to the authors' prior work are relevant and not excessive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"New here is the PPI/GPI/LOC evaluation framework and the observation that a method can have top effectiveness and flat exposure yet the worst preference consistency. That is a genuine contribution. The metrics are defined directly from sampled rankings with no fitted parameters, the experiments span three models and two datasets, and the code is public. The STELLA_LW counterexample is striking and, if it holds, supports the paper's main claim that marginal exposure correction does not guarantee ranking-function validity.\n\nThe soft spots are real. The biggest is the one flagged in the stress test. STELLA_LW's inference procedure uses internal shuffles to update its posterior. The paper says variation across evaluation rankings is 'induced by candidate permutation' but that statement appears to cover only the deterministic token-log-probability extraction, not the internal randomization of STELLA_LW. If the internal shuffles are not seeded per evaluation permutation, then each of the M=20 rankings for STELLA_LW contains algorithmic randomness. That would inflate PPI and deflate LOC on its own, independent of position bias. The paper does not report seed handling, so this confound is unresolved. This matters because STELLA_LW is the load-bearing counterexample.\n\nSecond, there are no error bars, confidence intervals, or significance tests anywhere in the paper. With M=20 permutations and position buckets, the PPI estimates for a given candidate pair can be based on a handful of observations. The consistent method ordering across six settings is suggestive, but we cannot tell if the differences between, say, bootstrapping and SGS are meaningful.\n\nThird, minor: GPI relies on a greedy approximation to Kemeny aggregation. It is probably fine, but it is not validated against exact computation on small instances.\n\nNone of this is unfixable, and the framework itself is a step forward. The paper deserves a serious referee. I would ask the authors to add variance reporting or bootstrapped CIs, to state explicitly how seeds are handled for all baseline internal shuffles (or better, show within-permutation stability for STELLA_LW), and to validate the greedy GPI on a subset of instances. If the STELLA_LW result survives those checks, this becomes a solid short paper. If not, the main claim needs substantial reworking.","headline":"Useful audit framework and a striking counterexample, but the STELLA_LW result may be confounded by unseeded internal shuffles and the paper lacks error bars.","tokens_in":9706,"tokens_out":4050,"would_cite":true,"duration_ms":35446,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"High relevance can hide an LLM reranker's worst order stability.","keywords":["listwise reranking","position bias","permutation invariance","preference consistency","pairwise preference instability","global preference inconsistency","listwise output consistency","large language models"],"falsifier":"Recompute the main table with substantially more permutations per query, or with bootstrap confidence intervals on PPI, GPI, and LOC, and check whether STELLA_LW still ranks worst on consistency and whether the overall method ordering is stable; if the ordering changes, the quantitative conclusions collapse even if the qualitative framework remains.","tokens_in":8652,"feed_emoji":"🎲","tokens_out":4451,"duration_ms":40007,"temperature":0.7,"pith_summary":"The paper argues that position bias in listwise LLM-based rerankers should be evaluated as a failure of the induced preference system, not just as changes in final ranked lists or marginal position exposure. It introduces three measures: pairwise preference instability (PPI), global preference inconsistency (GPI), and listwise output consistency (LOC). Across three decoder-only LLMs, two datasets, and list lengths of 15, 25, and 50, these three measures rank methods the same way, but they diverge from top-ranked relevance and from marginal exposure curves. The key demonstration is that STELLA_LW, a calibration method that flattens exposure and attains the highest HR@5, has the worst PPI, GPI, and LOC in every setting. The paper concludes that reducing marginal exposure skew is insufficient to establish ranking-function validity in LLM-based reranking.","feed_headline":"Best top-5 hits can mask the least stable LLM reranker","feed_subtitle":"New consistency metrics show that flattening position exposure does not restore stable pairwise preferences.","key_machinery":"The framework's central object is the induced preference system $Q$, whose entries $q(i,j)$ record the fraction of sampled permutations in which candidate $i$ is ranked above candidate $j$. On top of $Q$, PPI conditions pairwise preference probabilities on coarse input-position buckets and takes the maximum bucket-to-bucket swing; GPI computes the disagreement between $Q$ and the best-fitting global ranking under a weighted Kemeny objective; and LOC averages Kendall's $\\tau$ across the sampled rankings. Together they characterize candidate-order sensitivity at the pairwise, global, and output levels, and they are the measurement device that separates consistency from effectiveness and exposure.","core_discovery":"The central claim is that a reranker's validity under equivalent candidate permutations is a property of the preference system each serialized ranking induces, and that this property is not implied by relevance or exposure metrics. Treating rankings from $M$ sampled permutations as observations, the paper defines pairwise preference probabilities $q(i,j)$, then measures pairwise instability, global inconsistency via a weighted Kemeny disagreement with the best-fitting ranking, and output consistency via average Kendall's $\\tau$. The experiments show that these three consistency measures give the same method ordering across all settings, while effectiveness and marginal exposure can disagree with them; in particular, STELLA_LW improves top-ranked relevance and flattens exposure yet produces the most unstable and least coherent preferences. The paper therefore establishes that marginal exposure correction alone cannot certify permutation-consistent reranking.","pith_inferences":["Beyond the paper: if the observed alignment of PPI, GPI, and LOC generalizes, a lightweight consistency probe using a handful of permutations on a sample of queries could serve as a guardrail during model selection.","Beyond the paper: a control experiment that ties candidate relevance to near-equal quality would isolate pure position bias from content-driven preference, sharpening what PPI and GPI actually measure.","Beyond the paper: because STELLA_LW's calibration stage uses stochastic probing, its instability could originate in the calibration procedure rather than the base LLM; separating those sources would show where order sensitivity enters."],"forward_implications":["Evaluating LLM rerankers with HR or nDCG alone can miss severe permutation instability; PPI, GPI, and LOC provide a complementary axis.","Marginal position-exposure flattening should not be treated as evidence of permutation invariance.","SGS-style sequential selection can improve all three consistency measures at the cost of multiple sequential inference passes.","Because PPI, GPI, and LOC agree across models and datasets, one of them may suffice as a practical consistency probe.","Deployment decisions should weigh consistency against inference cost and effectiveness together."],"supporting_citations":[{"why":"Supplies the zero-shot LLM reranker and the bootstrapping aggregation baseline that the consistency framework evaluates.","marker":"[13]"},{"why":"Defines the SGS sequential-selection baseline, which produces the strongest consistency in the experiments.","marker":"[2]"},{"why":"Defines STELLA and its listwise adaptation STELLA_LW, the calibration-based method that flattens exposure yet fails the consistency measures.","marker":"[17]"},{"why":"Supplies the graph-based first-stage retriever that constructs the candidate sets used for reranking.","marker":"[12]"},{"why":"Provides the weighted Kemeny rank-aggregation objective underlying the global preference inconsistency measure.","marker":"[6]"},{"why":"Establishes log-probability marker extraction, used to make each reranker's ranking deterministic for a fixed serialized input.","marker":"[28]"},{"why":"Documents permutation self-consistency in listwise ranking, the phenomenon the paper formalizes into an evaluation framework.","marker":"[22]"}],"fun_headline_variants":["Position bias undermines LLM reranker preference stability","Flattening exposure fails to restore LLM reranker coherence","LLM rerankers reveal inconsistent preferences under shuffles","Exposure skew doesn't guarantee stable LLM reranker rankings","New consistency metrics expose LLM reranker position bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The measurement assumption that 20 sampled permutations per query give reliable estimates of pairwise preference probabilities and consistency scores is load-bearing, and the paper reports no variance, confidence intervals, or significance tests around its method ordering.","fun_headline_variants_meta":{"raw":{"variants":["Position bias undermines LLM reranker preference stability","Flattening exposure fails to restore LLM reranker coherence","LLM rerankers reveal inconsistent preferences under shuffles","Exposure skew doesn't guarantee stable LLM reranker rankings","New consistency metrics expose LLM reranker position bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000155,"raw_usage":{"total_tokens":1205,"prompt_tokens":930,"completion_tokens":275,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":194}},"tokens_in":546,"tokens_out":275,"duration_ms":3147,"temperature":1.0,"reasoning_tokens":194,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:52:28.193364+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the main table with substantially more permutations per query, or with bootstrap confidence intervals on PPI, GPI, and LOC, and check whether STELLA_LW still ranks worst on consistency and whether the overall method ordering is stable; if the ordering changes, the quantitative conclusions collapse even if the qualitative framework remains.","supporting_citations":[],"review_version":2}