{"id":"16ffda3c-cd96-4166-8ecb-d7e06d18019e","arxiv_id":"2502.08662","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"RoToR makes a frozen LLM order-invariant by circularly rotating a single global sort of segment position IDs, and Selective Routing combines it with the original model for mixed lists.","lead":"This paper presents RoToR, a zero-shot method that helps language models treat list-like inputs (retrieved documents, table rows, answer choices) as unordered by reassigning position IDs with a single circular global sort. It also adds Selective Routing, which switches between the order-invariant model and the original model based on confidence, improving robustness on MMLU and other benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Exact order-invariance is not established: RoToR's own shuffle rows show nonzero seed variance despite deterministic canonical sorting, and the paired t-test used to claim robustness is invalid.","rationale":"The paper has real strengths: RoToR is parameter-free and zero-shot, the efficiency analysis against PINE is concrete, and the LitM/KGQA improvements are consistent across several backbones. The MMLU overclaim identified by the reader is secondary to a deeper problem with the central claim. The method's own data show that the model's output is not invariant under shuffling, even though a deterministic canonical global sort should make it exactly invariant if the mechanism works as advertised. This discrepancy suggests that the actual implementation still depends on physical token order, so the 'guarantee' in Sec. 3.2 is unsupported. The paired t-test in Appendix I does not repair this: it pools heterogeneous tasks and models, uses averaged scores as single observations, and lands just below the conventional threshold, while the MMLU rows alone show the same reordering drop as the original model. A single token-level reproduction with shuffled permutations would settle whether exact invariance holds. Because the contribution may still be valuable as a variance-reduction technique, the appropriate outcome is conditional acceptance pending this check and a corrected statistical analysis.","tokens_in":24412,"tokens_out":12429,"duration_ms":119840,"concrete_test":"Pick one fixed Mintaka query and its 30 retrieved facts. Run RoToR-lexical with greedy decoding and a fixed seed on the initial order and on three shuffled permutations of those facts, and compare the generated output token-by-token. If the four outputs are not identical, RoToR is not exactly order-invariant, which would explain the nonzero seed standard deviations in Table 2 and contradict the Sec. 3.2 'guarantee' claim; the Sec. 5 shuffle-robustness conclusion would then need a properly paired, per-instance statistical test on a single benchmark.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"RoToR's central contribution is that one global sort plus circular position-ID assignment makes a frozen LM order-invariant (Sec. 3.2, 'our method guarantee invariance with a single forward pass'). If this were literally true, the method would canonicalize the input: for a fixed set of segments, every permutation maps to the same sorted sequence/ID assignment, so inference is deterministic and outputs must be identical across shuffled seeds. The paper's own numbers contradict this. In Table 2, RoToR rows report nonzero seed standard deviations (e.g., Llama-3.1-8B N=30: 0.05/0.05/0.02; Qwen1.5-7B N=50: 0.06/0.05/0.04), and Table 7 lists nonzero before-after differences (0.3, 0.1, 0.2, 0.1, 0, 0, 2.8, 1.1, 1.3). If segments were truly reordered (or if position IDs fully determined the attention pattern), these values would be exactly zero. The likely explanation is that RoToR reassigns position IDs while leaving physical token order and the causal mask intact, so the effective attention pattern still depends on the input permutation; invariance is therefore approximate, not guaranteed. The statistical test meant to support robustness is not probative: Table 8 pools nine heterogeneous observations (Mintaka at two ndoc values and MMLU), uses seed-averaged scores as single points, and gets t=2.23, df=8, p≈0.056, just below the 2.306 critical value; moreover the MMLU row shows RoToR+SR dropping 2.8 points after reordering, the same as the original model. Thus neither exact invariance nor even the claimed 'no significant difference' is established. The paper should either demonstrate token-level invariance or weaken the claim from 'guarantee' to 'reduce variance'.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes RoToR, a zero-shot method for making decoder-only LMs more order-invariant on listwise inputs. RoToR globally sorts input segments (e.g., lexically) and assigns position IDs via a circular arrangement, so that each segment can appear 'last' from its own perspective without per-query attention-based reordering. The paper also proposes Selective Routing, which routes between the original model and the invariant model based on maximum token probability plus a tuned bias term, aiming to handle benchmarks that mix order-invariant and order-sensitive inputs. Experiments cover Lost in the Middle (LitM), Mintaka KGQA, and MMLU, with baselines including PCW, Set-Based Prompting, PINE, and the original model, plus analyses of efficiency, perplexity, collision rate, long-context inputs, and template robustness.","tokens_in":24804,"tokens_out":7624,"duration_ms":63928,"significance":"If the central claim held, the contribution would be practical: RoToR is training-free, zero-shot, and avoids PINE's O(n^2 d) attention-computation overhead, using only a single global sort with O(k log k) cost; the efficiency comparison in Table 4 is detailed and useful. The paper also has several strengths: the deterministic global-sort mechanism has no fitted parameters; the only tuned quantity (alpha=0.2) is disclosed and selected on a validation split; the Oracle upper bound is explicitly labeled; and the code is released. However, the exact-invariance claim is contradicted by the paper's own shuffle-variance numbers, and the statistical support for robustness is methodologically weak. The empirical results are mostly favorable, but the current version overstates what is established, so the claims need to be revised or the experiments need to be strengthened.","major_comments":[{"comment":"The claim that RoToR 'guarantee[s] invariance with a single forward pass' (§3.2; also §2.2) is not supported by the reported data. Table 2 gives nonzero seed-to-seed standard deviations for RoToR (e.g., Llama-3.1-8B-Instruct, N=30: 0.05/0.05/0.02; Qwen1.5-7B-Chat, N=50: 0.06/0.05/0.04), and Table 7 lists before/after-shuffle differences for RoToR, including 2.8 points on MMLU with Selective Routing. If the global sort plus circular position assignment canonicalized the input, all shuffled runs would be identical and these numbers would be zero. Since the method as described reassigns position IDs while tokens retain their original physical order (Fig. 2 caption), the causal mask still depends on the input permutation, so invariance is at best approximate. Please either physically reorder segments to make the output exactly permutation-invariant, or revise the invariance claim and the related significance statements to describe approximate invariance.","section":"§3.2 and Tables 2, 7"},{"comment":"The robustness argument in Sec. 5 relies on a paired t-test that pools nine heterogeneous observations (Mintaka at two ndoc values for three backbones and MMLU for three backbones), uses seed-averaged scores as single points, and ignores within-condition variance. For RoToR the test gives t=2.23, df=8, p≈0.056, which is non-significant at α=0.05, but a non-significant result in this pooled design is not evidence of order-invariance; moreover Table 7 shows RoToR+SR dropping 2.8 points on MMLU, the same as the original model. Report per-condition confidence intervals, effect sizes, or a model that accounts for the repeated-measures structure, and do not claim statistical evidence of robustness from this table.","section":"Appendix I; Sec. 5"},{"comment":"The caption of Table 1 states that RoToR 'shows the best performance across all setups,' and Sec. 5 concludes that 'any type of global sorting, with the use of circular assignment is superior than PINE.' Both statements are contradicted by Table 1 for Qwen1.5-4B-Chat with ndoc=30: e.g., at gold index 0, PINE scores 46.4 while RoToR-lexical scores 44.6, and PINE is also ahead at most other gold positions for that configuration. Please qualify the claim to the specific sorting variants and model sizes where it actually holds.","section":"Table 1 and Sec. 5"}],"minor_comments":[{"comment":"The heading 'Statisticial significance' contains a typo; it should be 'Statistical significance.'","section":"Appendix I"},{"comment":"The method name is inconsistent: the title and abstract use 'RoToR' while many tables and appendix headings use 'ROTOR.' Please standardize the notation.","section":"Throughout"},{"comment":"The sentence 'We Except for the experiments on Llama-3.1-70B-Instruct, we only use a single A6000 GPU for all of the experiments' is ungrammatical and should be rewritten.","section":"Appendix F"},{"comment":"The word 'consistantly' is a typo for 'consistently' in the sentence about LongBench-2WikiMultihopQA.","section":"Appendix J"},{"comment":"The panel (a) label 'Overhead FLOPs, relative to original model' with a 'Reduction' column is ambiguous: clarify whether the reduction is relative to the PINE overhead or to the original model's FLOPs.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a venue on LLM reliability and positional bias. The main concern is that the headline invariance guarantee is not supported by the paper's own measurements; however, the issue is fixable by either physically reordering segments and re-running the affected experiments, or by reframing the contribution as approximate invariance with appropriate statistical support. No concerns about novelty disclosure or citation patterns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: this is a useful, simple technique for reducing positional bias in decoder-only LMs on listwise inputs, with real efficiency gains over PINE and mostly better performance on LitM and KGQA. The core idea—global sort plus circular position-ID assignment—is new relative to the cited order-invariant methods, and it is training-free. That is worth knowing.\n\nWhat the paper does well: PINE's per-query, per-head attention-based sorting is expensive and numerically unstable; RoToR's single global sort is a clean simplification. The experiments cover several backbones and task types, and the LitM/KGQA numbers mostly support RoToR over PINE, especially as segment count grows. Efficiency measurements are thorough (FLOPs, latency, perplexity, collision rates), and the Selective Routing alpha is tuned on validation and disclosed.\n\nSoft spots: First, Table 1's caption says RoToR is best in all setups, but the Qwen1.5-4B-Chat ndoc=30 row shows lexical RoToR below PINE. That is a factual overclaim. Second, Section 3.2 says the method guarantees invariance in a single forward pass. If invariance were exact, the shuffled-input rows in Table 2 would have zero standard deviation; they don't. The likely explanation is that RoToR reassigns position IDs but leaves physical token order and the causal mask intact, so the output still depends on the input permutation. The claim should be 'reduces variance,' not 'guarantees invariance.' Third, the significance test in Appendix I pools nine heterogeneous paired differences and uses seed-averaged scores as single points; the t=2.23, df=8 result does not establish robustness, and on MMLU RoToR+SR drops 2.8 points after reordering, the same as the original model. So the MMLU robustness story is weak.\n\nOverall: the method is simple, relevant, and likely useful in practice. The overclaim and the statistical analysis need fixing, but the central idea holds up. I would send this to a serious referee; accept conditionally.\n\nBest,\n[You]","headline":"A practical, training-free method for reducing positional bias that mostly delivers on LitM and KGQA, but the paper overstates both the invariance guarantee and the statistical evidence.","tokens_in":25365,"tokens_out":7048,"would_cite":true,"duration_ms":59554,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen LLM can be made order-invariant in a single forward pass by assigning position IDs from a rotated global ordering of the input segments.","keywords":["positional bias","order-invariant inputs","listwise inputs","rotational position ID assignment","lost in the middle","zero-shot order invariance","selective routing","language model reliability"],"falsifier":"Take an order-invariant task with $k$ distinct input segments, enumerate all $k!$ orderings, and check whether RoToR's greedy-decoded answers are identical across every ordering; any statistically reliable difference would refute the claimed invariance. A cheaper falsifier is to compare RoToR's hidden activations for two permutations of the same segment set at corresponding rotated positions—if they differ for any token, the mechanism is not exactly permutation-invariant.","tokens_in":24253,"feed_emoji":"🔄","tokens_out":8156,"duration_ms":671923,"temperature":0.7,"pith_summary":"Language models are biased by the order of list-like inputs: shuffling retrieved documents, table rows, or multiple-choice options can change answers. This paper claims that a frozen causal LLM can be made genuinely order-invariant with a single global sort of the input segments followed by circular position-ID assignment, so that each segment takes a turn appearing last when it acts as the query. The method, RoToR, is designed to avoid the train–test distribution mismatch and the per-query attention sorting of earlier zero-shot invariant models. Because real tasks mix order-invariant inputs with order-sensitive ones, the paper adds Selective Routing, which trusts whichever of the invariant or original model is more confident. If the claim is right, reliable answers on listwise inputs no longer require running many permutations or retraining.","feed_headline":"One global sort plus circular IDs cancels LLM order bias","feed_subtitle":"A single rotated ordering replaces per-query attention sorting, improving listwise accuracy and MMLU stability.","key_machinery":"The load-bearing mechanism is circular position-ID assignment over a single global segment ordering. Given a deterministic global order of the $k$ input segments, RoToR forms the same directed cycle for every query: starting from the query segment, the remaining segments follow in global order, and the query segment gets the largest position IDs so that it can attend to all others. Prefix and suffix tokens keep standard causal positions, and all suffix tokens see one shared rotation, so the added cost is one global sort, $O(k \\log k)$, with no extra attention computation. This replaces PINE's per-query, per-head, per-layer attention-score sorting, which the paper identifies as the source of distribution mismatch, numerical instability, and ordering collisions.","core_discovery":"The central claim is that order-invariance for decoder-only LMs is best achieved not by eliminating cross-segment attention or by re-sorting key segments for every query according to attention scores, but by choosing one deterministic global ordering of the segments and rotating it circularly when assigning position IDs. Under RoToR, the segment acting as query is placed at the end of the segment sequence, the other segments keep their relative global order, and suffix and generated tokens see a single fixed arrangement. The paper argues this keeps the model close to its pretrained positional distribution while still letting every segment attend to all others, and it reports that any global sorting strategy with this circular assignment outperforms the attention-based PINE scheme. On Lost in the Middle and knowledge-graph QA, RoToR improves accuracy and reduces variance across shuffles; on MMLU, RoToR alone is competitive but not always better, and adding Selective Routing makes the combination match or beat the original model across all answer orderings.","pith_inferences":["A direct corollary the paper does not pursue: if order-invariance is exact, RoToR's per-instance outputs should be identical across all permutations, so per-instance agreement across shuffles is a sharper test than average accuracy.","The global ordering is a free design choice, so one could sort by task-relevant keys (e.g., a numerical priority or semantic cluster) to combine order-invariance with a desired reading order—an untested extension.","Selective Routing's confidence comparison could serve as an order-sensitivity detector more generally: when the original and invariant models disagree, the input is likely order-sensitive or ambiguous, which may be useful for uncertainty estimation.","The circular-assignment idea is stated in terms of absolute position IDs, and an untested question is whether it transfers to relative positional encodings or to bidirectional models, where the same rotation could be implemented as a permutation of the key-value order."],"forward_implications":["A frozen, instruction-tuned LLM can be made order-invariant without any training, using one forward pass instead of evaluating multiple permutations.","Order-invariance no longer requires computing attention scores for sorting; the global sort is done once, making the method cheaper as the number of segments grows.","Collisions from tied attention scores are largely eliminated: lexical sorting only ties when two segment texts are literally identical.","Selective Routing lets a single deployment handle mixed listwise inputs, such as multiple-choice questions containing 'None of the above' options, without sacrificing the original model's accuracy.","The same mechanism extends beyond short lists: experiments with roughly 10k-token multi-hop QA contexts show RoToR remains effective and stable under severe order perturbations, where PINE runs out of memory."],"supporting_citations":[{"why":"Defines PINE, the per-query attention-based position re-assignment that RoToR identifies as causing distribution mismatch and replaces with a global circular ordering.","marker":"Wang et al., 2024"},{"why":"Defines PCW, the isolated-parallel-processing baseline whose cross-segment bottleneck RoToR avoids.","marker":"Ratner et al., 2023"},{"why":"Defines Set-Based Prompting, another zero-shot order-invariance baseline compared on LitM and MMLU.","marker":"McIlroy-Young et al., 2024"},{"why":"Provides the Lost in the Middle benchmark and documents the positional-bias problem that motivates the method.","marker":"Liu et al., 2024"},{"why":"Provides MMLU, the practical listwise benchmark whose mixture of invariant and sensitive options motivates Selective Routing.","marker":"Hendrycks et al., 2021"},{"why":"Supplies the KGQA preprocessing and evaluation setup on Mintaka used for the knowledge-graph experiments.","marker":"Baek et al., 2023"},{"why":"Supplies the finding that task difficulty correlates with order sensitivity, which underlies the confidence-based routing design.","marker":"Wei et al., 2024"},{"why":"Supplies the set/graph-ML precedent for using global sorting to obtain permutation-invariant functions.","marker":"Murphy et al., 2019a,b"}],"fun_headline_variants":["RoToR: one circular ordering makes LLMs order-invariant","Global sort + circular IDs: reliable listwise LLM answers","How to fix lost-in-the-middle: circular position IDs","Selective routing with RoToR improves LLM order robustness","Circular position IDs outperform attention sorting for order bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that rotating a single global ordering of position IDs is a small enough distributional change that a frozen pretrained LM keeps behaving sensibly, and that this rotation by itself makes the model truly order-invariant; on top of that, Selective Routing assumes that the maximum output-token probability is a reliable confidence signal for choosing between the two models.","fun_headline_variants_meta":{"raw":{"variants":["RoToR: one circular ordering makes LLMs order-invariant","Global sort + circular IDs: reliable listwise LLM answers","How to fix lost-in-the-middle: circular position IDs","Selective routing with RoToR improves LLM order robustness","Circular position IDs outperform attention sorting for order bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00026,"raw_usage":{"total_tokens":1596,"prompt_tokens":955,"completion_tokens":641,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":557}},"tokens_in":571,"tokens_out":641,"duration_ms":7193,"temperature":1.0,"reasoning_tokens":557,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T15:55:40.760703+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take an order-invariant task with $k$ distinct input segments, enumerate all $k!$ orderings, and check whether RoToR's greedy-decoded answers are identical across every ordering; any statistically reliable difference would refute the claimed invariance. A cheaper falsifier is to compare RoToR's hidden activations for two permutations of the same segment set at corresponding rotated positions—if they differ for any token, the mechanism is not exactly permutation-invariant.","supporting_citations":[{"cited_title":"Order-Independence Without Fine Tuning","cited_arxiv_id":"2406.06581","evidence_quote":"Defines Set-Based Prompting, another zero-shot order-invariance baseline compared on LitM and MMLU."}],"review_version":1}