Pith.

Artificial Intelligence · arXiv:2610.18233 v1 · submitted 13 Oct 2026 · 39 references

Consider, Then Choose: Order-Invariant Decisions in System One Models

Jianguo Huang, Yao Long Teng, Yanchen Deng, Bo An

The paper says

“System One models answer typed questions in a single forward pass and promise the same answer for the same input.”

more of the abstract

For choice questions, this promise fails: the answer depends on the order of the options, increasingly so as they grow in number, and averaging over their cyclic shifts reduces this dependence without removing it. Scoring each option alone, which the order cannot affect, shows that the order changes the answer mostly when the model is unsure among its top few options, even though the listwise answer looks confident. We therefore propose Runoff, which considers, then chooses: it scores each option alone to keep a few candidates, then compares them in listings where each candidate takes every position. It needs no training and can run in two forward passes, and its answer does not depend on the option order by construction. On two open models and five datasets, it changes no answer under shuffling, against 9.3% for cyclic averaging, and with 20 options it is about as accurate and three to four times faster. Wrapped around eight System One models, it removes the order dependence of the five benchmarks, while cyclic averaging changes up to 15.4%.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 14 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. A clean, practical method for removing order dependence in System One decision models, with proper analysis and honest limitations. This line is wrong

4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 39 references say.

4 minor objections

The paper says, at Appendix E, Assumption (b)

Pith says: minor objection 1 of 4

The proof of Proposition 1 assumes that π takes distinct values on O so that the canonical ranking is unique. In practice, ties could occur (e.g., with very similar pointwise scores). The paper may wish to state how ties are resolved (e.g., by option text) or note that the assumption is almost surely satisfied by floating-point computation.

The paper says, at Appendix A, packed forward pass

Pith says: minor objection 2 of 4

The description of the packed forward pass states that the K pointwise prompts share a common prefix encoded once. The fixed packing order (by option text) ensures invariance to σ, but the model's sensitivity to the relative position of prompts within a batch is not discussed. A brief remark on why this does not reintroduce order dependence would be helpful.

show 2 more

The paper says, at Table 6 and Section 5.2

Pith says: minor objection 3 of 4

The sensitivity analysis shows that k_min below 3 reduces accuracy as the gold answer is more often excluded from the consideration set. The paper reports that the gold answer is included in 90% of questions with k_min=3, but not how often it is included with k_min=1 or 2 beyond the accuracy drop. Reporting the inclusion rate would strengthen the justification for the default value.

The paper says, at Table 4, latency

Pith says: minor objection 4 of 4

The latency comparison uses mean over original and shuffled orders, but the standard deviation or a per-dataset breakdown (beyond the overall average) would give a clearer picture of variability. The current table already shows per-dataset figures, which is sufficient; consider adding a note about hardware or batching conditions for reproducibility.

What would settle it

For a closed-source API that leaves a non-zero residual flip rate under Runoff (e.g., Jev 1.13 with 0.9% residual), run a test-retest experiment: resend each Runoff pointwise yes/no request exactly once more without shuffling options. Compute the proportion of questions where the consideration set changes between the two identical requests. If this proportion matches or exceeds the reported flip rate under shuffling, the residual is fully explained by API nondeterminism; if it is substantially lower, some order-dependent signal remains.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 14 Oct 2026

  1. Reader holds up confidence high
  2. Skeptic agrees 1 min 11 s · 21,508 tokens read, 7,163 written
  3. Editor wrote the verdict above 10 s · 22,250 tokens read, 1,133 written
  4. Referee accept: 4 minor 25 s · 21,801 tokens read, 2,613 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This paper does two things well. First, it gives a clean analysis of where and why option-order dependence arises in System One models: the order mainly decides among the few options the model is uncertain about, and that uncertainty grows with the number of options. Second, it proposes Runoff, a training-free two-stage decision rule (pointwise scoring to build a consideration set, then cyclic averaging over only those few candidates) that is order-invariant by construction. The proof of invariance (Proposition 1) is correct under the stated assumptions. The experiments are thorough: ten models, five datasets, controlled shuffles, ablation studies, and a real agent deployment. The empirical claim that Runoff changes no answers under shuffling on deterministic open models holds up, and the flip rates on nondeterministic APIs are far below cyclic averaging.

The paper is honest about its soft spots. It acknowledges that on reasoning questions (MMLU-Pro) the consideration set sometimes misses the correct answer, so accuracy trails cyclic averaging by a few points. It also notes that for nondeterministic APIs a small residue of flips remains, and that Runoff only addresses option-order bias, not other biases like name preference. I have no major objections. The analysis of controlled shuffles in Appendix B is a nice touch: it shows that even the arrangement of non-candidate options can shift the answer, which explains why excluding them entirely (rather than averaging over all K listings) helps. That is a genuine insight.

If I had a minor quibble: the motivation analysis in Section 3 is done only on Qwen3-8B and Gemma 4 12B; the pattern is replicated later for other models in aggregate, but it would have been cleaner to show the relation between pointwise uncertainty and flip rates for each model individually. But this is a presentation preference, not a flaw.

The paper is a solid contribution to the subfield of decision-focused LLM research. It should be taken seriously. I would bring it to a reading group, and I would accept a peer review invitation.

The strongest objection

The paper presents a well-motivated, theoretically grounded, and empirically validated solution to order dependence in System One models. The proof of order invariance (Proposition 1, Appendix E) is sound under the stated assumptions (deterministic model, unique canonical ranking). The empirical results on open models (0% flip rate across five datasets) directly confirm the invariance. The method's limitations—reasoning questions where the consideration set can miss the correct answer, and residual flips from nondeterministic APIs—are explicitly stated and empirically bounded (Tables 2, 3). The controlled shuffle experiments (Appendix B) provide additional insight into where order sensitivity arises. The ablation study (Table 5) cleanly separates the contributions of Consider and Choose. No hidden assumption, undisclosed failure mode, or internal inconsistency was found.

The reader's weakest_assumption correctly identifies both the primary limitation (pointwise distribution may miss the correct answer on reasoning questions, acknowledged in Section 4 and Limitations) and the secondary one (deterministic model assumption, acknowledged in Limitations). My stress-test uncovered no additional load-bearing concern. The paper's central claim—that Runoff is order-invariant by construction, comparably accurate to cyclic averaging, and faster for many options—holds under the conditions it specifies. The empirical evidence (0% flip rate on deterministic models, comparable accuracy across five datasets, latency analysis in Table 4) is consistent with the theoretical guarantee. The paper is transparent about trade-offs (reasoning performance, nondeterminism, cost with few options) and does not overclaim. No adjustment to the verdict is warranted.

The referee's report

Order dependence is a practical obstacle to deploying System One models in agents where option order is inherited from tool registries or screen layouts. Runoff provides a principled, theoretically grounded solution that is training-free and demonstrably invariant. The analysis of where and when order matters (Section 3) is insightful and guides the method. The empirical validation is extensive, covering 10 models, 5 datasets, ablations, sensitivity analysis, and a computer-use agent deployment. If the results hold in broader settings, Runoff could become a standard post-processing method for choice decisions in System One models.

Pith's reading of the paper (a machine's summary, not the authors')

System One models, which answer choice questions in a single forward pass, produce answers that depend on the order of the options, with the problem worsening as the number of options grows. This paper traces the dependence to the model's uncertainty among a few top options and proposes Runoff, a two-step decision rule that first scores each option alone to form a small consideration set, then compares those candidates in all cyclic shifts of their ranking so that no position bias operates. Runoff requires no training, runs in two forward passes, and is order-invariant by construction. On open models and five datasets, it changes no answers under shuffling, compared to 9.3% for cyclic averaging, with comparable accuracy and three-to-four times faster speed when options are many. Its main limitation is on reasoning questions, where it trails cyclic averaging slightly.

What carries the argument. The key machinery is the combination of pointwise scoring to construct an order-invariant consideration set C (the smallest set of top options covering at least 99% of the pointwise probability, with at least 3 options) and cyclic-shift averaging over C: the model's listwise decisions on all m cyclic shifts of the canonical ranking of C are averaged in log-probability space, removing any position bias that would otherwise favor the first option in a fixed listing.

Core claim. The paper discovers that the option-order dependence of System One model choices is concentrated in the model's uncertainty among its top few options: when the model is unsure, shuffling the options changes the answer far more often, even though the listwise output appears confident. This insight leads to Runoff, which first obtains an order-invariant pointwise distribution by asking the model to score each option alone (yes/no), then keeps only the top options (default α=0.99, kmin=3) and compares them in all m cyclic shifts of their canonical ranking, averaging the listwise log-probabilities so every candidate appears in each position exactly once. The answer is thus independent of the input order by construction. Empirically, Runoff eliminates all answer flips under shuffling on deterministic models, matches the accuracy of cyclic averaging on classification tasks, and is faster by a factor of two to four.

The premise it rests on. Runoff assumes that the pointwise distribution correctly identifies the small set of options among which order can change the answer, and that the model is deterministic (returns identical outputs for identical requests).

If the paper is right (Pith's inferences, not the authors')

Runoff provides a training-free method to make any deterministic System One model order-invariant without retraining.

With many options (e.g., 20), Runoff is three to four times faster than cyclic averaging and achieves the same accuracy.

The method reveals that order dependence is largely a local phenomenon among uncertain top options, not a global sensitivity.

Runoff can be wrapped around closed-source APIs, reducing flips far below cyclic averaging, with only a small residue from model nondeterminism.

Using Runoff in a computer-use agent increases the number of tasks solved compared to native decisions.

The two-stage consider-then-choose framework could be applied to other domains where listwise decisions suffer from ordering biases, such as recommender systems or ranking tasks.

The pointwise scoring step does not require the full label set to be seen, so Runoff could scale to very large option sets by using efficient nearest-neighbor retrieval of candidate options before the pointwise scoring.

The method's reliance on deterministic models could be mitigated by running multiple pointwise passes and taking a majority, though that would increase cost.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

System One models answer typed questions in a single forward pass and promise the same answer for the same input. For choice questions, this promise fails: the answer depends on the order of the options, increasingly so as they grow in number, and averaging over their cyclic shifts reduces this dependence without removing it. Scoring each option alone, which the order cannot affect, shows that the order changes the answer mostly when the model is unsure among its top few options, even though the listwise answer looks confident. We therefore propose Runoff, which considers, then chooses: it scores each option alone to keep a few candidates, then compares them in listings where each candidate takes every position. It needs no training and can run in two forward passes, and its answer does not depend on the option order by construction. On two open models and five datasets, it changes no answer under shuffling, against 9.3% for cyclic averaging, and with 20 options it is about as accurate and three to four times faster. Wrapped around eight System One models, it removes the order dependence of the five benchmarks, while cyclic averaging changes up to 15.4%.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-14

The authors' main claim, as the classifier picked it out 0.73

“Wrapped around eight System One models, it removes the order dependence of the five benchmarks, while cyclic averaging changes up to 15.4%.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.34 · human participants 0.04 · controlled comparison 0.88 · randomized assignment 0.03 · states quantitative result 0.98 · compares with prior results 0.97 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
System One models answer typed questions in a single forward pass and promise the same answer for the same input. background 1.00 0.10 0.03 asserted 0.99
For choice questions, this promise fails: the answer depends on the order of the options, increasingly so as they grow in number, and averaging over their cyclic shifts reduces this dependence without removing it. result 0.60 0.69 0.15 asserted 1.00
Scoring each option alone, which the order cannot affect, shows that the order changes the answer mostly when the model is unsure among its top few options, even though the listwise answer looks confident. result 0.74 0.86 0.09 asserted 1.00
We therefore propose $\textbf{Runoff}$, which considers, then chooses: it scores each option alone to keep a few candidates, then compares them in listings where each candidate takes every position. method 0.85 0.77 0.23 asserted 0.94
It needs no training and can run in two forward passes, and its answer does not depend on the option order by construction. method 0.72 0.79 0.25 asserted 1.00
On two open models and five datasets, it changes no answer under shuffling, against 9.3% for cyclic averaging, and with 20 options it is about as accurate and three to four times faster. result 1.00 0.94 0.26 asserted 1.00
Wrapped around eight System One models, it removes the order dependence of the five benchmarks, while cyclic averaging changes up to 15.4%. result 1.00 0.94 0.34 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.78

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-14: 1 request, 5,671 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (8)
Figure 1 of the paper
Figure 1. Accuracy and flip rate against the number of options, Qwen3-8B. Flip rate: share of answers that change
Figure 2 of the paper
Figure 2. Order dependence of Listwise and its relation to the pointwise distribution for Qwen3-8B. Panels (b) and
Figure 3 of the paper
Figure 3. Overview of Runoff. probability of C according to r: P(o) = ( π(C) r(o) o ∈ C, π(o) otherwise. (4) Proposition 1. Runoff is order-invariant (Defini￾tion 1): the requests it sends to the model, and hence P, are the same for every order σ of O. Each pointwise decision sees one option and π normalizes over the set O, so π depends only on (x, q, O); C, its canonical ranking and the cyclic shifts are functions of π and the option texts (proof in Appendix E). The invariance holds whatever position bias the model has, and on a black-box model any remaining flip can only come from the model…
Figure 4 of the paper
Figure 4. OSWorld tasks solved by a System One computer-use agent with the native decisions and Runoff. takes 77 ms with Runoff against 308 ms with Any￾Jev, whereas on JevBench (3 to 6 options) it takes 77 ms against 67 ms. With many options, Choose compares only about four candidates instead of all K options in K listings. With few, the second for￾ward pass is a fixed cost that the smaller set does not repay. Averaged over the five datasets, Runoff is 2.1 times faster than AnyJev on both models, but remains slower than Listwise, which makes a single listwise decision. 6 Discussion Decisions…
Figure 5 of the paper
Figure 5. Prompt of the listwise decision for Qwen3-8B. Angle brackets: filled per question. The decision
Figure 6 of the paper
Figure 6. Prompt of the pointwise decision on an option

Showing 6 of the paper's 8 figures. Figures are the authors', reproduced from the paper.

References (39)
  1. [1] AnyJev Technical Report , author=. arXiv preprint arXiv:2610.00831 , year= on PithPith's reviewarXiv
  2. [2] Xu, Yinsong , title =. 2026 , note = on Pith
  3. [3] Lee, Theodore , title =. 2026 , note = on Pith
  4. [4] Large language models are not robust multiple choice selectors , author=. URL https://arxiv. org/abs/2309.03882 , year= on PithPith's reviewarXiv
  5. [5] Large language models sensitivity to the order of options in multiple-choice questions , author=. Findings of the Association for Computational Linguistics: NAACL 2024 , pages= on Pith
  6. [6] Efficient intent detection with dual sentence encoders , author=. Proceedings of the 2nd workshop on natural language processing for conversational AI , pages= on Pith
  7. [7] Newsweeder: Learning to filter netnews , author=. Machine learning proceedings 1995 , pages=. 1995 , publisher= on Pith
  8. [8] Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It , author=. 2026 , eprint= on Pith
  9. [9] Laya: Multilingual, Non-Autoregressive. 2026 , note = on Pith
  10. [10] Reid McIlroy-Young and Katrina Brown and Conlan Olson and Linjun Zhang and Cynthia Dwork , title =. Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024,. 2024 , url = on Pith
  11. [11] Palmer, Jared , title =. 2026 , note = on Pith
  12. [12] Wei, Sheng-Lun and Wu, Cheng-Kuang and Huang, Hen-Hsen and Chen, Hsin-Hsi. Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Models. Findings of the Association for Computational Linguistics: ACL 2024. 2024. doi:10.18653/v1/2024.findings-acl.333 on Pithdoi:10.18653/v1/2024.findings-acl.333
  13. [13] Wang, Peiyi and Li, Lei and Chen, Liang and Cai, Zefan and Zhu, Dawei and Lin, Binghuai and Cao, Yunbo and Kong, Lingpeng and Liu, Qi and Liu, Tianyu and Sui, Zhifang. Large Language Models are not Fair Evaluators. Proceedings of the 62nd Annual Meeting of the Association for ... on Pithdoi:10.18653/v1/2024.acl-long.511
  14. [14] Yu Sun and Junhao Xu and Jiajia Shi and Zijin Yang , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2609.26758 , eprinttype = on PithPith's reviewarXivdoi:10.48550/arxiv.2609.26758
  15. [15] Calibrate Before Use: Improving Few-shot Performance of Language Models , author =. Proceedings of the 38th International Conference on Machine Learning , pages =. 2021 , editor = on Pith
  16. [16] One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendation , author=. arXiv preprint arXiv:2604.27599 , year= on PithPith's reviewarXiv
  17. [17] Batch calibration: Rethinking calibration for in-context learning and prompt engineering , author=. International Conference on Learning Representations , volume= on Pith
  18. [18] Found in the middle: Permutation self-consistency improves listwise ranking in large language models , author=. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers... on Pith
  19. [19] Poe: Process of elimination for multiple choice reasoning , author=. Proceedings of the 2023 conference on empirical methods in natural language processing , pages= on Pith
  20. [20] Leveraging large language models for multiple choice question answering , author=. arXiv preprint arXiv:2210.12353 , year= on PithPith's reviewarXiv
show 19 more
  1. [21] Set-llm: A permutation-invariant llm , author=. Advances in Neural Information Processing Systems , volume= on Pith
  2. [22] RoToR: Towards More Reliable Responses for Order-Invariant Inputs , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
  3. [23] Eliminating position bias of language models: A mechanistic approach , author=. International Conference on Learning Representations , volume= on Pith
  4. [24] Atlas: Few-shot learning with retrieval augmented language models , author=. Journal of Machine Learning Research , volume= on Pith
  5. [25] Quantifying and Mitigating Selection Bias in LLMs: A Transferable LoRA Fine-Tuning and Efficient Majority Voting Approach , author=. Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the ... on Pith
  6. [26] Option-id based elimination for multiple choice questions , author=. arXiv preprint arXiv:2501.15175 , year= on PithPith's reviewarXiv
  7. [27] Accuracy and Order Sensitivity Diverge Under Label-Free Strategies , author=. arXiv preprint arXiv:2608.11947 , year= on PithPith's reviewarXiv
  8. [28] Fool your large (vision and) language models with embarrassingly simple permutations , author= on Pith
  9. [29] Skill Suggestion , howpublished =. 2026 , url = on Pith
  10. [30] Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year= on PithPith's reviewarXiv
  11. [31] Mmlu-pro: A more robust and challenging multi-task language understanding benchmark , author=. Advances in Neural Information Processing Systems , volume= on Pith
  12. [32] Standhartinger, Florian , title =. 2026 , note = on Pith
  13. [33] An evaluation cost model of consideration sets , author=. Journal of consumer research , volume=. 1990 , publisher= on Pith
  14. [34] The impact of candidate name order on election outcomes , author=. Public Opinion Quarterly , pages=. 1998 , publisher= on Pith
  15. [35] Learning probabilistic symmetrization for architecture agnostic equivariance , author=. Advances in Neural Information Processing Systems , volume= on Pith
  16. [36] Almeida, Diogo , title =. 2026 , note = on Pith
  17. [37] Guanxu Yu and Yuhang Yao , year=. 2609.25845 , archivePrefix= on PithPith's reviewarXiv
  18. [38] Tianbao Xie and Danyang Zhang and Jixuan Chen and Xiaochuan Li and Siheng Zhao and Ruisheng Cao and Toh Jing Hua and Zhoujun Cheng and Dongchan Shin and Fangyu Lei and Yitao Liu and Yiheng Xu and Shuyan Zhou and Silvio Savarese and Caiming Xiong and Victor Zhong and Tao Yu , y... on PithPith's reviewarXiv
  19. [39] typesafe-computer-use , author =. 2026 , howpublished = on Pith
Cited by 5 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment