Cryptography and Security · arXiv:2610.19856 v1 · submitted 14 Oct 2026 · 47 references
Lost in the Fallback: How Reliably Do Typed Decision Models Use Fallback Options?
The paper says
“Typed decision models (TDMs) are increasingly used in applications such as request routing and model selection, making reliable decisions critical to system performance. Fallback options accommodate categories outside regular choices, enabling deferral to human experts or more capable models.”
more of the abstract
While recent works have examined adversarial vulnerabilities in TDMs, their ability to follow fallback rules in non-adversarial settings remains underexplored. We introduce the Paired Fallback Robustness (PFR) Test to evaluate two aspects of fallback reliability: whether models correctly switch between regular and fallback options, and whether they make consistent decisions under logically equivalent definitions of the same rule. We evaluate open-source decision models on three benchmarks, with controlled comparisons across architectures and parameter counts. We find that correct recognition does not reliably transfer to correct fallback options, and errors persist even when the correct category is explicitly supplied.Moreover, logically equivalent formulations of the same fallback rule can also produce substantial disagreement in model decisions. These findings reveal a crucial reliability gap in current TDMs: fallback mechanisms designed to improve reliability can instead introduce errors. It highlights faithful execution of fallback rules and consistency under equivalent definitions as essential requirements for reliable typed decision-making.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 15 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. Well-executed empirical study revealing a genuine reliability gap in typed decision models; PFR test is a solid methodological contribution, but ecological validity is limited by the controlled setup. This line is wrong
5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 47 references say.
5 minor objections
The paper says, at Section 3.1, Eq. (2)–(4)
Pith says: minor objection 1 of 5
The metric F+ (false fallback rate) is defined as the fraction of gold-regular inputs where the model selects fallback. It would be helpful to explicitly state that this corresponds to unnecessary fallback triggers, as the current definition could be misinterpreted as a general false positive rate.
The paper says, at Section 3.1, paragraph on Joint correctness
Pith says: minor objection 2 of 5
The sentence 'J measures successful reassignment, while DR tests definition consistency' could be made more precise: J is successful reassignment only when both assignments yield correct decisions under the same definition, whereas DR measures any change in choice between U and C within the same assignment, regardless of correctness.
show 3 more
The paper says, at Figure 1
Pith says: minor objection 3 of 5
The hatched gap between the Union and Complement bars is described in the caption, but the exact numerical difference is not given. Consider adding the difference (e.g., in percentage points) to the caption or directing readers to Table 1 for the precise values.
The paper says, at Section 4.2, discussion of Figure 5
Pith says: minor objection 4 of 5
The statement that averaging across four positions and three choice sets leaves a gap of just +0.3 points is informative, but it could be misinterpreted as indicating overall stability. Consider adding an explicit note that this small average masks large opposing effects (e.g., from -41.4 to +35.2 pp) at different positions.
The paper says, at Section 7, Limitations
Pith says: minor objection 5 of 5
The authors appropriately acknowledge the controlled nature of the experiments. It would be beneficial to also note that the study focuses on a specific operationalization of fallback (category coverage) and that alternative definitions (e.g., quality-based thresholds) might yield different conclusions, although the policy experiments in Section 4.3 partially address this point.
What would settle it
Replicate the PFR evaluation on a set where the gold category is not known to the experimenter and the model must infer both the category and the fallback rule from ambiguous or partial descriptions, testing whether the same failures occur in more realistic conditions.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 15 Oct 2026
- Reader holds up
- Skeptic agrees
- Editor wrote the verdict above
- Referee accept: 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Worth a look. This paper does what good empirical work should: it identifies an underexplored problem—whether typed decision models correctly follow fallback rules under non-adversarial conditions—and builds a clean test suite (the Paired Fallback Robustness Test) around it. The central finding is that correct category recognition does not transfer to correct fallback selection, and that logically equivalent definitions of the same fallback rule can produce substantially different decisions. The evidence is consistent across three benchmarks, four model architectures, multiple parameter scales, and a stack of robustness checks (wording, position, naming, category information). The paper also shows that supplying the correct category explicitly does not fix the problem. That is a real result and worth knowing for anyone deploying these models.
The soft spot is the one the authors themselves flag in Section 7: the experimental setup uses known categories, gold labels, and exactly one valid destination per input. Real-world fallback often involves ambiguous requests, unknown intents, or multiple valid destinations. The paper does not claim to cover those cases, but the gap between this controlled setting and production deployment is large enough that the headline numbers (e.g., 23.1% fallback accuracy on correctly classified TREC inputs) should be read as a proof of concept rather than a direct estimate of field performance. A second limitation is that the paper is diagnostic only; it does not propose mitigations. That is fine for a study meant to raise awareness, but it does limit the practical takeaway.
Overall: a solid piece of work that argues convincingly that fallback reliability is not a solved problem and that standard metrics (accuracy, false fallback rate) are insufficient. The PFR test is a useful evalution tool. I would bring this to a reading group if the group works on decision models or structured output reliability. I would cite it for the empirical finding that equivalence formulations behave differently. The paper deserves a serious peer review; I expect it will be accepted with minor revisions, hopefully including a more careful discussion of the gap between lab and deployment conditions.
Serious_thinker: yes.
The strongest objection
The paper presents thorough and well-controlled experiments supporting its central claim that fallback reliability is poor and that standard metrics are insufficient. The main limitation—that the controlled setup (known categories, gold labels, exactly one valid destination) may not fully capture real-world ambiguity—is explicitly acknowledged by the authors in Section 7. I find no internal inconsistency, no hidden assumption that undermines the conclusion, and no reason to doubt the empirical findings within the stated scope.
The paper is rigorous and transparent. The concern about generalizability is acknowledged and does not affect the validity of the findings within the studied scope. No methodological flaw or overclaim is present.
The referee's report
If the findings hold generally, they have important implications for the design and evaluation of decision-making systems that rely on fallback mechanisms. The PFR test provides a systematic methodology for assessing fallback reliability beyond aggregate accuracy, and the documented failure modes (lost transfers, definition sensitivity) should inform both practical deployment and future research on robust decision models. The experiments are thorough, with multiple controlled comparisons, statistical testing, and sensitivity analyses, lending credibility to the conclusions.
Pith's reading of the paper (a machine's summary, not the authors')
Typed decision models (TDMs) are used for request routing and model selection, where fallback options handle cases outside regular choices. This paper shows that TDMs frequently fail to correctly switch between regular and fallback options, even on inputs they have correctly classified. Moreover, semantically equivalent definitions of the same fallback rule lead to different decisions, revealing a reliability gap: fallback mechanisms designed to improve handling can instead introduce errors. The paper introduces the Paired Fallback Robustness (PFR) test to jointly measure correct reassignment and definition consistency.
What carries the argument. The central object is the Paired Fallback Robustness (PFR) Test, which constructs four requests per input by crossing two category assignments (gold-regular vs gold-fallback) with two logically equivalent definitions (Union, which lists fallback members, and Complement, which excludes regular categories). It measures joint correctness (J) and disagreement rate (DR) to assess whether models correctly reassign inputs and remain consistent under equivalent definitions.
Core claim. The paper establishes that correct category recognition does not guarantee correct fallback selection: for example, on the TREC50 benchmark, Intern's accuracy drops from 97.9% to 23.1% when the required output changes from a regular option to fallback, even on inputs it previously classified correctly. These failures persist when the correct category is provided directly, and they occur across different model architectures and parameter counts. Furthermore, logically equivalent formulations of the same fallback rule—such as listing fallback categories explicitly versus defining fallback by excluding regular categories—produce substantial disagreement in individual decisions, even when aggregate accuracy remains similar.
The premise it rests on. The experimental setting assumes each input has exactly one valid fallback destination, all categories are known, and gold labels are available; in real deployments, fallback may involve ambiguous requests, unknown categories, or multiple valid destinations, as the authors acknowledge.
If the paper is right (Pith's inferences, not the authors')
Current evaluations that only measure classification accuracy or aggregate fallback rates miss systematic failures; the PFR test provides a more comprehensive evaluation.
Models cannot be trusted to handle fallback correctly even when they know the input category; additional safeguards or targeted training may be necessary.
Sensitivity to equivalent definitions suggests that phrasing affects deployment behavior, potentially leading to unpredictable outcomes in production systems.
The gap between individually recoverable errors and shared-threshold recovery implies that simple post-hoc scoring adjustments may not fix fallback failures reliably.
The findings suggest that TDM reliability may degrade further in real-world settings where fallback destinations are ambiguous or multiple valid destinations exist, a context the authors acknowledge is untested.
The PFR test could be extended to other domains such as content moderation or automated customer support, where fallback is critical for deferring to human experts.
The sensitivity to equivalent definitions points to a deeper issue in language models: they may not treat logically equivalent statements as having identical operational meaning, raising questions about compositional semantics in decision contexts.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Typed decision models (TDMs) are increasingly used in applications such as request routing and model selection, making reliable decisions critical to system performance. Fallback options accommodate categories outside regular choices, enabling deferral to human experts or more capable models. While recent works have examined adversarial vulnerabilities in TDMs, their ability to follow fallback rules in non-adversarial settings remains underexplored. We introduce the Paired Fallback Robustness (PFR) Test to evaluate two aspects of fallback reliability: whether models correctly switch between regular and fallback options, and whether they make consistent decisions under logically equivalent definitions of the same rule. We evaluate open-source decision models on three benchmarks, with controlled comparisons across architectures and parameter counts. We find that correct recognition does not reliably transfer to correct fallback options, and errors persist even when the correct category is explicitly supplied.Moreover, logically equivalent formulations of the same fallback rule can also produce substantial disagreement in model decisions. These findings reveal a crucial reliability gap in current TDMs: fallback mechanisms designed to improve reliability can instead introduce errors. It highlights faithful execution of fallback rules and consistency under equivalent definitions as essential requirements for reliable typed decision-making.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-15
The authors' main claim, as the classifier picked it out 0.89
“These findings reveal a crucial reliability gap in current TDMs: fallback mechanisms designed to improve reliability can instead introduce errors.”
How the main claim is stated: asserted 0.99
What the abstract itself states, as probabilities: claims first 0.12 · states limitation 0.16 · human participants 0.06 · controlled comparison 0.59 · randomized assignment 0.03 · states quantitative result 0.06 · compares with prior results 0.50 · mentions code or data availability 0.05
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Typed decision models (TDMs) are increasingly used in applications such as request routing and model selection, making reliable decisions critical to system performance. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| Fallback options accommodate categories outside regular choices, enabling deferral to human experts or more capable models. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| While recent works have examined adversarial vulnerabilities in TDMs, their ability to follow fallback rules in non-adversarial settings remains underexplored. | gap 1.00 | 0.04 | 0.03 | asserted 0.96 |
| We introduce the Paired Fallback Robustness (PFR) Test to evaluate two aspects of fallback reliability: whether models correctly switch between regular and fallback options, and whether they make consistent decisions under logically equivalent definitions of the same rule. | contribution 0.69 | 0.91 | 0.05 | asserted 0.99 |
| We evaluate open-source decision models on three benchmarks, with controlled comparisons across architectures and parameter counts. | method 0.97 | 0.95 | 0.02 | asserted 0.98 |
| We find that correct recognition does not reliably transfer to correct fallback options, and errors persist even when the correct category is explicitly supplied.Moreover, logically equivalent formulations of the same fallback rule can also produce substantial disagreement in model decisions. | result 1.00 | 0.97 | 0.75 | asserted 1.00 |
| These findings reveal a crucial reliability gap in current TDMs: fallback mechanisms designed to improve reliability can instead introduce errors. | interpretation 1.00 | 0.83 | 0.81 | asserted 0.99 |
| It highlights faithful execution of fallback rules and consistency under equivalent definitions as essential requirements for reliable typed decision-making. | interpretation 0.98 | 0.27 | 0.32 | asserted 0.98 |
What kind of paper the classifier reports this is: empirical study 0.73 · evidence from observation or experiment 0.98
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-15: 1 request, 6,240 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (22)
Showing 20 of the paper's 22 figures. Figures are the authors', reproduced from the paper.
References (47)
- [1] Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom a... on Pith
- [2] Ribeiro, Marco Tulio and Wu, Tongshuang and Guestrin, Carlos and Singh, Sameer , title=. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , year=. doi:10.18653/v1/2020.acl-main.442 , url= on Pithdoi:10.18653/v1/2020.acl-main.442
- [3] Guarded Query Routing for Large Language Models , year=. 2505.14524 , archivePrefix= on PitharXiv
- [4] Almeida, Diogo , title =. 2026 , howpublished = on Pith
- [5] Toolformer: Language Models Can Teach Themselves to Use Tools , author=. Thirty-seventh Conference on Neural Information Processing Systems , year= on Pith
- [6] Patil, Shishir G. and Zhang, Tianjun and Wang, Xin and Gonzalez, Joseph E. , title=. Advances in Neural Information Processing Systems , volume=. 2024 , url= on Pith
- [7] Geng, Saibo and Josifoski, Martin and Peyrard, Maxime and West, Robert , title=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , year=. doi:10.18653/v1/2023.emnlp-main.674 , url= on Pithdoi:10.18653/v1/2023.emnlp-main.674
- [8] Lu, Jiawen and Wu, Tongtong , title=. 2026 , eprint= on Pith
- [9] Larson, Stefan and Mahendran, Anish and Peper, Joseph J. and Clarke, Christopher and Lee, Andrew and Hill, Parker and Kummerfeld, Jonathan K. and Leach, Kevin and Laurenzano, Michael A. and Tang, Lingjia and Mars, Jason. An Evaluation Dataset for Intent Classification and Out-... on Pithdoi:10.18653/v1/d19-1131
- [10] Casanueva, I \ n igo and Tem c inas, Tadas and Gerz, Daniela and Henderson, Matthew and Vuli \'c , Ivan. Efficient Intent Detection with Dual Sentence Encoders. Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI. 2020. doi:10.18653/v1/2020.nlp... on Pithdoi:10.18653/v1/2020.nlp4convai-1.5
- [11] Li, Xin and Roth, Dan. Learning Question Classifiers. COLING 2002: The 19th International Conference on Computational Linguistics. 2002 on Pith
- [12] Tam, Zhi Rui and Wu, Cheng-Kuang and Lin, Chieh-Yen and Chen, Yun-Nung. None of the Above, Less of the Right Parallel Patterns in Human and LLM Performance on Multi-Choice Questions Answering. Findings of the Association for Computational Linguistics: ACL 2025. 2025. doi:10.18... on Pithdoi:10.18653/v1/2025.findings-acl.1031
- [13] When the Right Answer Is Missing: An Arithmetic-Dependent Rejection Bottleneck in Jev , author=. 2026 , eprint= on Pith
- [14] Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces , author=. 2026 , eprint= on Pith
- [15] Do System One Decisions Add Up? A Study of Probabilistic Coherence , author=. 2026 , eprint= on Pith
- [16] Calibrate Before Use: Improving Few-shot Performance of Language Models , author=. Proceedings of the 38th International Conference on Machine Learning , pages=. 2021 , volume= on Pith
- [17] Are Pre-trained Transformers Robust in Intent Classification? A Missing Ingredient in Evaluation of Out-of-Scope Intent Detection , author=. Proceedings of the 4th Workshop on NLP for Conversational AI , year=. doi:10.18653/v1/2022.nlp4convai-1.2 , url= on Pithdoi:10.18653/v1/2022.nlp4convai-1.2
- [18] Improved Out-of-Scope Intent Classification with Dual Encoding and Threshold-based Re-Classification , author=. 2024 , eprint= on Pith
- [19] Out-of-Distribution Intent Detection with Self-Supervision and Discriminative Training , author=. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long ... on Pithdoi:10.18653/v1/2021.acl-long.273
- [20] The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance , author=. 2024 , eprint= on Pith
show 27 more
- [21] Navigating the Prompt Space: Improving LLM Classification of Social Science Texts Through Prompt Engineering , author=. 2026 , eprint= on Pith
- [22] Mitigating Boundary Ambiguity and Inherent Bias for Text Classification in the Era of Large Language Models , author=. 2024 , eprint= on Pith
- [23] Readout Stability in Prefill-Only Decision Models: Zero-Label Prediction and Inference-Time Compute Allocation , author=. 2026 , eprint= on Pith
- [24] Decodable but Misrouted: Sparse Features Uncover a Readout Gap in Vision-Language Models for Harmful Meme Detection , author=. 2026 , eprint= on Pith
- [25] 2026 , howpublished= on Pith
- [26] Palmer, Jared , title=. 2026 , howpublished= on Pith
- [27] Serikbay, Alibi , title=. 2026 , howpublished= on Pith
- [28] Guo, Chuan and Pleiss, Geoff and Sun, Yu and Weinberger, Kilian Q. , title=. Proceedings of the 34th International Conference on Machine Learning , series=. 2017 , url= on Pith
- [29] Goren, Shani and Galil, Ido and El-Yaniv, Ran , title=. Advances in Neural Information Processing Systems , volume=. 2024 , url= on Pith
- [30] Li, Yinheng and Wagle, Justin , year=. 2610.02076 , archivePrefix= on PithPith's reviewarXiv
- [31] Conformal Intent Classification and Clarification for Fast and Accurate Intent Recognition , author=. Findings of the Association for Computational Linguistics: NAACL 2024 , year=. doi:10.18653/v1/2024.findings-naacl.156 , url= on Pithdoi:10.18653/v1/2024.findings-naacl.156
- [32] Test-Time Amendment with a Coarse Classifier for Fine-Grained Classification , author=. Advances in Neural Information Processing Systems , volume=. 2023 , url= on Pith
- [33] Sun, Yu and Xu, Junhao and Shi, Jiajia and Yang, Zijin , title=. 2026 , eprint=. doi:10.48550/arXiv.2609.26758 , url= on PithPith's reviewarXivdoi:10.48550/arxiv.2609.26758
- [34] Gangrade, Aditya and Kag, Anil and Saligrama, Venkatesh , title=. Proceedings of the 24th International Conference on Artificial Intelligence and Statistics , series=. 2021 , publisher= on Pith
- [35] Pezeshkpour, Pouya and Hruschka, Estevam. Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions. Findings of the Association for Computational Linguistics: NAACL 2024. 2024. doi:10.18653/v1/2024.findings-naacl.130 on Pithdoi:10.18653/v1/2024.findings-naacl.130
- [36] Wei, Sheng-Lun and Wu, Cheng-Kuang and Huang, Hen-Hsen and Chen, Hsin-Hsi. Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Models. Findings of the Association for Computational Linguistics: ACL 2024. 2024. doi:10.18653/v1/2024.findings-acl.333 on Pithdoi:10.18653/v1/2024.findings-acl.333
- [37] Lingjiao Chen and Matei Zaharia and James Zou , journal=. Frugal. 2024 , url= on Pith
- [38] Isaac Ong and Amjad Almahairi and Vincent Wu and Wei-Lin Chiang and Tianhao Wu and Joseph E. Gonzalez and M Waleed Kadous and Ion Stoica , booktitle=. Route. 2025 , url= on Pith
- [39] Consistent Estimators for Learning to Defer to an Expert , author=. Proceedings of the 37th International Conference on Machine Learning , series=. 2020 , publisher= on Pith
- [40] Interpretation of Natural Language Rules in Conversational Machine Reading , author=. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages=. 2018 , publisher= on Pith
- [41] Conformal Risk Control , author=. International Conference on Learning Representations , year= on Pith
- [42] Yu, Wenyi and Wang, Siyin and Chiba, Terumi and Chen, Xianzhao and Tian, Xiaohai and Zhang, Jun and Lu, Lu and Zhang, Chao , year=. 2609.34247 , archivePrefix= on PithPith's reviewarXiv
- [43] Geifman, Yonatan and El-Yaniv, Ran , title=. Proceedings of the 36th International Conference on Machine Learning , series=. 2019 , publisher= on Pith
- [44] Huang, Shihong and Wang, Shengjie and Ma, Hong and Xu, Zhou , title=. 2026 , eprint=. doi:10.48550/arXiv.2608.15817 , url= on PithPith's reviewarXivdoi:10.48550/arxiv.2608.15817
- [45] Jia, Zijun and Ye, Yuanchang and Jia, Sen and Qian, Yiyao and Wang, Haoning and Chen, Baojie and Tang, Diyin and Yu, Jinsong and Wang, Zhiyuan , title=. 2026 , eprint=. doi:10.48550/arXiv.2605.20084 , url= on PithPith's reviewarXivdoi:10.48550/arxiv.2605.20084
- [46] JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Models , author=. 2026 , eprint= on Pith
- [47] Decision Hijacking: Prompt Injection Attacks on Jev's Typed Probabilistic Decisions , author=. 2026 , eprint= on Pith
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment