Computation and Language · arXiv:2609.32160 v1 · submitted 26 Sep 2026 · 90 references
Typed Decision Models: An Early Evidence Audit and Evaluation Checklist
The paper says
“Typed decision models (TDMs) return probability distributions over caller-defined options without generating text. TypeSafe released Jev, a commercial typed decision model, on 15 September 2026, and a small body of evaluation and replication work appeared within days.”
more of the abstract
We review 28 papers posted between 19 and 24 September and relate their findings to earlier work on label-probability classification, constrained decoding, reranking, calibration, and model cascades. In this early literature, the typed readout itself has not shown an independent accuracy advantage over comparable label-probability readouts. Jev's clearest gains are in latency and cost, while accuracy gaps remain on harder tasks. In practical deployments, confidence is often used to decide when to defer to a stronger model or a human. We use recurring weaknesses in these studies to derive a 14-item evaluation checklist for future TDM work. Because the evidence covers only the first nine days after the release of one hosted model, the review should be read as an early evidence map rather than a settled assessment of the model class.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 29 Sep 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
The passes disagreed. Reader: holds up. Skeptic: holds up only with conditions. Referee: major revision.
A careful early-evidence audit of typed decision models; the checklist is the real contribution, and the negative accuracy finding is mostly supported but phrased too uniformly in one place. This line is wrong
1 major objection · 3 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 90 references say.
The paper says, at Section 5, Finding 1 (cf. Section 3.4)
The review did not quote this passage.
Pith says: Objection 1 of 1 · major
Finding 1 states that the five papers with label-probability baselines 'point in the same direction' [sentence beginning 'The five papers that do examine label probabilities...']. This over-consolidates the corpus's most controlled comparison, which the paper itself presents two sections earlier.
more of objection 1
In Section 3.4, Ren et al. [54] report a matched single-seed typed-readout model at AUROC .972 against .950 for the label-token readout of the same generative fine-tune, a gap numerically favoring the typed interface on the same backbone, data, and training budget; Finding 1 reports significance only for the three-seed ensemble (.974 vs .950, n.s.) and never states whether the .972-vs-.950 gap is significant. Xu [73] additionally report a clean-accuracy advantage for the typed interface concentrated in tool routing. If the .972-vs-.950 gap is significant, the paper's most controlled comparison would show a task-specific typed advantage and Finding 1's uniformity claim would need to be narrowed; if it is not significant, reporting that would strengthen the 'same direction' claim. The authors should either check and report the single-seed significance in the cited source (and add the .972 value to Table 4's Ren row, which currently omits it), or revise Finding 1 to acknowledge that the best-matched comparison numerically favors the typed readout and that the no-advantage conclusion rests on non-significance in that study plus the other four papers.
3 minor objections
The paper says, at References, [86]
Pith says: minor objection 1 of 3
The title reads 'An Study of Jev-Style Visual Choice Model'; 'An' should be 'A'.
The paper says, at Section 3.4
Pith says: minor objection 2 of 3
The calibration error is reported as 'ECE .050'; use '0.050' for consistency with the decimal style used elsewhere in the paper.
show 1 more
The paper says, at Tables 4 and 5
Pith says: minor objection 3 of 3
The tables are information-dense and rely on multi-purpose footnotes; a short legend row defining 'n.r.', '?', and the asterisk conventions directly beneath each table would improve readability for readers who enter via the tables rather than the text.
What would settle it
Re-read Ren et al. [54] at the seed level (three seeds per arm) and compute the two-sided significance of the matched single-seed AUROC difference, typed .972 vs label-token .950 on CallScreenBench, using a seeded or paired bootstrap consistent with the paper's reported design. If the difference is non-significant, Finding 1 stands as written and no revision is needed. If it is significant, revise Finding 1 to report that the corpus's best-matched comparison numerically and possibly significantly favors the typed readout, replace the �papers point in the same direction� phrasing with a per-paper breakdown, and state the ensemble comparison separately as seed-averaged.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 29 Sep 2026
- Reader holds up
- Skeptic partly agrees: says it holds up only with conditions Finding 1's uniformity claim that five papers 'point in the same direction' rests partly on Ren et al.'s matched single-seed comparison (typed AUROC .972 vs label-token .950), whose significance the review never reports; that point estimate numerically favors the typed readout.
- Editor wrote the verdict above
- Referee major revision: 1 major objection, 3 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Punchline first: this is a careful audit of a very young literature, and its real value is the 14-item checklist with compliance counts, not the negative accuracy finding. The checklist is a genuine organizational contribution; the six findings mostly restate what the reviewed papers already showed. That is fine for an audit, as long as it is priced as one.
The method is solid. Fixed extraction schema, a blind second pass that produced two factual corrections, explicit applicability-versus-compliance coding with unclear cases excluded, per-paper coding in the appendix. The limitations section is candid about the snapshot: nine days, mostly preprints, one hosted version, closed internals, arXiv-only search, and papers still appearing during the review. They do not overclaim mechanism, and they flag the sampling worry themselves.
The central claim — the typed readout has not yet shown an independent accuracy advantage ([b113]) — holds as an absence-of-evidence claim. Only five of 26 scored papers include a label-probability baseline, and those do not establish a readout benefit. But Finding 1 goes one phrase too far when it says the five “point in the same direction” ([b116]). The most controlled of them, Ren et al., has a matched single-seed comparison where the typed readout is numerically ahead (.972 vs .950), and the review does not report whether that gap is significant; the n.s. result it cites is the three-seed ensemble (.974 vs .950), which is not the same estimator. The headline claim survives, because “not yet shown” is an absence-of-evidence statement and a single benchmark cannot fill that absence. But the uniformity sentence over-consolidates the underlying comparisons and should be revised, with the single-seed significance reported or explicitly set aside as underpowered.
The other weak spots are minor and mostly acknowledged: the corpus is self-selected and could be skewed toward hastily evaluated work, and extraction was LLM-assisted, though the blind verification pass reduces that concern. Nothing in the data handling looks load-bearing broken. The citation pattern is appropriate; the lineage section correctly ties TDM components to verbalizer classification, constrained decoding, rerankers, and cascades.
Who should read this: anyone planning to evaluate a typed decision model, and anyone who needs a baseline map of the first-week literature. It deserves a serious referee. Send it to peer review; with the Finding 1 phrasing tightened and the single-seed comparison handled transparently, it is a publishable early evidence audit.
The strongest objection
The central negative claim de�the typed readout has not yet shown an independent accuracy advantage� is carried by five papers stated to �point in the same direction�. Two of them actually contain directional evidence favoring the typed interface, and one of those is the corpus's most controlled comparison. In Ren et al. (secs. 3.4 and 5, Table 4), a generative fine-tune read through label-token probabilities reaches AUROC .950, while a matched single-seed typed-readout model reaches .972 — a 0.022 gap numerically favoring the typed readout on the same backbone, data, and training budget. The review reports statistical significance only for the three-seed ensemble comparison (.974 vs .950, n.s.), which is confounded by seed averaging, and never reports whether the matched single-seed gap is significant. The omission is asymmetrically costly: if .972 vs .950 were n.s., reporting it would strengthen Finding 1, so the silence leaves open that the corpus's best-matched comparison may actually show a typed accuracy advantage. If that gap is significant, the �same direction� characterization over-consolidates heterogeneous evidence — Ren numerically typed-favoring, Xu [73] reporting a clean-accuracy advantage for the typed interface concentrated in tool routing — and Finding 1 would need to be narrowed to �some papers show no advantage, one matched comparison may show a task-specific one�. The abstract's �not a general accuracy advantage� hedge would survive a significant single-benchmark result, so the gist of the claim is not overturned; but the uniformity phrasing in Finding 1, as stated, is not secure without the missing significance test.
The reader's weakest assumption was corpus sampling representativeness and the thinness of the matched-baseline evidence, both of which the authors explicitly acknowledge in section 8. I read the argument in good faith and found the paper unusually careful: a blind second-pass verification produced explicit corrections, the claim is hedged as �not yet shown,� and the conclusion explicitly avoids asserting equivalence. However, my reading locates a more specific, unacknowledged interpretive gap than the reader flagged. Finding 1's supporting sentence says the five label-probability papers �point in the same direction,� but Ren et al.'s matched single-seed comparison (AUROC .972 typed vs .950 label-token) numerically favors the typed readout, and the review only reports significance for the seed-averaged ensemble comparison (.974 vs .950, n.s.), which is not the matched estimate. Whether the matched gap is significant is exactly the fact on which the uniformity characterization turns, and the paper never supplies it. Two of the five papers (Ren, and Xu's tool-routing clean-accuracy advantage) therefore point toward, rather than against, a typed benefit, which is compatible with the careful �general� hedge but inconsistent with the stronger �same direction� phrasing. Credit is due where it belongs: the review does not claim the readouts are equivalent, reports the .972 figure in a subordinate clause rather than suppressing it, and its latency/cost and calibration findings are well documented with source locations and a second-pass check. My concern is bounded — it affects the precision and uniformity of Finding 1, not the general negative conclusion, which survives even a significant single-benchmark result. That is why I recommend a conditional rather than a rejection: attach the single verification of Ren's matched significance and require the characterization to be adjusted accordingly, with per-paper reporting replacing the consolidated phrasing if the test is significant.
The referee's report
If the results hold, the paper is a useful and unusually transparent early map of a fast-moving area, with genuine methodological strengths: a fixed extraction schema; a blind second verification pass that produced two documented factual corrections; applicability-versus-compliance coding with unclear cases excluded from counts; a complete per-paper coding table (Table 8) from which the headline numbers (5/26, 15/27, 1/27) trace directly; and systematic hedging of every strong claim. The negative readout conclusion is grounded in external cited comparisons rather than in the paper's own fitted quantities, and there is no self-citation circularity. The main risk is the thinness of the base under the headline negative claim (five LP-baseline papers, several non-significant or overlapping in seed ranges), but this risk is largely acknowledged in Section 8. What remains unresolved is the accuracy of the 'point in the same direction' characterization in Finding 1, which sits in tension with the paper's own more careful presentation in Section 3.4.
Pith's reading of the paper (a machine's summary, not the authors')
The paper reviews the first 28 studies of typed decision models (TDMs), an interface in which a model returns probabilities over caller-defined options instead of generating text, published in the nine days after TypeSafe released Jev. Its central claim is negative and carefully hedged: the typed readout itself has not yet shown an independent accuracy advantage over an ordinary language model's label-probability readout, while Jev's clearest gains are in latency and cost. The review matters because TDMs are being deployed as cheap first stages that defer hard cases to stronger models, and the corpus shows that reported parity often occurs on saturated tests, that calibration is workload-dependent, and that confidence-based escalation pays off only when the two stages do not share the same errors. The paper converts the recurring weaknesses into a 14-item evaluation checklist with compliance counts, so that the next round of evidence can be compared and trusted. The authors are explicit that this is an early evidence map of a fast-moving area, not a settled assessment of the model class.
What carries the argument. The carrying object is the typed readout itself, defined by three properties: the option set is closed at request time, several isolated questions can share one state in a single forward execution, and the returned probability distribution is the product intended for downstream thresholding, ranking, or routing. The paper's analytical move is to separate this interface from the underlying model, asking whether the first wave of reported gains belongs to the typed interface or to the backbone, task training, or serving design behind it, and the load-bearing comparison is therefore the label log-probability baseline that reads the same options from an ordinary or same-backbone language model. The second machinery piece is the 14-item evaluation checklist (baselines B1-B3, test design T1-T4, reproducibility R1-R3, and confidence-and-deployment items C1-C4), which turns the recurring failures, such as missing baselines, saturated tests, post-hoc design choices, unreported intervals, thresholds tuned on evaluation data, and caching-unadjusted latency, into scorable requirements with corpus-level compliance counts.
Core claim. On the paper's own terms, the finding is a non-result with a practical payload: across the 27 evaluated papers, only five include a label log-probability baseline from an ordinary or same-backbone model, and those five all point the same way. A dedicated typed head and the ordinary LM head reach the same macro accuracy (.761 vs .761) when backbone, data, and budget are matched; a generative fine-tune read through label-token probabilities is not significantly worse than a three-seed typed ensemble (AUROC .950 vs .974); and a restricted option-logit readout equals greedy generation whenever the generation is valid, so its only gain is handling invalid generations. Because accuracy gains therefore cannot be attributed to the typed interface without a matched baseline, Jev's demonstrated value is assigned to speed and cost (median response times around 0.15 to 0.45 seconds, fees tens to hundreds of times below frontier models), while the corpus shows genuine accuracy gaps on harder tasks, such as 14.6 points on JudgeBench and 19.8 on hard RM-Bench pairs, and frequent parity claims on tests where every strong model is near ceiling. Calibration is inconsistent across workloads, with overconfident wrong answers and thresholds that fail to transfer, and the strongest deployment results use the TDM as an inexpensive first stage of a cascade rather than as a universal replacement. The deliverable is a 14-item checklist whose compliance counts show how thin the early evidence is: only 5 of 26 scored papers include the key label-probability baseline, only 1 of 5 cascade studies compares against a cheap generative cascade, and only 1 of 27 tests option-name invariance.
The premise it rests on. The load-bearing premise is that a nine-day, self-selected corpus of 28 arXiv preprints, in which only five studies include a same-backbone label-probability baseline and several of those are non-significant or overlapping in seed ranges, is representative enough to support both the negative readout claim and the checklist's compliance counts; if the snapshot skews toward hastily evaluated papers, the conclusion could be an artifact of sampling rather than a fact about typed decision models.
If the paper is right (Pith's inferences, not the authors')
Accuracy gains attributed to a TDM cannot be trusted unless a matched label-probability baseline from the same backbone is reported, since the five studies that include one find no significant advantage for the typed readout.
The defensible case for TDMs in production is currently latency and cost rather than accuracy, with reported median response times of 0.15-0.45 seconds and fees tens to hundreds of times below frontier models, while accuracy gaps remain on harder tasks.
Parity claims on saturated tests should be read as ceiling effects rather than model equivalence, since several high-profile parity results come from tests on which nearly every strong model scores at or near 100 percent.
Confidence-based escalation pays off only when the fallback model's errors are complementary to the TDM's, because when a stronger judge repeats the TDM's wrong answer, as it does in 96 percent of verdicts in one rubric study, escalation recovers almost nothing.
Calibration must be evaluated per model and workload on held-out data, since a single temperature or threshold chosen on a selection set can fail to transfer to held-out items, other workloads, or other fallback models.
If the next wave of studies follows the checklist, the question this corpus cannot answer, namely whether a well-trained typed head ever beats its own backbone's label probabilities on a hard, non-saturated benchmark, becomes directly testable, and the paper's own numbers suggest any such edge will be small.
The checklist transfers beyond TDMs: any model that returns scores for downstream gating, including rerankers, reward models, and verifiers, faces the same three traps the audit found, so the 14 items read naturally as a general protocol for score-based model claims.
The vendor-side implication the authors leave implicit is that comparisons asking LLMs to verbalize probabilities rather than expose token log-probabilities likely understate how close ordinary models already are to the typed readout, and a public log-probability benchmark would settle that gap quickly.
Because all hosted studies cover essentially one model version within one week, the durability of the latency and cost advantage across Jev versions, workloads, and prevalence shifts is untested, and a longitudinal re-run of a few corpus benchmarks after future releases would be the cheapest high-value follow-up.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Typed decision models (TDMs) return probability distributions over caller-defined options without generating text. TypeSafe released Jev, a commercial typed decision model, on 15 September 2026, and a small body of evaluation and replication work appeared within days. We review 28 papers posted between 19 and 24 September and relate their findings to earlier work on label-probability classification, constrained decoding, reranking, calibration, and model cascades. In this early literature, the typed readout itself has not shown an independent accuracy advantage over comparable label-probability readouts. Jev's clearest gains are in latency and cost, while accuracy gaps remain on harder tasks. In practical deployments, confidence is often used to decide when to defer to a stronger model or a human. We use recurring weaknesses in these studies to derive a 14-item evaluation checklist for future TDM work. Because the evidence covers only the first nine days after the release of one hosted model, the review should be read as an early evidence map rather than a settled assessment of the model class.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-29
The authors' main claim, as the classifier picked it out 0.91
“In this early literature, the typed readout itself has not shown an independent accuracy advantage over comparable label-probability readouts.”
How the main claim is stated: asserted 0.98
What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.97 · human participants 0.08 · controlled comparison 0.69 · randomized assignment 0.02 · states quantitative result 0.85 · compares with prior results 0.92 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Typed decision models (TDMs) return probability distributions over caller-defined options without generating text. | background 0.98 | 0.07 | 0.04 | asserted 1.00 |
| TypeSafe released Jev, a commercial typed decision model, on 15 September 2026, and a small body of evaluation and replication work appeared within days. | background 0.97 | 0.09 | 0.04 | asserted 1.00 |
| We review 28 papers posted between 19 and 24 September and relate their findings to earlier work on label-probability classification, constrained decoding, reranking, calibration, and model cascades. | method 0.57 | 0.91 | 0.04 | asserted 0.94 |
| In this early literature, the typed readout itself has not shown an independent accuracy advantage over comparable label-probability readouts. | result 0.98 | 0.35 | 0.51 | asserted 0.98 |
| Jev's clearest gains are in latency and cost, while accuracy gaps remain on harder tasks. | result 0.94 | 0.46 | 0.22 | asserted 1.00 |
| In practical deployments, confidence is often used to decide when to defer to a stronger model or a human. | background 0.99 | 0.06 | 0.03 | asserted 0.84 |
| We use recurring weaknesses in these studies to derive a 14-item evaluation checklist for future TDM work. | contribution 0.95 | 0.96 | 0.22 | asserted 1.00 |
| Because the evidence covers only the first nine days after the release of one hosted model, the review should be read as an early evidence map rather than a settled assessment of the model class. | limitation 1.00 | 0.44 | 0.17 | asserted 0.76 |
What kind of paper the classifier reports this is: review 1.00 · evidence from literature 1.00
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-29: 1 request, 6,197 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
References (90)
- [54] Simiao Ren, Kidus Zewde, Xingyu Shen, Yuchen Zhou, Dennis Ng, Ankit Raj, Tommy Duong, Yuxin Zhang, and Neo Tiangratanakul. Open-Jev Judgments on CallScreenBench: Calibrated 19 One-Pass Scam Screening with a Small Language Model. arXiv preprint arXiv:2609.23959, 2026 on PithPith's reviewarXiv
- [73] Zixiang Xu. JevOut: Natural Context Can Flip Decision Models. arXiv preprint arXiv:2609.30243, 2026 on PithPith's reviewarXiv
- [1] Arian Abbasi, Alan Aqrawi, and Ted Kwartler. Control the Harness, Control the Cost: Routing and Governing AI Coding Agents in the Enterprise. arXiv preprint arXiv:2609.28919, 2026 on PithPith's reviewarXiv
- [2] Pranjal Aggarwal, Aman Madaan, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang, Shyam Upadhyay, Manaal Faruqui, and Mausam. AutoMix: Automatically Mixing Language Models. InAdvances in Neural ... on Pith
- [3] Diogo Almeida. Introducing System One Models & Jev. TypeSafe AI blog,https://typesafe. ai/blog/introducing-system-one-models-and-jev, September 2026. Accessed 2026-09-24 on Pith
- [4] Anonymous Authors. Decisions, Not Tokens: A Survey of Typed, Calibrated, and Machine- Native Decision Models from Classical Classifiers to Jev.https://github.com/youzizzz1028/ Awesome-Jev, 2026. Full working draft, 21 September 2026. Accessed 2026-09-25 on Pith
- [5] Debangshu Banerjee, Tarun Suresh, Shubham Ugare, Sasa Misailovic, and Gagandeep Singh. CRANE: Reasoning with constrained LLM generation. InInternational Conference on Machine Learning (ICML), 2025 on Pith
- [6] David Bani-Harouni, Chantal Pellegrini, Paul Stangel, Ege Özsoy, Kamilia Zaripova, Nassir Navab, and Matthias Keicher. Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models. InInternational Conference on Learning Repres... on Pith
- [7] Joachim Baumann, Paul Röttger, Aleksandra Urman, Albert Wendsjö, Flor Miriam Plaza-del- Arco, Johannes B. Gruber, and Dirk Hovy. Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation. arXiv preprint arXiv:2509.08825, 2025 on PitharXiv
- [8] Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation. InInternational Conference on Machine Learning (ICML), pages 3658–3673, 2024 on Pith
- [9] Ralph Allan Bradley and Milton E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons.Biometrika, 39(3/4):324–345, 1952. doi: 10.2307/2334029 on Pithdoi:10.2307/2334029
- [10] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, ArielHerbert-Voss, GretchenKrueger, TomHenighan, RewonChild, AdityaRamesh, DanielM. Ziegler, Jeffrey W... on Pith
- [11] Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance.Transactions on Machine Learning Research, 2024 on Pith
- [12] Zehua Cheng, Wei Dai, and Jiahao Sun. this-that-model-1.0: A typed decision model that decides in 30 ms, for a millionth of a cent. arXiv preprint arXiv:2609.23886, 2026 on PithPith's reviewarXiv
- [13] C.-K. Chow. On Optimum Recognition Error and Reject Tradeoff.IEEE Transactions on Information Theory, 16(1):41–46, 1970. doi: 10.1109/TIT.1970.1054406 on PitharXiv
- [14] Volkan Dağlı, Zerrin Dağlı, and Dağhan Dağlı. Universal Fractal Natural Language Decision Map: Real-TimeEdgeTriageAcrossHeterogeneousDomains. arXivpreprintarXiv:2609.25498, 2026 on PithPith's reviewarXiv
- [15] Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld, Leshem Choshen, Yoon Kim, and Jacob Andreas. Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty. InInternational Conference on Learning Representations (ICLR), 2026 on Pith
- [16] Boyuan Deng, Shuyi Fan, Hongyang Zhang, and Xinhong Xie. Jev for Scientific Decisions: Evaluating Semantic Choices and Their Consequences. arXiv preprint arXiv:2609.24965, 2026. Version 2, 23 September 2026 on PithPith's reviewarXiv
- [17] Yixin Dong, Charlie F. Ruan, Yaxing Cai, Ziyi Xu, Yilong Zhao, Ruihang Lai, and Tianqi Chen. XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models. InProceedings of Machine Learning and Systems (MLSys), 2025 on Pith
- [18] Joas Antonio dos Santos Barbosa. Calibrated Decision Models for Autonomous Penetration- Testing Harnesses: JEV and Laya as System One Decision Layers for LLM-Driven Pentest Agents. arXiv preprint arXiv:2609.28940, 2026 on PithPith's reviewarXiv
show 70 more
- [19] Naoki Egami, Musashi Hinck, Brandon M. Stewart, and Hanying Wei. Using Imperfect Sur- rogates for Downstream Inference: Design-based Supervised Learning for Social Science Ap- plications of Large Language Models. InAdvances in Neural Information Processing Systems, volume 36, 2023. 16 on Pith
- [20] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few- shot learners. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors,Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Intern... on Pith
- [21] Yonatan Geifman and Ran El-Yaniv. Selective Classification for Deep Neural Networks. In Advances in Neural Information Processing Systems, 2017 on Pith
- [22] Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West. Grammar-constrained de- coding for structured NLP tasks without finetuning. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processi... on Pithdoi:10.18653/v1/2023.emnlp-main.674
- [23] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On Calibration of Modern Neural Networks. InInternational Conference on Machine Learning (ICML), 2017 on Pith
- [24] Ruoqi Guo, Yi Liu, Gelei Deng, Yuekang Li, Lida Zhao, Yutao Wu, Simin Chen, Ying Zhang, and Leo Yu Zhang. Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero- Shot Detector of AI Alignment Failures. arXiv preprint arXiv:2609.29429, 2026 on PithPith's reviewarXiv
- [25] Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Aditya Kr- ishna Menon, and Sanjiv Kumar. Language Model Cascades: Token-level uncertainty and beyond. InInternational Conference on Learning Representations (ICLR), 2024 on Pith
- [26] Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form competition: Why the highest probability answer isn’t always right. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors,Proceedings of the 2021 Conference ... on Pithdoi:10.18653/v1/2021.emnlp-main.564
- [27] Jiaju Huang, Hao Yang, Xinyu Ma, Xinglong Liang, Kunyan Cai, Junqiang Ma, Shaobin Chen, Yue Sun, and Tao Tan. Can Jev Judge Radiology Reports? Evaluating a System One Model for Clinical Factuality. arXiv preprint arXiv:2609.27607, 2026 on PithPith's reviewarXiv
- [28] Hazem Ibrahim and Yasir Zaki. Evaluating Decision Models for Text Annotation in Com- putational Social Science. arXiv preprint arXiv:2609.24574, 2026. Version 2, 23 September 2026 on PithPith's reviewarXiv
- [29] Dongming Jiang, Yi Li, and Bingzhe Li. Jev-Mem: System-One-Controlled Agentic Memory for Efficient AI Agents. arXiv preprint arXiv:2609.23986, 2026 on PithPith's reviewarXiv
- [30] Wittawat Jitkrittum, Neha Gupta, Aditya Krishna Menon, Harikrishna Narasimhan, Ankit Singh Rawat, and Sanjiv Kumar. When Does Confidence-Based Cascade Deferral Suf- fice? InAdvances in Neural Information Processing Systems, 2023. 17 on Pith
- [31] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav For... on PithPith's reviewarXiv
- [32] Daniel Kahneman.Thinking, Fast and Slow. Farrar, Straus and Giroux, New York, 2011 on Pith
- [33] Amita Kamath, Robin Jia, and Percy Liang. Selective question answering under domain shift. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors,Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5684–5696, Onlin... on Pithdoi:10.18653/v1/2020.acl-main
- [34] Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. Prometheus: Inducing Fine- grained Evaluation Capability in Language Models. InInternational Conference on Learning Representati... on Pith
- [35] Moritz Laurer, Wouter van Atteveldt, Andreu Casas, and Kasper Welbers. Building Efficient Universal Classifiers with Natural Language Inference. arXiv preprint arXiv:2312.17543, 2023 on PithPith's reviewarXiv
- [37] Delong Li, Xu Wang, Haochen Gong, Rui Lang, and Guangsheng Yu. Fast Intent-Driven Service Orchestration with Jev for 6G Edge Networks. arXiv preprint arXiv:2609.23136, 2026 on PithPith's reviewarXiv
- [38] Hengyu Li. KITE: Scaling Jev Population Experiments with Sparse Flagship Calibration. arXiv preprint arXiv:2609.27535, 2026 on PithPith's reviewarXiv
- [39] Yubo Li, Yidi Miao, Ramayya Krishnan, and Rema Padman. JEV-as-a-Judge: Accept When Confident, Escalate When Unsure. arXiv preprint arXiv:2609.26550, 2026 on PithPith's reviewarXiv
- [40] Guoming Ling, Muen Xue, and Zijian Ye. Jev in the Wild: A Data-Driven Analysis of the Jev Model’s Functionality, Applications and Ecosystem. arXiv preprint arXiv:2609.30216, 2026 on PithPith's reviewarXiv
- [41] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G- eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Pr... on Pithdoi:10.18653/v1/2023.emnlp-main.153
- [42] Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style. InInternational Conference on Learning Representations (ICLR), 2025. 18 on Pith
- [43] Meng Luo. Jev and Typed Decision Models: An Empirical Survey of Calibration, Selective Control, and Open Implementations.https://github.com/Eurekaleo/awesome-jev-survey, on Pith
- [44] Weiyu Ma, Liangbing Zhao, Yongcheng Zeng, and Jian Zhao. JEV-Star: Fast, Low-Cost StarCraft II Control with Language-Model Planning. arXiv preprint arXiv:2609.27331, 2026 on PithPith's reviewarXiv
- [45] David Madras, Toniann Pitassi, and Richard Zemel. Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer. InAdvances in Neural Information Processing Systems, pages 6150–6160, 2018 on Pith
- [46] Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Noisy channel language model prompting for few-shot text classification. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors,Proceedings of the 60th Annual Meeting of the Association for Computa... on Pith
- [47] HusseinMozannarandDavidSontag. ConsistentEstimatorsforLearningtoDefertoanExpert. InInternational Conference on Machine Learning (ICML), pages 7076–7087, 2020 on Pith
- [48] Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. Document ranking with a pretrained sequence-to-sequence model. In Trevor Cohn, Yulan He, and Yang Liu, edi- tors,Findings of the Association for Computational Linguistics: EMNLP 2020, pages 708–718, Online, Novembe... on Pithdoi:10.18653/v1/2020
- [49] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and... on Pith
- [50] Kanghee Park, Jiayu Wang, Taylor Berg-Kirkpatrick, Nadia Polikarpova, and Loris D’Antoni. Grammar-Aligned Decoding. InAdvances in Neural Information Processing Systems, 2024 on Pith
- [51] Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of options in multiple-choice questions. In Kevin Duh, Helena Gomez, and Steven Bethard, editors,Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006– 2017, Mexi... on Pith
- [52] Amir Rafe and Subasish Das. Calibrated Decisions at Scale: Converting Police Crash Nar- ratives into Probabilistic Crash Variables with a System One Model (Jev). arXiv preprint arXiv:2609.24052, 2026 on PithPith's reviewarXiv
- [53] Delip Rao and Chris Callison-Burch. JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places. arXiv preprint arXiv:2609.29769, 2026 on PithPith's reviewarXiv
- [55] Joshua Robinson and David Wingate. Leveraging Large Language Models for Multiple Choice Question Answering. InInternational Conference on Learning Representations (ICLR), 2023 on Pith
- [56] Werner Robitza. JEVQA - Video Quality from Metadata, Bitstream, and Pixel Features with a General-Purpose Decision Model. arXiv preprint arXiv:2609.24395, 2026 on PithPith's reviewarXiv
- [57] Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classifica- tion and natural language inference. In Paola Merlo, Jorg Tiedemann, and Reut Tsar- faty, editors,Proceedings of the 16th Conference of the European Chapter of the Associa- tion for Compu... on Pithdoi:10.18653/v1/2021.eacl-main.20
- [58] TimoSchickandHinrichSchütze. It’snotjustsizethatmatters: Smalllanguagemodelsarealso few-shot learners. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors,Proceedi... on Pith
- [59] Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors,Proceedings of the 2021 Conference on Empi... on Pithdoi:10.18653/v1/2021.emnlp-main.779
- [60] Ihor Stepanov, Mykhailo Shtopko, Dmytro Vodianytskyi, Oleksandr Lukashov, Alexander Ya- vorskyi, and Mykyta Yaroshenko. GLiClass: Generalist Lightweight Model for Sequence Clas- sification Tasks. arXiv preprint arXiv:2508.07662, 2025 on PithPith's reviewarXiv
- [61] Yu Sun, Junhao Xu, Jiajia Shi, and Zijin Yang. Type-Safe Is Not Error-Free: A Con- strained Decision Head Follows the Option Name, Not the Rubric Bound to It. arXiv preprint arXiv:2609.26758, 2026. Version 2, 23 September 2026 on PithPith's reviewarXiv
- [62] Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen. Let me speak freely? a study on the impact of format restrictions on large lan- guage model performance. In Franck Dernoncourt, Daniel Preoţiuc-Pietro, and Anastasia Shimorina, editors,Pro... on Pithdoi:10.18653/v1/2024.emn
- [63] Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y. Tang, Alejandro Cuadron, Chen- guang Wang, Raluca Ada Popa, and Ion Stoica. JudgeBench: A Benchmark for Evaluating LLM-based Judges. InInternational Conference on Learning Representations (ICLR), 2025. 20 on Pith
- [64] Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Houda Bouamor, ... on Pith
- [65] TypeSafe AI. System One Adapter (Python).https://github.com/typesafe-ai/ system-one-adapter-python, 2026. Accessed 2026-09-24 on Pith
- [66] TypeSafe AI. TypeSafe AI Documentation.https://docs.typesafe.ai/introduction, 2026. Accessed 2026-09-24 on Pith
- [67] Neeraj Varshney and Chitta Baral. Model cascading: Towards jointly improving efficiency and accuracy of NLP systems. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, ed- itors,Proceedings of the 2022 Conference on Empirical Methods in Natural Language Pro- cessing, pages 11... on Pithdoi:10.18653/v1/2022.emnlp-main.756
- [68] Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. “my answer is C”: First-token probabilities do not match text answers in instruction-tuned language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, edi... on Pithdoi:10.18653/v1/2024.findings-acl.441
- [69] Brandon T. Willard and Rémi Louf. Efficient Guided Generation for Large Language Models. arXiv preprint arXiv:2307.09702, 2023 on PithPith's reviewarXiv
- [70] Tiantong Wu and Wei Yang Bryan Lim. Decision Hijacking: Prompt Injection Attacks on Jev’s Typed Probabilistic Decisions. arXiv preprint arXiv:2609.28613, 2026 on PithPith's reviewarXiv
- [71] Tiantong Wu and Wei Yang Bryan Lim. REFLEX with Jev for Efficient Selective Control in LLM Agents. arXiv preprint arXiv:2609.26532, 2026 on PithPith's reviewarXiv
- [72] Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. InInternational Conference on Learning Representations (ICLR), 2024 on Pith
- [74] Weiwei Ye, Hangchen Liu, and Renhe Jiang. NumericJev: Jev-like LLM Numerical Decoding with Multiway Decision Trees. arXiv preprint arXiv:2609.28587, 2026 on PithPith's reviewarXiv
- [75] Wenpeng Yin, Jamaal Hay, and Dan Roth. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors,Proceedings of the 2019 Conference on Empirical Methods in Natural 21 Language Pro... on Pithdoi:10.18653/v1/d19-1404
- [76] Guanxu Yu and Yuhang Yao. Visual Jev: Accurate and Efficient Decisions from Shared Visual Context. arXiv preprint arXiv:2609.25845, 2026 on PithPith's reviewarXiv
- [77] Bianca Zadrozny and Charles Elkan. Transforming Classifier Scores into Accurate Multiclass Probability Estimates. InProceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 694–699, 2002. doi: 10.1145/775047.775151 on PitharXiv
- [78] Urchade Zaratiana, Nadi Tomeh, Pierre Holat, and Thierry Charnois. GLiNER: General- ist model for named entity recognition using bidirectional transformer. In Kevin Duh, He- lena Gomez, and Steven Bethard, editors,Proceedings of the 2024 Conference of the North American Chapte... on Pithdoi:10.18653/v1/2024
- [79] Fan Zhang, Yankai Chen, Zhuohan Xie, Yixi Zhou, Sijia Peng, Lei Fan, Xinhua Ji, Cunyuan Zheng, Huangyong Shan, Philip S. Yu, Xue Liu, Yu Chen, Preslav Nakov, and Songwei He. Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understanding. ... on PithPith's reviewarXiv
- [80] Linghua Zhang. Jev-Mobile: Jev as an Executor for Mobile GUI Agents. arXiv preprint arXiv:2609.30186, 2026 on PithPith's reviewarXiv
- [81] Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative Verifiers: Reward Modeling as Next-Token Prediction. InInternational Conference on Learning Representations (ICLR), 2025 on Pith
- [82] Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 Embed- ding: Advancing Text Embedding and Reranking Through Foundation Models. arXiv preprint arXiv:2506.05176, 2025 on PithPith's reviewarXiv
- [83] Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate Before Use: ImprovingFew-ShotPerformanceofLanguageModels. InInternational Conference on Machine Learning (ICML), pages 12697–12706, 2021 on Pith
- [84] Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large Language Models Are Not Robust Multiple Choice Selectors. InInternational Conference on Learning Represen- tations (ICLR), 2024 on Pith
- [85] Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine Heller, and Subhrajit Roy. Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering. InInternational Conference on Learning Representations (ICLR), 2024 on Pith
- [86] Xunlan Zhou, Xianliang Yang, and Li Zhao. From Text Decisions to Pixels: An Study of Jev-Style Visual Choice Model. arXiv preprint arXiv:2609.29283, 2026. 22 on PithPith's reviewarXiv
- [87] Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. Can large language models transform computational social science?Computational Linguistics, 50(1):237–291, March 2024. doi: 10.1162/coli_a_00502. URLhttps://aclanthology.org/ 2024.cl-1.8/. A Corpu... on PithPith's reviewarXiv
- [503] URLhttps://aclanthology.org/2020.acl-main.503/ on Pith
- [2021] Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.185. URL https://aclanthology.org/2021.naacl-main.185/ on Pithdoi:10.18653/v1/2021.naacl-main.185
- [2022] Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.365. URL https://aclanthology.org/2022.acl-long.365/ on Pithdoi:10.18653/v1/2022.acl-long.365
- [2026] Working draft, version 0.2.0, data cutoff 24 September 2026. Accessed 2026-09-25 on Pith
Cited by 13 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- Track or Follow? A Jev-Compatible Multi-Agent Trading System When Web News Lags the Market cs.CL · Nov 2026
- Confidence Is Not a Promise: A Reference Audit of Typed Decision Models cs.LG · Oct 2026
- Entity Matching using Decision Models cs.CL · Oct 2026
- Joint or Per-Option? Compact Encoders for Closed-Set Decisions from 17 Million to 1 Billion Parameters cs.CL · Oct 2026
- Ranked but Not Stopped: Detecting Prompt-Injection Hijacks from Typed Decision Models' Output Probabilities cs.CL · Nov 2026
- FlipJev: Fast Black-Box Red Teaming Finds Universal Adversarial Suffixes for Typed Decision Models cs.CR · Oct 2026
- Cross-Primitive Consistency in Jev cs.CV · Oct 2026
- Understanding Jev-Style System One Models: Does Structure Reduce Selection Bias? cs.CL · Oct 2026
- Measure Before You Train: Typed Decision Models vs. a Classical Baseline vs. a Generative LLM for Routing a Personal Multilingual Inbox cs.CL · Oct 2026
- Typed Decision Models under Prevalence Shift and Label Budgets: Decision Training, Calibration, and Implicit Priors cs.LG · Oct 2026
- StartLux-Decision: AutoThink Decision Models with Reinforcement Learning for Calibrated Deliberative Decisions cs.CL · Oct 2026
- Where Can a Decision Model Diagnose HVAC Faults? Reasoning Demand, Physical Representation, and Robustness Under Shift cs.AI · Oct 2026
- VisualJevBench: A Benchmark and Black-Box Attacks for Visual Typed Decision Models cs.CR · Nov 2026
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment