Artificial Intelligence · arXiv:2610.17205 v1 · submitted 12 Oct 2026 · 66 references
Releasing the Inner Jev: A Simple Recipe for Open Decision Models
The paper says
“LLM-based decision-making systems, also called System One models, have recently gained popularity, most notably through Jev (Almeida, 2026).”
more of the abstract
Given a typed input state, a question, and a set of options, they return a probability for each option in a single forward pass, without text generation. We show that existing open instruction-tuned language models can achieve comparable performance without any additional training, by reading out answers from letters (A, B, C, etc.) in the model's output logits at the last token position. Furthermore, we show how a simple supervised fine-tuning recipe can improve their performance further. We evaluate four open instruction-tuned language models: Gemma 4 31B, Gemma 4 12B, Qwen3.6-27B, and Qwen3.5-4B. We fine-tune each using LoRA and a decision head initialized from a slice of the language model head. Furthermore, we introduce OpenDecisions, a novel 2,352-question validation dataset designed to reflect the task composition of JevBench (JevBench main- tainers, 2026) and which computes a Capability score based on accuracy and calibration metrics. Without fine-tuning, Gemma 4 31B scores 79.6 on OpenDecisions, compared to Jev's 80.1. After undergoing our fine-tuning recipe, it scores 84.4 on OpenDecisions and 81.5 on JevBench v1.7, outperforming Jev by 4.3 and 4.4 Capability points respectively and setting a new state-of-the-art on OpenDecisions. On a suite of public decision datasets, our models achieve accuracy comparable to Jev, with lower calibration error. Our results show that open language models already possess System One capabilities, and with simple fine-tuning they outperform Jev on a wide range of decision tasks. We release our models, serving code, and OpenDecisions.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 14 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
The passes disagreed. Reader: holds up. Skeptic: holds up only with conditions. Referee: major revision.
Solid empirical paper showing open instruction-tuned LLMs can match a proprietary decision system with a simple letter-logit readout and cheap LoRA finetuning. The central claim holds up despite some caveats. This line is wrong
3 major objections · 5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 66 references say.
The paper says, at Abstract and Section 5.3 (Table14)
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The claim that Jeff-1.0-Large outperforms Jev by 4.4 Capability points on the JevBench leaderboard (v1.7) is based on comparing two different draws: Jev was measured on revision v1.6.1, Jeff-1.0-Large on v1.6.2 (Table14). The paper itself notes that 'each system is measured on its own draw, so a difference of 0.2 points is within noise' when comparing Jeff-1.0-Large to Quyet-1.0-Large.
more of objection 1
The same logic applies to the Jev comparison, yet the paper does not caution the reader. Moreover,Jev's own Intelligence score varies by8.3 points between OpenDecisions (71.9) and the leaderboard (63.6), indicating that question-set difficulty can substantially shift absolute scores. Without a system evaluated on both draws, part or all of the reported4.4-point advantage could reflect draw difficulty rather than model quality. This is a load-bearing issue for the most impactful claim in the paper.
The paper says, at Section5.3 (Table3 and leaderboard comparison)
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
The paper repeatedly states that the trained model 'outperforms Jev by4.3 and4.4 Capability points respectively' (Abstract, Figure2 caption, Section5.2), but the4.4-point leaderboard gap is not accompanied by the cross-draw caveat. The manuscript should either (a) provide an indirect comparison via a system evaluated on both draws (e.g., the zero-shot model on both draws) or (b) qualify the claim as 'exceeds Jev's published score on a different draw' and state the uncertainty explicitly. As written, the headline number is misleading.
The paper says, at Section3 (OpenDecisions validity)
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.75):
“Finally, because OpenDecisions was used to select the training recipe, it is a development set rather than an independent test set. 17 Releasing the Inner Jev: A Simple Recipe for Open Decision Models Table 14.Official JevBench leaderboard(JevBench maintainers, 2026): the systems with the highest Capability, as published in October 2026 (revision v1.6.7).”
Pith says: Objection 3 of 3 · major
OpenDecisions is acknowledged as a development set (Section C.1), and its construction involves two models related to the training supervision: Qwen3.6-27B provides teacher labels, and Qwen3.8-27B participates in the label-panel verification. The paper notes this potential bias (Section6) and shows high Spearman correlation (0.90) with the leaderboard for14 public systems.
more of objection 3
However, the0.90 correlation does not guarantee that improvements on OpenDecisions transfer to truly independent evaluations; the residual error of3.0 Capability points (Section3) is non-negligible. The main training recipe was selected on OpenDecisions, so the OpenDecisions comparison to Jev (84.4 vs80.1) is not a valid generalization claim by itself. The held-out datasets (Table4) provide stronger evidence, but they show the trained31B model essentially ties Jev (Macro89.0 vs88.0) rather than exceeding it. The paper should temper the claim that the recipe 'exceeds Jev on a wide range of decision tasks' to better reflect the evidence.
5 minor objections
The paper says, at Abstract and Introduction
Pith says: minor objection 1 of 5
The abstract states 'outperforming Jev by4.3 and4.4 Capability points respecively' without qualifying the draw difference for the leaderboard number. At the very least, a footnote should indicate that the leaderboard comparison uses different question draws.
The paper says, at Section5.2 (first paragraph)
Pith says: minor objection 2 of 5
The phrase 'leading Jev by4.3 ±2.5' on OpenDecisions is fine, but the next sentence should also note that OpenDecisions is a development set; this is acknowledged in Section C.1 but should be restated here.
show 3 more
The paper says, at Table14 and surrounding text
Pith says: minor objection 3 of 5
Table14 lists draws for each system. The text should explicitly state that Jev's draw (v1.6.1) and Jeff's draw (v1.6.2) are different and that direct comparison is confounded by draw difficulty. This is currently omitted.
The paper says, at Section6 (Discussion and limitations)
Pith says: minor objection 4 of 5
The limitations paragraph acknowledges the leaderboard comparison only indirectly (through the general caveat about OpenDecisions not representing all applications). It should also mention the cross-draw comparison issue.
The paper says, at General
Pith says: minor objection 5 of 5
The paper is otherwise thorough and well-written. The zero-shot results and ablations are particularly compelling. The training details and data checks are commendable.
What would settle it
Submit Jev and Jeff-1.0-Large to the same JevBench v1.7 draw (i.e., the same revision, with the same 1,500 items). If the Capability gap is 4.4 ± 1.96 SE (approximately 2.5 points or more), the draw-confound concern is resolved. If the gap shrinks to ≤2 points or flips, the headline claim about the leaderboard is overstated and the paper's central result rests more heavily on OpenDecisions, whose acknowledged limitations (development set, Qwen-based labels) would then become more consequential.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 14 Oct 2026
- Reader holds up
- Skeptic partly agrees: says it holds up only with conditions Leaderboard draws are not shared: the 4.4-point margin over Jev on JevBench v1.7 compares separate question sets without a common reference, and the paper's own estimate of per-draw variance (0.2-point noise for a similar gap) undermines the precision of the claim.
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
You should know two things about this paper. First, it is a competently executed study showing that existing open LLMs (Gemma 4, Qwen3.6) can match the proprietary Jev decision model zero-shot and exceed it after a cheap LoRA finetune. Second, the main caveats are honestly acknowledged but real: the leaderboard draw comparison is slightly weaker than advertised, abd OpenDecisions is a development set.
What's new? Not the letter-logit readout — that is in prior work (SemIf). Not the LoRA finetuning — that is standard. The novelty is the systematic demonstration that these tools together are sufficient: temperature scaling helps calibration, and a 5 GPU–hour LoRA run on an instruction–tuned model pulls ahead of the proprietary alternative. That is useful for practitioners who want to avpid vendor lock–in.
The evaluation is thorough: three seeds, error bars, overlap checks, seven held–out public datasets, and the independent JevBench leaderboard. The ablations cleanly attribute improvement to LoRa rather than the learnable head. The code, models, and OpenDecisions are released. These are genuine strengths.
Soft spots in proportion. The stress-test about leaderboard draws is osrt of right but overstated. Jeff–1.0–Large and Jev were measured on different draws (v1.6.2 vs v1.6.1), and the paper does not confirm equal difficulty. That matters for the headline 4.4–point claim. But the OpenDecisions comparison (84.4 vs 80.1 on the same 2,352 questions) gives a consistent 4.3–point advantage, so the robuster support comes from the authors' own development set, not the leaderboard. The paper acknowledges OpenDecisions is a development set (Section C.1), a point the reader and stress–test both repeat. The label quality concern (Qwen3.6–27B trains, Qwen3.8–27B eval labels) is also flagged by the authors. None of this sinks the paper, but a reasonable referee should ask for an independent held–out set for the final comparison, or at least a sensitivity analysis.
Smaller models benefit less; the 12B and 4B gains are modest or zero. That limits the generality of the recipe but does not invalidate the conclusions for the 30B class.
Wh o is this for? Practitioners who want to build a decision system on open models and need a straightforward recipe. It confirms that expensive specialized architecture s and RL–based training are unnecessary for reaching current leaderboard parity.
Would I accept for review? Yes. The paper is well–executed, the claims are (with the draw caveat) supported by mutiple lines of evidence, abd the realeased artifacts are useful. A smart referee can tighten the leaderboard discussion and ask for an independent validation set, but the core result will surviev moderate scrutiny.
The strongest objection
The paper's central evidence that a short LoRA fine-tune 'exceeds Jev by 4.4 Capability points' on the official JevBench leaderboard (v1.7) rests on comparing two different leaderboard draws: Jeff-1.0-Large was measured on revision v1.6.2, Jev on v1.6.1 (Table 14). The paper itself notes that 'each system is measured on its own draw, so a difference of 0.2 points is within noise' when comparing Jeff-1.0-Large to Quyet-1.0-Large (81.5 vs 81.7). This same design—different draws, no shared baseline—applies to the 4.4-point gap against Jev. The paper provides no evidence that the v1.6.1 and v1.6.2 draws are of comparable difficulty. In fact, Jev's own Intelligence score varies by 8.3 points between OpenDecisions (71.9, Table 3) and the leaderboard (63.6, Table 14), hinting that question-set difficulty can shift Capability substantially. Without a system evaluated on both draws, part or all of the reported 4.4-point advantage could reflect draw difficulty rather than model quality. The paper acknowledges this issue for small gaps but does not extend the same caution to the headline comparison, making the leaderboard-based outperformance claim less secure than it appears.
The reader correctly identified OpenDecisions as the weakest assumption but treated the JevBench leaderboard as a clean independent validation. The leaderboard comparison, however, shares a structural weakness: different draws without a common reference. The paper's own statement that a 0.2-point gap is 'within noise' due to separate draws implies that the 4.4-point gap carries similar, unquantified uncertainty—yet the paper does not apply that caveat to its headline comparison. The held-out datasets (Table 4) show a much smaller advantage (89.0 vs 88.0 macro accuracy), and the knowledge-heavy benchmarks show a clear deficit (88.8 vs 92.9). The 4.4-point JevBench margin is the strongest quantitative claim; if it is partially confounded, the paper's main conclusion softens from 'solidly exceeds Jev' to 'comparable on most metrics, leads on a carefully constructed development set, and shows a noisy advantage on a leaderboard with uncontrolled draws.' A single shared-draw evaluation would resolve this and strengthen the paper. Hence CONDITIONAL: the verdict should be finalized after this check.
The referee's report
If the core claims hold, this work would be significant: it demonstrates that open-weight instruction-tuned LLMs already possess decision-making capabilities comparable to a proprietary system, and that a simple supervised fine-tuning recipe can further improve them—all without specialized architectures or reinforcement learning. The zero-shot finding (Figure 2) is especially valuable as it suggests that expensive proprietary training may be unnecessary for many decision tasks. The ablations (Table 5) cleanly identify LoRA adaptation of the backbone, rather than head training, as the source of gain. However, the headline result that fine-tuning 'exceeds Jev by 4.4 Capability points' on the official leaderboard is less secure than it appears.
Pith's reading of the paper (a machine's summary, not the authors')
The paper addresses the problem of making fast, single-forward-pass decisions with calibrated probabilities using large language models, in the style of a system called Jev. The authors show that you do not need Jev's specialized architecture or proprietary training. Instead, if you format the decision as a multiple-choice prompt with letter options (A, B, C...), an instruction-tuned open LLM already gives competitive probabilities by taking the logits for those letters at the last token position. A simple temperature scaling fix (dividing logits by a number >1) solves the overconfidence issue, bringing Gemma 4 31B within 0.5 capability points of Jev on the authors' new benchmark, OpenDecisions.
The second result is that a very short supervised fine-tuning run - 400 steps, about 5 GPU hours - with low-rank adapters (LoRA) improves the 31B model by about 5 capability points, beating Jev by over 4 points. The improvement comes almost entirely from adapting the LLM itself, not from learning a new decision head: fixing the head at the original letter-rows and only training LoRA gives the same benefit. The authors release their best model (Jeff-1.0-Large) and the validation dataset OpenDecisions.
The paper includes thorough ablations on training length, LoRA rank, data sources, and calibration. It also evaluates on seven public decision datasets and on the official JevBench v1.7 leaderboard, where Jeff-1.0-Large scores 81.5 capability (second place, 0.2 behind the leader, 4.4 above Jev). The main limitation is that OpenDecisions was used to select the recipe, making it a development set; however, the independent leaderboard results reinforce the central claim.
Core claim. Existing open instruction-tuned language models, using only a letter-logit readout (Eq. 1) and temperature scaling, achieve performance comparable to the proprietary Jev model on decision tasks without any additional training; after a short supervised LoRA fine-tuning procedure (400 steps, ~5 GPU-hours), they exceed Jev on the OpenDecisions benchmark and the JevBench leaderboard (v1.7). Specifically, Gemma 4 31B scores 79.6 zero-shot (vs. Jev 80.1) and 84.4 after training (vs. Jev 80.1), and 81.5 on the JevBench leaderboard (vs. Jev 77.1).
The premise it rests on. The evaluation that drives recipe selection and the main comparison relies on OpenDecisions, a validation dataset constructed by the authors. While OpenDecisions shows high Spearman correlation (0.90) with the JevBench leaderboard for 14 public systems, its labels come from a three-model panel (including Qwen3.8-27B, a model related to the Qwen3.6-27B used for training supervision), and 60% of items are LLM-authored. The paper acknowledges that OpenDecisions was used to select the recipe, making it a development set (Section C.1). The independent JevBench v1.7 leaderboard results (second place, 81.5 vs. leader 81.7) partially mitigate this, but the training recipe was optimized for OpenDecisions, and the leaderboard win over Jev is more convincing than the win over the top public system (0.2 points behind Quyet-1.0-Large).
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
LLM-based decision-making systems, also called System One models, have recently gained popularity, most notably through Jev (Almeida, 2026). Given a typed input state, a question, and a set of options, they return a probability for each option in a single forward pass, without text generation. We show that existing open instruction-tuned language models can achieve comparable performance without any additional training, by reading out answers from letters (A, B, C, etc.) in the model's output logits at the last token position. Furthermore, we show how a simple supervised fine-tuning recipe can improve their performance further. We evaluate four open instruction-tuned language models: Gemma 4 31B, Gemma 4 12B, Qwen3.6-27B, and Qwen3.5-4B. We fine-tune each using LoRA and a decision head initialized from a slice of the language model head. Furthermore, we introduce OpenDecisions, a novel 2,352-question validation dataset designed to reflect the task composition of JevBench (JevBench main- tainers, 2026) and which computes a Capability score based on accuracy and calibration metrics. Without fine-tuning, Gemma 4 31B scores 79.6 on OpenDecisions, compared to Jev's 80.1. After undergoing our fine-tuning recipe, it scores 84.4 on OpenDecisions and 81.5 on JevBench v1.7, outperforming Jev by 4.3 and 4.4 Capability points respectively and setting a new state-of-the-art on OpenDecisions. On a suite of public decision datasets, our models achieve accuracy comparable to Jev, with lower calibration error. Our results show that open language models already possess System One capabilities, and with simple fine-tuning they outperform Jev on a wide range of decision tasks. We release our models, serving code, and OpenDecisions.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-14
The authors' main claim, as the classifier picked it out 1.00
“Our results show that open language models already possess System One capabilities, and with simple fine-tuning they outperform the classifier on a wide range of decision tasks.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.09 · states limitation 0.12 · human participants 0.04 · controlled comparison 0.89 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.98 · mentions code or data availability 0.98
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| LLM-based decision-making systems, also called System One models, have recently gained popularity, most notably through Jev (Almeida, 2026). | background 1.00 | 0.04 | 0.03 | asserted 1.00 |
| Given a typed input state, a question, and a set of options, they return a probability for each option in a single forward pass, without text generation. | background 0.97 | 0.06 | 0.03 | asserted 1.00 |
| We show that existing open instruction-tuned language models can achieve comparable performance \textit{without any additional training}, by reading out answers from letters (A, B, C, etc.) in the model's output logits at the last token position. | contribution 0.61 | 0.91 | 0.30 | asserted 1.00 |
| Furthermore, we show how a simple supervised fine-tuning recipe can improve their performance further. | contribution 0.66 | 0.85 | 0.07 | asserted 1.00 |
| We evaluate four open instruction-tuned language models: Gemma 4 31B, Gemma 4 12B, Qwen3.6-27B, and Qwen3.5-4B. | method 0.96 | 0.86 | 0.02 | asserted 0.85 |
| We fine-tune each using LoRA and a decision head initialized from a slice of the language model head. | method 1.00 | 0.95 | 0.02 | asserted 0.91 |
| Furthermore, we introduce OpenDecisions, a novel 2,352-question validation dataset designed to reflect the task composition of JevBench (JevBench main- tainers, 2026) and which computes a Capability score based on accuracy and calibration metrics. | contribution 0.68 | 0.95 | 0.04 | asserted 0.99 |
| Without fine-tuning, Gemma 4 31B scores 79.6 on OpenDecisions, compared to Jev's 80.1. | result 1.00 | 0.89 | 0.06 | asserted 1.00 |
| After undergoing our fine-tuning recipe, it scores 84.4 on OpenDecisions and 81.5 on JevBench v1.7, outperforming Jev by 4.3 and 4.4 Capability points respectively and setting a new state-of-the-art on OpenDecisions. | result 1.00 | 0.97 | 0.20 | asserted 1.00 |
| On a suite of public decision datasets, our models achieve accuracy comparable to Jev, with lower calibration error. | result 1.00 | 0.95 | 0.13 | asserted 1.00 |
| Our results show that open language models already possess System One capabilities, and with simple fine-tuning they outperform Jev on a wide range of decision tasks. | interpretation 0.97 | 0.95 | 0.93 | asserted 1.00 |
| We release our models, serving code, and OpenDecisions. | contribution 0.99 | 0.80 | 0.04 | asserted 0.79 |
What kind of paper the classifier reports this is: method or tool 0.79 · evidence from observation or experiment 0.98
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-14: 1 request, 9,175 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (5)
Figures are the authors', reproduced from the paper.
References (66)
- [1] allebee. JevK5 . https://github.com/allebee/jevk5, 2026 on Pith
- [2] Almeida, D. Introducing system one models & Jev . TypeSafe AI blog, https://typesafe.ai/blog/introducing-system-one-models-and-jev, 2026 on Pith
- [3] Anthropic . Claude Opus model family. https://www.anthropic.com/claude/opus, 2026 on Pith
- [4] blockbrain-ai . Cygnet recipe. https://github.com/blockbrain-ai/cygnet-recipe, 2026 on Pith
- [5] Borkan, D., Dixon, L., Sorensen, J., Thain, N., and Vasserman, L. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion Proceedings of the 2019 World Wide Web Conference, 2019. URL https://dl.acm.org/doi/10.1145/3308560.3317593 on Pithdoi:10.1145/3308560.3317593
- [6] Brier, G. W. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78 0 (1): 0 1--3, 1950. URL https://journals.ametsoc.org/view/journals/mwre/78/1/1520-0493_1950_078_0001_vofeit_2_0_co_2.xml on Pith
- [7] Cai, Z. Open-Jev-27B v1.1. https://huggingface.co/ZefanCai/Open-Jev-27B-v1.1, 2026 on Pith
- [8] Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., and Specia, L. SemEval -2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), 2017. URL https://aclant... on Pith
- [9] Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M., Gonzalez, J. E., and Stoica, I. Chatbot arena: An open platform for evaluating LLMs by human preference. In International Conference on Machine Learning (ICML), 2024. URL ... on Pith
- [10] Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ : Exploring the surprising difficulty of natural yes/no questions. In Proceedings of NAACL-HLT, 2019. URL https://aclanthology.org/N19-1300/ on Pith
- [11] Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try ARC , the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. URL https://arxiv.org/abs/1803.05457 on PithPith's reviewarXiv
- [12] Cloudflare . Clef . https://huggingface.co/Cloudflare/clef, 2026 on Pith
- [13] Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. URL https://arxiv.org/abs/2110.14168 on PithPith's reviewarXiv
- [14] Conover, M., Hayes, M., Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., and Xin, R. Free Dolly : Introducing the world's first truly open instruction-tuned LLM . Databricks blog, 2023. URL https://www.databricks.com/blog/2023/04/12/dolly-first-ope... on Pith
- [15] crh225. Plumb-4B . https://github.com/crh225/plumb, 2026 on Pith
- [16] de Marneffe, M.-C., Simons, M., and Tonhauser, J. The CommitmentBank : Investigating projection in naturally occurring discourse. In Proceedings of Sinn und Bedeutung 23, 2019. URL https://ojs.ub.uni-konstanz.de/sub/index.php/sub/article/view/601 on Pith
- [17] denis-pplx . AutoJev-27B . https://huggingface.co/denis-pplx/autojev-27b, 2026 on Pith
- [18] Desai, S. and Durrett, G. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. URL https://aclanthology.org/2020.emnlp-main.21/ on Pith
- [19] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT : Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo... on Pithdoi:10.18653/v1/n19-1423
- [20] EldanRing. Winnow-12B . https://huggingface.co/EldanRing/Winnow-12B, 2026 on Pith
show 46 more
- [21] Epstein, E. S. A scoring system for probability forecasts of ranked categories. Journal of Applied Meteorology, 8 0 (6): 0 985--987, 1969. URL https://journals.ametsoc.org/view/journals/apme/8/6/1520-0450_1969_008_0985_assfpf_2_0_co_2.xml on Pith
- [22] Gemma Team, Google DeepMind . Gemma 4 model card (gemma-4-31b-it, gemma-4-12b-it). https://huggingface.co/google/gemma-4-31B-it, 2026. Accessed October 2026 on Pith
- [23] Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In International Conference on Machine Learning (ICML), 2017. URL https://proceedings.mlr.press/v70/guo17a.html on Pith
- [24] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2021 a . URL https://openreview.net/forum?id=d7KBjmI3GmQ on Pith
- [25] Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021 b . ... on Pith
- [26] HopitAI. Hopper. https://huggingface.co/HopitAI/hopper, 2026 on Pith
- [27] Howard, J. and Ruder, S. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 328--339. Association for Computational Linguistics, 2018. doi:10.186... on Pithdoi:10.18653/v1/p18-1031
- [28] IkerMoel. open-alternative-jev. https://github.com/ikermoel/open-alternative-jev, 2026 on Pith
- [29] JevBench maintainers . JevBench : a benchmark for Jev -type decision models. https://benchmarkheaven.com/jev-models; harness: https://github.com/fstandhartinger/jevbench, 2026. The paper reports results for versions 1.6.x and 1.7; accessed October 10, 2026 on Pith
- [30] Kadavath, S. et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. URL https://arxiv.org/abs/2207.05221 on PithPith's reviewarXiv
- [31] Karandikar, A., Cain, N., Tran, D., Lakshminarayanan, B., Shlens, J., Mozer, M. C., and Roelofs, B. Soft calibration objectives for neural networks. In Advances in Neural Information Processing Systems, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/f8905bd3df64ace64... on Pith
- [32] Kennedy, C. J., Bacon, G., Sahn, A., and von Vacano, C. Constructing interval variables via faceted Rasch measurement and multitask deep learning: a hate speech application. arXiv preprint arXiv:2009.10277, 2020. URL https://arxiv.org/abs/2009.10277v1. Version 1, September 22, 2020 on PithPith's reviewarXiv
- [33] krishna-gogineni-765 . deck31b. https://github.com/krishna-gogineni-765/deck31b, 2026 on Pith
- [34] kshetrajna12. reflex. https://github.com/kshetrajna12/reflex, 2026 on Pith
- [35] Kumar, A., Sarawagi, S., and Jain, U. Trainable calibration measures for neural networks from kernel mean embeddings. In Proceedings of the 35th International Conference on Machine Learning, 2018. URL https://proceedings.mlr.press/v80/kumar18a.html on Pith
- [36] Li, X. and Roth, D. Learning question classifiers. In Proceedings of the 19th International Conference on Computational Linguistics (COLING), 2002. URL https://aclanthology.org/C02-1150/ on Pith
- [37] Li, Y. and Wagle, J. LLM-as-Jev : LLMs are already Jev -style decision models -- when and how to fine-tune them. arXiv preprint arXiv:2610.02076, 2026. URL https://arxiv.org/abs/2610.02076v2. Version 2 on PithPith's reviewarXiv
- [38] Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023. URL https://arxiv.org/abs/2305.20050 on PithPith's reviewarXiv
- [39] Liu, A., Swayamdipta, S., Smith, N. A., and Choi, Y. WANLI : Worker and AI collaboration for natural language inference dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2022, 2022. URL https://aclanthology.org/2022.findings-emnlp.508/ on Pith
- [40] Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7 on Pith
- [41] Marosi, M. decider: one-pass typed decisions with calibrated probabilities. https://github.com/Mapika/decider, 2026. Apache-2.0; accessed October 2026 on Pith
- [42] Mukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, P. H. S., and Dokania, P. K. Calibrating deep neural networks using focal loss. In Advances in Neural Information Processing Systems, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/aeb7b30ef1d024a76f21a1d40e3... on Pith
- [43] Nguyen, C. Quyet 1.0: calibrated decision models. https://huggingface.co/chinhnc/Quyet-1.0-Large, 2026. Model family includes Large, Medium, Small, and Tiny variants; accessed October 10, 2026 on Pith
- [44] Nie, Y., Zhou, X., and Bansal, M. What can we learn from collective human opinions on natural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. URL https://aclanthology.org/2020.emnlp-main.734/ on Pith
- [45] OpenAI . gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025. URL https://arxiv.org/abs/2508.10925 on PithPith's reviewarXiv
- [46] Pakdaman Naeini, M., Cooper, G. F., and Hauskrecht, M. Obtaining well calibrated probabilities using Bayesian binning. In AAAI Conference on Artificial Intelligence, 2015. URL https://ojs.aaai.org/index.php/AAAI/article/view/9602 on Pith
- [47] Pezeshkpour, P. and Hruschka, E. Large language models sensitivity to the order of options in multiple-choice questions. arXiv preprint arXiv:2308.11483, 2023. URL https://arxiv.org/abs/2308.11483 on PithPith's reviewarXiv
- [48] Qwen Team . Qwen3.5-4B model card. https://huggingface.co/Qwen/Qwen3.5-4B, 2026 a on Pith
- [49] Qwen Team . Qwen3.6-27B model card. https://huggingface.co/Qwen/Qwen3.6-27B, 2026 b on Pith
- [50] Qwen Team . Qwen3.8-27B model card. https://huggingface.co/Qwen/Qwen3.8-27B, 2026 c on Pith
- [51] Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training. OpenAI technical report, 2018. URL https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf on Pith
- [52] Robinson, J. and Wingate, D. Leveraging large language models for multiple choice question answering. In International Conference on Learning Representations (ICLR), 2023. URL https://openreview.net/forum?id=yKbprarjc5B on Pith
- [53] Sakaguchi, K., Le Bras, R., Bhagavatula, C., and Choi, Y. WinoGrande : An adversarial Winograd schema challenge at scale. In AAAI Conference on Artificial Intelligence, 2020. URL https://ojs.aaai.org/index.php/AAAI/article/view/6399 on Pith
- [54] Schick, T. and Sch \"u tze, H. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp.\ 255--269. Association fo... on Pithdoi:10.18653/v1/2021.eacl-main.20
- [55] SemIf contributors . SemIf : zero-shot decision scoring with letter-token logits. https://github.com/TheoLeeCJ/SemIf-OpenJev, 2026. MIT license; accessed October 2026 on Pith
- [56] Stepanov, I., Shtopko, M., Vodianytskyi, D., Lukashov, O., Yavorskyi, A., and Yaroshenko, M. GLiClass : Generalist lightweight model for sequence classification tasks. arXiv preprint arXiv:2508.07662, 2025. URL https://arxiv.org/abs/2508.07662 on PithPith's reviewarXiv
- [57] Tian, K., Mitchell, E., Zhou, A., Sharma, A., Rafailov, R., Yao, H., Finn, C., and Manning, C. D. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empir... on Pith
- [58] torchcast-ai . Torchcast-decision-12B . https://huggingface.co/torchcast-ai/torchcast-decision-12b, 2026 on Pith
- [59] Vicentino, C. Eikos : open, calibrated, single-pass typed-decision models for finance and trading. https://huggingface.co/caiovicentino1/Eikos-27B, 2026 on Pith
- [60] Wang, Y. et al. MMLU-Pro : A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/ad236edc564... on Pith
- [61] Welbl, J., Liu, N. F., and Gardner, M. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, 2017. URL https://aclanthology.org/W17-4413/ on Pith
- [62] Yin, W., Hay, J., and Roth, D. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pro... on Pithdoi:10.18653/v1/d19-1404
- [63] Yinsongxu. LLM2Jev . https://github.com/Yinsongxu/LLM2Jev, 2026 on Pith
- [64] Z.ai . GLM-4.7-Flash model card. https://huggingface.co/zai-org/GLM-4.7-Flash, 2026 on Pith
- [65] Zhang, J., Yang, T., Shi, Y., and Wu, L. AnyJev : Turn any LLM into a Jev -style decision model. https://github.com/nokia-applied-research/AnyJev, 2026 on Pith
- [66] Zheng, C., Zhou, H., Meng, F., Zhou, J., and Huang, M. Large language models are not robust multiple choice selectors. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=shr9PXz7T0 on Pith
Cited by 11 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- Joint or Per-Option? Compact Encoders for Closed-Set Decisions from 17 Million to 1 Billion Parameters cs.CL · Oct 2026
- From Large Language Models to Efficient System One Models via Adaptive Computation cs.CL · Nov 2026
- SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models cs.CL · Nov 2026
- TandemBench: Benchmarking Decision Models for Multi-Agent Collaboration cs.CL · Nov 2026
- Cross-Primitive Consistency in Jev cs.CV · Oct 2026
- Understanding Jev-Style System One Models: Does Structure Reduce Selection Bias? cs.CL · Oct 2026
- PathJev: Decision-Native Vision-Language Modelling for Histopathology cs.CV · Oct 2026
- Commit or Abstain? System One Decision Models, LLMs, and Missing Legal Authority cs.AI · Oct 2026
- EmbeddingGemma 2: Unified and Efficient Multimodal Representations cs.LG · Oct 2026
- Multimodal Reranking Is a System One Decision cs.IR · Oct 2026
- Decision Model Factory: Building Specialized Sovereign Decision Models cs.AI · Oct 2026
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment