Pith.

Artificial Intelligence · arXiv:2610.17205 v1 · submitted 12 Oct 2026 · 66 references

Releasing the Inner Jev: A Simple Recipe for Open Decision Models

Jonathan Geuter, Andrew Cheng, David Alvarez-Melis

The paper says

“LLM-based decision-making systems, also called System One models, have recently gained popularity, most notably through Jev (Almeida, 2026).”

more of the abstract

Given a typed input state, a question, and a set of options, they return a probability for each option in a single forward pass, without text generation. We show that existing open instruction-tuned language models can achieve comparable performance without any additional training, by reading out answers from letters (A, B, C, etc.) in the model's output logits at the last token position. Furthermore, we show how a simple supervised fine-tuning recipe can improve their performance further. We evaluate four open instruction-tuned language models: Gemma 4 31B, Gemma 4 12B, Qwen3.6-27B, and Qwen3.5-4B. We fine-tune each using LoRA and a decision head initialized from a slice of the language model head. Furthermore, we introduce OpenDecisions, a novel 2,352-question validation dataset designed to reflect the task composition of JevBench (JevBench main- tainers, 2026) and which computes a Capability score based on accuracy and calibration metrics. Without fine-tuning, Gemma 4 31B scores 79.6 on OpenDecisions, compared to Jev's 80.1. After undergoing our fine-tuning recipe, it scores 84.4 on OpenDecisions and 81.5 on JevBench v1.7, outperforming Jev by 4.3 and 4.4 Capability points respectively and setting a new state-of-the-art on OpenDecisions. On a suite of public decision datasets, our models achieve accuracy comparable to Jev, with lower calibration error. Our results show that open language models already possess System One capabilities, and with simple fine-tuning they outperform Jev on a wide range of decision tasks. We release our models, serving code, and OpenDecisions.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 14 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

The passes disagreed. Reader: holds up. Skeptic: holds up only with conditions. Referee: major revision.

Solid empirical paper showing open instruction-tuned LLMs can match a proprietary decision system with a simple letter-logit readout and cheap LoRA finetuning. The central claim holds up despite some caveats. This line is wrong

3 major objections · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 66 references say.

The paper says, at Abstract and Section 5.3 (Table14)

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The claim that Jeff-1.0-Large outperforms Jev by 4.4 Capability points on the JevBench leaderboard (v1.7) is based on comparing two different draws: Jev was measured on revision v1.6.1, Jeff-1.0-Large on v1.6.2 (Table14). The paper itself notes that 'each system is measured on its own draw, so a difference of 0.2 points is within noise' when comparing Jeff-1.0-Large to Quyet-1.0-Large.

more of objection 1

The same logic applies to the Jev comparison, yet the paper does not caution the reader. Moreover,Jev's own Intelligence score varies by8.3 points between OpenDecisions (71.9) and the leaderboard (63.6), indicating that question-set difficulty can substantially shift absolute scores. Without a system evaluated on both draws, part or all of the reported4.4-point advantage could reflect draw difficulty rather than model quality. This is a load-bearing issue for the most impactful claim in the paper.

The paper says, at Section5.3 (Table3 and leaderboard comparison)

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

The paper repeatedly states that the trained model 'outperforms Jev by4.3 and4.4 Capability points respectively' (Abstract, Figure2 caption, Section5.2), but the4.4-point leaderboard gap is not accompanied by the cross-draw caveat. The manuscript should either (a) provide an indirect comparison via a system evaluated on both draws (e.g., the zero-shot model on both draws) or (b) qualify the claim as 'exceeds Jev's published score on a different draw' and state the uncertainty explicitly. As written, the headline number is misleading.

The paper says, at Section3 (OpenDecisions validity)

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.75):

“Finally, because OpenDecisions was used to select the training recipe, it is a development set rather than an independent test set. 17 Releasing the Inner Jev: A Simple Recipe for Open Decision Models Table 14.Official JevBench leaderboard(JevBench maintainers, 2026): the systems with the highest Capability, as published in October 2026 (revision v1.6.7).”

Pith says: Objection 3 of 3 · major

OpenDecisions is acknowledged as a development set (Section C.1), and its construction involves two models related to the training supervision: Qwen3.6-27B provides teacher labels, and Qwen3.8-27B participates in the label-panel verification. The paper notes this potential bias (Section6) and shows high Spearman correlation (0.90) with the leaderboard for14 public systems.

more of objection 3

However, the0.90 correlation does not guarantee that improvements on OpenDecisions transfer to truly independent evaluations; the residual error of3.0 Capability points (Section3) is non-negligible. The main training recipe was selected on OpenDecisions, so the OpenDecisions comparison to Jev (84.4 vs80.1) is not a valid generalization claim by itself. The held-out datasets (Table4) provide stronger evidence, but they show the trained31B model essentially ties Jev (Macro89.0 vs88.0) rather than exceeding it. The paper should temper the claim that the recipe 'exceeds Jev on a wide range of decision tasks' to better reflect the evidence.

5 minor objections

The paper says, at Abstract and Introduction

Pith says: minor objection 1 of 5

The abstract states 'outperforming Jev by4.3 and4.4 Capability points respecively' without qualifying the draw difference for the leaderboard number. At the very least, a footnote should indicate that the leaderboard comparison uses different question draws.

The paper says, at Section5.2 (first paragraph)

Pith says: minor objection 2 of 5

The phrase 'leading Jev by4.3 ±2.5' on OpenDecisions is fine, but the next sentence should also note that OpenDecisions is a development set; this is acknowledged in Section C.1 but should be restated here.

show 3 more

The paper says, at Table14 and surrounding text

Pith says: minor objection 3 of 5

Table14 lists draws for each system. The text should explicitly state that Jev's draw (v1.6.1) and Jeff's draw (v1.6.2) are different and that direct comparison is confounded by draw difficulty. This is currently omitted.

The paper says, at Section6 (Discussion and limitations)

Pith says: minor objection 4 of 5

The limitations paragraph acknowledges the leaderboard comparison only indirectly (through the general caveat about OpenDecisions not representing all applications). It should also mention the cross-draw comparison issue.

The paper says, at General

Pith says: minor objection 5 of 5

The paper is otherwise thorough and well-written. The zero-shot results and ablations are particularly compelling. The training details and data checks are commendable.

What would settle it

Submit Jev and Jeff-1.0-Large to the same JevBench v1.7 draw (i.e., the same revision, with the same 1,500 items). If the Capability gap is 4.4 ± 1.96 SE (approximately 2.5 points or more), the draw-confound concern is resolved. If the gap shrinks to ≤2 points or flips, the headline claim about the leaderboard is overstated and the paper's central result rests more heavily on OpenDecisions, whose acknowledged limitations (development set, Qwen-based labels) would then become more consequential.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 14 Oct 2026

  1. Reader holds up confidence high
  2. Skeptic partly agrees: says it holds up only with conditions 1 min 34 s · 33,151 tokens read, 8,882 written Leaderboard draws are not shared: the 4.4-point margin over Jev on JevBench v1.7 compares separate question sets without a common reference, and the paper's own estimate of per-draw variance (0.2-point noise for a similar gap) undermines the precision of the claim.
  3. Editor wrote the verdict above 16 s · 34,129 tokens read, 1,504 written
  4. Referee major revision: 3 major objections, 5 minor 35 s · 35,322 tokens read, 1,550 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

You should know two things about this paper. First, it is a competently executed study showing that existing open LLMs (Gemma 4, Qwen3.6) can match the proprietary Jev decision model zero-shot and exceed it after a cheap LoRA finetune. Second, the main caveats are honestly acknowledged but real: the leaderboard draw comparison is slightly weaker than advertised, abd OpenDecisions is a development set.

What's new? Not the letter-logit readout — that is in prior work (SemIf). Not the LoRA finetuning — that is standard. The novelty is the systematic demonstration that these tools together are sufficient: temperature scaling helps calibration, and a 5 GPU–hour LoRA run on an instruction–tuned model pulls ahead of the proprietary alternative. That is useful for practitioners who want to avpid vendor lock–in.

The evaluation is thorough: three seeds, error bars, overlap checks, seven held–out public datasets, and the independent JevBench leaderboard. The ablations cleanly attribute improvement to LoRa rather than the learnable head. The code, models, and OpenDecisions are released. These are genuine strengths.

Soft spots in proportion. The stress-test about leaderboard draws is osrt of right but overstated. Jeff–1.0–Large and Jev were measured on different draws (v1.6.2 vs v1.6.1), and the paper does not confirm equal difficulty. That matters for the headline 4.4–point claim. But the OpenDecisions comparison (84.4 vs 80.1 on the same 2,352 questions) gives a consistent 4.3–point advantage, so the robuster support comes from the authors' own development set, not the leaderboard. The paper acknowledges OpenDecisions is a development set (Section C.1), a point the reader and stress–test both repeat. The label quality concern (Qwen3.6–27B trains, Qwen3.8–27B eval labels) is also flagged by the authors. None of this sinks the paper, but a reasonable referee should ask for an independent held–out set for the final comparison, or at least a sensitivity analysis.

Smaller models benefit less; the 12B and 4B gains are modest or zero. That limits the generality of the recipe but does not invalidate the conclusions for the 30B class.

Wh o is this for? Practitioners who want to build a decision system on open models and need a straightforward recipe. It confirms that expensive specialized architecture s and RL–based training are unnecessary for reaching current leaderboard parity.

Would I accept for review? Yes. The paper is well–executed, the claims are (with the draw caveat) supported by mutiple lines of evidence, abd the realeased artifacts are useful. A smart referee can tighten the leaderboard discussion and ask for an independent validation set, but the core result will surviev moderate scrutiny.

The strongest objection

The paper's central evidence that a short LoRA fine-tune 'exceeds Jev by 4.4 Capability points' on the official JevBench leaderboard (v1.7) rests on comparing two different leaderboard draws: Jeff-1.0-Large was measured on revision v1.6.2, Jev on v1.6.1 (Table 14). The paper itself notes that 'each system is measured on its own draw, so a difference of 0.2 points is within noise' when comparing Jeff-1.0-Large to Quyet-1.0-Large (81.5 vs 81.7). This same design—different draws, no shared baseline—applies to the 4.4-point gap against Jev. The paper provides no evidence that the v1.6.1 and v1.6.2 draws are of comparable difficulty. In fact, Jev's own Intelligence score varies by 8.3 points between OpenDecisions (71.9, Table 3) and the leaderboard (63.6, Table 14), hinting that question-set difficulty can shift Capability substantially. Without a system evaluated on both draws, part or all of the reported 4.4-point advantage could reflect draw difficulty rather than model quality. The paper acknowledges this issue for small gaps but does not extend the same caution to the headline comparison, making the leaderboard-based outperformance claim less secure than it appears.

The reader correctly identified OpenDecisions as the weakest assumption but treated the JevBench leaderboard as a clean independent validation. The leaderboard comparison, however, shares a structural weakness: different draws without a common reference. The paper's own statement that a 0.2-point gap is 'within noise' due to separate draws implies that the 4.4-point gap carries similar, unquantified uncertainty—yet the paper does not apply that caveat to its headline comparison. The held-out datasets (Table 4) show a much smaller advantage (89.0 vs 88.0 macro accuracy), and the knowledge-heavy benchmarks show a clear deficit (88.8 vs 92.9). The 4.4-point JevBench margin is the strongest quantitative claim; if it is partially confounded, the paper's main conclusion softens from 'solidly exceeds Jev' to 'comparable on most metrics, leads on a carefully constructed development set, and shows a noisy advantage on a leaderboard with uncontrolled draws.' A single shared-draw evaluation would resolve this and strengthen the paper. Hence CONDITIONAL: the verdict should be finalized after this check.

The referee's report

If the core claims hold, this work would be significant: it demonstrates that open-weight instruction-tuned LLMs already possess decision-making capabilities comparable to a proprietary system, and that a simple supervised fine-tuning recipe can further improve them—all without specialized architectures or reinforcement learning. The zero-shot finding (Figure 2) is especially valuable as it suggests that expensive proprietary training may be unnecessary for many decision tasks. The ablations (Table 5) cleanly identify LoRA adaptation of the backbone, rather than head training, as the source of gain. However, the headline result that fine-tuning 'exceeds Jev by 4.4 Capability points' on the official leaderboard is less secure than it appears.

Pith's reading of the paper (a machine's summary, not the authors')

The paper addresses the problem of making fast, single-forward-pass decisions with calibrated probabilities using large language models, in the style of a system called Jev. The authors show that you do not need Jev's specialized architecture or proprietary training. Instead, if you format the decision as a multiple-choice prompt with letter options (A, B, C...), an instruction-tuned open LLM already gives competitive probabilities by taking the logits for those letters at the last token position. A simple temperature scaling fix (dividing logits by a number >1) solves the overconfidence issue, bringing Gemma 4 31B within 0.5 capability points of Jev on the authors' new benchmark, OpenDecisions.

The second result is that a very short supervised fine-tuning run - 400 steps, about 5 GPU hours - with low-rank adapters (LoRA) improves the 31B model by about 5 capability points, beating Jev by over 4 points. The improvement comes almost entirely from adapting the LLM itself, not from learning a new decision head: fixing the head at the original letter-rows and only training LoRA gives the same benefit. The authors release their best model (Jeff-1.0-Large) and the validation dataset OpenDecisions.

The paper includes thorough ablations on training length, LoRA rank, data sources, and calibration. It also evaluates on seven public decision datasets and on the official JevBench v1.7 leaderboard, where Jeff-1.0-Large scores 81.5 capability (second place, 0.2 behind the leader, 4.4 above Jev). The main limitation is that OpenDecisions was used to select the recipe, making it a development set; however, the independent leaderboard results reinforce the central claim.

Core claim. Existing open instruction-tuned language models, using only a letter-logit readout (Eq. 1) and temperature scaling, achieve performance comparable to the proprietary Jev model on decision tasks without any additional training; after a short supervised LoRA fine-tuning procedure (400 steps, ~5 GPU-hours), they exceed Jev on the OpenDecisions benchmark and the JevBench leaderboard (v1.7). Specifically, Gemma 4 31B scores 79.6 zero-shot (vs. Jev 80.1) and 84.4 after training (vs. Jev 80.1), and 81.5 on the JevBench leaderboard (vs. Jev 77.1).

The premise it rests on. The evaluation that drives recipe selection and the main comparison relies on OpenDecisions, a validation dataset constructed by the authors. While OpenDecisions shows high Spearman correlation (0.90) with the JevBench leaderboard for 14 public systems, its labels come from a three-model panel (including Qwen3.8-27B, a model related to the Qwen3.6-27B used for training supervision), and 60% of items are LLM-authored. The paper acknowledges that OpenDecisions was used to select the recipe, making it a development set (Section C.1). The independent JevBench v1.7 leaderboard results (second place, 81.5 vs. leader 81.7) partially mitigate this, but the training recipe was optimized for OpenDecisions, and the leaderboard win over Jev is more convincing than the win over the top public system (0.2 points behind Quyet-1.0-Large).

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

LLM-based decision-making systems, also called System One models, have recently gained popularity, most notably through Jev (Almeida, 2026). Given a typed input state, a question, and a set of options, they return a probability for each option in a single forward pass, without text generation. We show that existing open instruction-tuned language models can achieve comparable performance without any additional training, by reading out answers from letters (A, B, C, etc.) in the model's output logits at the last token position. Furthermore, we show how a simple supervised fine-tuning recipe can improve their performance further. We evaluate four open instruction-tuned language models: Gemma 4 31B, Gemma 4 12B, Qwen3.6-27B, and Qwen3.5-4B. We fine-tune each using LoRA and a decision head initialized from a slice of the language model head. Furthermore, we introduce OpenDecisions, a novel 2,352-question validation dataset designed to reflect the task composition of JevBench (JevBench main- tainers, 2026) and which computes a Capability score based on accuracy and calibration metrics. Without fine-tuning, Gemma 4 31B scores 79.6 on OpenDecisions, compared to Jev's 80.1. After undergoing our fine-tuning recipe, it scores 84.4 on OpenDecisions and 81.5 on JevBench v1.7, outperforming Jev by 4.3 and 4.4 Capability points respectively and setting a new state-of-the-art on OpenDecisions. On a suite of public decision datasets, our models achieve accuracy comparable to Jev, with lower calibration error. Our results show that open language models already possess System One capabilities, and with simple fine-tuning they outperform Jev on a wide range of decision tasks. We release our models, serving code, and OpenDecisions.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-14

The authors' main claim, as the classifier picked it out 1.00

“Our results show that open language models already possess System One capabilities, and with simple fine-tuning they outperform the classifier on a wide range of decision tasks.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.09 · states limitation 0.12 · human participants 0.04 · controlled comparison 0.89 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.98 · mentions code or data availability 0.98

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
LLM-based decision-making systems, also called System One models, have recently gained popularity, most notably through Jev (Almeida, 2026). background 1.00 0.04 0.03 asserted 1.00
Given a typed input state, a question, and a set of options, they return a probability for each option in a single forward pass, without text generation. background 0.97 0.06 0.03 asserted 1.00
We show that existing open instruction-tuned language models can achieve comparable performance \textit{without any additional training}, by reading out answers from letters (A, B, C, etc.) in the model's output logits at the last token position. contribution 0.61 0.91 0.30 asserted 1.00
Furthermore, we show how a simple supervised fine-tuning recipe can improve their performance further. contribution 0.66 0.85 0.07 asserted 1.00
We evaluate four open instruction-tuned language models: Gemma 4 31B, Gemma 4 12B, Qwen3.6-27B, and Qwen3.5-4B. method 0.96 0.86 0.02 asserted 0.85
We fine-tune each using LoRA and a decision head initialized from a slice of the language model head. method 1.00 0.95 0.02 asserted 0.91
Furthermore, we introduce OpenDecisions, a novel 2,352-question validation dataset designed to reflect the task composition of JevBench (JevBench main- tainers, 2026) and which computes a Capability score based on accuracy and calibration metrics. contribution 0.68 0.95 0.04 asserted 0.99
Without fine-tuning, Gemma 4 31B scores 79.6 on OpenDecisions, compared to Jev's 80.1. result 1.00 0.89 0.06 asserted 1.00
After undergoing our fine-tuning recipe, it scores 84.4 on OpenDecisions and 81.5 on JevBench v1.7, outperforming Jev by 4.3 and 4.4 Capability points respectively and setting a new state-of-the-art on OpenDecisions. result 1.00 0.97 0.20 asserted 1.00
On a suite of public decision datasets, our models achieve accuracy comparable to Jev, with lower calibration error. result 1.00 0.95 0.13 asserted 1.00
Our results show that open language models already possess System One capabilities, and with simple fine-tuning they outperform Jev on a wide range of decision tasks. interpretation 0.97 0.95 0.93 asserted 1.00
We release our models, serving code, and OpenDecisions. contribution 0.99 0.80 0.04 asserted 0.79

What kind of paper the classifier reports this is: method or tool 0.79 · evidence from observation or experiment 0.98

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-14: 1 request, 9,175 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (5)
Figure 1 of the paper
Figure 1. A Jev-style decision in one forward pass. The request is rendered as a multiple-choice prompt with lettered options. The LLM reads it once; the decision head maps the last hidden state h to one logit per option, starting from the language-model head’s rows for the letters, so the untrained model is the zero-shot readout. Training adds LoRA adapters and a learned correction ∆ to the head. A softmax over the real options gives the probabilities. responding to the answer letter tokens (A, B, C, etc.). Both are trained jointly using cross-entropy on questions drawn from a pool of…
Figure 2 of the paper
Figure 2. Zero-shot reaches Jev; a short training run goes be￾yond. OpenDecisions Capability of the four open models without training (orange) and after training (blue, mean of three seeds; dots: seeds; Section 4). Dashed: Jev. Dotted: the strongest public system (Eikos-27B). The two models of about 30B parameters reach Jev’s level zero-shot, and training moves Gemma 4 31B clearly beyond it. the same training procedure for Gemma 4 31B, Qwen3.6- 27B and Gemma 4 12B. For Qwen3.5-4B, we added three changes that empirically helped this model: a LoRA learn￾ing rate of 10−4 ; as target, the average
Figure 3 of the paper
Figure 3. Training length for Gemma 4 31B. Capability on OpenDecisions (
Figure 4 of the paper
Figure 4. Training length for the smaller models. OpenDecisions Capability against training steps (65,536 tokens each), for checkpoints with a learning-rate decay at the end; each point is the mean of three seeds (
Figure 5 of the paper
Figure 5. Zero-shot accuracy against model size: macro accuracy (%) over the seven held-out decision sets of

Figures are the authors', reproduced from the paper.

References (66)
  1. [1] allebee. JevK5 . https://github.com/allebee/jevk5, 2026 on Pith
  2. [2] Almeida, D. Introducing system one models & Jev . TypeSafe AI blog, https://typesafe.ai/blog/introducing-system-one-models-and-jev, 2026 on Pith
  3. [3] Anthropic . Claude Opus model family. https://www.anthropic.com/claude/opus, 2026 on Pith
  4. [4] blockbrain-ai . Cygnet recipe. https://github.com/blockbrain-ai/cygnet-recipe, 2026 on Pith
  5. [5] Borkan, D., Dixon, L., Sorensen, J., Thain, N., and Vasserman, L. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion Proceedings of the 2019 World Wide Web Conference, 2019. URL https://dl.acm.org/doi/10.1145/3308560.3317593 on Pithdoi:10.1145/3308560.3317593
  6. [6] Brier, G. W. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78 0 (1): 0 1--3, 1950. URL https://journals.ametsoc.org/view/journals/mwre/78/1/1520-0493_1950_078_0001_vofeit_2_0_co_2.xml on Pith
  7. [7] Cai, Z. Open-Jev-27B v1.1. https://huggingface.co/ZefanCai/Open-Jev-27B-v1.1, 2026 on Pith
  8. [8] Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., and Specia, L. SemEval -2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), 2017. URL https://aclant... on Pith
  9. [9] Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M., Gonzalez, J. E., and Stoica, I. Chatbot arena: An open platform for evaluating LLMs by human preference. In International Conference on Machine Learning (ICML), 2024. URL ... on Pith
  10. [10] Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ : Exploring the surprising difficulty of natural yes/no questions. In Proceedings of NAACL-HLT, 2019. URL https://aclanthology.org/N19-1300/ on Pith
  11. [11] Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try ARC , the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. URL https://arxiv.org/abs/1803.05457 on PithPith's reviewarXiv
  12. [12] Cloudflare . Clef . https://huggingface.co/Cloudflare/clef, 2026 on Pith
  13. [13] Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. URL https://arxiv.org/abs/2110.14168 on PithPith's reviewarXiv
  14. [14] Conover, M., Hayes, M., Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., and Xin, R. Free Dolly : Introducing the world's first truly open instruction-tuned LLM . Databricks blog, 2023. URL https://www.databricks.com/blog/2023/04/12/dolly-first-ope... on Pith
  15. [15] crh225. Plumb-4B . https://github.com/crh225/plumb, 2026 on Pith
  16. [16] de Marneffe, M.-C., Simons, M., and Tonhauser, J. The CommitmentBank : Investigating projection in naturally occurring discourse. In Proceedings of Sinn und Bedeutung 23, 2019. URL https://ojs.ub.uni-konstanz.de/sub/index.php/sub/article/view/601 on Pith
  17. [17] denis-pplx . AutoJev-27B . https://huggingface.co/denis-pplx/autojev-27b, 2026 on Pith
  18. [18] Desai, S. and Durrett, G. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. URL https://aclanthology.org/2020.emnlp-main.21/ on Pith
  19. [19] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT : Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo... on Pithdoi:10.18653/v1/n19-1423
  20. [20] EldanRing. Winnow-12B . https://huggingface.co/EldanRing/Winnow-12B, 2026 on Pith
show 46 more
  1. [21] Epstein, E. S. A scoring system for probability forecasts of ranked categories. Journal of Applied Meteorology, 8 0 (6): 0 985--987, 1969. URL https://journals.ametsoc.org/view/journals/apme/8/6/1520-0450_1969_008_0985_assfpf_2_0_co_2.xml on Pith
  2. [22] Gemma Team, Google DeepMind . Gemma 4 model card (gemma-4-31b-it, gemma-4-12b-it). https://huggingface.co/google/gemma-4-31B-it, 2026. Accessed October 2026 on Pith
  3. [23] Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In International Conference on Machine Learning (ICML), 2017. URL https://proceedings.mlr.press/v70/guo17a.html on Pith
  4. [24] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2021 a . URL https://openreview.net/forum?id=d7KBjmI3GmQ on Pith
  5. [25] Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021 b . ... on Pith
  6. [26] HopitAI. Hopper. https://huggingface.co/HopitAI/hopper, 2026 on Pith
  7. [27] Howard, J. and Ruder, S. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 328--339. Association for Computational Linguistics, 2018. doi:10.186... on Pithdoi:10.18653/v1/p18-1031
  8. [28] IkerMoel. open-alternative-jev. https://github.com/ikermoel/open-alternative-jev, 2026 on Pith
  9. [29] JevBench maintainers . JevBench : a benchmark for Jev -type decision models. https://benchmarkheaven.com/jev-models; harness: https://github.com/fstandhartinger/jevbench, 2026. The paper reports results for versions 1.6.x and 1.7; accessed October 10, 2026 on Pith
  10. [30] Kadavath, S. et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. URL https://arxiv.org/abs/2207.05221 on PithPith's reviewarXiv
  11. [31] Karandikar, A., Cain, N., Tran, D., Lakshminarayanan, B., Shlens, J., Mozer, M. C., and Roelofs, B. Soft calibration objectives for neural networks. In Advances in Neural Information Processing Systems, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/f8905bd3df64ace64... on Pith
  12. [32] Kennedy, C. J., Bacon, G., Sahn, A., and von Vacano, C. Constructing interval variables via faceted Rasch measurement and multitask deep learning: a hate speech application. arXiv preprint arXiv:2009.10277, 2020. URL https://arxiv.org/abs/2009.10277v1. Version 1, September 22, 2020 on PithPith's reviewarXiv
  13. [33] krishna-gogineni-765 . deck31b. https://github.com/krishna-gogineni-765/deck31b, 2026 on Pith
  14. [34] kshetrajna12. reflex. https://github.com/kshetrajna12/reflex, 2026 on Pith
  15. [35] Kumar, A., Sarawagi, S., and Jain, U. Trainable calibration measures for neural networks from kernel mean embeddings. In Proceedings of the 35th International Conference on Machine Learning, 2018. URL https://proceedings.mlr.press/v80/kumar18a.html on Pith
  16. [36] Li, X. and Roth, D. Learning question classifiers. In Proceedings of the 19th International Conference on Computational Linguistics (COLING), 2002. URL https://aclanthology.org/C02-1150/ on Pith
  17. [37] Li, Y. and Wagle, J. LLM-as-Jev : LLMs are already Jev -style decision models -- when and how to fine-tune them. arXiv preprint arXiv:2610.02076, 2026. URL https://arxiv.org/abs/2610.02076v2. Version 2 on PithPith's reviewarXiv
  18. [38] Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023. URL https://arxiv.org/abs/2305.20050 on PithPith's reviewarXiv
  19. [39] Liu, A., Swayamdipta, S., Smith, N. A., and Choi, Y. WANLI : Worker and AI collaboration for natural language inference dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2022, 2022. URL https://aclanthology.org/2022.findings-emnlp.508/ on Pith
  20. [40] Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7 on Pith
  21. [41] Marosi, M. decider: one-pass typed decisions with calibrated probabilities. https://github.com/Mapika/decider, 2026. Apache-2.0; accessed October 2026 on Pith
  22. [42] Mukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, P. H. S., and Dokania, P. K. Calibrating deep neural networks using focal loss. In Advances in Neural Information Processing Systems, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/aeb7b30ef1d024a76f21a1d40e3... on Pith
  23. [43] Nguyen, C. Quyet 1.0: calibrated decision models. https://huggingface.co/chinhnc/Quyet-1.0-Large, 2026. Model family includes Large, Medium, Small, and Tiny variants; accessed October 10, 2026 on Pith
  24. [44] Nie, Y., Zhou, X., and Bansal, M. What can we learn from collective human opinions on natural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. URL https://aclanthology.org/2020.emnlp-main.734/ on Pith
  25. [45] OpenAI . gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025. URL https://arxiv.org/abs/2508.10925 on PithPith's reviewarXiv
  26. [46] Pakdaman Naeini, M., Cooper, G. F., and Hauskrecht, M. Obtaining well calibrated probabilities using Bayesian binning. In AAAI Conference on Artificial Intelligence, 2015. URL https://ojs.aaai.org/index.php/AAAI/article/view/9602 on Pith
  27. [47] Pezeshkpour, P. and Hruschka, E. Large language models sensitivity to the order of options in multiple-choice questions. arXiv preprint arXiv:2308.11483, 2023. URL https://arxiv.org/abs/2308.11483 on PithPith's reviewarXiv
  28. [48] Qwen Team . Qwen3.5-4B model card. https://huggingface.co/Qwen/Qwen3.5-4B, 2026 a on Pith
  29. [49] Qwen Team . Qwen3.6-27B model card. https://huggingface.co/Qwen/Qwen3.6-27B, 2026 b on Pith
  30. [50] Qwen Team . Qwen3.8-27B model card. https://huggingface.co/Qwen/Qwen3.8-27B, 2026 c on Pith
  31. [51] Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training. OpenAI technical report, 2018. URL https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf on Pith
  32. [52] Robinson, J. and Wingate, D. Leveraging large language models for multiple choice question answering. In International Conference on Learning Representations (ICLR), 2023. URL https://openreview.net/forum?id=yKbprarjc5B on Pith
  33. [53] Sakaguchi, K., Le Bras, R., Bhagavatula, C., and Choi, Y. WinoGrande : An adversarial Winograd schema challenge at scale. In AAAI Conference on Artificial Intelligence, 2020. URL https://ojs.aaai.org/index.php/AAAI/article/view/6399 on Pith
  34. [54] Schick, T. and Sch \"u tze, H. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp.\ 255--269. Association fo... on Pithdoi:10.18653/v1/2021.eacl-main.20
  35. [55] SemIf contributors . SemIf : zero-shot decision scoring with letter-token logits. https://github.com/TheoLeeCJ/SemIf-OpenJev, 2026. MIT license; accessed October 2026 on Pith
  36. [56] Stepanov, I., Shtopko, M., Vodianytskyi, D., Lukashov, O., Yavorskyi, A., and Yaroshenko, M. GLiClass : Generalist lightweight model for sequence classification tasks. arXiv preprint arXiv:2508.07662, 2025. URL https://arxiv.org/abs/2508.07662 on PithPith's reviewarXiv
  37. [57] Tian, K., Mitchell, E., Zhou, A., Sharma, A., Rafailov, R., Yao, H., Finn, C., and Manning, C. D. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empir... on Pith
  38. [58] torchcast-ai . Torchcast-decision-12B . https://huggingface.co/torchcast-ai/torchcast-decision-12b, 2026 on Pith
  39. [59] Vicentino, C. Eikos : open, calibrated, single-pass typed-decision models for finance and trading. https://huggingface.co/caiovicentino1/Eikos-27B, 2026 on Pith
  40. [60] Wang, Y. et al. MMLU-Pro : A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/ad236edc564... on Pith
  41. [61] Welbl, J., Liu, N. F., and Gardner, M. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, 2017. URL https://aclanthology.org/W17-4413/ on Pith
  42. [62] Yin, W., Hay, J., and Roth, D. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pro... on Pithdoi:10.18653/v1/d19-1404
  43. [63] Yinsongxu. LLM2Jev . https://github.com/Yinsongxu/LLM2Jev, 2026 on Pith
  44. [64] Z.ai . GLM-4.7-Flash model card. https://huggingface.co/zai-org/GLM-4.7-Flash, 2026 on Pith
  45. [65] Zhang, J., Yang, T., Shi, Y., and Wu, L. AnyJev : Turn any LLM into a Jev -style decision model. https://github.com/nokia-applied-research/AnyJev, 2026 on Pith
  46. [66] Zheng, C., Zhou, H., Meng, F., Zhou, J., and Huang, M. Large language models are not robust multiple choice selectors. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=shr9PXz7T0 on Pith
Cited by 11 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment