Pith.

Computation and Language · arXiv:2610.18529 v2 · submitted 13 Oct 2026

How Reliable Are System-One Decisions in Medicine? A Comparative Study of Jev, Jev-like Models, and LLMs

Yanrui Du, Danyang Zhao, Sendong Zhao, Wenhao Li, Yibo Gao, Haochun Wang, Zhuoer Xu, Shiwen Cui

show 1 more author

Bing Qin

The paper says

“Jev has attracted considerable attention for its rapid, generalizable decision-making, yet its medical reliability remains underexplored. To address this gap, our study compares various types of system-one decision models across 15 medical datasets comprising 8,742 instances under a unified protocol.”

more of the abstract

Beyond accuracy, we assess decision stability and the ability to recognize unreliable decisions. Our results show that Jev rivals leading models on primarily text-driven tasks but is less effective on numerically driven tasks, revealing a distinct capability boundary. Under question-format variations, Jev maintains strong decision consistency and small accuracy fluctuations. StartLux-27B exhibits comparable consistency, while D3-27B achieves higher overall accuracy and a lower mean decision reversal rate. But these strengths do not extend to reliable abstention awareness: most evaluated models show limitations in both recognizing missing correct answers and using optional abstention. Moreover, we find that Jev's probability-based answer selection can be improved by changing how decisions are scored: normalizing option-wise binary scores yields higher retained accuracy than choice scoring. Taken together, our findings provide early guidance for improving and applying decision models in medicine.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 14 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A solid, honest evaluation study whose OBS selection finding is the durable contribution; the unvalidated forced-prefix readout for LLMs is a real soft spot, but it mostly affects cross-model rank comparisons, not Jev's own results. This line is wrong

3 major objections · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at Section 3; Appendix D; Eq. (9)

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The load-bearing cross-model claims, namely that Jev rivals leading LLMs on text-driven tasks (Section 4.2) and has the lowest decision reversal rate (Section 5.2, Table 3), rest entirely on a single LLM readout: the evaluator forcibly appends 'The answer is \boxed{' and treats the next-token distribution, softmax-normalized over valid candidate labels only (Eq. 9), as each LLM's decision and confidence.

more of objection 1

This protocol is shared by all eleven LLMs but not by Jev or the Jev-like models, and it is not validated against any generation-based reference. Two mechanisms could bias the comparisons: (i) renormalizing over valid labels discards the probability the model assigns to any other response, so the resulting scores are not directly comparable to Jev's native joint-choice probabilities; and (ii) under relabeling the valid label set changes (e.g., A-D to 1-4), exposing the reversal-rate ranking to token-prior differences. The Limitations section does not flag this choice. I recommend adding a validation subset in which the same LLMs answer by unconstrained generation (greedy and sampled) and are scored by a parser, comparing those predictions with the forced-prefix ones, or at minimum reporting sensitivity to the prefill and explicitly conditioning the headline claims on this protocol.

The paper says, at Section 5.2, Table 3; Section 4.2

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

Headline comparisons are reported without uncertainty quantification. Jev's mean reversal rate of 6.17% versus StartLux-27B's 6.47% is a 0.30 percentage-point gap computed on 8,742 correlated paired entries; a paired McNemar test or a dataset-level bootstrap over the 15 datasets is needed to support the abstract's 'lowest decision reversal rate' claim.

more of objection 2

The same applies to the Section 4.2 Clinical Judgment comparison, where Jev's 74.91/74.31 exceeds Qwen3.8-27B's 72.88/72.52 by roughly two points; with dataset equal-weighting (see next comment) and dataset sizes as small as 41 entries, this gap may not be robust. I am not requesting significance tests for every table cell, but the comparisons that carry the paper's conclusions should be tested.

The paper says, at Section 4.1; Table 1; Tables 12-14

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

Dataset-level scores are averaged with equal weights, and the sensitivity of the conclusions to this weighting is not examined. Within Clinical Judgment, MedCalc (N=41) and SCT (N=78) receive the same weight as ICD-proxy and MIMIC-CDM (N=1,000 each). Recomputing the family mean with sample-size weighting, using the accuracy values in Tables 12-14, moves Jev from 74.9% to approximately 89.5% and Qwen3.8-27B from 72.9% to approximately 89.9%, so the claim that Jev exceeds Qwen3.8-27B in Clinical Judgment reverses under the alternative weighting.

more of objection 3

Because this feeds the abstract's claim that Jev rivals leading LLMs on text-driven tasks, the paper should report both weighting schemes (or dataset-level confidence intervals) and discuss which is appropriate given that the datasets are convenience samples rather than a stratified population.

5 minor objections

The paper says, at Section 6.2

Pith says: minor objection 1 of 5

The paragraph ending 'the explicit option alone does not reliably achieve both' is followed by the stray word 'Coling', which appears to be a leftover artifact and should be deleted.

The paper says, at Table 11; Appendix D

Pith says: minor objection 2 of 5

The term 'noul' (e.g., 'native noul probabilities', 'native noul judgments') is used for the binary-judgment interface that supplies the OBS affirmative probabilities but is never defined. Please define the term or replace it with a standard phrase such as 'native binary (yes/no) judgment probabilities'.

show 3 more

The paper says, at Figure 1

Pith says: minor objection 3 of 5

In the Probability-based reliability panel, the expression 'P(A) = 0.84 / 0.90 approximately 0.93' is not explained; the 0.90 denominator is the sum of affirmative probabilities across candidates and should be labeled as such.

The paper says, at Section 5.2

Pith says: minor objection 4 of 5

The sentence 'Its within-dataset variance of 1.70 pp2 is below every evaluated LLM, though StartLux-4B and -9B are lower still' is confusing because StartLux is a Jev-like model, not an LLM; rephrase to 'below every evaluated LLM and all but the two smallest StartLux configurations'.

The paper says, at Abstract; Section 5.1

Pith says: minor objection 5 of 5

The abstract describes OBS as one of the 'question-format variations', but Section 5.1 correctly characterizes OBS as a change of decision formulation (from joint choice to per-candidate binary judgments); the terminology should be aligned so readers do not conflate surface-format interventions with scoring-formulation changes.

What would settle it

Re-run the comparison on Clinical Judgment (2,501 entries) and Safety Review (1,812 entries) with a second independent readout for the same LLMs: greedy decoding with exact-match scoring of the full answer, and also full-response log-likelihood scoring of each candidate answer string, instead of the one-token post-brace softmax of Eq. 9. Recompute Jev versus Gemma4-31B-IT and Qwen3.8-27B accuracy, and Jev's reversal-rate rank, under each readout. If the 74.91 versus 80.14 and 59.76 versus 60.04 gaps, or the 6.17% versus 7.41% reversal gap, move by more than a paired bootstrap standard error, the cross-model claims are readout-dependent; if they are stable, the shared-prefix objection is closed.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 14 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 36 s · 48,382 tokens read, 4,863 written
  3. Editor wrote the verdict above 55 s · 49,400 tokens read, 8,206 written
  4. Referee major revision: 3 major objections, 5 minor 1 min 24 s · 48,805 tokens read, 13,865 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Two things to know about arXiv:2610.18529. The durable result is the probability-selection finding: normalizing option-wise binary scores (OBS) beats direct Choice scoring for Jev across a full threshold sweep and at matched coverage (94.05% to 99.31% at top-5%). OBS is a defined transform with no fitted parameters, the thresholds are reported in full, and the authors disclose that part of the gain is concentration on Jev's stronger datasets. The second thing is the readout: every LLM is evaluated by force-prefixing "The answer is \boxed{" and reading candidate-normalized next-token probabilities as its decision score and confidence. Nothing in the paper validates that this equals what these models would decide, and Jev and the Jev-like models use their native interfaces instead.

The stress-test note calls this readout load-bearing. I agree for the cross-model comparisons, but the blast radius is narrower than the note suggests. The OBS result, the CTG-21 collapse (20.70% against a 53.50% majority baseline), and the abstention failure (accuracy drops from 77.62% to 65.60% when the correct answer is replaced) are all measured on Jev's native interface, so they survive any readout correction. What does not survive is the fine-grained ranking. The 6.17% versus 6.47% reversal gap between Jev and StartLux-27B is roughly 26 decisions out of 8,742 with no confidence intervals, and every LLM number in that table passes through the forced-prefix protocol. The big differences are clearly real — the small LLMs sit at 20-40% reversal — but "lowest reversal rate" should be softened to "among the lowest, alongside StartLux," or given error bars.

What the paper does well: 21 models, 15 datasets, three reliability dimensions, a clearly described protocol, and an honest limitations section. The reversal-rate metric and the matched-subset abstention deltas are well designed, and the proxy labels (recorded ESI as historical nurse labels, two-accepted-answer items) are disclosed rather than hidden.

Soft spots in proportion: the unvalidated LLM readout is moderate and fixable with a generation-based sanity check; the missing significance tests are moderate because the gaps are often small; no released artifacts is minor but makes verification harder.

This paper deserves a serious referee — the OBS finding alone justifies it, and the benchmark will be reused. My recommendation: engage, with a conditional-accept shape, requesting the readout validation, error bars on headline comparisons, and artifact release.

The strongest objection

The central cross-model claims — Jev 'rivals leading LLMs' on text-driven tasks and has the lowest decision reversal rate — rest entirely on a single readout applied to all eleven LLMs: the evaluator forcibly appends 'The answer is \boxed{' and treats the next-token distribution, softmax-normalized over valid candidate labels (Section 3, Appendix D, Eq. 9), as the LLM's decision score and confidence. This protocol is not validated as a faithful proxy for how these LLMs make decisions. LLMs are not trained to answer through this one-token post-brace distribution, and renormalizing over labels discards the model's own probability of producing any other response. If Gemma4-31B-IT's 80.14% Clinical Judgment score or Qwen3.8-27B's relative ranking depends on the exact prefill and token-level readout, then the headline boundary between Jev and 'leading LLMs', and the 6.17% versus 7.41% reversal-rate comparison, describe properties of the interface rather than of the models. Because the protocol is shared by all LLMs but not by Jev or the Jev-like models, a systematic bias in it directly contaminates the paper's primary comparative conclusions. The paper's limitations section does not flag this; Appendix D presents it as the chosen protocol.

The reader identified the shared forced-prefix readout as the weakest assumption, and that is also the load-bearing point in my reading. The paper's two headline comparative claims — that Jev 'rivals leading LLMs' on text-driven tasks and that it has the lowest reversal rate — are computed from LLM scores obtained by a single next-token softmax after a fixed prefill. Appendix D makes the protocol explicit but gives no evidence that it is a valid or calibrated representation of LLM decisions. A secondary concern, the absence of significance testing and error bars, is real and compounds the issue: several headline gaps, such as 59.76 versus 60.04 in Safety Review and 6.17% versus 6.47% reversal, are small relative to plausible sampling noise. The OBS selection result is better supported internally because it compares two protocols on the same entries, and the paper honestly notes that part of the global gain reflects concentration on stronger tasks. I therefore focus the concern on the cross-model readout rather than on the OBS finding. This does not move the verdict: the reader's CONDITIONAL remains the right disposition, with the readout validation as the decisive condition. If the proposed generation-based check reproduces the rank order, the comparative claims would be substantially strengthened; if not, the central 'rivals leading LLMs' conclusion would need to be scoped to the forced-prefix interface.

The referee's report

If the results hold, this is a useful early reliability map of a rapidly expanding model family in a high-stakes domain. The study has concrete methodological strengths: the OBS normalization (Eq. 10) is a defined transform with no fitted parameters; threshold results are reported as full sweeps (Tables 21-22) rather than cherry-picked operating points; the coverage analysis in Appendix K explicitly shows that global OBS gains partly reflect concentration on tasks where Jev is already strong; the abstention analysis controls for sample selection via a same-subset baseline (Delta_S, Section 6.1); and the paper reports that OBS does not help all models (Appendices H-J), which is a sign of non-selective reporting. The central claims about Jev's competitiveness and robustness are, however, conditional on the shared LLM readout protocol and on the absence of significance testing, as detailed below.

Pith's reading of the paper (a machine's summary, not the authors')

This paper tries to establish where a new class of fast, training-free decision models, Jev and its imitators, can be trusted in medicine. These models score candidate answers directly from probabilities instead of generating reasoning, and the paper compares 21 of them across 15 medical datasets with 8,742 instances under one protocol. It finds a sharp capability boundary: Jev rivals the strongest open LLMs on text-driven tasks such as clinical judgment and safety review, but collapses on numerically driven physiological assessment, scoring 35.95% accuracy where the best LLM reaches 58.10% and falling below the majority-class baseline on fetal-monitoring triage. Beyond accuracy, Jev shows the strongest resistance to question-format changes of all models tested, with a 6.17% mean decision-reversal rate, yet it shares the general weakness of these models at knowing when to abstain or recognize a missing correct answer. The paper also shows that how decision scores are constructed matters: normalizing per-candidate binary judgments instead of direct joint-choice probabilities raises Jev's retained accuracy from 94.05% to 99.31% at top-5% confidence coverage.

What carries the argument. The engine of the comparison is a unified system-one decision protocol: every model receives the same evidence, instruction, and candidate set, and must score candidates directly without producing reasoning traces. Two scoring constructions are contrasted: Choice, which uses the joint probability of each candidate, and OBS, option-wise binary scoring, which asks whether each candidate is correct and normalizes the affirmative probabilities across candidates by their sum. Reliability is measured through paired format interventions, relabeling, reordering, and OBS reformulation, summarized by reversal rates and within-dataset accuracy variance; through abstention interventions that replace the correct answer with NONE or add an ABSTAIN option, summarized by retained accuracy and coverage; and through probability-based selection at fixed thresholds and matched coverage. LLMs are read through an evaluator-forced assistant prefill ending in 'The answer is boxed left-brace' with candidate-normalized next-token probabilities, so all model families share one readout.

Core claim. The central claim is that Jev's medical reliability is task-shaped rather than uniform. It competes with leading LLMs where decisions rest on clinical text, with Clinical Judgment accuracy and macro-F1 of 74.91 and 74.31 against Gemma4-31B-IT's 80.14 and 79.82, and Safety Review accuracy of 59.76 against 60.04 with higher macro-F1. It is markedly worse where the evidence is numerical, scoring 35.95 and 27.19 on Physiological Assessment against 58.10 and 50.37, with CTG-21 accuracy of 20.70% below the 53.50% majority-class baseline. Under the same protocol, Jev is the most format-stable model, with a 6.17% mean reversal rate across relabeling, reordering, and binary reformulation, while only StartLux-27B comes close at 6.47%. But no model family exercises abstention reliably: replacing the correct answer with none of the above drops Jev's accuracy by 12.02 percentage points, and optional abstention adds only 0.33 points while slightly degrading answers on the retained subset. The paper further claims that Jev's probability signal is poorly suited to selection as originally scored, and that normalizing option-wise binary scores (OBS) instead of direct joint probabilities yields higher retained accuracy at every tested threshold and every matched coverage level, because OBS removes unwarranted confidence-one assignments and flattens the score distribution.

The premise it rests on. The entire cross-model ranking assumes that an LLM's probability for the single next token after the forced prefill 'The answer is boxed left-brace' is a fair decision score comparable to Jev's native joint-choice probabilities; if that readout does not represent how LLMs choose, the headline comparisons are artifacts of the interface.

If the paper is right (Pith's inferences, not the authors')

If the capability boundary holds, system-one models like Jev can be used with relative confidence on text-driven medical tasks such as diagnosis, safety review, and care routing, but must not be trusted for numerical physiological assessment without a guardrail.

Jev's low reversal rate means its answers are stable across plausible rephrasings, a useful property for auditable deployment, while OpenJev's comparable accuracy with roughly 2.2 times the reversal rate shows that aggregate accuracy is a poor proxy for decision consistency.

The OBS result implies a cheap, training-free improvement: medical systems that route or defer based on Jev's probabilities should score candidates by normalized binary judgments rather than joint choice probabilities, raising top-5% retained accuracy from 94.05% to 99.31%.

Since adding an abstain option barely changes retained accuracy across most models, deferral policies should rely on probability thresholds rather than on the model's own willingness to abstain.

High-confidence OBS selection concentrates on Jev's strongest tasks, with ICD-proxy and MIMIC-CDM making up 78.08% of the retained pool at the 0.90 threshold, so the routing benefit is partly a task-concentration effect that deployments must monitor.

One testable extension the paper leaves implicit: the OBS-versus-Choice advantage should be evaluated per task family rather than on the pooled cohort, since the pooled gain is partly driven by concentration on Jev's strong datasets; an item-level, task-stratified comparison would show whether OBS sharpens discrimination within weak tasks or merely re-ranks strong tasks first.

The shared LLM readout assigns confidence from a single next token after a forced prefix; a natural stress test is whether Jev's text-driven advantage and lowest reversal rate survive when LLM scores come from full answer-sequence likelihoods or multiple samples, since their disappearance would indict the interface rather than the models.

If the weak-abstention finding generalizes, it suggests abstention awareness should be treated as a trained capability rather than an emergent one: models that never saw a missing correct answer during alignment cannot be expected to flag one, pointing to targeted data rather than prompting.

The paper's acknowledged snapshot scope means the text-versus-numbers boundary is a property of current model generations; re-running the same protocol as Jev-like models adopt numerical backbones could show whether the boundary moves or persists.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Jev has attracted considerable attention for its rapid, generalizable decision-making, yet its medical reliability remains underexplored. To address this gap, our study compares various types of system-one decision models across 15 medical datasets comprising 8,742 instances under a unified protocol. Beyond accuracy, we assess decision stability and the ability to recognize unreliable decisions. Our results show that Jev rivals leading models on primarily text-driven tasks but is less effective on numerically driven tasks, revealing a distinct capability boundary. Under question-format variations, Jev maintains strong decision consistency and small accuracy fluctuations. StartLux-27B exhibits comparable consistency, while D3-27B achieves higher overall accuracy and a lower mean decision reversal rate. But these strengths do not extend to reliable abstention awareness: most evaluated models show limitations in both recognizing missing correct answers and using optional abstention. Moreover, we find that Jev's probability-based answer selection can be improved by changing how decisions are scored: normalizing option-wise binary scores yields higher retained accuracy than choice scoring. Taken together, our findings provide early guidance for improving and applying decision models in medicine.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-14

The authors' main claim, as the classifier picked it out 0.52

“Our results show that the classifier rivals leading models on primarily text-driven tasks but is less effective on numerically driven tasks, revealing a distinct capability boundary.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.71 · human participants 0.12 · controlled comparison 0.31 · randomized assignment 0.02 · states quantitative result 0.88 · compares with prior results 0.91 · mentions code or data availability 0.02

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Jev has attracted considerable attention for its rapid, generalizable decision-making, yet its medical reliability remains under explored. gap 0.67 0.04 0.02 asserted 0.99
To address this gap, our study compares various types of system-one decision models across 15 medical datasets comprising 8,742 instances under a unified protocol. contribution 0.54 0.90 0.03 asserted 0.93
Beyond accuracy, we assess decision stability and the ability to recognize unreliable decisions. contribution 0.53 0.87 0.03 asserted 0.94
Our results show that Jev rivals leading models on primarily text-driven tasks but is less effective on numerically driven tasks, revealing a distinct capability boundary. result 0.98 0.94 0.42 asserted 1.00
Under question-format variations, Jev demonstrates strong robustness, achieving the lowest decision reversal rate and relatively small accuracy fluctuations. result 1.00 0.94 0.09 asserted 1.00
Among evaluated open-source alternatives, only StartLux exhibits comparable robustness. result 1.00 0.88 0.04 asserted 1.00
But these strengths do not extend to reliable abstention awareness: most evaluated models show limitations in both recognizing missing correct answers and using optional abstention. result 0.97 0.89 0.22 asserted 1.00
Moreover, we find that Jev's probability-based answer selection can be improved by changing how decisions are scored: normalizing option-wise binary scores yields higher retained accuracy than choice scoring. result 1.00 0.95 0.09 asserted 1.00
Taken together, our findings provide early guidance for improving and applying decision models in medicine. interpretation 0.89 0.26 0.17 asserted 0.60

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 0.89

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-14: 1 request, 6,801 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (8)
Figure 1 of the paper
Figure 1. Overview of the comparative evaluation. Three model families, four medical task families, and four evaluation dimensions. A shared clinical judgment case illustrates performance, format changes, missing-answer recognition, optional abstention, and probability-based selection. version to option-wise binary judgments. Absten￾tion awareness assesses whether models recog￾nize missing correct answers and abstain effectively when given the option. Probability-based reliabil￾ity examines whether higher decision probabilities identify more accurate decisions at fixed probabil￾ity thresholds…
Figure 2 of the paper
Figure 2. Jev’s probability-based selection. Choice (blue) and OBS (orange) compared through (a) threshold accuracy, (b) accuracy and coverage at 0.90, (c) accuracy at matched global coverage, and (d) confidence distributions (bin width 0.05). Accuracy pools retained entries; coverage uses all 8,742 entries. These are selection results, not calibration curves. retention levels (
Figure 3 of the paper
Figure 3. Dataset-level original-choice accuracy (top) and macro-F1 (bottom) (%) for all 21 model configurations
Figure 4 of the paper
Figure 4. Three distinct views of each interface change for all models, with equal dataset weighting. Left: accuracy
Figure 5 of the paper
Figure 5. Retained accuracy across confidence thresholds for all 21 models. Blue circles denote Choice; orange
Figure 6 of the paper
Figure 6. Rounded coverage percentages at every threshold for all models;
Figure 7 of the paper
Figure 7. OBS minus Choice accuracy (percentage points) at matched global coverage for all 21 models. Positive
Figure 8 of the paper
Figure 8. Choice versus normalized OBS confidence distributions for all 21 models on the same 8,742 entries. Confidence is the probability assigned to the predicted label. Lines show the percentage of entries in confidence bins of width 0.05. Blue solid lines denote Choice; orange dashed lines denote OBS

Figures are the authors', reproduced from the paper.

Cited by 3 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment