Pith.

Machine Learning · arXiv:2610.00831 v1 · submitted 30 Sep 2026 · 59 references

AnyJev Technical Report

Jiamu Zhang, Tianze Yang, Yucheng Shi, Evan Chen, Zixiang Nie, Kelly Wan, Liangjie Hong, Ninghao Liu

show 1 more author

Liang Wu

The paper says

“A typed decision is a choice among a fixed set of options, returned as a probability rather than as text. Systems that need typed decisions today use models trained for that purpose.”

more of the abstract

This report describes AnyJev, which reads a typed decision from one prefill of a pretrained instruction-tuned language model. The readout restricts the next-token distribution at the answer position to the option tokens. It has two defects: the model assigns higher probability to some labels whatever the input, and to some positions in the option list. AnyJev corrects both with no gradient steps and no parameter changes: it divides out a label prior estimated from unlabelled inputs, and it averages log-probabilities over the K cyclic rotations of the option list. On two 20-option tasks the rotations lower the order-flip rate from 0.33 to 0.14 and from 0.33 to 0.18, and raise accuracy on 11 of 11 models on both. Reading every rotation requires K prefills. A stopping rule selected against the full-rotation decision on unlabelled states cuts that. Selecting the threshold on one unlabelled split and bounding its disagreement on a second, it reads 10.6 rotations of 18 at a verified 0.008 bound on two of four cells; selected and bounded on one split, as our serving run did, it reads 7.3 and serves 2.2 times as many decisions per second on vLLM. The code is open source.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 2 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Rotation averaging measurably cuts order dependence on frozen LLMs (11/11, p=4.9e-4) and the paper is unusually honest about its limits; the +7.2pt accuracy claim is listing-dependent until the canonical-order baseline they name is actually run. This line is wrong

3 major objections · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 59 references say.

The paper says, at Section 4.2 / Eq. (3) and Section 8

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The exactness proof at Eq. (3) requires the additive, content-independent model c_i + b_j for the logit of option i at position j. The paper's own measurements contradict this model as the operative mechanism: the residual flip rate after averaging is 0.138/0.183 (Tables 4-5), an order of magnitude above the model's prediction of zero, and the Section 8 exchangeable-mask experiment …

more of objection 1

… shows accuracy collapsing from 0.633 to 0.042 when options cannot attend to one another, demonstrating that position-dependent inter-option attention, a direct violation of the additive assumption, is load-bearing for the decisions. The abstract's claim that the readout "removes" position bias should be restated as an empirical reduction in order sensitivity; the additive model should be presented as an idealized limit that the measured residuals contradict rather than as the mechanism that produces the accuracy gain.

The paper says, at Section 7.1, Table 1, and Appendix C (Figure 4)

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.79):

“Therunsreportedhererotatethecaller’slisting, sowedonotmeasureit. 4.3 Confidence Neither correction makes the model’s confidence match its accuracy, and we did not measure a label-free procedure that detects this.”

Pith says: Objection 2 of 3 · major

The accuracy arm compares the rotation average only to a single raw readout of the caller's selected listing; the canonical-order baseline that Section 4.2 argues would make the prompt set a function of the option set alone is named but not run ("we do not measure it").

more of objection 2

Appendix C shows the rotation average never exceeds the best single rotation (0.3-2.3 points below) and sits above the worst, which is consistent with the gain being hedging against one particular submitted listing rather than removal of a content-independent position bias. Since Section 9 states the 20-option results have no per-item intervals, the "raises accuracy" claim as evidence of debiasing is not separable from the effect of choosing one reference listing. The report should either run the canonical-order experiment or, at minimum, restrict the 20-option accuracy claim to "improvement over a single caller-selected listing".

The paper says, at Section 4.1, Table 1 vs. Table 14

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.67):

“Selecting on one unlabelled split and measuring on a second would settle it; we did not do that here. • The efficiency and accuracy measurements use different tasks.The JevBench subset’s𝐾 is 2 to 6, so a rotation budget has little to save there. • Hidden-state readouts need a local backend.The affine map of Appendix D and the per-question head need hidden states, which a hosted API does not expose.”

Pith says: Objection 3 of 3 · major

The label prior at alpha=0.75 is fitted once against gold labels on a separate sweep (Section 6), and the paper's own full-split transfer evaluation shows it lowers accuracy relative to rotation averaging alone (Table 14: 65.01 vs 68.01 on Qwen3.5-9B and 65.55 vs 67.59 on Qwen3-32B), even where it improves ECE.

more of objection 3

The paper does disclose this transfer failure in the Status paragraph and Section 7.3, and Section 6 names alpha as a label-entering quantity; what remains unresolved is presentation: the abstract still advertises prior division as one of the two headline corrections, and the +prior columns in Table 1 (0.752, 0.675) appear without a pointer to the contradictory transfer result. The abstract should either drop the prior from the headline correction list or qualify it with the observed task-dependence, and the +prior columns should carry an explicit annotation that alpha is gold-fitted and does not transfer to the evaluated large split.

This review says the paper's mathematics is wrong. Pith did not check that against the paper's source.

5 minor objections

The paper says, at Section 7.1 and Table 1, injection column

Pith says: minor objection 1 of 5

The "0.000" order-flip rate after rotation averaging on the binary task is a structural consequence of the K=2 cyclic rotation: reversing the list merely swaps the two rotations, so the average is invariant under reversal by construction. The text "the rotations remove order dependence completely" presents a tautology as a finding; it should state that the K=2 average is order-invariant by construction and that the relevant evidence here is only the accuracy null result (3/7, p=0.77).

The paper says, at Section 3, Eq. (1) vs. Eq. (3)

Pith says: minor objection 2 of 5

The notation would benefit from an explicit mapping between the position-indexed restricted probability p_tilde_j of Eq. (1) and the option-indexed q_t(i) of Eq. (3), since a reader must invert the layout pi_t by hand to see that q_t(i) is p_tilde at the position occupied by option i under rotation t.

show 3 more

The paper says, at Section 3 and Table 6

Pith says: minor objection 3 of 5

The answer-mass gate is stated as excluding rows below 0.99 for the 20-option sweep, while Section 3 says every benchmark row has median m >= 0.995 and Table 6 prints the three excluded rows (SmolLM2 0.408, Granite 0.976, OLMo 0.710). Please state the gate value used for the benchmark run explicitly, and note whether the 0.99 threshold is applied to the median or to each cell, so the exclusion rule is reproducible.

The paper says, at References and Related Work

Pith says: minor objection 4 of 5

The related-work section leans heavily on 2026 arXiv preprints and GitHub repositories, several of which appear contemporaneous with this report; please confirm that the most load-bearing citations (the Jev interface [2] and JevBench [42]) are publicly accessible and have versioned identifiers, and trim references that are only loosely connected to the readout mechanism (e.g., [16], [28], [45], [47]).

The paper says, at Table 3 and Section 7.4

Pith says: minor objection 5 of 5

The serving row labeled "budget, wave 1" for Transformers reports 7.2 rotations while the text says the shipped default reads two rotations per round; the sentence explaining that "the shipped default reads two rotations before it may stop, which moves the leftmost bar" (Figure 4) should be applied to Table 3 as well so the wave-1 column is not read as the shipped configuration.

What would settle it

Run the canonical-order control the paper names in Section 4.2: for each item in the same 300-item banking20 and newsgroups cells, sort the 20 options lexicographically, then (a) read the raw readout from that canonical listing only, and (b) compute the log-space rotation average over cyclic rotations of the canonical listing. Compare (a) and (b) against each other and against Table 1's caller-listing rows for flip rate and accuracy. Decision rule: if (b) still beats (a) by roughly 7 points and (a) matches the caller-listing raw accuracy, the +7.2 gain is genuine position-bias removal robust to listing choice; if (a) is close to (b), the headline gain is an artifact of an unlucky caller listing; if (b) falls materially below the caller-listing rotation average, non-additive adjacency effects contribute a large share of the reported benefit.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 2 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 1 min 6 s · 24,191 tokens read, 9,682 written
  3. Editor wrote the verdict above 29 s · 25,286 tokens read, 4,580 written
  4. Referee major revision: 3 major objections, 5 minor 1 min 7 s · 24,517 tokens read, 10,877 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The useful thing here is the order-dependence result: log-space cyclic rotation averaging cuts the flip rate from 0.33 to 0.14–0.18 on both 20-option tasks across 11/11 models (sign test p=4.9e-4), and they measure the residual rather than assume it away. The paper is also genuinely honest — Section 8 reports what did not work, and the limitations list names their own selection-multiplicity flaw in the stopping certificate. That candor is real and uncommon.

What is actually new is the packaging. Per-rotation log-space averaging for typed readouts is in Zheng et al. 2024, the prior is batch calibration, the temperature is Guo et al.; the contribution is the systematic combination plus a stopping rule that certifies agreement with the full rotation at a Clopper-Pearson bound, reading 10.6 of 18 rotations. Practitioners who want thresholdable probabilities from frozen LLMs get something deployable.

The soft spot is the headline accuracy gain, and the stress-test concern lands. Their exactness model (logit = ci + bj) predicts a zero residual flip rate after averaging; they measure 0.138–0.183. Their own exchangeable-mask experiment (Section 8) shows why: options attend to each other, and the model uses that signal. The paper states the additive model is an approximation rather than hiding it, so this is an acknowledged gap, not a concealed flaw — but it means the debias is approximate precisely where claimed. Second, the +7.2 point accuracy gain is against one caller-made listing, and the canonical-order baseline (sort by text, then rotate) named in Section 4.2 is explicitly not run. So the magnitude is listing-dependent. I would not call it fatal: 11/11 with the effect on two different tasks makes pure artifact unlikely. But the right reading is "this setup showed a gain," not "rotation averaging guarantees +7 points."

Other items, in proportion: alpha=0.75 is fitted against gold labels and they show it lowers accuracy on the BEV split, which they acknowledge; the injection-task rows gated out are disclosed and that result is honestly non-significant (p=0.77); the abstract's "0.008 bound" is a self-agreement guarantee, not accuracy, though the body states this plainly. Code is claimed open but no commit hash is given, so verification is on the authors' shelf.

This is an early technical report, an engineering artifact rather than a settled ranking — useful for anyone building typed readouts on frozen models, and a worthwhile referee. Send it out.

The strongest objection

The central transferable claim is that rotation averaging removes position bias in log space and thereby raises accuracy on 20-option tasks (11/11 models, +7.2/+5.9 points, Table 1, Section 7.1). The stated model for exactness is the additive, content-independent logit c_i + b_j (Section 4.2, Eq. 3 context): under it, averaging over cyclic rotations gives softmax(c) exactly, i.e., a residual flip rate of zero. The measured residuals are 0.138 (banking20) and 0.183 (newsgroups), an order of magnitude larger than the model predicts. The paper's own Section 8 experiment demonstrates why the additive model fails in a mechanism-relevant way: with an exchangeable mask that makes options attend only to the prefix and themselves, output moves by only 3.7e-6 under reshuffling but accuracy collapses to chance (0.042 vs 0.633 on the routing question), while the causal mask shifts outputs by 2.2e-3. So position-dependent inter-option attention is load-bearing for the decisions, which is a direct violation of c_i + b_j; the debias is approximate, not exact, precisely where it is claimed (K=20). Two consequences follow. First, the flip-rate drop (0.328 to 0.138) mixes genuine position-bias removal with adjacency and attention interaction effects that the additive model would not predict. Second, and more importantly for the headline, accuracy is compared against a single raw readout taken from the caller's listing; nothing in the paper controls for the possibility that this particular listing systematically penalizes correct options, which would inflate the measured gain. The canonical-order protocol (sort options by text, then rotate; prompt set becomes a function of the option set) is explicitly named in Section 4.2 and just as explicitly not run ('The runs reported here rotate the caller's listing, so we do not measure it').

The reader's weakest_assumption and my concern are the same underlying point: the rotation average is exact only under the additive c_i + b_j bias model, and the paper's own measurements show that model is violated in a way that matters, with the residual flip rate at 13.8-18.3% and Section 8's exchangeable-mask experiment demonstrating load-bearing, non-additive inter-option attention. The reader already connected this to the headline accuracy benefit and named a canonical-order baseline as an addressable fix; I agree fully and have sharpened the concern into a single refutable experiment with a decision rule. I keep the verdict at CONDITIONAL rather than raising or lowering it, for two reasons. Credit where due: the flip-rate reduction (0.328 to 0.138) is a measured, label-free readout property; the sign test over 11 models is a clean non-parametric statement; and the paper is unusually honest, disclosing the residual order dependence, the selection-multiplicity flaw in the stopping certificate, the failure of alpha=0.75 on BEV, and the absence of per-item intervals on the 20-option headline. None of my concern impeaches those facts. What the concern does is hold the strongest accuracy claim to its own standard: the +7.2 point gain at K=20 compared against a caller-selected listing, with no canonical-order control, in a regime where the additive model is demonstrably approximate, while the same correction shows no measurable accuracy effect at K=2-6 (JevBench) and only +0.16 to +0.56 points on BEV. The canonical-order experiment is cheap, uses the released code, and directly tests whether the transferable accuracy benefit survives a listing-robust comparison; until it is run, CONDITIONAL is the right verdict.

The referee's report

If the empirical claims hold, this is a useful and unusually honest contribution: a parameter-free debiasing readout (the rotation average needs no labels under its stated additive model) that reduces order sensitivity on every one of 11 models, plus a label-free stopping rule with a correctly computed selection-verification split. Strengths that should be credited: the paper reports its failed mechanisms in full (Section 8), names the two label-entering quantities (alpha and temperature) in Section 6, computes the sign test correctly (one-sided p=4.9e-4 for 11/11), runs per-item bootstrap intervals on JevBench, evaluates on the full 46,320-decision BEV split with intervals, and publishes the code. The importance is moderate: the paper is an early technical report, the accuracy mechanism is empirically robust but not mechanistically explained by its stated model, and the label prior does not transfer to the large-split evaluation.

Pith's reading of the paper (a machine's summary, not the authors')

This paper claims that the ability to make a typed decision, a choice among a fixed set of options returned as a probability, is already present in a pretrained instruction-tuned language model, and what needs correcting is the readout rather than the model. AnyJev reads the decision from one prefill, restricts the next-token distribution at the answer position to the option tokens, and then corrects two defects with no gradient steps and no parameter changes. It divides out a label prior estimated from unlabelled inputs to fix a bias toward certain labels, and it averages log-probabilities over the K cyclic rotations of the option list to remove position bias. Averaged over eleven models, this rotation average lowers the order-flip rate from 0.328 to 0.138 on banking20 and from 0.334 to 0.183 on newsgroups, and raises accuracy on 11 of 11 models on both tasks (sign test $p = 4.9 \times 10^{-4}$). A stopping rule selected against the full-rotation decision on unlabelled states reads 10.6 of 18 rotations at a verified 0.008 bound on two of four cells, cutting the prefills needed per decision.

What carries the argument. The log-space cyclic rotation average is the object that carries the argument. The caller lists the K options in a printed order and the model reads a probability for each position; the paper reads the same decision under every cyclic rotation of that list, so each option occupies each position exactly once, and averages the log-probabilities before renormalising. The average is exact under the additive position-bias model $c_i + b_j$, which is what turns a debiasing heuristic into a stated identity. A second mechanism is the label-prior division of Eq. (2), a batch-calibration estimate of a content-free label preference, divided out with strength $\alpha = 0.75$ using no gold labels. A third mechanism is the stopping rule: a log-odds margin statistic in log space, with a threshold picked on unlabelled states and its disagreement with the full-rotation decision bounded by a Clopper-Pearson interval, decides how many of the K rotations to read.

Core claim. The paper's central claim is that the order dependence of a language model's choice among listed options is a defect of the readout, and it can be cancelled exactly by a layout symmetry: read the decision under all K cyclic rotations of the option list and average the restricted probabilities in log space. Under the model that the logit for option i at position j separates as $c_i + b_j$, summing over all rotations sends each option through every position once, so the position terms collapse to a constant and softmax of the averaged log-probabilities recovers the model's true preference exactly; averaging probabilities instead of their logarithms does not have this property. The paper measures the effect: reversing the option list changes the answer on 32.8% of banking20 items and 33.4% of newsgroups items on the raw readout, and on 13.8% and 18.3% after the average, across 11 of 11 models, with the sign test at $p = 4.9 \times 10^{-4}$. The paper separates what the rotation average does from what it does not: on a benchmark whose items have 2 to 6 options it leaves accuracy unchanged by a measurable amount, on the binary two-option task it removes order dependence but does not raise accuracy, and a fitted label prior even lowers accuracy on a larger 46320-decision evaluation; the measured benefit of rotations is order stability and calibrated confidence rather than a general accuracy gain.

The premise it rests on. The position bias must be additive and content-independent, so that the logit for option $i$ at position $j$ separates as $c_i + b_j$, for the rotation average to cancel it exactly; the paper itself measures a residual order dependence after averaging (flip rates of 13.8% and 18.3%) and notes that options attend to one another inside the prompt.

If the paper is right (Pith's inferences, not the authors')

Any frozen instruction-tuned model can serve typed decisions with an order-stable probability, without training a dedicated decision model, changing weights, or generating tokens.

The layout-symmetry principle generalizes: any additive per-position bias in the logit can be removed by averaging over the full set of layout permutations at no parameter cost.

The K-prefill cost can be cut by a label-free stopping rule that certifies agreement with the full-rotation decision at a verified $1-\varepsilon$ bound, reading about 10.6 of 18 rotations.

Where position bias is large, as on many-option tasks, the rotation average materially raises accuracy; where it is small, as on few-option benchmarks, it buys order stability and calibration rather than accuracy.

The paper proposes but does not measure a canonical ordering, sorting the option list by text before rotating, which would remove the residual dependence on which options are adjacent; a testable extension is whether that closes the residual 13.8% and 18.3% flip rates it reports after averaging.

Because the paper documents that options attend to one another inside the prompt, the additive-bias model is likely to be weakest for semantically overlapping options; one can test whether rotation averaging is less effective when two options are near-duplicates.

The label prior helped on the paper's own 20-option tasks but hurt on the larger full-split evaluation, which points to a deployment rule: apply the prior only when the input-label marginal is measured to be non-extreme, an assumption the paper itself flags.

The mechanism suggests a recipe for other order-sensitive readouts, such as few-shot exemplar ordering: average in log space over the symmetry group of the layout to remove ordering artifacts, at the price of more forward passes.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

A typed decision is a choice among a fixed set of options, returned as a probability rather than as text. Systems that need typed decisions today use models trained for that purpose. This report describes AnyJev, which reads a typed decision from one prefill of a pretrained instruction-tuned language model. The readout restricts the next-token distribution at the answer position to the option tokens. It has two defects: the model assigns higher probability to some labels whatever the input, and to some positions in the option list. AnyJev corrects both with no gradient steps and no parameter changes: it divides out a label prior estimated from unlabelled inputs, and it averages log-probabilities over the K cyclic rotations of the option list. On two 20-option tasks the rotations lower the order-flip rate from 0.33 to 0.14 and from 0.33 to 0.18, and raise accuracy on 11 of 11 models on both. Reading every rotation requires K prefills. A stopping rule selected against the full-rotation decision on unlabelled states cuts that. Selecting the threshold on one unlabelled split and bounding its disagreement on a second, it reads 10.6 rotations of 18 at a verified 0.008 bound on two of four cells; selected and bounded on one split, as our serving run did, it reads 7.3 and serves 2.2 times as many decisions per second on vLLM. The code is open source.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-02

The authors' main claim, as the classifier picked it out 0.71

“AnyJev corrects both with no gradient steps and no parameter changes: it divides out a label prior estimated from unlabelled inputs, and it averages log-probabilities over the K cyclic rotations of the option list.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.95 · human participants 0.04 · controlled comparison 0.76 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.70 · mentions code or data availability 0.98

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
A typed decision is a choice among a fixed set of options, returned as a probability rather than as text. background 1.00 0.06 0.03 asserted 1.00
Systems that need typed decisions today use models trained for that purpose. background 1.00 0.05 0.02 asserted 1.00
This report describes AnyJev, which reads a typed decision from one prefill of a pretrained instruction-tuned language model. contribution 0.99 0.79 0.17 asserted 1.00
The readout restricts the next-token distribution at the answer position to the option tokens. method 0.99 0.68 0.04 asserted 1.00
It has two defects: the model assigns higher probability to some labels whatever the input, and to some positions in the option list. limitation 0.92 0.75 0.05 asserted 1.00
AnyJev corrects both with no gradient steps and no parameter changes: it divides out a label prior estimated from unlabelled inputs, and it averages log-probabilities over the K cyclic rotations of the option list. method 0.88 0.92 0.50 asserted 1.00
On two 20-option tasks the rotations lower the order-flip rate from 0.33 to 0.14 and from 0.33 to 0.18, and raise accuracy on 11 of 11 models on both. result 1.00 0.93 0.23 asserted 1.00
Reading every rotation requires K prefills. limitation 0.99 0.59 0.03 asserted 1.00
A stopping rule selected against the full-rotation decision on unlabelled states cuts that. method 0.86 0.82 0.06 asserted 1.00
Selecting the threshold on one unlabelled split and bounding its disagreement on a second, it reads 10.6 rotations of 18 at a verified 0.008 bound on two of four cells; selected and bounded on one split, as our serving run did, it reads 7.3 and serves 2.2 times as many decisions per second on vLLM. The code is open source. result 0.95 0.95 0.13 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.83

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-02: 1 request, 7,618 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (5)
Figure 1 of the paper
Figure 1. The same base model, three decision paths. (a) Asking for the answer in text costs one forward pass per generated token and returns prose the caller has to parse, with no probability attached. (b) AnyJev L0 runs one prefill per cyclic rotation. It restricts the next-token distribution to the option-label tokens and renormalises, corrects the label prior, maps each position back to its option, and averages in log space across the 𝐾 rotations before renormalising. A, B and C are position labels whose option assignments change across rotations; the final distribution is indexed by…
Figure 2 of the paper
Figure 2. Rotation averaging on two 20-option tasks. Each row is one model, with separate subpanels for banking20 and 20 newsgroups under each metric. Hollow points show the raw readout; filled points show the rotation average of Eq. (3). Left: the order-flip rate, the share of items whose answer changes when the option list is reversed. Right: accuracy. Axes are shown as percentages. All 22 model–task pairs have a lower order-flip rate and higher accuracy after averaging. The label prior is switched off, so the changes are due to rotations alone. Settings and hardware. The main tables read…
Figure 3 of the paper
Figure 3. Differences on the JevBench public subset with 95% bootstrap intervals over 2 000 resamples, paired over items, one row per model. Left: debiased minus raw accuracy; every interval contains zero. Right: the change in calibration error from adding the temperature; four of the eight intervals exclude zero, and all four are improvements. Coloured intervals exclude zero. 7.2 The benchmark Accuracy on this subset spans 0.549 for Qwen3-1.7B to 0.798 for Qwen3-32B (
Figure 4 of the paper
Figure 4. Left: every cyclic rotation scored on its own, against the full-rotation average, on four (model, task) cells. On all four the average falls at or below the best single rotation, by 0.3 to 2.3 points, and above the worst by 3.5 to 9.3 points. Right: where the budget stops on one 18-option cell, over 600 decisions, for a rule with no minimum-rotation floor; the shipped default reads two rotations before it may stop, which moves the leftmost bar. The left panel states what the rotations do. They do not raise accuracy above the best single rotation. They remove the caller’s ability to…
Figure 5 of the paper
Figure 5. Agreement against depth, eight models. Markers are the depths we ran, and the curve is drawn through them rather than smoothed, because the intermediate depths were not measured. Hollow markers show the first depth reaching 0.95. Dashed lines mark 0.95 and 0.98. AnyJev 19

Figures are the authors', reproduced from the paper.

References (59)
  1. [1] Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-LoRAs. arXiv preprint arXiv:2503.01743, 2025 on PithPith's reviewarXiv
  2. [2] Diogo Almeida. Introducing System One models & Jev. TypeSafe AI blog,https://typesafe. ai/blog/introducing-system-one-models-and-jev,September2026. Published15September 2026 on Pith
  3. [3] avbiswas. BEV Decision Mix. Hugging Face dataset, https://huggingface.co/datasets/ avbiswas/bev-decision, 2026. Default configuration; accessed September 30, 2026 on Pith
  4. [4] Nora Belrose, Igor Ostrovsky, Lev McKinney, Zach Furman, Logan Smith, Danny Halawi, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112, 2023 on PithPith's reviewarXiv
  5. [5] Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, et al. SmolLM2: When smol goes big – data-centric training of a small language model. arXiv preprint arXiv:2502.02737, 2025 on PithPith's reviewarXiv
  6. [6] Bespoke Labs and Maheswaran Sathiamoorthy. Nimble. GitHub repository,https://github.com/ bespokelabsai/nimble, 2026. Accessed 29 September 2026 on Pith
  7. [7] Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient intent detection with dual sentence encoders. InProceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38–45. Association for Computational Linguist... on Pith
  8. [8] Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance.Transactions on Machine Learning Research, 2024. Code: https://github.com/stanford-futuredata/FrugalGPT on Pith
  9. [9] Yilong Chen, Xiao Qin, Chenghao Liu, Liang Wu, Noelle I. Samia, and Kaize Ding. LLM agents for time-series: A survey. arXiv preprint arXiv:2608.26226, 2026. Accepted to Findings of EMNLP 2026 on PithPith's reviewarXiv
  10. [10] C. J. Clopper and E. S. Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial.Biometrika, 26(4):404–413, 1934 on Pith
  11. [11] dzhng. jevgrep. GitHub repository, https://github.com/dzhng/jevgrep, 2026. Accessed 29 September 2026 on Pith
  12. [12] fast-jev-compaction. GitHub repository, https://github.com/tamaratran/ fast-jev-compaction, 2026. Accessed 29 September 2026 on Pith
  13. [13] feder-cr. jevos. GitHub repository, https://github.com/feder-cr/jev, 2026. Accessed 29 September 2026 on Pith
  14. [14] Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natarajan. MASSIVE: A 1M-example multilingual ... on Pith
  15. [15] Tianyu Gao, Adam Fisch, and Danqi Chen. Making Pre-trained Language Models Better Few-shot Learners. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: ... on Pith
  16. [16] Paimon Goulart, Liang Wu, Kelly Wan, Evangelos E. Papalexakis, and Liangjie Hong. Field-aware agent skill retrieval. arXiv preprint arXiv:2608.02880, 2026 on PithPith's reviewarXiv
  17. [17] Granite Team, IBM. Granite 3.0 language models. Technical report, https://github.com/ ibm-granite/granite-3.0-language-models/, October 2024 on Pith
  18. [18] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024 on PithPith's reviewarXiv
  19. [19] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. InProceedings of the 34th International Conference on Machine Learning, volume 70 ofProceedings of Machine Learning Research, pages 1321–1330. PMLR, 2017 on Pith
  20. [20] Ari Holtzman, Peter West, Vered Shwartz, YejinChoi, and Luke Zettlemoyer. Surfaceformcompetition: Why the highest probability answer isn’t always right. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7038–7051. Association for C... on Pith
show 39 more
  1. [21] Jev Ultrafast. GitHub repository, https://github.com/browser-use/jev-ultrafast, 2026. Accessed 29 September 2026 on Pith
  2. [22] Jevlike. GitHub repository,https://github.com/vinnylarouge/jevlike, 2026. Accessed 29 September 2026 on Pith
  3. [23] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, DiegodelasCasas,FlorianBressand,GiannaLengyel,GuillaumeLample,LucileSaulnier,LélioRenard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothé... on PithPith's reviewarXiv
  4. [24] Kev. GitHub repository,https://github.com/jaredpalmer/kev, 2026. Accessed 29 September 2026 on Pith
  5. [25] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles ... on Pith
  6. [26] Ken Lang. NewsWeeder: Learning to filter netnews. InMachine Learning Proceedings 1995: Proceedings of the Twelfth International Conference on Machine Learning, pages 331–339. Morgan Kaufmann, 1995. Original source of the 20 Newsgroups dataset on Pith
  7. [27] Laya: Multilingual,non-autoregressivesystem1decisionengine. GitHubrepository, https://github. com/NandhaKishorM/laya, 2026. Accessed 29 September 2026. AnyJev 13 on Pith
  8. [28] Delong Li, Xu Wang, Haochen Gong, Rui Lang, and Guangsheng Yu. Fast Intent-Driven Service Orchestration with Jev for 6G Edge Networks. arXiv preprint arXiv:2609.23136, 2026 on PithPith's reviewarXiv
  9. [29] Keyi Li, Yihao He, and Quanyi Li. Beyond Calibration: Do a Typed-Decision Model’s Probabilities Obey the Probability Axioms? arXiv preprint arXiv:2609.33209, 2026. Code:https://github. com/bro789/typed-decision-coherence on PithPith's reviewarXiv
  10. [30] Yubo Li, Yidi Miao, Ramayya Krishnan, and Rema Padman. JEV-as-a-Judge: Accept When Confident, Escalate When Unsure. arXiv preprint arXiv:2609.26550, 2026 on PithPith's reviewarXiv
  11. [31] LLM2Jev: Turn LLMs into Jev-Style Decision Models. GitHub repository,https://github.com/ Yinsongxu/LLM2Jev, 2026. Accessed 29 September 2026 on Pith
  12. [32] llm2jev.GitHubrepository, https://github.com/tic-top/llm2jev,2026.Accessed29September 2026 on Pith
  13. [33] Mark Marosi. decider: one-pass typed decisions with calibrated probabilities. GitHub repository, https://github.com/Mapika/decider, 2026 on Pith
  14. [34] Guanghui Min, Liang Wu, Mayank Darbari, Chen Chen, and Liangjie Hong. Toward reliable context compression for long-horizon agents: An empirical study of execution instability. arXiv preprint arXiv:2608.06503, 2026. Code:https://github.com/nokia-applied-research/Trace on PithPith's reviewarXiv
  15. [35] NanoJev: A nano replica of Jev. GitHub repository, https://github.com/TianyuCodings/ NanoJev, 2026. Accessed 29 September 2026 on Pith
  16. [36] nostalgebraist. interpreting GPT: the logit lens. LessWrong,https://www.lesswrong.com/posts/ AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens, August 2020. Published 31 August 2020 on Pith
  17. [37] OpenAI. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025 on PithPith's reviewarXiv
  18. [38] Qwen, An Yang, Baosong Yang, Beichen Zhang, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024 on PithPith's reviewarXiv
  19. [39] Delip Rao and Chris Callison-Burch. JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places. arXiv preprint arXiv:2609.29769, 2026 on PithPith's reviewarXiv
  20. [40] Timo Schick and Hinrich Schütze. Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference. InProceedings of the 16th Conference of the European Chapter of the AssociationforComputationalLinguistics: MainVolume,pages255–269.AssociationforComput... on Pith
  21. [41] SemIf (formerly OpenJev). GitHub repository,https://github.com/TheoLeeCJ/SemIf-OpenJev, on Pith
  22. [42] Florian Standhartinger. JevBench: a benchmark for typed decision models. Hugging Face dataset, https://huggingface.co/datasets/fstandhartinger/jevbench, 2026. Public subset used here: 231 items indatasets/public/ on Pith
  23. [43] Yu Sun, Junhao Xu, Jiajia Shi, and Zijin Yang. Type-Safe Is Not Error-Free: A Constrained Decision Head Follows the Option Name, Not the Rubric Bound to It. arXiv preprint arXiv:2609.26758, 2026. AnyJev 14 on PithPith's reviewarXiv
  24. [44] Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, et al. 2 OLMo 2 furious. arXiv preprint arXiv:2501.00656, 2024 on PithPith's reviewarXiv
  25. [45] XinyuanWang,LiangWu,DongjieWang,andYanjieFu. PageLLM:Amulti-grainedrewardframework for whole-page optimization with large language models. arXiv preprint arXiv:2506.09084, 2026 on PithPith's reviewarXiv
  26. [46] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, PierricCistac,TimRault,RemiLouf,MorganFuntowicz,JoeDavison,SamShleifer,PatrickvonPlaten, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Que... on Pith
  27. [47] LiangWu,KellyWan,MayankDarbari,andLiangjieHong. Towardsresilientandautonomousnetworks: A BlueSky vision on AI-native 6G. arXiv preprint arXiv:2605.21395, 2026. Accepted at KDD 2026 on PithPith's reviewarXiv
  28. [48] Tiantong Wu and Wei Yang Bryan Lim. REFLEX with Jev for Efficient Selective Control in LLM Agents. arXiv preprint arXiv:2609.26532, 2026 on PithPith's reviewarXiv
  29. [49] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025 on PithPith's reviewarXiv
  30. [50] Tianze Yang, Liang Wu, Ruitong Sun, Yucheng Shi, Yanqiao Wang, Mayank Darbari, Ninghao Liu, Jin Sun, and Liangjie Hong. Self-evolving code-with-image reasoning. arXiv preprint arXiv:2608.11292, 2026 on PithPith's reviewarXiv
  31. [51] Weiwei Ye, Hangchen Liu, and Renhe Jiang. NumericJev: Jev-like LLM Numerical Decoding with Multiway Decision Trees. arXiv preprint arXiv:2609.28587, 2026 on PithPith's reviewarXiv
  32. [52] Guanxu Yu and Yuhang Yao. Visual Jev: Accurate and Efficient Decisions from Shared Visual Context. arXiv preprint arXiv:2609.25845, 2026. Code:https://github.com/guanxuyu-sv/Visual-Jev on PithPith's reviewarXiv
  33. [53] Jiamu Zhang, Liang Wu, Mayank Darbari, and Liangjie Hong. WiSP: A working-set view of mixture- of-experts serving on extremely low-resource hardware. arXiv preprint arXiv:2606.21868, 2026. Code: https://github.com/nokia-applied-research/WiSP on PithPith's reviewarXiv
  34. [54] Jiamu Zhang, Liang Wu, Kelly Wan, Hanjie Chen, and Liangjie Hong. SPECTRA: Pushing the KV cache beyond the 2-bit cliff via spectral transform coding. arXiv preprint arXiv:2608.07915, 2026. Code:https://github.com/nokia-applied-research/SPECTRA on PithPith's reviewarXiv
  35. [55] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. InProceedings of the 38th International Conference on Machine Learning, volume 139 ofProceedings of Machine Learning Research, pages 12697–... on Pith
  36. [56] Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. InInternational Conference on Learning Representations (ICLR), 2024 on Pith
  37. [57] Zaiyi Zheng, Liang Wu, Guanghui Min, Yaochen Zhu, Liangjie Hong, Chen Chen, and Jundong Li. SAPO:Step-alignedpolicyoptimizationforreasoning-basedgenerativerecommendation. arXivpreprint arXiv:2605.17648, 2026. AnyJev 15 on PithPith's reviewarXiv
  38. [58] Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine Heller, and Subhrajit Roy. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. In International Conference on Learning Representations (ICLR), 2024. A Per-model result... on Pith
  39. [2026] Accessed 29 September 2026 on Pith
Cited by 13 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment