Pith.

Computation and Language · arXiv:2610.17309 v2 · submitted 12 Oct 2026

StartLux-Decision: AutoThink Decision Models with Reinforcement Learning for Calibrated Deliberative Decisions

Xialie Zhuang, Zunhai Su, Jia Pan, Ngai Wong, Guanhua Chen, Chuan-Wei Kuo

The paper says

“We introduce StartLux-Decision, decision models from 0.8B to 27B parameters that combine efficient single-pass prediction with adaptive, confidence-guided reasoning. The Instant models use bidirectional attention in full-attention layers to improve decision accuracy while keeping single-pass inference.”

more of the abstract

The AutoThink models invoke explicit reasoning for uncertain decisions and answer confident cases directly. A three-stage training pipeline establishes shared decision capabilities, specializes the two variants, and equips AutoThink with reasoning through reasoning supervision and reinforcement learning. We introduce Reinforcement Learning for Calibrated Deliberative Decisions (RLCD2), which uses a frozen model and proper scoring rules to reward improvements in decision probability quality and applies a soft penalty to overlong reasoning traces. RLCD2 raises the share of reasoning traces that terminate within the generation limit by up to 48 percentage points and cuts average reasoning length by up to 65%, while largely preserving decision accuracy. On the public subset of Decision Index 0.3, the Instant models outperform the strongest evaluated comparable-scale open-weight decision models by 7.4-16.9 points, and the 27B AutoThink model reaches 73.4. We also introduce DecisionPivot, a programmatically verifiable benchmark of computational decisions built from contrast pairs that differ in a single answer-changing detail. On DecisionPivot, AutoThink outperforms its Instant counterparts by 11.5-18.4 points in pair accuracy and leads among evaluated systems. The models can be run using frameworks such as vLLM and MLX, with quantized versions available to reduce memory usage.Inference code and links to model weights are available at https://github.com/StartLuxLabs/Startlux-Decision .

from the abstract

Pith says

Machine review · deepseek-v4-flash · 14 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. A thorough, well-engineered paper with strong empirical support, though the frozen-reward staleness concern is acknowledged but not investigated. This line is wrong

3 major objections · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at §3.1, Eq. (5); §3.2; §3.3

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The RLCD² reward uses a frozen copy of the starting model's probability readouts (p0 and p1). As the reasoning policy shifts during optimization, the distribution of generated traces may diverge from that of the frozen model, potentially making the reward signal stale.

more of objection 1

The paper does not analyze whether the frozen model's probabilities remain aligned with the current policy's decision quality, nor does it discuss the stability of the reward under distribution shift. The empirical improvements on held-out and DecisionPivot are encouraging, but the theoretical concern about reward validity under policy drift remains unaddressed. Please add a discussion or analysis of this potential issue, either by measuring alignment drift across training or by bounding it.

The paper says, at §4.1; §4.2

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.65):

“Nevertheless, calibration remains sensitive to distribution shift: maps fitted on the held-out distribution exhibit residual overconfidence of up to 6.1 percentage points when transferred to unseen datasets.”

Pith says: Objection 2 of 3 · major

The public subset of Decision Index 0.3 served as a development benchmark for model selection (data recipes, attention variants, seed averages). The authors acknowledge this, stating that DI scores reflect model selection. This fact reduces the strength of the DI 0.3 claims of outperforming reference models by 7.4–16.9 points, because the comparison is on a set that was used to guide develoment.

more of objection 2

The independent evaluation is on DecisionPivot, which was not used for any development choice. The manuscript should be explicit about this interpretation in the abstract and conclusion, not just in §4.1, to avoid overclaiming.

The paper says, at §5.2, Figure 13; §3.4, Table 3

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The fusion weight α=0.75 and the soft reasoning budget B=3,072 are optimized on held-out data from the training distribution. While this is standard practice, the sensitivity of the results to the choice of held-out set is not explored. For instance, the fusion weight that minimizes Brier on held-out data reduces DI 0.3 accuracy at the 27B scale (calibration improves but accuracy changes little).

more of objection 3

The trade-off between calibration and accuracy is acknowledged, but the authors should verify that the chosen parameters are not brittle across different held-out distributions (e.g., a separate validation set from DecisionPivot). Please provide a brief sensitivity analysis or discuss the stability of these hyperparameters.

6 minor objections

The paper says, at Figure 5, y-axis label 'Long computations'

Pith says: minor objection 1 of 6

The axis label 'Long computations' is ambiguous; it should read 'DecisionPivot pair accuracy (%)' or similar to match the table and text.

The paper says, at Table 3, column header 'Default accuracy'

Pith says: minor objection 2 of 6

The caption says 'Default-answer accuracy' but the table header abbreviates it. Please be consistent.

show 4 more

The paper says, at §5.4, first paragraph

Pith says: minor objection 3 of 6

The sentence 'Selection changes the conditional distribution of returned predictions.' could be improved by adding a reference to the optimizer's curse mentioned later.

The paper says, at Appendix D.1, Table D.1

Pith says: minor objection 4 of 6

The table footnot that 'best scores are bolded' but some benchmarks have ties. Consuder clarifying that bold indicates the best score among the listed models, with ties marked if necessary.

The paper says, at §2.3.3

Pith says: minor objection 5 of 6

The paragraph on 'Reasoning Supervision' mentions 'approximately 4,000–21,000 supervised reasoning examples per model'. The large range suggests different model sizes; please specify which size uses which approximate count.

The paper says, at §3.4, Table 3

Pith says: minor objection 6 of 6

The row 'Decision-weighted score, budget 3,072 (release)' appears twice for the 9B model (once after the 4B section and again in the 9B section). The formatting suggests the table may have been misaligned; check that all rows correspond to the correct model scale.

What would settle it

On the 2,986 held-out examples used in Table 2, compute two reward variants for each sampled trace: (1) the original frozen-model reward, and (2) a reward computed using the current (post-RL) policy’s own probability readouts instead of the frozen copy. Compare the rankings of traces within each group by the two rewards using Spearman correlation. If the average within-group Spearman ρ falls below 0.85, or if the fraction of items where the top reward trace changes exceeds 10%, the frozen reward introduces systematic bias that could degrade the policy.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 14 Oct 2026

  1. Reader holds up confidence high
  2. Skeptic agrees 29 s · 38,671 tokens read, 3,193 written
  3. Editor wrote the verdict above 15 s · 39,467 tokens read, 1,130 written
  4. Referee minor revision: 3 major objections, 6 minor 27 s · 39,009 tokens read, 2,632 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This is a substantial piece of work. The central idea—a single-pass decision model with a confidence gate that routes hard cases to explicit reasoning, trained with an RL method that rewards improvements in probability quality rather than just answer correctness—is novel and well-motivated. The engineering is extensive: five model scales, bidirectional attention for Instant, a three-stage training pipeline, and a new benchmark (DecisionPivot) that tests contrastive reasoning. The empirical work is the paper’s strongest suit: thorough ablations, calibration studies, latency comparisons, and quantized deployment support.

What the paper does well: it ships. Models on HF, code on GitHub, inference across three backends. The RLCD² analysis (Table 2, Figure 5) convincingly shows gains in termination rate and reasoning length while holding accuracy. The policy-aware calibration section (Section 5.5) is a nice bonus—it shows the authors thought beyond benchmark scores.

Soft spots. The stress-test concern about the frozen reward model in RLCD² is real and the paper does not address it. The reward is a proper-score difference from a frozen copy; if the policy’s trace distribution shifts far enough, the frozen model’s probability readouts could become misleading guides. The paper shows the method works in the examined settings, but it would be stronger with an analysis of reward stability under drift. Also, the DI public subset was used as a development set (the authors are transparent about this), meaning those scores reflect model selection, not independent test results. DecisionPivot is independent, but it is the authors’ own benchmark—that does not undermine the results, but it means the strongest reported numbers come from a self-constructed test.

Overall, the paper is solid, not revolutionary. The architecture and training recipe are practical contributions that will be useful to the decision-model community. The paper deserves a serious referee. I would bring it to reading group and would cite it if I work on reasoning-efficient decision models.

The strongest objection

The RLCD^2 reward is computed using a frozen copy of the starting model’s probability readouts (Section 3.1, Eq. 5). The reasoning policy is optimized based on how much the frozen model’s probabilities improve after reading the generated trace. As the policy parameters change, the distribution of generated traces shifts away from that of the starting model. The frozen model’s probability estimates may then become misaligned with the actual decision quality of the current policy. The paper does not measure or discuss this drift. If the reward becomes stale, the policy could be optimized toward improving the frozen model’s scores without corresponding gains on the current model’s own outputs. The held-out accuracy and termination improvements are encouraging, but they do not rule out a latent degradation in the reliability of the reward signal. This concern is distinct from the length penalty or score choice; it concerns the fundamental validity of using an unchanging scorer on a moving distribution.

The reader correctly identified the frozen reward model as the weakest assumption. The paper’s central claim of state-of-the-art performance is well supported by independent evaluation on DecisionPivot and strong results on the public DI subset. The concern about reward staleness is theoretical and does not undermine the presented empirical results, which show clear improvements from RLCD^2. However, it is a genuine gap that limits understanding of the method’s robustness. I agree with the reader’s assessment that the overall merits justify acceptance, but the medium correctness risk is warranted. No verdict change is needed; the existing verdict already reflects this nuance.

The referee's report

If the results hold, StartLux-Decision advances the state of open-weight decision models by combining efficient single-pass inference with adaptive reasoning through a unified probability interface. RLCD² provides a principled method for optimizing reasoning policies for decision quality rather than answer correctness, and demonstrates significant gains in reasoning termination and efficiency. The DecisionPivot benchmark offers a useful complement to existing benchmarks by emphasizing sensitivity to single-detail changes. The policy-aware calibration analysis is thorough and addresses a real practical issue. The release of models and code facilitates independent scrutiny and application.

Pith's reading of the paper (a machine's summary, not the authors')

The paper introduces StartLux-Decision, a family of decision models (0.8B to 27B) that pair an efficient single-pass predictor (Instant) with a variant that selectively invokes explicit reasoning for uncertain decisions (AutoThink). To train the reasoning variant, the authors propose Reinforcement Learning for Calibrated Deliberative Decisions ($RLCD^{2}$), which rewards improvements in probabilistic decision quality—measured via proper scoring rules—rather than answer correctness alone, and imposes a soft penalty on overly long reasoning traces. $RLCD^{2}$ increases termination rates by up to 48 percentage points and reduces average reasoning length by up to 65% while largely preserving held-out accuracy. On the public subset of Decision Index 0.3, the Instant models outperform all comparable-scale open-weight decision models by 7.4–16.9 points. On the new DecisionPivot benchmark, which uses contrast pairs that differ in a single answer-changing detail, the 27B AutoThink model achieves 86.9% pair accuracy, the highest among all evaluated systems, including much larger models.

What carries the argument. The key machinery is $RLCD^{2}$ (Reinforcement Learning for Calibrated Deliberative Decisions), which extends decision-oriented RL to explicit reasoning. It uses a frozen copy of the model to compute a reward equal to the improvement in a decision-weighted proper scoring rule between pre- and post-reasoning probabilities, minus a soft length penalty. The policy is optimized with group-relative policy gradients (based on GRPO). The AutoThink variant employs a confidence gate (threshold 0.9) to decide whether to reason; when reasoning is invoked, the initial and post-reasoning probability distributions are fused via a weighted average (75% post-reasoning). The Instant variant uses bidirectional attention in full-attention layers to improve single-pass accuracy.

Core claim. The central discovery is that a decision model can be trained to efficiently allocate reasoning only when its initial confidence is low, using a reinforcement learning reward that explicitly measures improvement in probabilistic calibration after reasoning. This approach, $RLCD^{2}$, uses a frozen copy of the model to compute the difference in a decision-weighted proper scoring rule between pre- and post-reasoning probability distributions, combined with a soft penalty on tokens exceeding a budget. The resulting AutoThink models preserve the speed of single-pass prediction on confident cases and deliver substantial accuracy gains on computation-intensive decisions, all while operating at parameter scales (0.8B to 27B) that outperform larger reference models. The authors also introduce DecisionPivot, a programmatically verifiable benchmark that tests sensitivity to subtle answer-changing variations.

The premise it rests on. The training reward in $RLCD^{2}$ uses a frozen copy of the model's probability estimates as a constant signal throughout policy optimization; if the actively trained policy's distribution diverges from the frozen model's, this signal may become stale or misleading, potentially degrading decision quality or calibration, and the paper does not analyze the stability of this setup under distribution shift.

If the paper is right (Pith's inferences, not the authors')

The 4B Instant model outperforms all reference decision models on Decision Index 0.3, including much larger systems, showing that efficient single-pass architectures can be competitive when trained with bidirectional attention and the right data.

AutoThink consistently beats its Instant counterpart by 11.5–18.4 percentage points on DecisionPivot, demonstrating that adaptive reasoning is especially valuable for computation-intensive decisions like arithmetic, scheduling, and procedural verification.

RLCD^2 improves termination rate by up to 48 percentage points and reduces mean reasoning length by up to 65% at 9B scale, offering a practical method to deploy reasoning models without excessive latency or resource use.

Policy-aware calibration reduces confidence-accuracy gaps to within 2.7 percentage points across four decision pipelines, addressing a previously overlooked source of miscalibration from selection and routing.

The RLCD^2 reward relies on a frozen model's probability estimates; if the trained policy diverges substantially, the frozen readout may become a stale signal, an open stability question not addressed in the paper.

DecisionPivot's programmatic generation framework could be extended to create unlimited contrastive decision pairs, potentially providing more robust and less cheat-prone benchmarks for decision model evaluation.

The confidence gate threshold and fusion weight are fixed after tuning on held-out data; adapting these parameters dynamically per task or per instance could further improve the accuracy-efficiency trade-off.

The strong performance of small Instant models suggests that many decision tasks do not require reasoning; a hybrid deployment could use a fast Instant model as a first pass and only escalate to larger or reasoning models when confidence is low.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

We introduce StartLux-Decision, decision models from 0.8B to 27B parameters that combine efficient single-pass prediction with adaptive, confidence-guided reasoning. The Instant models use bidirectional attention in full-attention layers to improve decision accuracy while keeping single-pass inference. The AutoThink models invoke explicit reasoning for uncertain decisions and answer confident cases directly. A three-stage training pipeline establishes shared decision capabilities, specializes the two variants, and equips AutoThink with reasoning through reasoning supervision and reinforcement learning. We introduce Reinforcement Learning for Calibrated Deliberative Decisions (RLCD2), which uses a frozen model and proper scoring rules to reward improvements in decision probability quality and applies a soft penalty to overlong reasoning traces. RLCD2 raises the share of reasoning traces that terminate within the generation limit by up to 48 percentage points and cuts average reasoning length by up to 65%, while largely preserving decision accuracy. On the public subset of Decision Index 0.3, the Instant models outperform the strongest evaluated comparable-scale open-weight decision models by 7.4-16.9 points, and the 27B AutoThink model reaches 73.4. We also introduce DecisionPivot, a programmatically verifiable benchmark of computational decisions built from contrast pairs that differ in a single answer-changing detail. On DecisionPivot, AutoThink outperforms its Instant counterparts by 11.5-18.4 points in pair accuracy and leads among evaluated systems. The models can be run using frameworks such as vLLM and MLX, with quantized versions available to reduce memory usage.Inference code and links to model weights are available at https://github.com/StartLuxLabs/Startlux-Decision .

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-14

The authors' main claim, as the classifier picked it out 0.41

“On the public subset of Decision Index 0.3, the Instant models outperform the strongest evaluated comparable-scale open-weight decision models by 7.4-16.9 points, and the 27B AutoThink model reaches 73.4.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.17 · human participants 0.04 · controlled comparison 0.77 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.96 · mentions code or data availability 0.14

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
We introduce StartLux-Decision, decision models from 0.8B to 27B parameters that combine efficient single-pass prediction with adaptive, confidence-guided reasoning. contribution 1.00 0.94 0.16 asserted 1.00
The Instant models use bidirectional attention in full-attention layers to improve decision accuracy while keeping single-pass inference. method 0.95 0.71 0.04 asserted 1.00
The AutoThink models invoke explicit reasoning for uncertain decisions and answer confident cases directly. method 0.84 0.60 0.06 asserted 1.00
A three-stage training pipeline establishes shared decision capabilities, specializes the two variants, and equips AutoThink with reasoning through reasoning supervision and reinforcement learning. method 0.98 0.91 0.05 asserted 1.00
We introduce Reinforcement Learning for Calibrated Deliberative Decisions (RLCD$^2$), which uses a frozen model and proper scoring rules to reward improvements in decision probability quality and applies a soft penalty to overlong reasoning traces. method 0.65 0.93 0.08 asserted 1.00
RLCD$^2$ raises the share of reasoning traces that terminate within the generation limit by up to 48 percentage points and cuts average reasoning length by up to 65%, while largely preserving decision accuracy. result 1.00 0.95 0.22 asserted 0.97
On the public subset of Decision Index 0.3, the Instant models outperform the strongest evaluated comparable-scale open-weight decision models by 7.4-16.9 points, and the 27B AutoThink model reaches 73.4. result 1.00 0.94 0.22 asserted 1.00
We also introduce DecisionPivot, a programmatically verifiable benchmark of computational decisions built from contrast pairs that differ in a single answer-changing detail. contribution 0.96 0.94 0.05 asserted 0.99
On DecisionPivot, AutoThink outperforms its Instant counterparts by 11.5-18.4 points in pair accuracy and leads among evaluated systems. result 1.00 0.94 0.15 asserted 1.00
The models run on vLLM, llama.cpp and MLX, with quantized variants for lower memory use. method 0.81 0.51 0.02 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.84

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-14: 1 request, 7,757 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (15)
Figure 1 of the paper
Figure 1. StartLux-Decision leads at every evaluated scale on the public subset of Decision Index 0.3. *Corresponding authors. 1 arXiv:2610.17309v1 [cs.CL] 12 Oct 2026
Figure 2 of the paper
Figure 2. Training and inference workflows of StartLux-Decision. (a) Three-stage training establishes shared decision capabilities, specializes the Instant and AutoThink variants, and develops explicit reasoning in AutoThink through reasoning supervision and reinforcement learning. Instant adopts bidirectional attention in full-attention layers, while AutoThink retains causal attention. (b) Instant produces decision probabilities in a single forward pass, whereas AutoThink first obtains an initial prediction p0 and selectively invokes reasoning based on its confidence. When reasoning is…
Figure 3 of the paper
Figure 3. AutoThink selectively invokes reasoning for uncertain decisions. Top: Two cases from the same DecisionPivot policy illustrate confidence-guided inference, with direct prediction for a confident case and reasoning-assisted correction for an uncertain case. Bars show decision probabilities, with correct options highlighted in green. Bottom: Across 7,800 core DecisionPivot items, the 27B AutoThink model makes direct predictions on 39% of cases with 96% accuracy and invokes reasoning for the remaining 61%. On reasoning-invoked cases, accuracy improves from 57% before reasoning to 93%…
Figure 4 of the paper
Figure 4. Bidirectional attention improves single-pass decision performance across most evaluated model scales. (a) Causal and bidirectional attention patterns in full-attention layers, with linear-attention layers remaining causal and decision probabilities read from the final prompt position. (b) DI 0.3 performance of Stage 2 models trained on identical data from the same Stage 1 checkpoint, differing only in their attention configurations. Runs with identical training configurations exhibit variations of 0.2–0.7 index points across random seeds. options that follow them. At 4B and 9B, this…
Figure 5 of the paper
Figure 5. RLCD2 improves termination and efficiency while largely preserving held-out accuracy. The 4B, 9B, and 27B AutoThink models before RL (gray) and after RLCD2 (violet), using the decision-weighted score and a soft penalty budget of 3,072 tokens (distinct from the 4,096-token generation limit). Held-out metrics are measured on 2,986 examples with reasoning invoked for every example: default-answer accuracy, termination rate, mean trace length, and post-reasoning calibration error before temperature scaling. DecisionPivot pair accuracy is measured on 7,800 core items at the default…
Figure 6 of the paper
Figure 6. RLCD2 improves decisions with previously unterminated traces but can reduce accuracy on long computations. Change in DecisionPivot accuracy at the default gate, after RL minus before RL, grouped by pre-RL trace length. Error bars show 95% intervals, and n denotes the number of items in each group. Effects on Longer Computations. To identify where these trade-offs arise, we group DecisionPivot examples by their pre-RL reasoning lengths (
Figure 7 of the paper
Figure 7. The Instant models outperform comparable-scale reference decision models by 7.4–16.9 points on the public subset of DI 0.3. Scores are plotted against parameter count. The gray line connects the strongest evaluated third-party model at each scale, all from the Decision 3.0 family of vLLM Semantic Router. Jev is hosted, with an undisclosed parameter count. 4. Experimental Evaluation We evaluate decision accuracy and probability reliability across tasks with varying computational demands. Decision Index 0.2.1 and 0.3 assess performance on established benchmarks, while DecisionPivot…
Figure 8 of the paper
Figure 8. The 27B Instant model outperforms all evaluated reference decision models across the five DI 0.3 domains. Overall and domain scores are reported in points on the public subset, with domain weights matching the leaderboard. Solid bars show Instant performance, while hatched extensions represent the score differences between AutoThink and Instant at each scale. When AutoThink scores lower, the dashed line marks the higher Instant score. Reference scores come from the public leaderboard, except for Decision 3.0 d3, which we evaluate using the same pipeline as StartLux-Decision. as many…
Figure 9 of the paper
Figure 9. The 27B AutoThink model achieves the highest pair accuracy among evaluated systems on DecisionPivot. Results cover 7,800 examples across 13 programmatically generated task families, with 95% bootstrap confidence intervals. Solid bars show single-pass performance, while hatched extensions indicate the gains of AutoThink over Instant at the default confidence threshold of 0.9. answerable facts clause change dependency resolution filter match ledger reconcile log count check permission inheritance policy case record rule schedule conflict state goal tool call valid triage order Clef…
Figure 10 of the paper
Figure 10. AutoThink excels on computation-intensive decisions, while strong single-pass models remain competitive on structural tasks. Pair accuracy is reported in percent across the 13 DecisionPivot core families, with darker cells indicating higher scores. StartLux-Decision models are shown in violet and reference systems in gray. 4.4 Probability Calibration Reliable decision-making requires well-calibrated probabilities, particularly when downstream systems use con￾fidence to act or defer. On 7,800 DecisionPivot core examples, we compare predicted confidence, defined as the maximum option…
Figure 11 of the paper
Figure 11. StartLux-Decision achieves lower calibration error than evaluated reference decision models on DecisionPivot. Left: empirical accuracy versus predicted confidence (maximum option probability) for the 4B, 9B, and 27B AutoThink models and hosted Jev across 7,800 core examples. Points below the diagonal indicate overconfidence. Right: calibration error based on confidence–accuracy gaps across 15 bins, reported in percentage points. Bars show AutoThink at the default confidence threshold of 0.9, with dashed lines marking the corresponding Instant models. The evaluated StartLux-Decision…
Figure 12 of the paper
Figure 12. AutoThink outperforms Instant on computation-intensive tasks across all three evaluated scales. Solid bars show Instant accuracy, while hatched extensions indicate accuracy differences between the complete model variants. AutoThink uses the default confidence threshold of 0.9. Results cover the 4B, 9B, and 27B models on 2,638 reconstructed GSM8K problems from DI 0.3, along with CRUXEval, GPQA Diamond, and MMLU-Pro from DI 0.2.1 (Table D.1). Effects of Bidirectional Attention. Bidirectional prompt attention improves performance beyond task families represented in Stage 2 training.…
Figure 13 of the paper
Figure 13. Confidence-guided reasoning captures most of the attainable accuracy gains, while RLCD2 reduces reasoning cost at 4B and 9B. Accuracy is plotted against average reasoning tokens per example, including those answered without reasoning, as the confidence threshold varies from 0 (never reason) to 1 (always reason). Stars indicate the default threshold of 0.9. The top row evaluates 2,986 held-out examples from the training distribution; the bottom row reports pair accuracy on 7,800 DecisionPivot core examples. Each model scale uses its own token axis. Solid curves represent the…
Figure 14 of the paper
Figure 14. Decision probabilities respond to supporting evidence and become more consistent across related question formulations as model scale increases. Results use single-pass readouts of the released models. (a) Mean change in the supported answer’s probability after removing a supporting sentence from HoVer or a gold paragraph from HotpotQA. Control perturbations add an irrelevant sentence or remove a distractor paragraph, producing mean changes within 0.015 of zero. (b) Mean absolute deviation of the summed affirmative probabilities from one when each candidate option is independently…
Figure 15 of the paper
Figure 15. Selection and routing induce miscalibration that policy-aware calibration mitigates without retraining the underlying models. (a) Accuracy of responses selected by a 9B verifier from N candidates, restricted to reported confidence of 80–90%. Shaded bands indicate variation across 20 splits. (b) Mean absolute confidence–accuracy gaps across 184 settings spanning best-of-N, think-gate, cascade, and select-then-think pipelines. Policy-aware calibration reduces these gaps across all four pipelines, with signed gaps ranging from −2.7 to +2.1 percentage points. individual responses does…

Figures are the authors', reproduced from the paper.

Cited by 7 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment