Pith.

Artificial Intelligence · arXiv:2611.16499 v1 · submitted 12 Nov 2026 · 30 references

Recalibrating a Typed-Decision Model in an Afternoon: A Hands-On Recipe for Laya's Confidence Scores

Mohammed Nayeem Khan, Aliyas Ahmad Misri

The paper says

“A typed-decision model such as Laya reads a text and a list of options that the caller defines, for example ticket categories, and returns a probability for each option.”

more of the abstract

Code around it compares the top probability with a threshold to answer, abstain or escalate to a person. Earlier work reports, for a Turkish model, that temperature scaling can raise held-out calibration error (ECE). This tutorial gives a five-step recipe (measure, diagnose, fit, validate, deploy) in 130 lines of numpy with standard methods, none new. On 1,065 synthetic, LLM-labelled items the released Laya checkpoint has a pooled ECE of 0.073 but a bucket-macro ECE of 0.143 (0.127-0.184), the mean over buckets (questions grouped by type and option count): the 12-option bucket is over-confident by +0.22, the three-to-five-option bucket under-confident by 0.05, and they cancel. One global temperature makes pooled ECE worse (0.173); a cross-fitted temperature per bucket brings macro-ECE to 0.073 (0.078 over 10 fold assignments) and reaches 0.089 with 10 labelled items per bucket, but holding out one of seven evaluation sets gives 0.099. After the refit, a gate aimed at 10% error answers 46.3% of items (10.5% wrong) against 6.9% on the published confidences, but one cut-off per bucket on the published confidences answers 45.4%, so most of the gain comes from separating buckets. Confidence-only maps (Platt, histogram, isotonic) need 20 or more labelled items per bucket to approach the temperature; with 320 per bucket the lowest macro-ECE belongs to Platt. In these files the retrained checkpoints store no per-bucket table and the stored 12-option value 0.1006 is clamped to 0.5 at runtime; the refit helps 31 of 33 checkpoints on random folds and 21 with a set held out. Fit and test share one generator and LLM labels, so these are findings about these checkpoints and this data.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. A thorough, self-aware tutorial on recalibrating Laya's confidence scores; the empirical finding about the pooling trap is the main contribution, but the scope is narrow. This line is wrong

5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 30 references say.

5 minor objections

The paper says, at §7.1

Pith says: minor objection 1 of 5

The sensitivity of the choice:11+ multiplier to the probability floor (1e-9 vs. 1e-6 vs. 5e-5) is acknowledged but not discussed in terms of how a practitioner should choose this floor. A brief recommendation (e.g., 'set the floor to the smallest positive probability that can be represented by your probability storage format') would improve practical usability.

The paper says, at Table 7

Pith says: minor objection 2 of 5

The bootstrapped confidence intervals for macro-ECE are computed after cross-fitting and therefore ignore the variance of the fitted maps. The fold-seed column partly addresses this, but the caption should explicitly note that the intervals are conditional on the fitted maps, not unconditional estimates of the method's performance.

show 3 more

The paper says, at Figure 12

Pith says: minor objection 3 of 5

The y-axis label appears truncated or unclear in the description (likely 'Macro-ECE' is intended). Ensure the figure labels are legible at print resolution.

This objection may describe the copy Pith gave the reviewer, not the paper.

The paper says, at Listing 2

Pith says: minor objection 4 of 5

The grid is set as np.geomspace(0.2, 20, 60), giving ~8% steps. Briefly justify why this range and granularity are sufficient (e.g., 'sampling noise dominates steps finer than 8%).

The paper says, at Appendix C

Pith says: minor objection 5 of 5

The SHA-256 hash of the base results file is given; also state where the file can be obtained (e.g., a permanent repository link) so readers can verify the hash.

What would settle it

Reproduce the core experiment on a small corpus of real human-labeled traffic (e.g., 500 customer support tickets with human labels) using the same per-bucket temperature recipe. If the macro-ECE reduction (e.g., from ~0.14 to ~0.07) does not replicate within the paper's bootstrap intervals, the claim's generalizability would be weakened. If it does replicate, the recipe's practical utility is strengthened.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 13 Nov 2026

  1. Reader holds up confidence high
  2. Skeptic agrees 15 s · 26,615 tokens read, 1,606 written
  3. Editor wrote the verdict above 9 s · 27,406 tokens read, 855 written
  4. Referee accept: 5 minor 18 s · 26,911 tokens read, 2,108 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The paper is a well-structured, empirically careful tutorial that diagnoses and partially repairs miscalibration in Laya’s confidence scores. The core finding—that pooled ECE 0.073 hides a bucket-macro ECE of 0.143 because the 12-option bucket is overconfident and the 3–5 option bucket is underconfident, and they cancel—is a genuine observation that is not present in the cited literature. The ablation of how many labeled items each method needs (temperature already plateaus at 10 per bucket; Platt needs about 20) is practically useful for anyone doing this in an afternoon. The validation is thorough: cross-fitting, multiple fold seeds, binning sensitivity, clean-label subset, perturbation tests, and 33 checkpoints. The code is provided and frozen, which is good practice.

The soft spots are all acknowledged by the authors. The data are synthetic and LLM-labelled; accuracy is agreement with a teacher model. Calibration fitted here would not necessarily transfer to real traffic with different label noise or distribution shift. The authors explicitly state this in the limitations and in the abstract. The hold-out-set test gives a macro-ECE of 0.099 instead of 0.073, which is honest but narrows the practical improvement. Also, most of the gain in coverage at a 10% error target comes from treating buckets separately, not from the temperature itself—the paper is transparent about this, but it means the recipe is more about per-bucket thresholds than calibration per se.

Who is this for? Practitioners running Laya in production who want a quick, no-GPU fix for confidence calibration. It is not a new method and makes no theoretical claim. The paper deserves a serious referee because it is internally sound, reproducible, and the problem is real for anyone depending on Laya's confidence scores. I would recommend acceptance with the understanding that the scope is narrow and the authors have been transparent about limitations.

The strongest objection

The paper is transparent about its scope: labels are LLM-generated, data is synthetic, and findings are limited to the specific checkpoints and pool. It explicitly discusses the impact of label noise (Section 8.3), the effect of the probability floor (Section 7.1), and the fact that most gains come from bucket separation rather than calibration itself (Section 9.2). The reader's weakest_assumption (label quality and transferability) is already addressed by the authors. No hidden assumptions or internal inconsistencies were found. The central claim—that per-bucket temperature scaling reduces bucket-macro ECE on this pool—is supported by cross-validation, hold-out set validation, and multiple robustness checks (binning sensitivity, clean-label subset, perturbation). The results are reproducible from the provided code and frozen data. Therefore, no load-bearing concern beyond what the authors already acknowledge.

The paper's central finding is an empirical demonstration of a calibration improvement on a specific model and dataset. The authors comprehensively document limitations and test sensitivity to key assumptions (binning, label noise, fold assignment, floor handling). The reader's identified weakest assumption—that findings may not transfer to real-world data—is explicitly stated by the authors as a limitation (§15). The stress-test finds no additional load-bearing concerns that would alter the reader's verdict. The concrete test is a natural next step for assessing external validity, but its absence does not undermine the paper's internal soundness.

The referee's report

The paper is a thorough, transparent tutorial that demonstrates a practical pitfall—pooled ECE can hide opposing per-bucket errors—and offers a simple, inexpensive fix. Its strengths include: (i) every number is computed from frozen, SHA-256-verified data and code; (ii) cross-fitting, multiple fold seeds, bootstrap intervals, binning sensitivity, clean-label checks, hold-out set, and perturbation tests are all performed; (iii) limitations are stated explicitly (synthetic data, LLM labels, scope limited to these checkpoints); (iv) the recipe is directly implementable by practitioners. While the findings are specific to the studied checkpoints and data, the methodology transfers and the paper sets a high standard for reproducibility in applied calibration work. For these reasons, the paper makes a useful contribution to the community.

Pith's reading of the paper (a machine's summary, not the authors')

The paper provides a five-step tutorial for recalibrating the confidence scores of Laya, a typed-decision model that classifies text into predefined option sets. The authors measure calibration using expected calibration error (ECE), which compares average confidence to accuracy in bins. They discover that the overall ECE of 0.073 masks a much larger per-bucket average ECE of 0.143: the bucket for 11+ options is overconfident (confidence 0.947 vs. 72.2% accuracy), while the 3-5 option bucket is underconfident (0.633 vs. 68.7%). Because these errors point in opposite directions, they cancel in the pooled average. The core fix is to fit a separate temperature scaling factor per bucket, which flattens or sharpens the probability distribution without changing the predicted label. The authors cross-validate, compare with Platt scaling, histogram binning, and isotonic regression, and show that temperature is effective with as few as 10 labelled items per bucket. They also show how to set abstention thresholds after recalibration, which substantially improves selective prediction coverage at a fixed error rate. The work is thoroughly caveated: the data is synthetic, LLM-labelled, and fit and test share the same generator, so the numbers apply only to these checkpoints and this pool. All code is provided in plain numpy.

Core claim. The pooled ECE of 0.073 on the released Laya checkpoint hides a bucket-macro ECE of 0.143 because the 12-option bucket is overconfident by +0.22 and the 3-5 option bucket is underconfident by -0.05, and these opposite errors cancel. Per-bucket temperature scaling (cross-fitted) reduces macro-ECE to 0.073, and with only 10 labelled items per bucket it reaches 0.089.

The premise it rests on. The labels used to fit and evaluate calibration are generated by an LLM (Qwen3.8-27B), with accuracy measured as agreement with that teacher; the paper's findings are limited to these checkpoints on this synthetic, LLM-labeled pool, and real-world traffic with different label noise and distribution shift may invalidate the fitted calibration. The authors explicitly state this as a limitation.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

A typed-decision model such as Laya reads a text and a list of options that the caller defines, for example ticket categories, and returns a probability for each option. Code around it compares the top probability with a threshold to answer, abstain or escalate to a person. Earlier work reports, for a Turkish model, that temperature scaling can raise held-out calibration error (ECE). This tutorial gives a five-step recipe (measure, diagnose, fit, validate, deploy) in 130 lines of numpy with standard methods, none new. On 1,065 synthetic, LLM-labelled items the released Laya checkpoint has a pooled ECE of 0.073 but a bucket-macro ECE of 0.143 (0.127-0.184), the mean over buckets (questions grouped by type and option count): the 12-option bucket is over-confident by +0.22, the three-to-five-option bucket under-confident by 0.05, and they cancel. One global temperature makes pooled ECE worse (0.173); a cross-fitted temperature per bucket brings macro-ECE to 0.073 (0.078 over 10 fold assignments) and reaches 0.089 with 10 labelled items per bucket, but holding out one of seven evaluation sets gives 0.099. After the refit, a gate aimed at 10% error answers 46.3% of items (10.5% wrong) against 6.9% on the published confidences, but one cut-off per bucket on the published confidences answers 45.4%, so most of the gain comes from separating buckets. Confidence-only maps (Platt, histogram, isotonic) need 20 or more labelled items per bucket to approach the temperature; with 320 per bucket the lowest macro-ECE belongs to Platt. In these files the retrained checkpoints store no per-bucket table and the stored 12-option value 0.1006 is clamped to 0.5 at runtime; the refit helps 31 of 33 checkpoints on random folds and 21 with a set held out. Fit and test share one generator and LLM labels, so these are findings about these checkpoints and this data.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-13

The authors' main claim, as the classifier picked it out 0.57

“One global temperature makes pooled ECE worse (0.173); a cross-fitted temperature per bucket brings macro-ECE to 0.073 (0.078 over 10 fold assignments) and reaches 0.089 with 10 labelled items per bucket, but holding out one of seven evaluation sets gives 0.099.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.95 · human participants 0.05 · controlled comparison 0.85 · randomized assignment 0.28 · states quantitative result 0.99 · compares with prior results 0.96 · mentions code or data availability 0.75

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
A typed-decision model such as Laya reads a text and a list of options that the caller defines, for example ticket categories, and returns a probability for each option. background 0.99 0.08 0.03 asserted 1.00
Code around it compares the top probability with a threshold to answer, abstain or escalate to a person. background 0.96 0.11 0.03 asserted 0.99
Earlier work reports, for a Turkish model, that temperature scaling can raise held-out calibration error (ECE). background 1.00 0.03 0.04 asserted 0.89
This tutorial gives a five-step recipe (measure, diagnose, fit, validate, deploy) in 130 lines of numpy with standard methods, none new. contribution 0.57 0.72 0.09 asserted 1.00
On 1,065 synthetic, LLM-labelled items the released Laya checkpoint has a pooled ECE of 0.073 but a bucket-macro ECE of 0.143 (0.127-0.184), the mean over buckets (questions grouped by type and option count): the 12-option bucket is over-confident by +0.22, the three-to-five-option bucket under-confident by 0.05, and they cancel. result 1.00 0.91 0.14 asserted 1.00
One global temperature makes pooled ECE worse (0.173); a cross-fitted temperature per bucket brings macro-ECE to 0.073 (0.078 over 10 fold assignments) and reaches 0.089 with 10 labelled items per bucket, but holding out one of seven evaluation sets gives 0.099. result 1.00 0.94 0.33 asserted 1.00
After the refit, a gate aimed at 10% error answers 46.3% of items (10.5% wrong) against 6.9% on the published confidences, but one cut-off per bucket on the published confidences answers 45.4%, so most of the gain comes from separating buckets. result 0.91 0.93 0.21 asserted 0.99
Confidence-only maps (Platt, histogram, isotonic) need 20 or more labelled items per bucket to approach the temperature; with 320 per bucket the lowest macro-ECE belongs to Platt. result 0.99 0.85 0.11 asserted 1.00
In these files the retrained checkpoints store no per-bucket table and the stored 12-option value 0.1006 is clamped to 0.5 at runtime; the refit helps 31 of 33 checkpoints on random folds and 21 with a set held out. result 0.89 0.90 0.09 asserted 1.00
Fit and test share one generator and LLM labels, so these are findings about these checkpoints and this data. limitation 0.98 0.88 0.05 asserted 0.97

What kind of paper the classifier reports this is: empirical study 0.55 · evidence from observation or experiment 0.80

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 8,166 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

References (30)
  1. [1] Laya: multilingual non-autoregressive decision model. huggingface.co/convaiinnovations/laya; code github.com/NandhaKishorM/laya on Pith
  2. [2] Guo et al. On Calibration of Modern Neural Networks. arXiv:1706.04599, 2017 on PithPith's reviewarXiv
  3. [3] Naeini, Cooper and Hauskrecht. Obtaining Well Calibrated Probabilities Using Bayesian Binning. AAAI, 2015 on Pith
  4. [4] Nixon et al. Measuring Calibration in Deep Learning. arXiv:1904.01685, 2019 on PithPith's reviewarXiv
  5. [5] Kull et al. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with Dirichlet calibration. arXiv:1910.12656, 2019 on PithPith's reviewarXiv
  6. [6] Ovadia et al. Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. arXiv:1906.02530, 2019 on PithPith's reviewarXiv
  7. [7] Desai and Durrett. Calibration of Pre-trained Transformers. arXiv:2003.07892, 2020 on PithPith's reviewarXiv
  8. [8] Geifman and El-Yaniv. Selective Classification for Deep Neural Networks. arXiv:1705.08500, 2017 on PithPith's reviewarXiv
  9. [9] Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review 78, 1950 on Pith
  10. [10] Warner et al. Smarter, Better, Faster, Longer: ModernBERT. arXiv:2412.13663, 2024 on PithPith's reviewarXiv
  11. [11] Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers, 1999 on Pith
  12. [12] Zadrozny and Elkan. Transforming classifier scores into accurate multiclass probability estimates. KDD, 2002 on Pith
  13. [13] Niculescu-Mizil and Caruana. Predicting good probabilities with supervised learning. ICML, 2005 on Pith
  14. [14] Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory 16(1), 1970 on Pith
  15. [15] Kumar, Liang and Ma. Verified Uncertainty Calibration. NeurIPS, 2019. arXiv:1909.10155 on PithPith's reviewarXiv
  16. [16] Minderer et al. Revisiting the Calibration of Modern Neural Networks. NeurIPS, 2021. arXiv:2106.07998 on PithPith's reviewarXiv
  17. [17] Vaicenavicius et al. Evaluating model calibration in classification. AISTATS, 2019. arXiv:1902.06977 on PithPith's reviewarXiv
  18. [18] DeGroot and Fienberg. The comparison and evaluation of forecasters. The Statistician 32, 1983 on Pith
  19. [19] Błasiok et al. Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing. arXiv:2309.12236, 2023 on PithPith's reviewarXiv
  20. [20] Berta et al. CalArena: A Large-Scale Post-Hoc Calibration Benchmark. arXiv:2605.30188, 2026 (preprint) on PithPith's reviewarXiv
show 10 more
  1. [21] Sharma, Duncan, Marceau Caron and Vollmer. TypedBench: A Benchmark for Calibration, Framing Sensitivity, and Cost in System One Decision Models. arXiv:2610.11392, 2026 (preprint) on PithPith's reviewarXiv
  2. [22] Teke. ufakzeka-karar: An Open Turkish Typed-Decision Model with Order-Invariant Option Scoring. arXiv:2610.06744, 2026 (preprint) on PithPith's reviewarXiv
  3. [23] Li, He and Li. Beyond Calibration: Do a Typed-Decision Model’s Probabilities Obey the Probability Axioms? arXiv:2609.33209, 2026 (preprint) on PithPith's reviewarXiv
  4. [24] Joy. Do System One Decisions Add Up? A Study of Probabilistic Coherence. arXiv:2609.33971, 2026 (preprint) on PithPith's reviewarXiv
  5. [25] Rafe and Das. Benchmarking System One decision models against trained classifiers and language models for automated decision gates. arXiv:2610.00346, 2026 (preprint) on PithPith's reviewarXiv
  6. [26] Porcedda. Jev thinks “I don’t know”, but doesn’t say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration. arXiv:2609.35342, 2026 (preprint) on PithPith's reviewarXiv
  7. [27] Dai et al. ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty. arXiv:2610.17016, 2026 (preprint) on PithPith's reviewarXiv
  8. [28] Santillana. VX-One: Typed, Calibrated Decision Heads for Real-Time Perimeter Triage. arXiv:2610.30090, 2026 (preprint) on PithPith's reviewarXiv
  9. [29] Sudjianto et al. Teacher Supervision, Fixed Representations and Calibration: Exact Risk Calculations for Candidate-Scoring Decision Models. arXiv:2610.26797, 2026 (preprint) on PithPith's reviewarXiv
  10. [30] Efron and Tibshirani. An Introduction to the Bootstrap. Chapman & Hall, 1993 on Pith

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment