{"id":"e94ff9b3-9bdc-4bb7-86a9-f1cda2639ccc","arxiv_id":"2611.16499","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Per-bucket temperature scaling removes the cancellation that hides a 0.143 macro-ECE under a 0.073 pooled ECE on Laya's confidence scores, needing only 10 labels per bucket.","lead":"On the released Laya typed-decision model, the overall calibration error (ECE) is 0.073, but per-bucket ECE averages 0.143 because a 12-option bucket is overconfident (+0.22) and a 3-5 option bucket is underconfident (-0.05), cancelling in the pool. A simple per-bucket temperature scaling, using only 130 lines of numpy and 10 labelled items per bucket, reduces the macro-ECE to 0.089, providing a practical afternoon recipe for practitioners.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"No significant objection identified.","rationale":"The paper's central finding is an empirical demonstration of a calibration improvement on a specific model and dataset. The authors comprehensively document limitations and test sensitivity to key assumptions (binning, label noise, fold assignment, floor handling). The reader's identified weakest assumption—that findings may not transfer to real-world data—is explicitly stated by the authors as a limitation (§15). The stress-test finds no additional load-bearing concerns that would alter the reader's verdict. The concrete test is a natural next step for assessing external validity, but its absence does not undermine the paper's internal soundness.","tokens_in":26615,"tokens_out":1606,"duration_ms":14997,"concrete_test":"Reproduce the core experiment on a small corpus of real human-labeled traffic (e.g., 500 customer support tickets with human labels) using the same per-bucket temperature recipe. If the macro-ECE reduction (e.g., from ~0.14 to ~0.07) does not replicate within the paper's bootstrap intervals, the claim's generalizability would be weakened. If it does replicate, the recipe's practical utility is strengthened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper is transparent about its scope: labels are LLM-generated, data is synthetic, and findings are limited to the specific checkpoints and pool. It explicitly discusses the impact of label noise (Section 8.3), the effect of the probability floor (Section 7.1), and the fact that most gains come from bucket separation rather than calibration itself (Section 9.2). The reader's weakest_assumption (label quality and transferability) is already addressed by the authors. No hidden assumptions or internal inconsistencies were found. The central claim—that per-bucket temperature scaling reduces bucket-macro ECE on this pool—is supported by cross-validation, hold-out set validation, and multiple robustness checks (binning sensitivity, clean-label subset, perturbation). The results are reproducible from the provided code and frozen data. Therefore, no load-bearing concern beyond what the authors already acknowledge.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper presents a five-step recipe (measure, diagnose, fit, validate, deploy) for recalibrating the confidence scores of Laya, a typed-decision model, using standard methods such as temperature scaling, Platt scaling, histogram binning, and isotonic regression. On 1,065 synthetic, LLM-labelled items, the released Laya checkpoint has a pooled ECE of 0.073 but a bucket-macro ECE of 0.143, because the 12-option bucket is overconfident (+0.22) and the 3–5 option bucket is underconfident (−0.05), cancelling in the pool. A cross-fitted per-bucket temperature reduces macro-ECE to 0.073 (0.078 over 10 fold assignments) and reaches 0.089 with only 10 labelled items per bucket. The paper provides the complete numpy code, ablations on sample size, comparisons with alternative calibration maps, validation on 33 checkpoints, and explicit discussion of limitations (synthetic data, LLM labels, bucket proxy, binned ECE).","tokens_in":26911,"tokens_out":2108,"duration_ms":17967,"significance":"The paper is a thorough, transparent tutorial that demonstrates a practical pitfall—pooled ECE can hide opposing per-bucket errors—and offers a simple, inexpensive fix. Its strengths include: (i) every number is computed from frozen, SHA-256-verified data and code; (ii) cross-fitting, multiple fold seeds, bootstrap intervals, binning sensitivity, clean-label checks, hold-out set, and perturbation tests are all performed; (iii) limitations are stated explicitly (synthetic data, LLM labels, scope limited to these checkpoints); (iv) the recipe is directly implementable by practitioners. While the findings are specific to the studied checkpoints and data, the methodology transfers and the paper sets a high standard for reproducibility in applied calibration work. For these reasons, the paper makes a useful contribution to the community.","major_comments":[],"minor_comments":[{"comment":"The sensitivity of the choice:11+ multiplier to the probability floor (1e-9 vs. 1e-6 vs. 5e-5) is acknowledged but not discussed in terms of how a practitioner should choose this floor. A brief recommendation (e.g., 'set the floor to the smallest positive probability that can be represented by your probability storage format') would improve practical usability.","section":"§7.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The bootstrapped confidence intervals for macro-ECE are computed after cross-fitting and therefore ignore the variance of the fitted maps. The fold-seed column partly addresses this, but the caption should explicitly note that the intervals are conditional on the fitted maps, not unconditional estimates of the method's performance.","section":"Table 7","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The y-axis label appears truncated or unclear in the description (likely 'Macro-ECE' is intended). Ensure the figure labels are legible at print resolution.","section":"Figure 12","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The grid is set as np.geomspace(0.2, 20, 60), giving ~8% steps. Briefly justify why this range and granularity are sufficient (e.g., 'sampling noise dominates steps finer than 8%).","section":"Listing 2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The SHA-256 hash of the base results file is given; also state where the file can be obtained (e.g., a permanent repository link) so readers can verify the hash.","section":"Appendix C","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"accept"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The paper is a well-structured, empirically careful tutorial that diagnoses and partially repairs miscalibration in Laya’s confidence scores. The core finding—that pooled ECE 0.073 hides a bucket-macro ECE of 0.143 because the 12-option bucket is overconfident and the 3–5 option bucket is underconfident, and they cancel—is a genuine observation that is not present in the cited literature. The ablation of how many labeled items each method needs (temperature already plateaus at 10 per bucket; Platt needs about 20) is practically useful for anyone doing this in an afternoon. The validation is thorough: cross-fitting, multiple fold seeds, binning sensitivity, clean-label subset, perturbation tests, and 33 checkpoints. The code is provided and frozen, which is good practice.\n\nThe soft spots are all acknowledged by the authors. The data are synthetic and LLM-labelled; accuracy is agreement with a teacher model. Calibration fitted here would not necessarily transfer to real traffic with different label noise or distribution shift. The authors explicitly state this in the limitations and in the abstract. The hold-out-set test gives a macro-ECE of 0.099 instead of 0.073, which is honest but narrows the practical improvement. Also, most of the gain in coverage at a 10% error target comes from treating buckets separately, not from the temperature itself—the paper is transparent about this, but it means the recipe is more about per-bucket thresholds than calibration per se.\n\nWho is this for? Practitioners running Laya in production who want a quick, no-GPU fix for confidence calibration. It is not a new method and makes no theoretical claim. The paper deserves a serious referee because it is internally sound, reproducible, and the problem is real for anyone depending on Laya's confidence scores. I would recommend acceptance with the understanding that the scope is narrow and the authors have been transparent about limitations.","headline":"A thorough, self-aware tutorial on recalibrating Laya's confidence scores; the empirical finding about the pooling trap is the main contribution, but the scope is narrow.","tokens_in":27406,"tokens_out":855,"duration_ms":8656},"pith_extraction":null,"created_at":"2026-11-13T14:47:14.261178+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":null,"supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}