{"id":"d648e7a0-ce26-4afe-9af1-bab80deec51d","arxiv_id":"2611.16498","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A typed decision encoder (Laya, ModernBERT-large) needs about 859 rows (24 per question-option cell) to reach 82% accuracy on in-distribution synthetic data and tolerates 10% symmetric label noise without detectable loss.","lead":"A study of a decision encoder (Laya) finds that accuracy rises from 69.4% to 86.9% when trained on 2,818 LLM-written rows, and that 10% symmetric label noise causes no detectable drop but 20% confusable-pair noise costs 7.4 points. The results provide practical budgets (e.g., 24 rows per option cell for 82% accuracy) but come with strong warnings: everything is synthetic, in-distribution, and from one backbone and trainer.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Central estimate for 10% symmetric label noise relies on a single run; 95% interval does not exclude a cost up to 1.7 points, so claim of no detectable change is not settled.","rationale":"The paper is transparent about the single-seed limitation and includes the wide interval. The overall conclusions (accuracy growth, noise effects at 20% and 40%, advantage over TF-IDF) are based on multiple runs or have larger effects that are less likely to be due to seed variability. The 10% symmetric result is the most fragile because it is the only one with a non-significant cost, and it supports the narrative that low noise is harmless. A replication with multiple seeds would strengthen or qualify this claim, but the existing evidence is not strong enough to change the verdict from ACCEPT, given the paper's explicit caveats. The reader's weakest assumption (synthetic in-distribution data) is separate from this concern about seed replication for noise. Hence, no change to the reader's ACCEPT verdict is warranted, but the concern is real.","tokens_in":22478,"tokens_out":3644,"duration_ms":34566,"concrete_test":"Repeat the 10% symmetric noise condition on the full pool (2,818 rows) with at least 5 independent seeds. Compute mean accuracy drop and 95% confidence interval across seeds. If the mean drop exceeds 1.0 points or the interval excludes zero, the claim of no detectable change is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper reports that 10% symmetric label noise causes no detectable accuracy loss (-0.2 points, 95% interval -1.7 to +1.4). This result is based on one fine-tune (single seed). At full data, seed variance is only 0.1 points, but noise runs may have different variance, and the 20% symmetric noise run shows a 3.4-point drop. The paper explicitly states 'the 10% symmetric run is the only one without a detectable cost, and that is a statement about one run.' However, this single run is used to argue that low noise is well-tolerated. Without replication, the central estimate is unreliable. If the true cost were 1.0 points (within the interval), the conclusion about noise tolerance would weaken. The paper's exchange rate and practical budget rely on noise results; though the 10% run is not directly used in the exchange rate, the comparative claim that symmetric noise at 10% is harmless is a key part of the narrative.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper empirically studies the data efficiency and label noise tolerance of a fine-tuned typed-decision encoder (Laya, based on ModernBERT-large) trained on LLM-generated synthetic data. The authors fine-tune on nested subsets of a 2,818-row pool, test on 1,065 in-distribution items, and measure learning curves, calibration, per-task budgets, and the impact of symmetric and confusable-pair label noise. Key findings: accuracy rises from 69.4% to 86.9% with all rows; 10% symmetric label noise shows no significant drop in a single run (-0.2 points, 95% CI -1.7 to +1.4); 20% symmetric noise costs 3.4 points; 20% confusable-pair noise costs 7.4 points. The encoder outperforms TF-IDF on small data but ties at full size. The paper is narrowly scoped to synthetic, in-distribution, single-backbone settings and includes careful uncertainty quantification and explicit limitations.","tokens_in":22839,"tokens_out":4092,"duration_ms":55732,"significance":"If the results are reproducible, they provide practical guidance for teams deploying decision encoders: roughly 24 rows per question-option cell approach saturation, and moderate symmetric label noise (10%) may be well-tolerated, while confusable-pair noise is twice as damaging. The paper is methodologically careful—bootstrap intervals, McNemar tests, seed repeats for some conditions, and transparent reporting of limitations. The finding that confusable-pair noise at 20% costs 7.4 points is new for typed-decision models. The scope is narrow (synthetic, in-distribution, one backbone), but the paper honestly and thoroughly delineates what it does not claim, making it a useful benchmark for future work on data scaling and noise robustness in decision models.","major_comments":[{"comment":"The 10% symmetric noise result is based on a single seed. The paper states 'no detectable change' and provides the paired 95% CI (-1.7 to +1.4). However, this is the only noise level without a statistically significant effect, and the interval does not exclude a cost of up to 1.7 points. The abstract and key findings present this as 'no detectable change' without emphasizing the single-run limitation. Given that the narrative around noise tolerance may be taken as a headline result, I recommend adding an explicit qualification (e.g., 'in one run, we did not detect a cost; replication is needed') in the abstract and key findings to align presentation with the detailed caveats in Section 13.","section":"Section 9.1, Table 10; Abstract; Key Findings","authors_acknowledge":"This rests on one run and an inversion of the fitted curve, so only the order of magnitude is reliable.","authors_acknowledge_p":0.66},{"comment":"The confusable-pair partner map is computed from the clean models' predictions on the held-out test items. The authors acknowledge this in Section 13 and argue it overestimates harm. While the leave-one-out analysis in Section 11 partially addresses this for the labeller comparison, the noise sweep itself remains informed by test-set information. A more explicit discussion of how this design choice may limit generalizability to new distributions (where the confusable pairs may differ) would strengthen the limitations section.","section":"Section 3 and Section 13","authors_acknowledge":"No trained model saw a held-out item, but the confusable-pair sweep is not blind, and the same items define the partner map and the labeller comparison in Section 11 (which the leave-one-out count there addresses only in part).","authors_acknowledge_p":0.8},{"comment":"The saturating fit yields a plateau of 99.9% with a bootstrap interval [97.8–100.0], which the authors correctly note cannot be right because the independent labeller bounds accuracy near 97%. Despite this, the fit is inverted in Section 10 to compute 'equivalent clean rows' for noisy runs (Table 11). The paper states 'only the order of magnitude is reliable,' but the inversion of a non-informative curve may give a false sense of precision (e.g., ratios like 0.34, 0.39, 0.16). I suggest adding a more prominent caution in Section 10 that these exchange-rate numbers are rough extrapolations from a poorly constrained fit and should not be taken as precise estimates.","section":"Section 4 (Table 7) and Section 10 (Table 11)","authors_acknowledge":"This rests on one run and an inversion of the fitted curve, so only the order of magnitude is reliable.","authors_acknowledge_p":0.89}],"minor_comments":[{"comment":"The sentence 'A single run with 10% symmetric label noise shows no detectable change' could be rephrased as 'In a single run, 10% symmetric label noise showed no statistically detectable change' for clarity. The current phrasing may be misinterpreted as a definitive statement.","section":"Abstract and Key Findings","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Table 3 reports the 100% row as '86.9 ± 0.1' with range '86.6-86.9'. From the text, the four seeds are 86.9, 86.9, 86.9, 86.8; the mean is 86.875, s.d. is 0.05, and range 86.8-86.9. The reported s.d. of 0.1 is slightly generous but acceptable. Minor: consider reporting s.d. to one significant digit consistently.","section":"Table 3 (and Table 5)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The 'gap closed' percentages (15% and 23%) are based on a single seed and are not statistically significant (McNemar p > 0.10). Suggest adding a footnote stating that these percentages are from one seed and should be interpreted as descriptive only.","section":"Section 5, Table 5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The task named 'negation / churn / outage' is abbreviated in some figure labels as 'minor' (e.g., Figure 9). Please ensure that the figure legends use the full task names or define abbreviations clearly in the caption.","section":"Figure 8 and Table 6","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The description of Zhang et al. [4] states 'large networks can fit arbitrary labels.' That paper studies training from scratch with randomized labels on CIFAR/ImageNet, a different regime from fine-tuning a pre-trained encoder on a few hundred rows. Consider clarifying that the context differs, as this could mislead readers about the relevance.","section":"Section 14 (References)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The fingerprint table (Table A-2) lists only three shared input files. For full reproducibility, consider adding SHA-256 fingerprints of the core result files (e.g., the per-run JSON files or the summary tables) so that readers can verify the numbers without rerunning the pipeline.","section":"Appendix B","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The study design figure (Figure 1) would benefit from a small note that the calibration slice is taken from the corrupted file in noisy runs, as this is important for understanding the calibration results in Section 9.2.","section":"Section 3, Figure 1","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"minor_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This is a solid, bounded empirical study. The paper asks a practical question—how many LLM-written rows does a decision encoder need, and how wrong can the labels be?—and answers it with transparent, well-executed measurements. The learning curves, seed noise characterization, equal-update control, and noise sweep with confusable-pair structure are all done carefully, with bootstrap intervals and McNemar tests. The authors consistently flag limitations: synthetic in-distribution data, single seeds for many conditions, low seed counts, and that the 10% symmetric noise run is a single run with an interval that does not exclude a cost of up to 1.7 points. They are not overclaiming. The stress-test concern about the 10% symmetric noise result is valid but fully acknowledged in the paper. The authors explicitly say it is a statement about one run and that the interval does not rule out a cost. So there is no deception or overreach. The practical budgets (e.g., ~24 rows per question-option cell for 82% accuracy) are appropriately caveated. Soft spots: the data are entirely synthetic and in-distribution, so the curves are likely steeper than what real traffic would yield. The authors say this themselves. The equal-update control is only one seed and not significant; again, stated. The confusable-pair noise design uses held-out information to set the partner map, which is acknowledged and argued to overestimate harm. These are not hidden flaws. This paper is for practitioners deploying LLM-labelled data for routing/triage systems who need concrete budgets. It is also a good example of honest reporting for the ML community. It deserves serious peer review; I would accept it and recommend publication with minor clarifications (the single-run limitation is already clear). I would not cite it in my own work because the scope is narrow and synthetic, but I would recommend it to someone asking about data efficiency for similar models.","headline":"A careful, honest empirical study that measures learning curves and label noise for a typed-decision encoder on LLM-labelled synthetic data; the paper's main strength is its transparent reporting of its own limitations.","tokens_in":23916,"tokens_out":484,"duration_ms":21924},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"143 synthetic rows buy 28% of Laya's accuracy gain; 10% label noise has no cost","keywords":["data efficiency","learning curves","label noise","confusable-pair noise","LLM-generated supervision","decision encoders","calibration","annotation budget"],"falsifier":"Run the same fine-tuning protocol on human-labelled data from a comparable decision task (e.g., ticket routing) with at least 1,000 items, and measure whether the learning curve from 143 to 2,818 rows shows the same shape and whether 10% symmetric label noise still causes no detectable accuracy drop. If the early gains are smaller or the noise tolerance is lower, the paper's conclusions are specific to synthetic LLM data.","tokens_in":23986,"feed_emoji":"🤖","tokens_out":7627,"duration_ms":86551,"temperature":0.7,"pith_summary":"This paper measures how many LLM-written training rows a decision encoder (Laya, based on ModernBERT-large) needs and how much label noise it can tolerate, using a pool of 2,818 synthetic rows and 1,065 held-out items from the same generator. Accuracy rises from 69.4% (zero-shot) to 86.9% with all rows, with the biggest gain from the first 143 rows (74.3%). Symmetric label noise at 10% shows no detectable loss (-0.2 points, 95% interval -1.7 to +1.4); at 20% the loss is 3.4 points, and confusable-pair noise at 20% costs 7.4 points. The encoder outperforms a TF-IDF classifier strongly at small data sizes but ties at full size. All results are on synthetic, in-distribution data and are optimistic for real traffic, as the authors acknowledge.","feed_headline":"143 rows give 28% of Laya's accuracy gain; 10% label noise has no cost","feed_subtitle":"Synthetic data budgets: 24 rows per option cell reach 82% accuracy; 10% label noise undetectable","key_machinery":"Laya, a non-autoregressive decision encoder based on ModernBERT-large, fine-tuned with a calibration-aware loss on nested subsets of an LLM-written synthetic pool. The two noise structures—uniform label flips (symmetric) and flips to the most confusable option (confusable-pair)—are the main instruments for measuring label-quality tolerance. The curve is fitted with a saturating function err(f) = e∞ + a $f^{{-b}}$ to extrapolate the plateau.","core_discovery":"The paper establishes that for the fine-tuned decision encoder Laya (ModernBERT-large) on LLM-written synthetic rows, the learning curve saturates such that 143 rows (4 per option cell) achieve 74.3% accuracy, and 2,818 rows achieve 86.9%, with the step from 75% to 100% inside binomial error. Under label noise, a single run with 10% symmetric flips shows no detectable accuracy change (-0.2 points); 20% symmetric costs 3.4 points and 20% confusable-pair costs 7.4 points. The encoder's advantage over a bag-of-n-grams logistic regression is a small-data advantage: +21.1 points at 5% of rows, dropping to +0.9 points (interval includes zero) at full size. The paper claims these figures apply only to synthetic, LLM-labelled data from the same generator, and that real traffic curves would be shallower.","pith_inferences":["Because the independent labeller agrees with gold labels on only 96.5% of items, the ceiling for accuracy on this synthetic pipeline is around 97%, and about half of the remaining error is confusable-pair in nature. This suggests that the label noise inherent in LLM-generated data may already be at the boundary where symmetric noise is tolerable but confusable-pair noise is not.","The extreme flatness of the learning curve beyond 1,400 rows implies that for many practical applications, collecting far fewer synthetic rows may be optimal, contradicting the common practice of scaling up synthetic data to thousands.","The study uses only one backbone (ModernBERT-large) and one trainer; if these results hold across other architectures, they would provide a general rule-of-thumb for budget planning with LLM-supervised decision encoders. If not, the specific numbers may not transfer, but the method for measuring them does."],"forward_implications":["Teams deploying Laya can budget about 24 rows per question-option cell for ~82% accuracy on synthetic in-distribution tasks.","Label noise up to 10% symmetric is effectively free; confusable-pair noise above 10% should be monitored because it costs roughly double the symmetric rate.","The sample-efficiency ratio over TF-IDF shrinks from 5.1× at 143 rows to 1.0× at 2,818 rows, so the pre-trained encoder is only worthwhile when fewer than about 1,000 rows are available.","Per-task learning curves vary widely: multilingual routing gains little from more data, while JSON tickets and urgency benefit more. Budgets should be set per task.","Training longer on small subsets closes only 15–23% of the gap to full-data accuracy and degrades calibration, so repeating epochs is not a substitute for more distinct rows."],"supporting_citations":[{"why":"Establishes power-law scaling of language-model loss with data, which motivates the curve-fitting approach of this paper.","marker":"[1]"},{"why":"Provides the confident-learning framework that inspired the confusable-pair noise design and the class-conditional noise structure.","marker":"[2]"},{"why":"Shows that deep networks are robust to massive label noise when enough clean data are present, which provides background for the noise tolerances observed here.","marker":"[5]"},{"why":"Benchmarks typed-decision models and reports that fine-tuned encoders are the most accurate; this paper extends their work by measuring the data budget.","marker":"[6]"},{"why":"Quantifies the scaling of fine-tuning with pre-training, which relates to the equal-update control and the shape of the learning curve at small sizes.","marker":"[9]"},{"why":"Compares label noise to other noise types in LLM fine-tuning but does not vary noise structure; this paper fills that gap for decision encoders.","marker":"[10]"}],"fun_headline_variants":["143 rows give 28% of Laya's accuracy gain","10% label noise has no cost for Laya","At 5% rows, Laya beats TF-IDF by 21 points","24 rows per option cell reach 82% accuracy","20% confusable-pair noise costs 7.4 points"],"cache_read_input_tokens":0,"weakest_assumption_plain":"All training and held-out items come from the same LLM generator and screening pipeline, so the measured learning curves and noise tolerances are in-distribution and optimistic; they may not transfer to human-labelled data or real user traffic.","fun_headline_variants_meta":{"raw":{"variants":["143 rows give 28% of Laya's accuracy gain","10% label noise has no cost for Laya","At 5% rows, Laya beats TF-IDF by 21 points","24 rows per option cell reach 82% accuracy","20% confusable-pair noise costs 7.4 points"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000267,"raw_usage":{"cost":0.000266728,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000266728,"upstream_inference_prompt_cost":0.000144368,"upstream_inference_completions_cost":0.00012236},"total_tokens":1673,"prompt_tokens":1236,"completion_tokens":437,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":349}},"tokens_in":1236,"tokens_out":437,"cost_source":"openrouter_usage","duration_ms":4936,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":349,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T14:41:52.515281+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Run the same fine-tuning protocol on human-labelled data from a comparable decision task (e.g., ticket routing) with at least 1,000 items, and measure whether the learning curve from 143 to 2,818 rows shows the same shape and whether 10% symmetric label noise still causes no detectable accuracy drop. If the early gains are smaller or the noise tolerance is lower, the paper's conclusions are specific to synthetic LLM data.","supporting_citations":[{"cited_title":"huggingface.co/convaiinnovations/laya; code github.com/NandhaKishorM/laya","cited_arxiv_id":null,"evidence_quote":"Establishes power-law scaling of language-model loss with data, which motivates the curve-fitting approach of this paper."},{"cited_title":"Probable inference, the law of succession, and statistical inference","cited_arxiv_id":null,"evidence_quote":"Provides the confident-learning framework that inspired the confusable-pair noise design and the class-conditional noise structure."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}