{"id":"7264391c-0247-4f38-91dd-5f1ca197286d","arxiv_id":"2607.19847","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An ensemble of three post-trained small specialist language models (knowledge, reasoning, coding) with calibrated confidence and abstention outperforms frontier reasoning models on high-precision missing-value prediction at a fraction of the cost.","lead":"The paper builds Auto-Fill, an ensemble of three small specialist language models — one for factual knowledge, one for step-by-step reasoning, one for code — that fills missing cells in tables and abstains when unsure. On 11 benchmarks it reports higher precision-controlled recall than frontier models such as o3-pro and Gemini 3 Pro, at roughly one percent of their query cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Max-of-calibrated confidences is used as final confidence without post-selection recalibration; conditioning on being the max biases the score upward, so the §6.2 precision-at-τ guarantee is not supported.","rationale":"The reader's random-masking concern is a real external-validity issue, but the selection-bias concern is more directly about the paper's stated mechanism. The ensemble's final confidence is the maximum of three separately calibrated scores. A maximum of calibrated noisy predictors is not itself calibrated; this is a classic selection effect. The paper's evaluation of R@P=0.9 does not test calibration, and the ECE analysis in §B.2 is per-specialist only. The central problem definition (Def. 1) requires precision≥τ with abstention; if the final confidence is biased, a deployed τ=0.9 threshold will not deliver the promised precision. The fix is straightforward — recalibrate after selection — and the paper is otherwise strong: specialist training, ablations, and error audit are valuable. Therefore the verdict should remain CONDITIONAL: the authors need to add a post-selection calibration check and report ensemble-level ECE/precision-at-τ, including on OOD. I partially overlap with the reader's concern because both are about calibration validity, but the more internal calibration gap seems more load-bearing; both should be addressed.","tokens_in":54439,"tokens_out":18933,"duration_ms":203222,"concrete_test":"Run the full ensemble on a held-out validation set disjoint from the 2,000-case calibration set; record (selected specialist, final \\hat conf, correctness). Fit a second-stage isotonic calibrator h to the final \\hat conf. Compare ECE of raw final \\hat conf vs h on OOD, and compute actual precision at τ=0.9 under both. Also recompute R@P=0.9 using h-ranked scores. If raw ECE is materially worse and/or precision at τ<0.9, the precision guarantee fails; if h changes R@P, current selection is suboptimal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6.1 fits an isotonic calibrator g_i separately for each specialist using pairs (c_j, y_j) from Dval; Section 6.2 then selects i* = argmax_i \\hat conf_i and abstains if \\hat conf_{i*} < τ. Even if each g_i is marginally calibrated, the selected score is conditionally biased: the event that a specialist has the largest of three noisy calibrated scores is correlated with over-estimation of its true correctness probability. Hence P(correct | i*=i, \\hat conf_i=p) need not equal p; the advertised 'precision ≥ τ' at threshold τ (Definition 1) is not established. The paper reports per-specialist ECE (Table 9 / Fig. 8) but no ECE for the ensemble's final confidence. Appendix D's 10 'mis-routes' — cases where a non-selected specialist was individually correct — are symptomatic of a selection/calibration failure. Since R@P=0.9 is rank-based, the headline benchmark numbers can survive this flaw, but the central 'calibrated abstention ensures high precision' claim cannot be verified without a second-stage calibration of the selected output.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses high-precision missing-value prediction in tables: a model must either predict the exact missing value or abstain, with precision above a user threshold τ. It proposes Auto-Fill, which post-trains three specialist small language models — a knowledge specialist (direct SFT, log-probability confidence), a reasoning specialist (distillation from DeepSeek-R1, two-stage sampling-based confidence), and a coding specialist (distillation of column-level code, execution-based confidence). Each specialist’s confidence is calibrated with isotonic regression, and the ensemble selects the most confident specialist or abstains if the maximum calibrated confidence is below τ. Experiments on 11 benchmarks with 2,200 tables (5 in-distribution, 6 out-of-distribution) compare Auto-Fill against frontier reasoning models, retrieval imputation, classical repair, and tabular ML baselines. The paper reports mean R@P=0.9 of 0.628 (Auto-Fill-Qwen) and 0.656 (Auto-Fill-GPT), above the best baseline o3-pro (0.56), at substantially lower cost. It also includes extensive ablations, negative results (modest RL gains, hybrid model collapse), calibration comparisons, and a manual error audit.","tokens_in":54564,"tokens_out":5128,"duration_ms":54418,"significance":"If the empirical claims hold, this is a practically valuable contribution: it demonstrates that a calibrated ensemble of small specialized models can achieve high-precision table repair at a fraction of the cost of frontier models, and the explicit abstention formulation is well matched to real spreadsheet use. The paper’s strengths include a publicly released benchmark suite, open-source code, systematic ablation of the design space, honest documentation of negative results, a manual failure audit, and an explicit comparison of calibration methods. The core capability-decomposition idea is well supported by the ablation tables. However, the central ‘calibrated abstention ensures precision ≥ τ’ claim is not fully verified because the ensemble selection step itself is not recalibrated, and the cost claim in the abstract is stronger than the reported tables support. These are fixable but load-bearing issues.","major_comments":[{"comment":"The max-of-calibrated-confidences selection is not recalibrated. Even if each g_i is marginally calibrated (Eq. 8), the event that specialist i has the largest of three noisy calibrated scores is correlated with over-estimation of its true correctness probability. Thus P(correct | i*=i, conf_i=p) need not equal p, and the Definition 1 precision guarantee at threshold τ is not established. Table 9 reports per-specialist ECE but no ECE for the ensemble’s final selected confidence. The error audit in Appendix D is symptomatic: the 10 mis-routed failures and 11 unrecoverable failures are, by definition, cases where the ensemble made a confident wrong prediction rather than abstaining; the audit’s claim that unrecoverable cases are ‘silent deferrals’ is inconsistent with them being failures. The authors should provide a second-stage isotonic calibration of the selected output, or equivalently","section":"§6.2, Eq. (9); §6.1, Eq. (8); Appendix D"},{"comment":"The training and most test protocols use uniformly random cell masking; A.1 states that for non-relational datasets cells are ‘uniformly sampled’ and only one cell per table is masked. Real spreadsheet missingness is likely not MCAR — missingness often depends on the row/column context, and the paper’s deployment target is Excel/Sheets at scale. Under MNAR, the advertised precision-at-τ guarantee, which is calibrated on random masking, may not transfer. The paper’s own Appendix D shows 11% of sampled failures are values not deducible from context, supporting the concern that real missing data can violate the learnable-rule assumption. The authors should evaluate on naturally missing cells (e.g., cells that were originally empty and held out) or provide a direct argument, with data, that random masking is representative of the missingness mechanism in their target deployment. Without this","section":"§5, Example 4; §7.1, A.1"},{"comment":"The abstract and Figure 5 state that Auto-Fill operates at ‘less than 1%’ of the cost of frontier models. From Table 2, Auto-Fill-Qwen’s mean cost is $1.48e-3, which is about 0.84% of o3-pro’s $1.77e-1, but it is 3.4% of Gemini 3 Pro’s $4.30e-2. Auto-Fill-GPT, the higher-quality variant, costs $1.13e-2, which is 6.4% of o3-pro and 26% of Gemini 3 Pro. Thus the ‘less than 1%’ claim holds only for Auto-Fill-Qwen against o3-pro, not for the other frontier baselines reported in the same sentence. Please qualify the cost claim to match the data, or present the cost comparison separately by baseline.","section":"Abstract; Figure 5; Table 2"}],"minor_comments":[{"comment":"Eq. (5) mixes units: in Example 5, the verbalized confidence is an integer 0–100, producing conftrain = 76, while later sections treat confidence in [0,1]. Please state the normalization explicitly.","section":"§5.2, Eq. (5)"},{"comment":"The hybrid model’s R@P=0.9 of 0.000 on ID while its full accuracy is 0.558 (Table 14) is striking; a sentence clarifying that this is a confidence-calibration collapse rather than a pure prediction failure would help readers interpret the zero entry.","section":"§7.4, Table 6"},{"comment":"The ECE values in Table 9 are per-specialist. Please add a line reporting ECE for the ensemble’s selected confidence, since that is what governs the abstention decision.","section":"Appendix B.2, Table 9"},{"comment":"In the residual error labels, the ‘within-M_K’ category includes cases that the authors propose to fix with retrieval augmentation, but retrieval is not implemented in the current system. This makes the ‘89/100 absorbed by the decomposition’ statement optimistic. Consider reporting the subset that is recoverable without adding new machinery.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The paper is substantive and well executed, and the core specialization idea is likely to be influential. The main blockers are the missing recalibration of the max-selected confidence (which is load-bearing for the precision guarantee) and the overbroad cost claim. A naturally missing evaluation, even on a small sample, would also materially increase confidence in the deployment claim. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Auto-Fill is a solid systems paper that does what it claims on its own benchmarks: three capability-specialist SLMs with mode-specific confidence beat frontier reasoning models on high-precision missing-cell prediction at a fraction of the cost. The design-space exploration is thorough, the ablations are honest, and the error audit is a nice touch. If you work on tabular data cleaning or LLM confidence, this is worth reading.\n\nThe genuinely new part is the combination: logprob-based confidence for a direct-answer knowledge model, teacher-sampling-based confidence for a reasoning model, and execution-based column validation for a coding model, all mapped through isotonic calibration and combined by max-selection with abstention. The paper also contributes a new 11-benchmark suite and shows that small specialists can be competitive with o3-pro at ~1% the cost (for the Qwen variant). The negative results — RL gives modest gains, the hybrid model collapses, classical ensembles don't generalize — are reported without spin.\n\nThe soft spots are real but not fatal. The stress-test note is correct: each specialist's calibrated confidence is marginally calibrated, but the max-of-three is conditionally overconfident because selection on the score biases the estimate. The paper never recalibrates the ensemble score, so the claim that abstaining below tau guarantees precision >= tau isn't actually established. The rank-based R@P=0.9 metric survives this, but the deployment story relies on the guarantee. That should be fixed with a second-stage calibrator on the selected outputs.\n\nAlso, the abstract's \"less than 1% of cost\" only holds for Auto-Fill-Qwen; Auto-Fill-GPT is about 6% of o3-pro's price. And the evaluation masks cells uniformly at random in most benchmarks but strategically in the relational ones, which flatters the coding specialist; the enterprise OOD data is proprietary, so the headline numbers can't be independently reproduced. The random-mask assumption could also matter for real missingness, though that's a distributional concern rather than a flaw in the method.\n\nOverall: the core result is credible, the paper is transparent, and the issues are addressable. It deserves a serious referee, and I'd send it out. For my own work, I'd cite it as a strong example of specialist ensembles with abstention, though with a caveat about the calibration guarantee.","headline":"Auto-Fill is a strong systems paper with a real calibration gap: the ensemble confidence isn't recalibrated after max-selection, so the precision-at-threshold guarantee isn't supported, but the empirical rank-based results are credible.","tokens_in":55252,"tokens_out":3550,"would_cite":true,"duration_ms":37882,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","68T50","62D10"],"pacs":[],"model":"deepseek-v4-flash","headline":"Three small specialist language models — one for knowledge, one for reasoning, one for code — fill missing spreadsheet cells more accurately than frontier reasoning models, at under 1% of the cost.","keywords":["missing value prediction","data cleaning","small language models","model specialization","confidence calibration","abstention","table reasoning","code-based reasoning"],"falsifier":"Take naturally missing cells from real spreadsheets, recover their true values from source documents, run the released system, and measure precision on the predictions it does not abstain from: if precision falls materially below the calibrated 0.9 threshold, the random-masking assumption fails. A cheaper probe is to re-mask the same 2,200 benchmark tables with a non-random missingness model — e.g., masking cells conditional on row or column features — and compare R@P=0.9.","tokens_in":54148,"feed_emoji":"🧹","tokens_out":14669,"duration_ms":117468,"temperature":0.7,"pith_summary":"The paper sets out to show that filling missing cells in tables is a three-mode task — it needs world knowledge for factual cells, text-based reasoning for pattern inference, and code-based reasoning for arithmetic or formula relationships — and that each mode is best handled by a separate small language model trained for that one capability. Its central claim is that a calibrated ensemble of these specialists, which picks the most confident model or abstains entirely, beats frontier reasoning models such as o3-pro, Gemini 3 Pro, and DeepSeek R1 on high-precision missing-value prediction, at under 1% of their cost. On 11 benchmarks built from 2,200 real tables, the best variant reports mean R@P=0.9 of 0.656 against 0.56 for the strongest frontier baseline. A sympathetic reader would take away that specialized, abstention-aware small models can make spreadsheet-scale data cleaning both accurate and affordable.","feed_headline":"Three specialized models beat frontier AI at filling missing cells","feed_subtitle":"Three specialists trained for knowledge, reasoning, and code beat the best frontier model at a hundredth of the cost.","key_machinery":"The load-bearing mechanism is the specialist ensemble plus calibrated abstention. Three small models are post-trained so that each handles one mode: the knowledge specialist (direct supervised fine-tuning; confidence from geometric-mean token log-probabilities), the reasoning specialist (distilled on the teacher's shortest correct chain-of-thought trace; confidence from a two-stage product of correctness rate × mean verbalized confidence), and the coding specialist (distilled on teacher-generated column-level code; confidence from executing the code and measuring how many observed column values it reconstructs). Because the three raw scores are not comparable, isotonic regression maps each t","core_discovery":"This paper claims that high-precision missing-value prediction in tables is best solved by a calibrated ensemble of three specialist small language models. A knowledge specialist answers directly from memory and formats output to match the column; a reasoning specialist is distilled on chain-of-thought traces; a coding specialist emits column-level Python that reconstructs the target column. Each carries a mode-appropriate confidence signal — token log-probabilities, teacher sampling agreement, execution-based column reconstruction — mapped by isotonic regression to one probability scale. The ensemble returns the most confident prediction only above a precision threshold, else abstains.","pith_inferences":["Editorial: the masking assumption is the transfer risk. Training and evaluation both use uniformly random cell masking; real-world missingness is often not random — cells tend to be absent because of the very column context the models attend to. If that distribution shift is material, the advertised 0.9 precision at deployment would drop below the measured level.","Editorial: the coding specialist's execution-based self-validation is a transferable idea — any prediction expressible as a column transformation (SQL, formula, unit conversion) can be validated against observed rows without labels, which may apply to error detection and schema repair beyond imputation.","Editorial: the error audit points to an obvious paper-adjacent improvement: 52 of 100 sampled failures are knowledge-specialist misses, many of them long-tail facts a web lookup would settle. Augmenting the knowledge specialist with retrieval would likely raise recall, at the cost of the paper's strict per-query budget.","Editorial: the 'exact match or abstain' framing may be the paper's most transferable contribution — it reframes imputation from statistical closeness to decision-making under a cost of error, a framing that could benefit other data-cleaning tasks such as error detection and schema matching."],"forward_implications":["Missing-value prediction can be deployed at spreadsheet scale: the best Auto-Fill variant returns a 0.9-precision segment with mean recall 0.656 at $0.0113 per query, while o3-pro's comparable segment reaches 0.56 at $0.177 per query.","Calibrated abstention, not raw accuracy, is what separates usable from hallucinating predictors: frontier models reach near-perfect recall on deterministic relational tables but collapse on benchmarks that require saying 'I don't know'.","Separate specialists beat a single hybrid model: the hybrid collapses to near-zero recall under abstention, while the ensemble is best on every dataset, confirming capability interference in shared parameters.","The three-mode decomposition is near-exhaustive: the paper's error audit attributes 89 of 100 sampled failures to capability gaps within the three specialists and only 11 to genuinely unrecoverable cells.","Cost scales down gracefully: smaller backbones (Qwen 1.7B/4B, GPT-4.1 nano) retain most of the quality, so practitioners can trade recall for cost within the same framework."],"fun_headline_variants":["Three mini models fill blanks better than big AI","Specialist team beats frontier AI on missing data","Auto-Fill: tiny models, huge accuracy for table gaps","Calibrated abstention makes small models win on tables","Knowledge, text, code: three specialists fix tables cheaply"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that uniformly random cell masking produces training and test cases that represent how values really go missing in real tables; if actual missingness is systematic rather than random, the calibrated precision guarantees may not hold in deployment.","fun_headline_variants_meta":{"raw":{"variants":["Three mini models fill blanks better than big AI","Specialist team beats frontier AI on missing data","Auto-Fill: tiny models, huge accuracy for table gaps","Calibrated abstention makes small models win on tables","Knowledge, text, code: three specialists fix tables cheaply"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00065,"raw_usage":{"total_tokens":2826,"prompt_tokens":761,"completion_tokens":2065,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":1986}},"tokens_in":505,"tokens_out":2065,"duration_ms":15026,"temperature":1.0,"reasoning_tokens":1986,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T11:31:52.026942+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take naturally missing cells from real spreadsheets, recover their true values from source documents, run the released system, and measure precision on the predictions it does not abstain from: if precision falls materially below the calibrated 0.9 threshold, the random-masking assumption fails. A cheaper probe is to re-mask the same 2,200 benchmark tables with a non-random missingness model — e.g., masking cells conditional on row or column features — and compare R@P=0.9.","supporting_citations":[],"review_version":1}