{"id":"f5578c05-471b-47a7-811b-4d94a8c0049b","arxiv_id":"2508.12905","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"TCUQ turns short-horizon output instability into an on-device uncertainty score with a streaming quantile threshold, claiming calibrated abstention for TinyML without online labels.","lead":"This paper proposes TCUQ, a small on-device monitor that flags uncertain predictions on microcontrollers by checking how consistent a model's recent outputs are, then abstains when a streaming threshold is exceeded. It is worth reading for anyone deploying neural nets on battery-powered devices with scarce memory and labels, though the calibration claims and reported numbers need scrutiny.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The streaming conformal layer is asserted to calibrate abstention, but the only reported calibration metric—Eq. 11—is tautological, and no risk-on-accepted-predictions test is given.","rationale":"The reader's weakest_assumption identifies the same gap: the streaming conformal layer is not shown to control accepted-set error. That is the load-bearing concern because the abstract and Section 3 present calibrated abstention as a central contribution, and the reported detection results are scored on the raw uncertainty signal, not on the accept/abstain policy. A temporal score with strong AUPRC is plausible and useful, but it is not calibrated abstention. Nothing in the appendices closes the gap: Appendix B.5 is about temporal-assistance exits, and the quantile tracker and rate controller are left unspecified. The internal numerical inconsistencies (latency and severity AUPRC discrepancies) are secondary; the primary issue is the missing calibration guarantee. I therefore see no reason to change the reader's REJECT verdict, and the proposed accepted-risk test would settle whether a revised version can be accepted.","tokens_in":26849,"tokens_out":2867,"duration_ms":32088,"concrete_test":"Withheld-label end-to-end check: on a held-out ID+CID stream (e.g., CIFAR-10-C), run TCUQ online with defaults lambda=0.7, alpha=0.1, b=0.15, recording r_t, q_alpha,t, abstain decisions, and ground-truth correctness. Compute (i) empirical exceedance Delta_alpha, (ii) error rate among accepted predictions, and (iii) coverage. If accepted error is not <= alpha (within sampling error) while Delta_alpha is near zero, the calibration claim fails. Repeat at severity 5 and with W in {16,24,32}; also compare against an oracle batch-quantile threshold to separate tracker error from a failure of the decision rule itself.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that TCUQ's streaming conformal layer turns the temporal score r_t into a 'calibrated accept/abstain rule' (abstract and Section 3). For this claim to hold, abstaining when r_t >= q_alpha,t should control the error rate of accepted predictions near alpha, not merely make the abstention rate equal alpha. The paper's streaming calibration metric, Eq. 11, measures the exceedance deviation Delta_alpha = |mean(1[r_t >= q_alpha,t]) - alpha|. If q_alpha,t is an online estimate of the (1-alpha) quantile of {r_1,...,r_t}, this quantity is approximately zero by construction; it says nothing about whether the accepted subset is correct. The paper never specifies the 'memory-constant quantile tracker' mentioned in Section 3, never reports accepted-set risk/coverage on CID streams, and defers the budget controller to Appendix B.5, which actually analyzes temporal-assistance exit counts, not a rate controller. Thus the paper's claimed label-free calibrated abstention is not established. The temporal score may rank errors usefully, and the AUPRC results in Tables 1 and 2 are consistent with that weaker claim, but the calibration layer's contribution is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TCUQ, an on-device uncertainty monitor for streaming TinyML. It computes four temporal-consistency signals from a short ring buffer (multi-lag predictive divergence, feature stability, decision persistence, and a confidence/margin proxy), combines them with a logistic model fitted offline on a labeled development set, forms a scalar nonconformity score r_t, and maintains an online quantile q_{α,t} to decide between prediction and abstention. The authors claim that this yields a calibrated, budgeted accept/abstain rule without online labels, and they report MCU size/latency comparisons, CID accuracy-drop detection AUPRC, failure-detection AUROC, and proper-score/ECE comparisons on MNIST, SpeechCommands, CIFAR-10, and TinyImageNet.","tokens_in":27115,"tokens_out":8826,"duration_ms":84716,"significance":"If the calibration claim were valid, TCUQ would be a meaningful TinyML contribution: a single-pass, kilobyte-scale, label-free abstention mechanism with O(W) memory and O(1) per-step updates is genuinely useful, and the temporal-consistency score shows promising empirical performance as a ranking signal (best CID AUPRC in Tables 1 and 4, competitive AUROC in Table 2). However, the central novelty—streaming conformal calibration—is not established. The only streaming calibration metric is tautological, the quantile tracker and budget controller are unspecified, no accepted-set risk or coverage is reported, and the resource tables contain internal contradictions. The empirical detection results may survive a reframing as a lightweight error-ranking score, but the current claims about calibrated, budgeted abstention are unsupported.","major_comments":[{"comment":"The only streaming calibration metric reported, Δ_α = |(1/T)Σ 1[r_t ≥ q_{α,t}] − α|, measures the rate at which the label-free nonconformity score r_t exceeds its own maintained quantile. For any reasonable quantile estimator this rate is approximately α by construction, so Eq. (11) is a consistency check on the estimator, not an independent test of calibrated abstention. Calibrated selective classification requires controlling the error rate among accepted predictions (equivalently, coverage of the accepted set), and the paper reports no such quantity; Section 4 says risk–coverage curves are plotted, but no such figure appears in the results. The abstract's claim that the streaming conformal layer yields 'calibrated behavior' is therefore unsupported.","section":"Section 3; Eq. (11)"},{"comment":"The 'simple rate controller' said to enforce the abstention budget b is never specified, and the cross-reference to Appendix B.5 is incorrect: Appendix B.5 analyzes the number K of temporal-assistance exits and does not discuss an abstention-rate controller. Without the controller's rule, the interaction between quantile-based abstention and budget enforcement cannot be analyzed, and the claimed budgeted abstention is not reproducible.","section":"Section 3; Appendix B.5"},{"comment":"The paper relies on a 'memory-constant quantile tracker' to maintain q_{α,t}, but no algorithm, state size, update rule, or convergence property is given; the only description elsewhere is 'a lightweight stochastic estimator' in Section 4. Because this tracker is the entire basis of the streaming conformal layer, the central mechanism cannot be implemented or verified from the manuscript.","section":"Section 3"},{"comment":"The resource-savings numbers are internally inconsistent. The Section 5 summary claims flash reductions of 50%/52% versus EE-ens and 38%/62% versus DEEP on Big-MCU, and the abstract says 'typically about 50 to 60% smaller', but Table 11 shows reductions of 33% (300 vs 450 KB) and 34% (356 vs 541 KB) versus EE-ens, and 28% (300 vs 414 KB) and 38% (356 vs 578 KB) versus DEEP. Section 5.1's latency claims (27% and 22%) also disagree with Table 11 (43%/31% and 29%/38%). These discrepancies affect the headline deployment claim.","section":"Abstract; Section 5; Table 11"},{"comment":"The Small-MCU latency measurements are mutually contradictory: Table 12 reports SpeechCommands latency for TCUQ of 169 ms (with EE-ens at 352 ms and DEEP at 296 ms), while Table 13 reports 48.0 ± 0.7 ms for TCUQ on Small-MCU SpeechCommands (EE-ens 98 ms, DEEP 86 ms). The relative speedups are similar, but the absolute numbers differ by about a factor of 3.5, so at least one table cannot be correct. The energy/latency protocol in Appendix D.2 must be reconciled with the main MCU results.","section":"Table 12; Table 13"}],"minor_comments":[{"comment":"The symbol α is used both for the confidence-margin blend in Eq. (2) and for the target quantile level in the streaming calibration setup; rename one to avoid ambiguity.","section":"Equation (2); Section 4"},{"comment":"Appendix Table 4 reports TCUQ MNIST-C AUPRC of 0.62 and SpeechCmd-C AUPRC of 0.62, while Table 1 reports 0.66 and 0.63 for the same settings; the discrepancy should be explained.","section":"Table 1; Appendix Table 4"},{"comment":"The text says latency measurements are averaged over 1,000 inferences and the standard deviation is reported, but Tables 11 and 12 contain no standard deviations; either add the intervals or adjust the text.","section":"Section 4; Tables 11–12"},{"comment":"The reproducibility statement says measurement scripts and board configurations are provided as an artifact bundle, but no link or repository is given.","section":"Appendix D.2"}],"recommendation":"reject","confidential_remarks":"The manuscript's headline claim—label-free calibrated abstention via streaming conformal calibration—is not supported by the reported evidence, and the numerical inconsistencies in the MCU tables suggest the experimental section needs a full recheck. The temporal-consistency detection results are promising, but the current framing and evidence do not meet the bar for publication in this journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"TCUQ is a plausible TinyML uncertainty monitor with a genuinely useful temporal-consistency feature set, but the headline streaming conformal calibration claim does not survive contact with the paper's own equations.\n\nWhat is actually new: combining four lightweight temporal signals — multi-lag JSD divergence, cosine feature stability, label persistence, and a confidence-margin proxy — into a single logistic score maintained with an O(W) ring buffer and O(1) updates. That specific combination is new for TinyML, and the ablations (Tables 6 and 7) suggest the temporal signals carry real signal. The MCU footprint and latency results (Tables 11–12) also look plausible: single-pass inference, small state, and roughly 50% smaller binaries than ensemble baselines. Those are useful engineering contributions worth acknowledging.\n\nWhere it falls down: the paper claims the streaming quantile gives 'calibrated' abstention without online labels. The only calibration metric offered is Eq. 11, the exceedance deviation between r_t and its own online quantile. That quantity is approximately alpha by construction; it says nothing about whether the accepted predictions are correct. The paper never reports selective risk or accepted-set error rates, and the 'memory-constant quantile tracker' from Section 3 is never specified. The budget controller is also missing: Section 3 points to Appendix B.5, but that appendix is about the number of temporal-assistance exits, not a rate controller. On top of that, the numbers have internal contradictions: Big-MCU latency reductions differ between Section 5.1 and Table 11 (27% vs. 43%), CIFAR-10-C severity-5 AUPRC is 0.80 in Section 5.2 but 0.77 in Tables 4 and 6, and Appendix D.2 reports Big-MCU latencies roughly seven times smaller than Table 11 for the same setup. These do not necessarily invalidate the temporal feature, but they make it hard to trust the headline empirical claims.\n\nWho this is for: TinyML practitioners interested in lightweight uncertainty signals, and UQ researchers who want a cautionary example of conflating quantile exceedance with risk control. A serious referee could get the authors to fix the framing — drop the word 'conformal' or define precisely what is being calibrated, specify the quantile tracker and the budget controller, report accepted-set risk, and reconcile the tables. As is, the central calibration claim is the centerpiece and it is unsupported. I would not desk-reject it, but I would not accept it in this form.","headline":"The temporal-consistency uncertainty signals and MCU measurements are useful, but the paper's central 'streaming conformal calibration' claim is not established by the evidence it provides.","tokens_in":27688,"tokens_out":3478,"would_cite":false,"duration_ms":34820,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A TinyML monitor turns short-horizon temporal instability into calibrated abstention without labels or extra forward passes.","keywords":["temporal consistency","uncertainty quantification","TinyML","streaming conformal calibration","abstention","label-free monitoring","accuracy-drop detection"],"falsifier":"On a labeled stream that interleaves ID, CID, and OOD segments, record the empirical error rate among predictions the rule accepts; if that rate exceeds $\\alpha$ (say, 0.1) on any segment, or if the rejection rate drifts away from $\\alpha$ because the quantile tracker does not follow the distribution, the streaming conformal calibration claim fails.","tokens_in":26598,"feed_emoji":"📉","tokens_out":7813,"duration_ms":70782,"temperature":0.7,"pith_summary":"TCUQ proposes that a lightweight device can monitor its own reliability in a streaming setting by watching how its predictions and features wobble over a short window. Four cheap signals—multi-lag predictive divergence, feature instability, label-flip rate, and a confidence-margin proxy—are combined into a single uncertainty score with $O(W)$ ring-buffer state and $O(1)$ per-step updates, and a streaming conformal quantile converts that score into a budgeted accept/abstain rule. The paper argues this yields calibrated behavior without online labels or extra forward passes, fits in kilobyte-scale memory on microcontrollers, and detects accuracy drops under corrupted streams earlier and more reliably than early-exit and deep-ensemble baselines (up to 0.86 AUPRC at high severity). If right, it gives TinyML deployments a practical way to refuse predictions when conditions degrade, without the memory and latency cost of ensembles or multi-pass methods.","feed_headline":"TinyML gets calibrated abstention without labels or extra passes","feed_subtitle":"A single-pass, label-free monitor converts short-window instability into a calibrated abstain rule, reaching 0.86 AUPRC on corrupted…","key_machinery":"The load-bearing object is the streaming conformal layer combined with the temporal-consistency signal. The signal vector $\\mathbf{s}_t = [D_t, (1-S_t), (1-c_t), m_t]^\\top$ aggregates multi-lag posterior divergence, feature instability, decision inconsistency, and a near-tie-sensitive confidence proxy; the logistic combiner $\\sigma(\\mathbf{w}^\\top \\mathbf{s}_t + b)$ trained once offline maps it to an uncertainty score $U_t$. The streaming conformal layer maintains a memory-constant estimate of the $(1-\\alpha)$ quantile $q_{\\alpha,t}$ of the nonconformity scores $r_t = \\lambda U_t + (1-\\lambda)(1-C_\\phi(x_t))$ and abstains when $r_t \\ge q_{\\alpha,t}$; this is what converts the score into a calibrated, budgeted accept/abstain rule without online labels or extra forward passes. The ring buffer of size $W$ supplies the history for the lagged signals, keeping all per-step updates $O(1)$ and total state $O(W(d+L))$.","core_discovery":"The paper's central claim is that short-horizon temporal consistency is a sufficient label-free signal for calibrated uncertainty monitoring on microcontrollers. Concretely, TCUQ computes, from a frozen backbone's posteriors and features over a small ring buffer, four signals—multi-lag Jensen–Shannon divergence, cosine feature instability, decision-persistence violation, and a blended inverse-confidence/inverse-margin proxy—and merges them through a logistic combiner fitted once offline into a scalar $U_t$. The nonconformity score $r_t = \\lambda U_t + (1-\\lambda)(1-C_\\phi(x_t))$ is fed to a streaming conformal layer that maintains an online estimate of the $(1-\\alpha)$ quantile; the device abstains when $r_t \\ge q_{\\alpha,t}$, subject to a rate controller for the abstention budget $b$. The paper reports that this single-pass monitor fits in kilobytes, reduces flash and latency by roughly 50–60% and 30–45% versus early-exit ensembles and deep ensembles while keeping accuracy parity, and improves corrupted-stream accuracy-drop detection by 3–7 AUPRC points, reaching 0.66 on MNIST-C and up to 0.86 at high severity on TinyImageNet-C, with failure detection up to 0.92 AUROC.","pith_inferences":["The calibration guarantee is only as strong as the assumption that the online quantile estimator actually tracks the $(1-\\alpha)$ quantile under stream shift; the paper does not specify the memory-constant tracker, so a stress test on a drifting stream would settle whether the exceedance rate matches $\\alpha$ over time.","The measured $\\Delta_\\alpha$ (Eq. 11) is the deviation of the rejection rate from $\\alpha$, which follows from the quantile definition; it is not a measure of error control among accepted predictions. A fair reader should infer that the paper establishes quantile exceedance calibration, not selective-risk control, unless the rate controller also bounds accepted-prediction error.","The paper defers the rate controller that reconciles $\\alpha$ with the abstention budget $b$ to Appendix B.5, which actually discusses the number of temporal-assistance exits, not the controller; the budgeted-abstention guarantee is therefore not fully demonstrated in the main text.","A testable extension: use TCUQ on a long, labeled deployment stream and compute the empirical error rate among accepted predictions; if it exceeds $\\alpha$ when the stream drifts, the conformal layer needs explicit risk-controlling updates rather than a drifting threshold."],"forward_implications":["If TCUQ's claims hold, a TinyML device can run a calibrated abstention policy in a single forward pass, needing only a few kilobytes of extra state, where early-exit ensembles and deep ensembles are out of memory.","Corrupted-stream accuracy drops would be detectable 25–35% sooner than with the compared baselines, with AUPRC gains of 3–7 points and up to 0.86 at high corruption severity.","The monitor would remain label-free online: calibration comes from the streaming quantile, so no ground-truth labels are needed after deployment.","The same recipe extends to audio keyword spotting and to larger backbones (ResNet-50 experiments) without architectural changes, and the authors argue it transfers to transformer-based backbones.","Budgeted abstention becomes practical: a user-set abstention budget $b$ can be enforced while still responding to bursts of high uncertainty."],"supporting_citations":[{"why":"Supplies the CIFAR-10-C and TinyImageNet-C corrupted benchmarks that define the accuracy-drop detection task.","marker":"Hendrycks & Dietterich (2019)"},{"why":"Supplies the MNIST-C benchmark used for corrupted in-distribution accuracy-drop detection.","marker":"Mu & Gilmer (2019)"},{"why":"Documents overconfidence under shift and provides the calibration metrics (ECE, temperature scaling) used to frame the problem and evaluate.","marker":"Guo et al. (2017)"},{"why":"Establishes that neural network uncertainty degrades under dataset shift, motivating a label-free streaming monitor.","marker":"Ovadia et al. (2019)"},{"why":"Provides the conformal prediction and distribution-free risk control background that the streaming conformal layer builds on.","marker":"Angelopoulos & Bates (2021)"},{"why":"Adaptive conformal inference under distribution shift, the reference for the paper's streaming quantile approach.","marker":"Gibbs & Candès (2021)"},{"why":"Deep ensembles baseline whose memory and latency costs TCUQ is designed to avoid.","marker":"Lakshminarayanan et al. (2017)"},{"why":"Early-exit ensembles baseline, the main cost/quality comparator in the microcontroller experiments.","marker":"Qendro et al. (2021)"},{"why":"Supplies the failure detection evaluation protocol (correct vs. incorrect within ID and ID vs. OOD) adopted in Section 5.3.","marker":"Xia & Bouganis (2023)"}],"fun_headline_variants":["Single-pass label-free abstention for TinyML","TCUQ: calibrated abstention from temporal consistency","Calibrated abstention without labels or extra passes","Kilobyte-scale calibrated abstention for streaming TinyML"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central premise is that an online, memory-constant estimate of the $(1-\\alpha)$ quantile of the label-free scores yields a calibrated accept/abstain rule—but the paper never specifies the tracker, and the measured exceedance deviation $\\Delta_\\alpha$ matches $\\alpha$ by the quantile's definition rather than by demonstrating control of the error rate among accepted predictions.","fun_headline_variants_meta":{"raw":{"variants":["Single-pass label-free abstention for TinyML","TCUQ: calibrated abstention from temporal consistency","Calibrated abstention without labels or extra passes","Kilobyte-scale calibrated abstention for streaming TinyML"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000915,"raw_usage":{"total_tokens":3972,"prompt_tokens":1029,"completion_tokens":2943,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":645,"completion_tokens_details":{"reasoning_tokens":2880}},"tokens_in":645,"tokens_out":2943,"duration_ms":22849,"temperature":1.0,"reasoning_tokens":2880,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:18:48.955790+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a labeled stream that interleaves ID, CID, and OOD segments, record the empirical error rate among predictions the rule accepts; if that rate exceeds $\\alpha$ (say, 0.1) on any segment, or if the rejection rate drifts away from $\\alpha$ because the quantile tracker does not follow the distribution, the streaming conformal calibration claim fails.","supporting_citations":[],"review_version":2}