{"id":"c9e329f5-3d7e-4fcc-9e8d-56982478172d","arxiv_id":"2411.15805","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Uncertainty-guided active learning selects homes for appliance sensor installation, reducing labeled data needs in NILM while beating random selection on the Pecan Street dataset.","lead":"This paper tests whether active learning can pick which homes to install appliance monitors in, to train a home-energy disaggregation model with less labeled data. Uncertainty-guided selection beat random selection on a public dataset and came close to full-data accuracy with fewer monitors.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The active-learning advantage in the headline results is evaluated with a dynamic pool window that includes T+7 days of future mains data (Section 5.3.7); because the paper does not run a corrected lagged or previous-year evaluation, the claimed outperformance over random is not yet established.","rationale":"The paper's central claim is that uncertainty-based house selection beats random sampling. The experimental protocol computes acquisition scores using mains data from a seven-day future window; in the last iterations this window reaches into the test period. This leaks information that a real deployment would not have, and it biases the comparison in favor of active learning. The paper is transparent about the look-ahead but explicitly declines to run a corrected evaluation (Section 5.3.7). Because the entire empirical contribution rests on this comparison, the concern is load-bearing. Secondary issues (single split, no error bars for AL, inconsistent data-fraction claims) are real but less fundamental. The correct fix is a past-only or lagged dynamic pool; the paper already has the infrastructure to do this, so a revised evaluation would be straightforward. I agree with the reader's conditional verdict: the idea is plausible, but the headline numbers are not yet trustworthy.","tokens_in":15666,"tokens_out":6411,"duration_ms":54421,"concrete_test":"Re-run the query-singly and query-all-at-once experiments with the dynamic pool window restricted to past data only (e.g., T-14 to T, no future timestamps), and also separately with the pool window shifted back by 7 days, comparing RMSE curves to the random +/- sigma envelope in Figures 6 and 10-14. If entropy/MI curves do not stay below the random envelope in early iterations, the reported advantage is driven by look-ahead.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.3.7 defines a dynamic pool window spanning T-7 to T+7 when scoring pool-house uncertainty (Eq. 14, triangle kernel). At query time this future mains data is unavailable in any real deployment, and the paper concedes this in Section 5.3.7. Even worse, the final iterations use pool windows that overlap the test period: starting the first query on March 11 and stepping every five days, the last iteration has T approximately April 25, so the T+7 bound reaches May 2, inside the May 1-10 test window. Thus the acquisition function has access to test-period mains for pool houses, a direct protocol violation. The main figures (6-14) and the '2x reduction' claim all use this dynamic, triangle-weighted window. The static-window comparisons in Figures 7 and 8 do not repair this: the headline results include no past-only dynamic pool. If the AL advantage persists when the window is restricted to T-14 to T (or lagged), the claim is saved; if not, the central comparison to random is invalid.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an active learning framework for non-intrusive load monitoring (NILM), in which a model actively selects houses to instrument with appliance-level sensors. The authors build on Seq2Point networks with heteroskedastic outputs and MC-dropout uncertainty, and compare entropy and mutual-information acquisition functions against a random baseline and a full-data 'total' baseline. Experiments on Pecan Street Dataport cover a single-appliance ('query singly') setting and a multi-appliance ('query all at once') setting, using static and dynamic temporal pooling. The headline claims are that active learning matches full-data performance with roughly 30% of the data and reduces disaggregation error by up to 2x relative to random. The manuscript includes code and is framed as a benchmark.","tokens_in":15913,"tokens_out":6554,"duration_ms":57601,"significance":"If the empirical claims survive a causally correct evaluation, this would be a useful contribution to NILM and to active learning for time-series regression. The paper is clearly written, uses a public dataset, reports a random baseline over 10 seeds, and evaluates two standard acquisition functions. However, the main result currently relies on a dynamic pool window that uses future mains data at acquisition time, which is not available in any real deployment; until this is corrected, the central comparison is not trustworthy. The paper also lacks repeated runs of the AL methods themselves, so the claimed 'significant' improvement over random is not statistically quantified. These issues are fixable within the manuscript's scope, which is why I recommend major revision rather than rejection.","major_comments":[{"comment":"The dynamic pool window includes T+7 days of future mains data when scoring pool-house uncertainty. At query time, this future data is unavailable, and the paper itself concedes in §5.3.7 that 'we will never have access to future data for evaluating pool uncertainty.' More seriously, because the final acquisition iterations occur in late April, the T+7 window overlaps the May 1–10 test period, so the acquisition function uses test-period mains data for pool houses. All headline results (Figures 6–14 and the abstract claims of '2x reduction' and '~30% data') use this dynamic triangle-weighted window. The static-window comparisons in Figures 7–8 do not repair this, since they are not the basis of the headline claims. Please re-run the experiments with a causal window (e.g., T−14 to T, or previous-year data as suggested in §5.3.7) and report whether the advantage over random persists. Without this, the central empirical claim is not established.","section":"§5.3.7 and Eq. (14)"},{"comment":"The active-learning acquisition curves (entropy and MI) are presented as single runs, while only the random baseline is repeated across 10 seeds and shown as mean ± σ. The text states that the AL results are 'well below the sigma interval' of random, but without repeated AL runs or a statistical test, the claim that AL 'significantly outperforms' random is not fully supported. Please provide multiple seeds (or at least confidence intervals) for the AL strategies, or a paired significance test across the same random seeds.","section":"§5.4.1, Figures 6–14"},{"comment":"The query-all-at-once setting, which the paper argues is the realistic one, does not include a total baseline ('Due to resource constraints, we have not evaluated the total baseline error for query all at once AL'). Therefore the abstract's claim of achieving 'performance comparable to models trained on the entire dataset' is only demonstrated in the query-singly setting, which the paper itself shows is impractical because different appliances select different houses (Figure 9). Please either add the total baseline to the query-all-at-once experiments or restrict the comparable-to-full-data claim accordingly.","section":"§5.4.2"}],"minor_comments":[{"comment":"There is a small inconsistency in the start date of the first acquisition: §5.3.4 says 'starting from 11 March 2018', while §5.3.7 says 'the dynamic window starts from 5−20 March 2018'. Please clarify whether the first query occurs on 11 March and how the dynamic window is centered on that date.","section":"§5.3.4 vs §5.3.7"},{"comment":"Equation (6) approximates the differential entropy of the predictive mixture by log(σ_ensemble) plus a constant, but for a Gaussian mixture (Eq. (1)) the entropy is not exactly that of the single Gaussian with the aggregate variance; there is an additional term due to the spread of component means. Please state that this is an approximation and, if possible, assess its effect on acquisition rankings.","section":"§4.2.1, Eq. (6)"},{"comment":"The notation around Eqs. (1), (9), and (10) is confusing: the mixture components are written as N(x | μ_i, σ_i) but the variable being predicted is y, and x_s is used both as a sample and as the conditioning input. Please make the distinction between input and output variables consistent.","section":"§2.2.3 and §4.2.2"},{"comment":"Typos: 'Additionaly' should be 'Additionally' in Section 5.1; §5.3.6 contains 'i.e i.e.' with a missing comma; §5.3.8 says 'we use4 X NVidia A100 GPUs' (missing space). Please update these.","section":"§5.1"},{"comment":"The sentence 'For the purposes of a practical implementation, one could use the data from the previous year (if available) or introduce a lag in adding pool data (till the end of the pool window)' actually describes the correct causal protocol, but no such evaluation is reported. Highlighting the corrected protocol in the main results would strengthen the paper.","section":"§5.3.7"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be an ACM-style submission; the repository link is anonymized as 'anonn23/anon.git', which is fine for review but should be replaced in the final version. The central issue—the T+7 look-ahead in the dynamic pool—is a correctable experimental design flaw, not a fundamental problem with the proposed approach. I recommend major revision with a request to rerun the experiments under a causal acquisition protocol and to add error bars for the AL methods. If the advantage over random disappears under the corrected protocol, the paper's claims would need to be substantially revised."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is the first pool-based, regression-setting benchmark for active learning in NILM, and it ships code and a reproducible setup on Pecan Street Dataport. Second, the headline advantage over random is not yet established, because the dynamic pool window used for acquisition includes seven days of future mains data (T+7), and in later iterations that window overlaps the test period. The paper discloses the issue in Section 5.3.7 but never runs the corrected evaluation.\n\nWhat is genuinely good: the problem framing is practical — choose which homes to instrument, with mains data available for all pool homes. The distinction between query-singly (one house per appliance) and query-all-at-once (one house for all appliances) is useful, and the finding that MI beats entropy for sparse appliances is consistent with the epistemic-uncertainty story. The random baseline gets ten seeds and the authors are explicit about limitations. The static vs dynamic pool comparison, while incomplete, is a sensible first look at temporal context.\n\nThe soft spots, in order of severity. The look-ahead is load-bearing: Eq. 14 scores pool houses over T−7 to T+7, and since the first query is March 11 with five-day steps, the final iterations score using mains data from early May — that is the test window (May 1–10). So the acquisition function sees data it would not have at deployment time. The static-window results (Figs. 7, 8) cover only two appliances and are not the headline. A lagged or previous-year dynamic pool would settle this, but it is not run. Also, only the random baseline has error bars; the AL curves are single seeds. There is one fixed split, and the data-reduction claims are inconsistent (abstract says ~30%, later says 50% for AC, 40% for furnace/dishwasher, 25% in Table 1). Query-all-at-once never compares to the total baseline, so the 'comparable to full data' claim is demonstrated only in the impractical query-singly setting.\n\nThis paper is for the NILM/BuildSys community, and for anyone working on active learning for time series. It deserves a serious referee: the benchmark is reproducible, the question is timely, and the authors are honest. But the revision must replace the dynamic pool with a past-only window, add error bars to AL methods, and ideally run multiple splits. I would engage with it as a referee, and I would want to see the corrected numbers before believing the 2x claim.","headline":"Clear, reproducible first benchmark for pool-based active learning in NILM, but the headline advantage over random relies on a dynamic pool window that leaks future mains data into query selection.","tokens_in":16458,"tokens_out":3090,"would_cite":false,"duration_ms":26903,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Uncertainty-aware active learning can select the few households whose appliance data most improves NILM, matching full-data performance with roughly 30% of the labeled homes.","keywords":["active learning","non-intrusive load monitoring (NILM)","Bayesian neural networks","uncertainty quantification","sensor placement","energy disaggregation","Monte Carlo dropout","mutual information acquisition"],"falsifier":"Run the benchmark with uncertainty scores computed only from data available at the query time (e.g., past seven days, or a one-week lag) and compare against random sampling; if the gap disappears or reverses, the reported gains depend on look-ahead.","tokens_in":15460,"feed_emoji":"🎯","tokens_out":7539,"duration_ms":60958,"temperature":0.7,"pith_summary":"The paper asks whether appliance-level training data for non-intrusive load monitoring (NILM) can be collected more cheaply by installing sensors only in the most informative households. It claims that uncertainty-based active learning selects those households well: models trained on data from roughly 30% of homes reach accuracy comparable to models trained on every home, and for a fixed sensor budget the disaggregation error is up to twice as low as with random house selection. The authors benchmark this on five common appliances and conclude that querying houses by predictive uncertainty, especially mutual information for sparsely used appliances, is a practical way to cut labeling cost. They also argue that a dynamic temporal window, which scores houses based on recent usage patterns, matters most for appliances with seasonal behavior like air conditioners.","feed_headline":"Smart sensor placement matches full-data NILM with 30% of homes","feed_subtitle":"Uncertainty-based querying cuts disaggregation error up to 2x versus random sampling for a fixed sensor budget.","key_machinery":"The mechanism is an active-learning loop built on Monte Carlo dropout uncertainty estimates. The disaggregation model is a sequence-to-point neural network with heteroskedastic Gaussian outputs, so each forward pass yields a mean and variance; running F stochastic passes gives a Gaussian mixture whose entropy or mutual information becomes the acquisition score. Candidate houses are ranked by these scores, and the highest-ranked house is chosen for sensor installation. The temporal context is handled by aggregating acquisition scores over a sliding window, using a static or dynamic window with uniform or triangle weighting; the dynamic window with triangle weighting is used for the final experiments.","core_discovery":"The paper's central claim is that uncertainty-based active learning can identify the subset of households whose appliance-level data is most valuable for training NILM models. On a public residential energy dataset with five appliances, the authors report that entropy and mutual-information acquisition functions both outperform random sampling; mutual information is particularly effective for appliances that are used rarely and irregularly, such as dishwashers and clotheswashers. They report that roughly 30% of the labeled data suffices to match the accuracy of a model trained on the full pool, and that for a fixed number of installed sensors the strategy reduces root mean squared error by up to a factor of two in the early iterations of active learning.","pith_inferences":["The reported advantage is measured under a look-ahead: each house's uncertainty score uses seven days of future mains data, which is unavailable at query time in practice. A corrected protocol that uses only past data (or previous-year data as a proxy) is the natural next test, and could shrink the gap.","The same uncertainty-guided sensor placement logic could transfer to other building-level metering tasks—water, gas, or room-level electricity—where a cheap aggregate signal is available but per-device instrumentation is expensive.","The query-singly results show near-zero overlap in selected houses across appliances in early iterations, implying that a single fixed house rarely dominates uncertainty for all appliances; a batch-selection method that accounts for shared information across houses could recover some of the per-appliance gains in the practical query-all-at-once setting.","A stopping criterion based on uncertainty (when the acquisition score falls below a threshold) could further reduce sensor count; the paper lists this as future work, but the reported error curves suggest diminishing returns after the first few queries."],"forward_implications":["If the claim holds, energy utilities can install appliance monitors in only about 30% of homes and still train disaggregation models that match full-data performance, cutting sensor hardware and installation cost substantially.","For sparsely used appliances, acquisition functions based on mutual information (epistemic uncertainty) are more cost-efficient than entropy-based ones, because they avoid spending budget on houses whose data noise is high.","For seasonally varying appliances, scoring houses over a dynamic recent window (as opposed to a fixed past window) yields better models, because it tracks the appliance's current usage regime.","Jointly querying a single house for all appliances ('query all at once') is a practical approximation: it requires more sensors than the per-appliance ideal, but still considerably fewer than random selection, e.g., four sensors instead of eight for air conditioners."],"supporting_citations":[{"why":"Supplies the sequence-to-point neural network architecture used as the base disaggregation model.","marker":"[38]"},{"why":"Provides Monte Carlo dropout, the technique used to estimate predictive uncertainty for acquisition scoring.","marker":"[7]"},{"why":"Supplies the public residential energy dataset used for all benchmarks.","marker":"[26]"},{"why":"Establishes uncertainty quantification for neural network NILM and informs the heteroskedastic output design.","marker":"[2]"},{"why":"Defines the train/test/pool house split from prior NILM work that the experiments reuse.","marker":"[31]"},{"why":"Provides the general active learning framework and acquisition function definitions (entropy, mutual information).","marker":"[30]"},{"why":"The prior active-learning-for-NILM work in a classification setting; the paper benchmarks against it and claims to be the first regression setting.","marker":"[34]"},{"why":"Supplies the approximation for mutual information in deep Bayesian active learning, used here to estimate the acquisition score.","marker":"[19]"}],"fun_headline_variants":["Active learning picks best homes for NILM data, matches full set with 30%","Uncertainty-guided sensor placement halves NILM error with 30% data","Active learning matches full-data NILM using 30% of homes","Smart sensor choice cuts NILM data needs to 30% without accuracy loss","Uncertainty-based data selection rivals full NILM training with third of data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The active learning scores each candidate house using a seven-day window of future mains data; if that future data is not available at query time, the advantage over random may shrink or vanish.","fun_headline_variants_meta":{"raw":{"variants":["Active learning picks best homes for NILM data, matches full set with 30%","Uncertainty-guided sensor placement halves NILM error with 30% data","Active learning matches full-data NILM using 30% of homes","Smart sensor choice cuts NILM data needs to 30% without accuracy loss","Uncertainty-based data selection rivals full NILM training with third of data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000661,"raw_usage":{"total_tokens":2994,"prompt_tokens":889,"completion_tokens":2105,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":1999}},"tokens_in":505,"tokens_out":2105,"duration_ms":14039,"temperature":1.0,"reasoning_tokens":1999,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:54:27.160854+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the benchmark with uncertainty scores computed only from data available at the query time (e.g., past seven days, or a one-week lag) and compare against random sampling; if the gap disappears or reverses, the reported gains depend on look-ahead.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the sequence-to-point neural network architecture used as the base disaggregation model."},{"cited_title":"I do not know","cited_arxiv_id":null,"evidence_quote":"Establishes uncertainty quantification for neural network NILM and informs the heteroskedastic output design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The prior active-learning-for-NILM work in a classification setting; the paper benchmarks against it and claims to be the first regression setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the approximation for mutual information in deep Bayesian active learning, used here to estimate the acquisition score."}],"review_version":1}