{"id":"2146dddd-1116-4d62-8cd0-df80e654bd21","arxiv_id":"2505.06330","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Prompt-only LLMs give basic but transferable NILM state detection (F1 0.44-0.61, below deep learning's 0.73-0.88) with readable explanations and zero retraining.","lead":"This paper asked whether large language models can tell which home appliances are running by reading only the house's total electricity use, with no appliance data used for training. The models fall short of deep learning on accuracy, but they transfer to unseen homes and produce plain-language reasons for each on/off call.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-region generalization claim is not actually tested: UK-DALE knowledge and test both come from house 2, so RQ2's 'new region' evidence is an in-house temporal split.","rationale":"The reader's CONDITIONAL verdict already flags the same-house UK-DALE issue and the lack of variance reporting. My analysis sharpens this into the single most load-bearing flaw in the central claim. The paper's distinct contribution is the assertion that prompt-only LLMs generalize across houses and regions; if the cross-region part is untested, the contribution reduces to 'cross-house transfer within one region,' which is a materially weaker claim. This does not undermine RQ1 (accuracy lag) or RQ3 (qualitative explainability), and the REDD cross-house result remains genuinely informative. The issue is experimental design and interpretation rather than internal inconsistency of the framework, so the appropriate response is to require an unseen-house UK-DALE test before accepting RQ2's broad wording. That is exactly what a CONDITIONAL verdict expresses, so no change to the reader's verdict is needed.","tokens_in":14670,"tokens_out":4124,"duration_ms":39413,"concrete_test":"Evaluate the optimized prompt on an unseen UK-DALE house (e.g., house 5) with knowledge extracted exclusively from other UK-DALE houses, and separately with knowledge extracted from REDD houses 2/3 to test true cross-region transfer without any target-house data. Compare the resulting F1 to the reported 0.6120 on the house 2 split. If F1 remains comparable, the cross-region generalization claim survives; if it drops substantially or the model fails to detect the kettle/dishwasher states, RQ2 should be narrowed to within-region cross-house transfer plus within-house temporal generalization. Run at least three independent API calls per condition and report mean and standard deviation, since the current single-run numbers cannot separate prompt design from sampling noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central value proposition is zero-shot generalization: RQ2 claims 'robust generalization ... across diverse and previously unseen environments,' and the abstract highlights 'strong generalization across different houses and even regions.' The REDD half supports the cross-house half: knowledge is built from houses 2/3 and evaluated on unseen house 1. The cross-region half does not. Section 6 states that for UK-DALE, 'we use data from house 2, splitting it into training (70%), validation (10%), and testing (20%)'; Section 3.1 says appliance knowledge is 'extracted from historical data using statistical methods,' i.e., from that same house 2 data. The UK-DALE test set is therefore a later time slice of the same household, with the same appliances and occupants, not an unseen house in a new region. The text in Section 6.2 that the framework adapts 'solely by modifying the prompt's knowledge injection' describes adaptation using target-house data; this is not zero-shot transfer to a new region. Consequently, the headline generalization claim rests on one dataset/house combination for cross-region transfer, and 'generalization across regions' is confounded with within-house temporal generalization. The weakest point of the central claim is not merely the representativeness of knowledge features; it is the absence of any experimental condition in which an unseen UK household is tested.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LLM4NILM, a prompt-based framework that uses general-purpose large language models (LLMs) for non-intrusive load monitoring (NILM) without any model training. The framework injects appliance knowledge (power range, standby power, duration, usage pattern), one-shot examples, timestamps, and contextual information from previous windows into a prompt. The authors evaluate state-detection performance on REDD and UK-DALE, compare against Seq2Seq and Seq2Point baselines, and report that LLMs lag specialized DL models in F1 score but claim strong zero-shot generalization across houses and regions, as well as human-readable explainability. The paper also includes ablations of prompt components, knowledge-injection factors, window size, context length, and model capacity, concluding that LLMs are not ready to replace DL methods but may complement them in adaptable, transparent systems.","tokens_in":14882,"tokens_out":2450,"duration_ms":26060,"significance":"If the claims hold, this is a useful capability-boundary study: it is among the first systematic evaluations of prompt-only LLMs for NILM, and it honestly documents a clear accuracy gap while exploring generalization and explainability. The ablations are informative, particularly the finding that power range is the most influential knowledge component and that context length has a non-monotonic effect. The headline negative result (LLMs lag specialized models) is consistently supported by Table 3 across both datasets. The main advertised positive claim, however, is cross-region generalization, and that claim is currently under-supported by the experimental design.","major_comments":[{"comment":"The cross-region generalization claim (RQ2, abstract, Section 6.2) is not actually tested. Section 6 states that for UK-DALE, \"we use data from house 2, splitting it into training (70%), validation (10%), and testing (20%),\" while Section 3.1 says appliance knowledge is \"extracted from historical data using statistical methods.\" Thus the UK-DALE test set is a later time slice of the same household used to derive the prompt knowledge, not an unseen house in a new region. The REDD experiment (knowledge from houses 2/3, test on unseen house 1) supports cross-house generalization, but no experiment evaluates an unseen UK household. The claim of \"generalization ... across different houses and even regions\" therefore rests on a single in-house temporal split and should be revised, or the experiment should be rerun on a held-out UK-DALE house.","section":"Section 6, Section 6.2, Section 3.1"},{"comment":"The LLM results are stochastic API outputs, but each reported F1 score appears to come from a single run with no variance, repetition, temperature setting, or seed information. Given the large gaps between models (e.g., GPT-4.1-mini vs. DeepSeek-V3-0324 on REDD), single-run point estimates are insufficient to establish stable model rankings or reliable performance levels. The paper should report mean and standard deviation over multiple calls, or at least specify the sampling parameters and justify that the reported values are representative.","section":"Section 6, Table 3"},{"comment":"It is unclear whether the UK-DALE appliance knowledge (standby power, power range, duration, usage pattern) is computed from the training split only or from the full house-2 dataset. If the knowledge statistics are computed before the temporal split, information from the test period could leak into the prompts. The manuscript should state explicitly which portion of house 2 is used for knowledge extraction and, if necessary, recompute the statistics on the training split.","section":"Section 6, Section 3.1"},{"comment":"The explainability evaluation is purely qualitative. The paper states that explanations are \"verifiable\" and compares them against ground truth and the aggregate signal, but no systematic protocol, metric, or inter-annotator assessment is provided. As explainability is one of the three central research questions, the RQ3 claim would be stronger with a defined evaluation procedure, even a simple correctness-rate measure over a sample of explanations.","section":"Section 6.3, Appendix B"}],"minor_comments":[{"comment":"The caption states that the dishwasher is omitted from the figure, yet the overall F1 score includes it; this makes the overall trend hard to interpret because the composition of the average changes with appliance coverage. A note explaining how the overall score is computed in the figure would help.","section":"Figure 6a"},{"comment":"The rows of Table 2 are not labeled, so the reader must infer which knowledge combinations correspond to each row. Adding row labels or checkmarks for Power, Duration, and Pattern would improve readability.","section":"Table 2"},{"comment":"The Output Normalizer uses forward padding or truncation for misaligned outputs. Since the LLM is penalized for length errors, the choice of padding direction could affect appliance-state boundaries; a brief discussion of this effect would be useful.","section":"Section 4.3"},{"comment":"The prompt template is helpful but omits the concrete appliance knowledge values used for REDD and UK-DALE (e.g., the actual power ranges and durations). Providing the exact knowledge entries in an appendix or supplementary file would make the study reproducible.","section":"Appendix A"},{"comment":"The phrase \"unseen REDD test set\" in the caption of Figure 4 is slightly misleading because the knowledge is derived from houses 2 and 3, so the house is unseen but the appliance types and their statistical features are known. The wording could be clarified.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about the accuracy gap and the ablation study is solid. The main issue is that the flagship generalization claim is over-stated relative to the experimental design; fixing this with an unseen UK-DALE house or a carefully hedged claim is feasible within the manuscript's scope. I would also encourage the editor to ask for repeated API runs or variance reporting, as single-run stochastic evaluations are a recurring concern in this line of work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about arXiv:2505.06330 is that it is a useful, honest benchmark of prompt-only LLMs for NILM state detection, but its headline generalization claim overreaches the experiments. The authors themselves show LLMs are clearly below specialized deep learning models (F1 0.44–0.61 vs 0.73–0.88 on REDD/UK-DALE). That negative result is credible and worth having.\n\nWhat is genuinely new: this is the first systematic study I know of that applies LLMs via prompts alone to NILM, and the prompt design is thorough. The ablations (Table 1, Table 2) are well structured; the finding that power range is the single most informative knowledge feature is a concrete, useful takeaway. The REDD cross-house test (knowledge from houses 2/3, test on house 1) is a legitimate zero-retraining transfer experiment.\n\nThe soft spots are in the stronger claims. First and most important: 'cross-region generalization' is not actually tested. For UK-DALE they use house 2, split into train/val/test, and the prompt knowledge is extracted from that same house's training portion. The test set is a later time slice of the same household, not an unseen UK house. So the 'even regions' headline rests on a within-house temporal split, which is far weaker than what is claimed. Second, the LLM API outputs are scored from single runs with no variance or repetition. Given that LLM responses are stochastic, this makes the headline numbers less stable than they appear. Third, the 'zero-data' framing is overstated: the prompts are populated with statistics estimated from training-house data, so the approach is data-dependent even if no gradients are updated. Fourth, explainability is only qualitatively inspected; there is no systematic check of whether the generated rationales actually match ground truth. Minor: no code or API configuration is released, so reproducibility is limited.\n\nOn balance, the paper's central claim—that prompt-only LLMs offer basic NILM capability with a large accuracy gap to DL—holds up. The generalization and explainability advantages are plausible but not established at the level claimed. I would send it to peer review, because the first systematic study of this question deserves referee time, and the fix is straightforward: add a true cross-house UK test (e.g., train knowledge on house 2, test on house 5), report variance over multiple runs, and soften the 'zero-data' language. With those changes it would be a solid contribution.\n\nFor your reading group: worth discussing, because the paper defines a capability boundary for LLM-based disaggregation and sets up the next round of experiments.","headline":"Prompt-only LLMs for NILM are honestly benchmarked and fall short of DL baselines, but the cross-region generalization claim is not supported by the experiments as designed.","tokens_in":15505,"tokens_out":3203,"would_cite":true,"duration_ms":29345,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompt-only LLMs can perform basic non-intrusive load monitoring without training, but their accuracy trails specialized deep-learning models; their strengths are cross-house transfer and readable explanations.","keywords":["non-intrusive load monitoring","energy disaggregation","large language models","prompt engineering","in-context learning","zero-shot transfer","explainability","appliance state detection"],"falsifier":"Run the same framework with knowledge extracted from REDD houses 2–3 but test on an unseen UK-DALE house (house 1 or 5) rather than house 2; if overall F1 collapses toward chance, the cross-region generalization claim fails. As a second check, replace the injected power ranges with deliberately wrong values and confirm the F1 falls back to the no-knowledge level of about 0.30, which would show the prompt, not some latent model knowledge, is carrying the capability.","tokens_in":14411,"feed_emoji":"⚡","tokens_out":9537,"duration_ms":87353,"temperature":0.7,"pith_summary":"This paper asks whether a general-purpose large language model, fed only a text prompt and given no training on power data, can do non-intrusive load monitoring: inferring which appliances are on from a home's aggregate electricity draw. The answer it defends is a boundary claim rather than a breakthrough in accuracy. With a carefully assembled prompt—appliance statistics, one example, and the previous window's predicted states—the best LLM reaches F1 scores of about 0.44 on an unseen US house and 0.61 on a UK house, well below the 0.73–0.87 scores of CNN baselines. The model's distinctive value appears in what trained models lack: it transfers to a new house and a new regional appliance set by rewriting the prompt's knowledge block, and it emits a human-readable reason for every ON/OFF decision. The paper concludes that for now LLMs should complement, not replace, accurate but opaque disaggregation models.","feed_headline":"Prompt-only LLMs can disaggregate home power, but lag trained models","feed_subtitle":"Zero-training LLM disaggregation transfers across houses and emits readable explanations; deep models still win on accuracy.","key_machinery":"The carrying mechanism is the prompt itself, specifically the knowledge-injection and context blocks of the framework. Appliance features estimated by statistics from a few training houses are written into the prompt, and the state predictions from the previous window are appended as context; the LLM is instructed to output JSON state sequences. Ablations show that adding context delivers the largest single performance boost and that power range is the most decisive appliance feature: removing it drops the F1 score from 0.5376 to 0.3008, while removing pattern description only lowers it to 0.5725. The framework's sensitivity analyses establish that window length and context length must be tuned to model capacity—longer windows help larger LLMs but cause smaller ones to violate the required output format.","core_discovery":"The paper's central discovery is a capability boundary for prompt-only LLMs in NILM. Using the proposed framework, a prompt composed of a role/task description, statistical appliance knowledge (standby power, power range, typical duration, usage pattern), a one-shot example, and the previous window's predicted states turns a general LLM into a zero-shot appliance-state detector. On REDD house 1 and UK-DALE house 2 the optimized prompt yields overall F1 scores of 0.4397 and 0.6120 for the stronger LLM, versus 0.7289 and 0.8721 for the Seq2Point CNN baseline; power estimation is weaker still, with signal aggregate error of about 0.425 versus 0.170 for the baseline. Appliances with sharp, high-power signatures or strong periodicity are detected most reliably, while multi-stage devices such as dishwashers are frequently masked by other loads. The same prompt, with only the knowledge block rewritten, transfers to the unseen US house and to the UK appliance set, and the model supplies natural-language justifications that can be checked against the aggregate signal. The paper reads these results as defining the boundary: prompt-only LLMs are not accuracy-competitive, but they offer training-free deployment, easy adaptation, and explainability.","pith_inferences":["A strict cross-region test remains open: the UK knowledge and the UK test set both come from house 2, so applying REDD-derived knowledge to an unseen UK house would be the decisive experiment for the regional generalization claim.","Because power range alone carries most of the gain, automatically mining sharper appliance signatures (e.g., per-cycle envelopes or duration distributions) from unlabeled aggregate data could cheaply raise zero-shot accuracy.","The emitted rationales could be used as machine-checkable evidence rather than decoration: a threshold rule can verify each explanation against the aggregate signal and flag disagreements, turning explainability into a quality-control loop.","The context block acts as a hand-written state smoother; an agent that maintains a longer internal belief state across windows might close part of the gap with trained models without fine-tuning."],"forward_implications":["A zero-training NILM deployment is possible: adapting to a new house or a different regional appliance set means editing the prompt's knowledge block, not retraining a model.","LLM-based disaggregation is usable where transparency matters more than peak accuracy, since every ON/OFF decision comes with a textual rationale that can be compared with the aggregate power trace.","The accuracy gap quantifies what prompt-only knowledge can capture: distinctive signatures (peak power, periodic cycling) transfer, while multi-stage signatures like a dishwasher's are lost among other loads.","Practitioners should pair window length and context length with model capacity; beyond a window of about 30 samples, smaller models' output format stability degrades sharply.","Power-value estimation is not yet practical by prompting alone, so near-term applications should treat appliance state detection as the deliverable."],"supporting_citations":[{"why":"Supplies the REDD dataset: aggregate mains and appliance channels from six US homes, with houses 2–3 used for knowledge extraction and baselines and unseen house 1 for testing.","marker":"[3]"},{"why":"Supplies the UK-DALE dataset: UK whole-house and appliance data from house 2, used for the cross-region prompt adaptation test.","marker":"[16]"},{"why":"Supplies the Seq2Seq/Seq2Point CNN baselines and the 6-second resampling preprocessing that the LLM comparison and evaluation protocol follow.","marker":"[8]"},{"why":"Frames neural-network NILM and its labeled-data dependence, the contrast that motivates the training-free LLM approach.","marker":"[6]"},{"why":"Provides the survey of LLM prompting principles and in-context learning capabilities that the prompt design draws on.","marker":"[12]"},{"why":"Supports treating time series as token sequences for zero-shot LLM forecasting, the precedent for feeding aggregate power windows to an LLM.","marker":"[32]"},{"why":"Motivates the test of reasoning-style prompts and the view of disaggregation as sequential reasoning, which shaped the final prompt choice.","marker":"[34]"}],"fun_headline_variants":["Zero-shot LLM disaggregation: no training, but lags deep nets","LLM NILM: no training, cross-house transfer, lower accuracy","Training-free LLM power reading: explainable, but not as precise","Prompt-only LLMs split homes, but CNN baselines win on F1","LLMs take on NILM untrained, yet deep models hold accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that the approach generalizes rests on the assumption that appliance characteristics measured in a few known houses describe the same appliance types in unseen houses well enough for a text-only prompt to recognize them.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot LLM disaggregation: no training, but lags deep nets","LLM NILM: no training, cross-house transfer, lower accuracy","Training-free LLM power reading: explainable, but not as precise","Prompt-only LLMs split homes, but CNN baselines win on F1","LLMs take on NILM untrained, yet deep models hold accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000404,"raw_usage":{"total_tokens":2133,"prompt_tokens":1004,"completion_tokens":1129,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":1028}},"tokens_in":620,"tokens_out":1129,"duration_ms":10560,"temperature":1.0,"reasoning_tokens":1028,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:48:03.683155+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same framework with knowledge extracted from REDD houses 2–3 but test on an unseen UK-DALE house (house 1 or 5) rather than house 2; if overall F1 collapses toward chance, the cross-region generalization claim fails. As a second check, replace the injected power ranges with deliberately wrong values and confirm the F1 falls back to the no-knowledge level of about 0.30, which would show the prompt, not some latent model knowledge, is carrying the capability.","supporting_citations":[{"cited_title":"Redd: A public data set for energy disag- gregation research","cited_arxiv_id":null,"evidence_quote":"Supplies the REDD dataset: aggregate mains and appliance channels from six US homes, with houses 2–3 used for knowledge extraction and baselines and unseen house 1 for testing."},{"cited_title":"The uk-dale dataset, domestic appliance- level electricity demand and whole-house demand from five uk homes","cited_arxiv_id":null,"evidence_quote":"Supplies the UK-DALE dataset: UK whole-house and appliance data from house 2, used for the cross-region prompt adaptation test."},{"cited_title":"Sequence-to-point learning with neural networks for non-intrusive load monitoring","cited_arxiv_id":null,"evidence_quote":"Supplies the Seq2Seq/Seq2Point CNN baselines and the 6-second resampling preprocessing that the LLM comparison and evaluation protocol follow."},{"cited_title":"Neural nilm: Deep neural networks applied to energy disaggregation","cited_arxiv_id":null,"evidence_quote":"Frames neural-network NILM and its labeled-data dependence, the contrast that motivates the training-free LLM approach."}],"review_version":1}