{"id":"b9ef959d-d66e-4b18-bd45-f0d4d971ab61","arxiv_id":"2505.11417","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"EdgeWisePersona is a new synthetic dataset and benchmark for reconstructing structured smart-home user routines from multi-session dialogues, on which large LLMs clearly outperform small on-device models.","lead":"The authors release EdgeWisePersona, a synthetic dataset of 10,000 smart-home conversations generated from 200 structured user profiles, plus a benchmark that asks language models to reconstruct those profiles from the conversations. It tests whether small on-device models can infer user routines and preferences privately, and reports that they lag large cloud models by a wide margin.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No control condition separates profile recovery from template priors; high large-model scores may not reflect reading the sessions.","rationale":"The reader identified the synthetic, human-reviewed generation as the load-bearing premise. My concern sharpens this: even granting that the sessions are fluent and faithful to the authored profiles, the evaluation protocol does not demonstrate that the scores depend on reading the sessions. The 'self-validating' argument in Section 3 is circular in effect, and the prediction-count leak in Section 4.1 plus the lack of no-session baselines makes the benchmark's construct validity unproven. This is consistent with the reader's CONDITIONAL verdict, so I do not recommend changing it. The proposed control experiment is cheap and would settle whether the central claim transfers to deployment. I chose partial agreement because the reader's focus is on data realism and diversity, whereas my focus is on the evaluation's information-flow control; both concern benchmark validity but are not identical.","tokens_in":11246,"tokens_out":3068,"duration_ms":33837,"concrete_test":"Run each model under two conditions: (a) with the actual session transcript as prompt, and (b) with an empty or masked session placeholder but identical instructions and the same number of routines to predict. If Jaccard or exact-match scores in condition (b) remain close to condition (a) (within 20% relative), the sessions are not the source of the scores, and the benchmark would need redesign. Additionally, add a majority baseline that always predicts the dataset's most frequent routine components. This control separates content-based reconstruction from prior-based guessing and directly tests whether the evaluation protocol measures what the abstract claims.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that EdgeWisePersona measures reconstruction of user routines from interaction history, and that large models outperform small ones on this task. The weakest point is not syntheticness alone, but the absence of any control that ties reported scores to session content. The routines are drawn from a small space of categorical fields and common device settings, so a model could produce plausible routines from priors alone. Section 3 calls the dataset 'inherently self-validating' because large models recover routines, but this only shows that the generator's priors align with the evaluator's priors; it does not show the sessions carry the information. Section 4.1 worsens this by prompting models with the exact number of ground-truth routines, and no baseline is reported (e.g., majority-class routine, no-session prompt, or shuffled sessions). Without such a baseline, high Jaccard values (Gemini 90%, GPT-4o 85%, even Gemma-3-4B 66%) could largely reflect generic template matching rather than successful profiling. The small-vs-large gap may be real, but the claim that the benchmark measures profile reconstruction from interactions is not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EdgeWisePersona, a synthetic dataset of smart-home interaction sessions in which each user is associated with a structured profile of routines (context-triggered device-action patterns). Sessions are generated by DeepSeek-V3 from hand-designed profiles, then human-reviewed. The core benchmark task is profile reconstruction: given a user's 50 natural-language sessions, a model must output the user's routines. The authors benchmark three large models (GPT-4o, Gemini-2.5-Flash, DeepSeek-V3) and four compact edge-deployable models (Gemma-3-4B, Qwen-2.5-3B, Llama-3.2-3B, Phi-4-mini-4B), reporting exact-match, Jaccard, MAE on scalar fields, and categorical accuracy. They conclude that small models lag large models substantially, especially on exact action recovery, and that the dataset is a realistic testbed for on-device profiling.","tokens_in":11456,"tokens_out":2844,"duration_ms":31165,"significance":"If the benchmark measures what it claims, this is a useful public resource: it provides a structured, multi-session, profile-annotated dataset in a domain where such resources are scarce, and it ships open-source evaluation code. The reported gap between small and large models is qualitatively consistent across exact-match, Jaccard, MAE, and categorical metrics, which is a genuine empirical finding. The paper's main weakness is that the evaluation protocol leaks information about the ground truth and lacks control conditions that would show the scores reflect session content rather than generic priors; these issues affect the validity of the central claim that the benchmark measures profile reconstruction from interaction history.","major_comments":[{"comment":"The protocol states that 'models are prompted to predict exactly the same number of routines as the ground truth contains.' This leaks the routine count, which is part of the profile structure and is not available in a realistic setting where a model sees only the interaction history. Providing the count can materially inflate scores and weakens the claim that routines are inferred 'solely from interactions history.' The authors should either not provide the count, evaluate sensitivity to it, or justify it as a fixed deployment design choice with evidence that it does not drive the main conclusions.","section":"Section 4.1, Evaluation Protocol"},{"comment":"No control condition is reported that ties the scores to the session content. Because the routine space is small (categorical triggers and a limited set of device actions), a model could produce plausible routines from priors alone, without reading the sessions. The paper's 'self-validating' argument in Section 3 only shows that the generator's priors align with the evaluator's priors. I request baselines such as: a majority-class routine predictor, a no-session prompt, a session-shuffled prompt, or a prompt containing random sessions. These controls are necessary to support the claim that high Jaccard values (e.g., Gemini 90%) reflect successful profiling rather than template matching.","section":"Section 3 and Section 4 (no control condition)"},{"comment":"The evaluation does not specify how predicted routines are aligned to ground-truth routines for the routine-level exact-match and Jaccard metrics. Section 4.1 says there is 'one-to-one correspondence' after truncation/padding, but routines are not naturally ordered, and if predictions are compared positionally, any permutation would be penalized. Section 4.3 uses 'most similar ground-truth routine' for trigger/action breakdown, which suggests a matching step, but the routine-level metrics in Section 4.2 must use the same or an explicitly defined matching rule. Please describe the alignment algorithm precisely (e.g., greedy matching by Jaccard, Hungarian assignment) and report whether the conclusions are robust to the choice.","section":"Section 4.1 and Section 4.3 (matching procedure)"},{"comment":"DeepSeek-V3 generated all sessions and is also one of the benchmarked models. The paper even calls the dataset 'inherently self-validating.' This creates a circularity risk for DeepSeek-V3's scores and for the claim that strong large-model performance confirms 'internal coherence and task alignment.' The authors should discuss this explicitly and, if possible, report whether DeepSeek-V3's results change when it is evaluated on a held-out subset or on sessions it did not generate. At minimum, the 'self-validating' terminology should be replaced with a more cautious description of what the large-model scores demonstrate.","section":"Section 3.2 and Section 4 (generator-as-benchmarked-model circularity)"}],"minor_comments":[{"comment":"The human review step is described only as 'minor corrections are made when needed'; please quantify the number of annotators, the fraction of sessions edited, and any inter-annotator agreement measure, since the dataset's realism claim depends partly on this step.","section":"Section 3.2 (Human Review)"},{"comment":"The limitations section does not mention that the benchmark is fully synthetic and that no real-world validation is provided; given the paper's claims about 'realistic, structured testbed,' a sentence on this is warranted.","section":"Section 5 (Limitations)"},{"comment":"Reference [41] contains a typo: 'Adaptive Layer Tuning and V oting' should be 'Adaptive Layer Tuning and Voting.'","section":"References"},{"comment":"The appendix documents the routines and sessions schemas but not the characters.jsonl schema, although the dataset structure in Section 3.2 lists it as a core aligned file; please include it for completeness.","section":"Appendix A (Schemas)"}],"recommendation":"major_revision","confidential_remarks":"The paper is from an industry lab and the dataset is publicly released. The main concern is not syntheticness per se but whether the evaluation demonstrates that scores are driven by session content; the missing control conditions and the leaked routine count are fixable within the manuscript's scope. If the authors add controls and clarify the matching protocol, the central small-vs-large finding would be on much firmer ground. I would also encourage the editor to ask the authors to moderate the 'self-validating' language, as it conflates generator/evaluator prior alignment with evidence of task validity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: EdgeWisePersona is a genuinely useful public resource — 200 synthetic smart-home users, 10,000 multi-session dialogues, structured routines, and open code/data. The gap it fills is real: no existing public dataset combines persistent identity, multi-session continuity, numeric routines, and the smart-home domain. The paper positions itself well against related work, and the qualitative finding that large models beat small models is consistent across exact-match, Jaccard, MAE, and categorical metrics. That directional result is probably robust.\n\nThe soft spots are real, though. The central claim — that this benchmark measures reconstructing user routines from interaction history — is not established, because there is no control condition. A no-session prompt or a shuffled-sessions prompt would tell you whether high Jaccard scores (Gemini 90%, GPT-4o 85%) reflect reading the sessions or just matching the generator's priors, since routines live in a small space of categorical fields and common device settings. Section 3 even calls the dataset 'inherently self-validating' because large models recover routines; that only shows the generator's priors align with the evaluator's priors. Second, the protocol leaks ground truth: Section 4.1 tells models to predict exactly the same number of routines as the reference. That turns part of the task into filling a known fixed number of slots and inflates small-model Jaccard scores. Third, there are no error bars, no variance estimates, and the matching procedure for ordered routines is not described, which weakens confidence in Tables 2 and 3.\n\nThe circularity with DeepSeek-V3 being both generator and a benchmarked model is real but mild, since the main comparison is between independent model families. The syntheticness itself is not the problem; the missing control condition is what prevents the central conclusion from being accepted as stated.\n\nCredit where it is due: the dataset itself is the contribution, the schemas are clear, the human review step is disclosed (though its extent is unspecified), and the benchmarking code is open. The paper is not a takedown target — it is a solid resource attached to an evaluation protocol that needs repair.\n\nWho this is for: researchers in edge AI and personalized smart-home systems who want a testbed for compact models. They should use it, but they should not cite the reported numbers as evidence of profile reconstruction until the protocol is fixed.\n\nMy recommendation: send this to peer review, but require the authors to add no-session and shuffled-session baselines, remove the ground-truth routine-count leak, report variance, and either describe the matching procedure or switch to a set-based metric. With those fixes, the paper could be a dependable community resource; without them, the benchmark results overclaim.","headline":"Useful synthetic dataset and benchmark for on-device user profiling, but the evaluation lacks a no-session baseline, which leaves the central claim about reconstructing routines from sessions unproven; worth engaging with as a resource, not as stated results.","tokens_in":11950,"tokens_out":1903,"would_cite":true,"duration_ms":19907,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces a 10,000-session synthetic smart-home dataset and reports that compact, edge-deployable models recover almost none of a user's routines exactly, while large models reach 44 percent.","keywords":["user profiling","smart home","on-device LLM","routine reconstruction","synthetic dataset","benchmark","natural language interactions","edge AI"],"falsifier":"Run the same profile-reconstruction benchmark on real opt-in smart-home interaction logs for which the user's actual routines are known; if the gap between small and large models narrows or reverses, the synthetic benchmark's conclusion does not transfer to deployment. A cheaper probe is to feed the session generator only the personality descriptions and compare recovery rates, which would reveal whether routines are encoded in the dialogues or leak through stylistic cues.","tokens_in":11041,"feed_emoji":"🏠","tokens_out":11689,"duration_ms":107544,"temperature":0.7,"pith_summary":"The paper introduces EdgeWisePersona, a synthetic dataset of 10,000 smart-home dialogues from 200 users, each with a structured profile built from routines: repeatable, context-triggered behaviors such as 'when it is rainy, warm, and evening, dim the lights and raise the AC.' The central claim is that this dataset is a realistic, structured testbed for profile reconstruction—inferring a user's routines and preferences solely from their interaction history, with the true routines available as ground truth. Benchmarking on this task, the paper reports that compact models intended for on-device deployment recover full routines essentially never (best exact-match score around 2%), while large cloud models reach 27–44%, and even partial-overlap scores (Jaccard similarity) show a wide gap (23–66% versus 82–90%). The authors argue this gap matters because on-device profiling would keep personal data local, lower latency, and enable smart-home personalization without depending on the cloud.","feed_headline":"Best small model scores 2% exact match on smart-home routines","feed_subtitle":"A 10,000-session synthetic benchmark shows large models reach 44%, exposing the edge-AI gap.","key_machinery":"The central unit is a routine—a context-triggered rule of the form 'when [time, weather, sun phase, and temperature conditions] hold, adjust these devices to these settings.' A user's profile is the set of such routines plus a short personality description. The dataset is built by sampling realistic contexts, then asking a large language model to generate a dialogue consistent with the profile and context, so every session is anchored to known ground-truth routines; the authors call this self-validating because a model that recovers the routine from the dialogue demonstrates that the dialogue encodes it. The evaluation protocol then prompts a model to predict exactly as many routines as the ground truth contains and scores each prediction by exact match, partial overlap (Jaccard), and component-level accuracy for triggers, actions, scalar values, and categorical values. This mechanism—routine as ground truth and evaluation target—is what lets the benchmark separate the task of recognizing when a behavior occurs from the task of reproducing precisely how a user acts.","core_discovery":"The paper's central discovery is that profile reconstruction from ordinary smart-home conversation is possible in principle but not yet attainable with current small, edge-deployable models. In the authors' evaluation, large models can capture most of a routine's components—trigger conditions are recovered correctly over 90% of the time—but exact, full-routine recovery remains hard even for them, with the best large model at 44%. Small models fall far behind: the strongest compact candidate reaches only 2% exact-match accuracy and its partial-overlap score is 66%, while the others are near zero on exact matches. The hardest component is the action sequence, particularly numeric device settings, where small models deviate by an order of magnitude more than large ones. The paper presents this large-versus-small gap, together with the dataset's self-validating design, as the empirical case for a benchmark that can drive work on privacy-preserving, on-device personalization.","pith_inferences":["A reader might infer that the bottleneck for small models is structured output rather than context length: the small models can paraphrase routines but cannot emit them in the exact schema, so fine-tuning on schema-constrained generation could close more of the gap than adding parameters.","The ease of trigger prediction suggests a cheap screening test: if a candidate on-device model can recover triggers from a short dialogue sample, researchers could fast-filter models before running the full structured decoding benchmark.","The same routine-as-truth design could transfer to other device domains where behavior is defined by context and settings, such as automotive assistance or wearable health automation, with minimal changes to the protocols.","A testable extension of the self-validating logic is to isolate the influence of the personality description: if removing it from the session prompts changes large-model recovery substantially, part of the measured profiling performance may come from stylistic cues rather than from behavior inferred from interaction."],"forward_implications":["Compact, edge-deployable models must move beyond recognizing routine triggers (which they already do reasonably well) to generating exact action sequences and numeric parameters, the components where they currently fail almost completely.","Because the best large model still achieves only 44% exact-match routine recovery, exact full-structure prediction is not yet solved at any scale, so the benchmark defines a clear target for future work.","The dataset's line-aligned structure—personality, routines, and sessions for each user stored in parallel files—supports both supervised training and standardized benchmarking of user profiling.","The self-validating property gives researchers a quality check: if a large model recovers a routine from a generated session, that session demonstrably contains the behavior, so improvements in recovery reflect real progress in reading behavior from language.","The benchmark's uniform prediction-count policy and component breakdowns provide a reusable template for evaluating other behavior-reconstruction tasks beyond smart homes."],"supporting_citations":[{"why":"The large language model used to generate all synthetic sessions from user profiles; the dataset's realism and consistency claim depends on this generator.","marker":"[7]"},{"why":"A compact model benchmarked on the reconstruction task; its 2% exact-match result anchors the small-model side of the reported gap.","marker":"[34]"},{"why":"A compact benchmarked model whose near-zero exact-match performance shows the gap is not specific to one architecture.","marker":"[26]"},{"why":"A compact benchmarked model that also lands near zero on exact-match, supporting the claim that small models generally fail at full routine recovery.","marker":"[21]"},{"why":"A compact benchmarked model whose large scalar-field errors quantify the numeric-precision gap the paper reports.","marker":"[1]"},{"why":"The large cloud model whose high partial-overlap and exact-match scores set the large-model reference point in the benchmark.","marker":"[24, 25]"},{"why":"The large model that achieves the highest exact-match accuracy (44%), providing the strongest evidence that routines are recoverable from the generated sessions.","marker":"[11, 33]"},{"why":"The leaderboard used to select models by accuracy and context length, shaping which small and large models were compared.","marker":"[19]"}],"fun_headline_variants":["Edge AI profiling gap: 2% vs 44% exact routine match","Small models only 2% exact match on smart-home profiles","10,000-session benchmark shows on-device profiling limits","Privacy-preserving user profiling still far off for edge models","Small language models lag big on routine reconstruction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's load-bearing premise is that sessions generated by a large language model from hand-designed profiles, after only light human correction, are realistic and varied enough to stand in for real smart-home user interactions; if that premise fails, the benchmark measures recovery of synthetic templates rather than actual user profiling.","fun_headline_variants_meta":{"raw":{"variants":["Edge AI profiling gap: 2% vs 44% exact routine match","Small models only 2% exact match on smart-home profiles","10,000-session benchmark shows on-device profiling limits","Privacy-preserving user profiling still far off for edge models","Small language models lag big on routine reconstruction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1514,"prompt_tokens":992,"completion_tokens":522,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":438}},"tokens_in":608,"tokens_out":522,"duration_ms":5133,"temperature":1.0,"reasoning_tokens":438,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:53:22.828449+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same profile-reconstruction benchmark on real opt-in smart-home interaction logs for which the user's actual routines are known; if the gap between small and large models narrows or reverses, the synthetic benchmark's conclusion does not transfer to deployment. A cheaper probe is to feed the session generator only the personality descriptions and compare recovery rates, which would reveal whether routines are encoded in the dialogues or leak through stylistic cues.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A compact benchmarked model that also lands near zero on exact-match, supporting the claim that small models generally fail at full routine recovery."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The leaderboard used to select models by accuracy and context length, shaping which small and large models were compared."}],"review_version":1}