{"id":"b0d1202f-85c1-447e-b2b7-431e773f65c9","arxiv_id":"2501.04974","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SensorQA is a new crowdsourced benchmark showing that current AI models answer only about 28% of daily-life sensor-data questions correctly.","lead":"This paper introduces SensorQA, a dataset of 5,648 human-written questions and answers about daily activities, built on long-term sensor data from 60 users. It benchmarks AI models on this dataset and finds they answer correctly only about 28% of the time, highlighting a gap in sensor-data question answering.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SensorQA's ground-truth answers derive from unvalidated ExtraSensory self-report labels and unverified AMT worker answers; if either is noisy, the reported 28% best accuracy and the 'gap' claim are not trustworthy.","rationale":"The reader's weakest assumption—that ground-truth answers are correct because they come from ExtraSensory labels and AMT worker readings—is precisely the load-bearing condition I identify. This is the most upstream and consequential uncertainty in the paper. If the reference answers are wrong, every benchmark number in Table 4 (e.g., Llama-Adapter at 28% exact match) and the central conclusion that current models perform poorly cease to be meaningful. The concern is concrete: Section 3.1 provides no evidence of label quality beyond 'after cleaning', and the ExtraSensory dataset itself is known to contain noisy self-reports; Section 3.2 provides no inter-annotator agreement, no validation, and no procedure for filtering answers that cannot be derived from the graph. The paper explicitly invites subjective questions ('work-life balance'), which makes the ground truth even more vulnerable. My proposed test directly measures the reliability of the reference answers by having independent annotators re-derive them from the same inputs. A high agreement rate would support the benchmark; a low rate would require re-annotation and could change the verdict to REJECT or UNVERDICTED. Because the reader already made this the basis of a CONDITIONAL verdict, my stress-test confirms rather than changes the verdict, so I recommend UNCHANGED. I considered other potential concerns—lack of a human baseline, noisy keyword distillation, and the label-versus-raw-signal mismatch—but they are secondary or downstream of the primary concern about ground-truth validity. The absence of a human baseline would weaken the 'optimal performance' phrasing but not invalidate the dataset; the keyword extraction issue matters only if the full answers are themselves correct. No other objection rises to the same level of load-bearing importance.","tokens_in":12175,"tokens_out":5320,"duration_ms":51558,"concrete_test":"Sample 200 QA pairs uniformly. Recruit two expert annotators (or a second AMT batch with strict instructions) to independently write answers from the same activity graphs and label files, without seeing the original answers. Compute exact-match and semantic-similarity agreement between original and re-derived answers, and have each annotator flag answers not entailed by the graph/labels. If pairwise agreement is below 85%, or if more than 10% of original answers are flagged as not graph-entailed, the ground truth is unreliable; the authors should clean or re-annotate and rerun all baseline metrics. As a secondary check, re-derive the short-answer keywords for these 200 pairs by human extraction and compare with the GPT-3.5-Turbo keywords used for exact-match.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing condition for SensorQA's central claim is that its 5,648 reference answers are correct. This is not established. Section 3.1 states that graphs are built from ExtraSensory's activity/context labels 'after cleaning' but gives no cleaning protocol and no accuracy or agreement metrics for those labels. ExtraSensory labels are self-reported by participants in-the-wild and are known to be noisy and sparse (Vaizman et al., 2017, 2018). Section 3.2 describes AMT workers generating QA pairs from these graphs, with no inter-annotator agreement, no second-pass validation, and no check against the labels. The instruction to role-play and ask what 'would interest someone' invites subjective inference, and the examples include evaluative questions (e.g., work-life balance) whose answers are not uniquely determined by the graph. The abstract's 'accurate answers derived from sensor data' is inaccurate: answers are derived from labels, not raw signals. Because all metrics (Rouge, Meteor, Bleu, exact-match) are computed against these reference answers, any label error or worker mistake propagates directly into the reported 28% best accuracy and the claimed gap between models and optimal performance. If references are noisy, model scores are not interpretable as task performance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces SensorQA, a benchmark of 5,648 question-answer pairs built by AMT workers from activity/context-label graphs derived from the ExtraSensory dataset. The authors benchmark text-only, vision+text, and sensor+text baselines on both full-answer and short-answer versions, reporting a best exact-match accuracy of 28% (Llama-Adapter), and measure memory and latency on the Jetson TX2. The paper's main claims are that SensorQA is the first human-created QA benchmark for long-term time-series sensor data and that current AI models exhibit a substantial gap to optimal QA performance and efficiency.","tokens_in":12390,"tokens_out":4579,"duration_ms":46747,"significance":"If the reference answers and evaluation splits are validated, SensorQA would fill a real gap: existing sensor QA datasets are either template-generated (DeepSQA) or limited to short, fixed-duration signals (AnyMAL, OneLLM). The dataset is open-sourced, covers multiple timescales and diverse question/answer categories, and includes both accuracy and edge-deployment measurements, which are useful for the community. However, the quantitative conclusions currently rest on unvalidated reference answers, a potentially leaking train/test split, and no human performance baseline, so the significance of the reported numbers is not yet established.","major_comments":[{"comment":"The ground-truth answers are not validated: the paper reports no inter-annotator agreement, no second-pass review, and no consistency check against the label intervals in the activity graphs. Workers were instructed to ask questions that 'would interest someone' with a smart device, which invites subjective and evaluative questions such as the work-life balance example in Section 1, whose answers are not uniquely determined by the graph. Because every reported metric is computed against these reference answers, any noise in the ExtraSensory labels (self-reported and only described as 'after cleaning' in Section 3.1, without a protocol) or any worker error propagates directly into the benchmark numbers. Please add an annotation-quality study, including agreement metrics, a sample of questions verified against the raw label timestamps, and a discussion of rejected or corrected pairs.","section":"§3.2 and §5.2"},{"comment":"The random 80/20 train/test split is performed at the QA-pair level, not at the user or activity-graph level. Since multiple QA pairs are generated from the same graph and the same user can appear in multiple graphs, the same underlying activity pattern can appear in both training and testing, potentially inflating the fine-tuned model accuracies. Please evaluate on a user-disjoint or graph-disjoint split, or at minimum report results for both a random split and a disjoint split and discuss any performance difference.","section":"§5.1"},{"comment":"The claim that the results reveal 'a gap between current models and optimal QA performance' is not supported by the evidence reported in the paper, because no human performance baseline is provided. A human baseline on a representative sample of test questions, scored with the same exact-match protocol, would calibrate what 'optimal' means for this benchmark and is needed to quantify how large the gap actually is. Please add such a baseline or substantially soften the claim.","section":"Abstract and §5.2"},{"comment":"The abstract states that answers are 'derived from sensor data,' but the QA pairs are actually generated from activity/context label graphs, not from raw sensor signals. This distinction changes what the benchmark tests: reading summarized activity labels versus reasoning over raw multimodal time-series data. It also affects the interpretation of the Sensor+Text baselines, which are evaluated on label-derived answers rather than raw signal inference. Please state this precisely, describe the label-cleaning protocol in Section 3.1, and discuss the implications for the claimed sensor-based QA contribution.","section":"§1, §3, and Table 4"},{"comment":"The exact-match accuracy is computed from keywords extracted by GPT-3.5-Turbo from the full answers, with no validation of the extraction quality. The reported accuracy therefore depends on an unverified intermediate step, and the metric name 'exact-match' is misleading because the check is keyword containment rather than exact string or semantic equivalence. Please validate the distilled short answers on a human-annotated sample, report the extraction accuracy, and either rename or redefine the metric to match what is actually computed.","section":"§5.1 and §5.2"}],"minor_comments":[{"comment":"DeepSQA reports an exact-match accuracy of 27.4% while achieving a Bleu score of 0.0; this unusual combination should be explained, since it is not typical for n-gram metrics to be zero when keywords are present.","section":"Table 4"},{"comment":"The statement that Llama performs 'only slightly better than random guessing, with an accuracy of 58%' needs a defined random baseline; for Yes/No questions the chance level depends on the label distribution, which should be reported.","section":"Figure 3"},{"comment":"The paper refers to the GitHub repository for 'more details' on splits, prompts, and hyperparameters; for reproducibility these details should be included in the paper or a supplementary document, including seeds and the exact few-shot prompts.","section":"§5.1"},{"comment":"The AMT collection description lacks basic quality-control information, such as the number of workers, qualification requirements, payment, and how many collected pairs were discarded before the final 5,648; reporting these numbers would help assess the reliability of the crowdsourced content.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable dataset/benchmark contribution for a venue that values resources and empirical evaluation. The central concerns—reference-answer validation, disjoint splitting, and the absence of a human baseline—are addressable with additional experiments, so I would not recommend rejection if the authors can supply those analyses."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nSensorQA is worth a look. It's the first human-created QA benchmark for long-term daily-life sensor data, and that claim holds: prior work like DeepSQA is template-generated, and AnyMAL/OneLLM handle short fixed signals. The dataset—5,648 QA pairs over ExtraSensory-derived activity graphs, using 14 label subsets and multi-timescale visualizations—is a real resource, and they've open-sourced it along with the benchmark code. The edge-device latency/memory measurements are a useful addition.\n\nThe paper does several things well. The collection protocol is described concretely: workers see Gantt-style graphs of activity labels over time, are asked to role-play as the device owner, and write first-person questions with second-person answers. The category analysis (six question types, seven answer types) shows deliberate design for diversity. The benchmark setup spans text-only, vision+text, and sensor+text baselines, and the finding that sensor+text models underperform text-only models is an interesting signal for the community.\n\nThe soft spots are real but addressable. The ground-truth answers are derived from ExtraSensory's self-reported activity labels and from unvalidated AMT worker readings of the graphs. There is no inter-annotator agreement, no second-pass check, and no expert review of the QA pairs. The abstract's 'accurate answers derived from sensor data' overstates things—the answers come from labels, not raw signals, and labels in ExtraSensory are known to be noisy. Some questions (e.g., 'How good was my work-life balance?') are not uniquely determined by the graph; the worker's own judgment is part of the answer. Second, there is no human performance baseline, so the claimed 'gap between current models and optimal QA performance' is unquantified. Third, benchmark numbers come without confidence intervals; with a 20% test split (~1,100 pairs), a 2–3 point accuracy difference between baselines may be noise.\n\nNone of these are fatal. The dataset is valuable as a first-of-its-kind resource even if some answers are imperfect. The paper needs a validation study—e.g., a second AMT pass or expert sample review—and a human baseline before the accuracy numbers are taken at face value.\n\nThis deserves a serious referee. I'd send it to review with the expectation of revision addressing validation and human performance. For someone working on sensor-LLM fusion, it's a cite-worthy resource.\n\nBest,\n[Your name]","headline":"A genuinely new human-authored QA benchmark for long-term sensor data, with thoughtful collection design; but unvalidated ground-truth answers and a missing human baseline make the headline accuracy numbers provisional.","tokens_in":12968,"tokens_out":2399,"would_cite":true,"duration_ms":23059,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SensorQA is the first human-created question-answering benchmark for long-term time-series sensor data, and existing AI models answer at most 28% of its questions exactly.","keywords":["question answering","sensor data","benchmark","human-created dataset","daily-life monitoring","large language models","multimodal reasoning","wearable sensors"],"falsifier":"Take a random sample of SensorQA answer pairs and re-derive the answers directly from the raw ExtraSensory sensor streams, such as accelerometer, location, and audio data, rather than from the pre-computed activity labels; if a substantial fraction disagree, the benchmark's ground truth is the weak link.","tokens_in":11979,"feed_emoji":"📊","tokens_out":6197,"duration_ms":57943,"temperature":0.7,"pith_summary":"The paper introduces SensorQA, which it presents as the first human-created benchmark for question answering over long-term time-series sensor data in daily-life monitoring. The dataset contains 5,648 question-answer pairs built by showing Amazon Mechanical Turk workers activity graphs derived from the ExtraSensory dataset and asking them to write first-person questions and second-person answers. Benchmarking state-of-the-art models shows a large accuracy gap: the best baseline, Llama-Adapter, reaches 28% exact-match accuracy, and sensor-plus-text models do worse than text-only models. The authors argue this gap, together with impractical latency on edge devices, shows that current AI is not ready to serve as a conversational personal sensing assistant.","feed_headline":"Best AI model scores just 28% on sensor-life QA test","feed_subtitle":"5,648 human-written questions about weeks of wearable data expose a big gap in AI sensor understanding.","key_machinery":"The mechanism that carries the argument is the QA-collection protocol: workers receive activity graphs, which are color-coded, Gantt-style visualizations of activity and context labels over time, and are instructed to role-play as the device owner, generating a first-person question and a second-person answer. Multi-time-scale graphs steer workers toward both short quantitative queries and long-horizon qualitative queries, while 14 label subsets cover postures, location, diet, sleep, commute, exercise, electronics, social life, work, and work-life balance. This protocol converts raw sensor streams into human-readable ground truth, which is then used to train and evaluate models.","core_discovery":"On its own terms, the paper's central discovery is that today's AI models cannot reliably answer realistic human questions about weeks of wearable sensing data. By visualizing ExtraSensory activity and context labels as multi-time-scale Gantt-like graphs (daily graphs with a timetable, and multi-day graphs) and using 14 label subsets, the authors collect 5,648 human-authored QA pairs spanning six question categories and seven answer categories. They then benchmark text-only, vision+text, and sensor+text models and find the best exact-match accuracy is only 28%, with sensor+text fusion underperforming text-only baselines and time-related questions being hardest. This is taken as evidence that the missing ingredient is a realistic, long-duration QA benchmark and that new model architectures or training methods are needed.","pith_inferences":["Because the ground truth is derived from ExtraSensory labels, a label-quality audit or a second round of human verification against raw sensor streams would materially strengthen the benchmark; the paper does not report one.","The paper's result that sensor+text fusion hurts performance suggests current adapters compress long time series into lossy text, so models that reason over event intervals or temporal graphs might beat the 28% ceiling.","The same graph-plus-label-subset recipe could create QA benchmarks for other long-term time-series domains, such as energy meters, sleep clinics, or vehicle telematics.","A practical corollary the authors leave implicit is that a viable edge assistant may need to run small models on distilled summaries rather than billion-parameter LLMs to meet latency and memory constraints."],"forward_implications":["Any deployable personal AI assistant over wearable data must handle multi-day temporal context, not just short clips.","New sensor-text fusion methods are needed because feeding sensor information to models currently hurts performance relative to text-only prompting.","Time-related questions, such as durations, timestamps, and day comparisons, are the hardest and most distinctive category, so future work should target temporal reasoning.","Fine-tuning on the target data matters: Llama-Adapter goes from 0% exact-match accuracy without fine-tuning to 28% with it.","Billion-parameter LLMs are impractical for edge deployment, with average answer generation latency over 57 seconds on a Jetson TX2."],"supporting_citations":[{"why":"Supplies the raw sensor measurements and the 51 activity and context labels from which all activity graphs are built.","marker":"[44, 45]"},{"why":"The prior template-generated sensor QA dataset that SensorQA contrasts with and uses as a baseline.","marker":"[48]"},{"why":"AnyMAL represents the prior sensor-plus-LLM approach limited to short, fixed-duration signals and is used as related work and a baseline.","marker":"[29]"},{"why":"IMU2CLIP supplies the retrieval-based sensor+text baseline (IMU2CLIP-GPT4) evaluated on SensorQA.","marker":"[30]"},{"why":"OneLLM is the multimodal sensor+text baseline that fuses sensor data through a CLIP encoder and mixture of projection experts.","marker":"[16]"},{"why":"Llama-Adapter is the vision+text baseline that achieves the best exact-match accuracy of 28%.","marker":"[55]"},{"why":"LLaVA-1.5 is the vision+text baseline that combines a visual encoder with Vicuna for answering from activity graphs.","marker":"[25]"},{"why":"The GPT-4 model family provides the text-only and vision+text few-shot baselines (GPT-4, GPT-4-Turbo, GPT-4o).","marker":"[3]"}],"fun_headline_variants":["AI fails sensor-life quiz: best score 28%","SensorQA benchmark stumps AI: 28% accuracy","AI scores 28% on human sensor questions","New sensor QA test reveals AI's 28% ceiling","Wearable data queries crush AI models at 28%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the answers in the 5,648 QA pairs are correct because they are read, by crowd workers, from activity graphs built on ExtraSensory labels; if those labels or readings are wrong, the benchmark numbers collapse.","fun_headline_variants_meta":{"raw":{"variants":["AI fails sensor-life quiz: best score 28%","SensorQA benchmark stumps AI: 28% accuracy","AI scores 28% on human sensor questions","New sensor QA test reveals AI's 28% ceiling","Wearable data queries crush AI models at 28%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00011,"raw_usage":{"total_tokens":1014,"prompt_tokens":868,"completion_tokens":146,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":66}},"tokens_in":484,"tokens_out":146,"duration_ms":2317,"temperature":1.0,"reasoning_tokens":66,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:20:39.608317+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of SensorQA answer pairs and re-derive the answers directly from the raw ExtraSensory sensor streams, such as accelerometer, location, and audio data, rather than from the pre-computed activity labels; if a substantial fraction disagree, the benchmark's ground truth is the weak link.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The prior template-generated sensor QA dataset that SensorQA contrasts with and uses as a baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"OneLLM is the multimodal sensor+text baseline that fuses sensor data through a CLIP encoder and mixture of projection experts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LLaVA-1.5 is the vision+text baseline that combines a visual encoder with Vicuna for answering from activity graphs."}],"review_version":1}