{"id":"9e257cd0-89dc-4bdd-8bf2-1e94fd00cc36","arxiv_id":"2501.16864","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper describes iLogCal, a calendar-based scheduling and monitoring methodology for personal data collection that adds situational and temporal context to sensor and questionnaire data, demonstrated on a 170-participant subset of a 350-student study.","lead":"This paper presents a data collection platform, built on the iLog app, that combines smartphone sensor data with participant questionnaire answers and gives researchers and participants tools to monitor and adjust the experiment in real time. The authors show the platform on a study of 350 university students and report a machine learning model that predicts when a participant is likely to answer within 30 minutes.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 7's ML component predicts a 30-minute response-time proxy, not answer quality; the cutoff is taken from [67] without validation on this dataset, so the paper's quality-prediction claim is unsupported.","rationale":"The Reader's verdict is CONDITIONAL at moderate confidence, and I agree with the weakest assumption. The strongest claim is the abstract's promise of 'much improved quality and richness,' and Section 7 is the only place where that promise is given an empirical, quantitative form. That section's design conflates a convenient, measurable proxy (response within 30 minutes) with the construct the paper actually cares about (correct, meaningful answers). This is an empirical-validity concern, not a disagreement with current consensus: the paper may be right that shorter reaction times correlate with better answers, but it does not demonstrate this correlation in its own data, and it is exactly the kind of claim that can fail in a specific population (e.g., students who delay answering because they are busy but still provide correct answers later). The fact that the authors have GPS-based validation available in the same dataset and do not use it makes the proxy an unnecessary risk. Additionally, the evaluation uses random 80/20 splits over all participant instances, so answers from the same individual can appear in both training and test, potentially inflating the reported accuracy; the per-participant analysis avoids leakage but still inherits the proxy problem and reports no baseline. These issues are fixable, which is why CONDITIONAL rather than REJECT is appropriate. A revision that validates the threshold against correctness labels and, ideally, tests the adaptive scheduling loop against a fixed schedule would turn the central empirical claim into a supported one. The platform description itself is credible and the case study is substantial, so this is not a rejection of the system; it is a request for evidence that the advertised benefit exists.","tokens_in":20458,"tokens_out":7298,"duration_ms":66023,"concrete_test":"Leverage the dataset's GPS and self-reported location answers to build a correctness label for 'Where are you?' questions (the paper describes this validation capability in Section 3). Compute the agreement between the 30-minute response window and this correctness label; then retrain the Random Forest from Table 8 on the subset of questions that have GPS-validated correctness labels, using participant-grouped cross-validation. If agreement is low or the retrained model fails to beat a majority-class baseline, the Section 7 claim that the component predicts answer quality is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that giving researchers, participants, and the platform control over collection improves data quality and richness. The only empirical support for this is the Section 7 machine-learning result, which claims to 'predict answer quality' from context. But the target label is not the answer-quality construct defined in Section 3 ('an answer which has been meaningfully provided and which is correct'). It is a binary indicator of whether the participant answered within 30 minutes, a threshold imported from reference [67] with no evidence shown that it tracks correctness in this Trento student population. The authors themselves note in Section 3 that correctness can be validated for at least some questions (e.g., comparing self-reported 'Where are you?' answers with GPS-derived location), but Section 7 does not use such validation. If the 30-minute proxy is not actually a proxy for correctness, then the 0.758 accuracy in Table 8 and the per-participant predictions in Fig. 13 demonstrate only the predictability of response latency, not of answer quality. This matters because the abstract's 'much improved quality and richness' claim rests, in part, on this ML component. The evaluation also lacks a comparison to a fixed-schedule control, so even a valid proxy would not establish that the platform's adaptive control improves outcomes.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a methodology and platform, built on the existing iLog data-collection app, intended to improve the quality and richness of combined sensor and questionnaire data by giving researchers, participants, and the platform itself more control over data collection. Four functionalities are described: a situational context model with five dimensions (where, what, internal state, who, utensils), a temporal context language called iLogCal based on iCal, a calendar-based dashboard for real-time monitoring, and a mechanism for run-time revision of the data collection plan. The platform is demonstrated on a case study with 350 University of Trento students (170 analyzed), and the paper reports an ML component that aims to predict answer quality from context features, achieving 0.758 accuracy with Random Forest. The central claims are that these functionalities enable 'much improved quality and richness' of collected data and that the ML component can predict answer quality.","tokens_in":20639,"tokens_out":3647,"duration_ms":34122,"significance":"If the claims are substantiated, the paper would make a useful contribution to mobile sensing and EMA/ESM practice by proposing a concrete integration of sensor data, questionnaire data, and process-level meta-context in a single platform, deployed at nontrivial scale. The introduction of QA sensors, the iLogCal scheduling language, and the dashboard-based monitoring workflow are potentially reusable design artifacts. The ML evaluation uses standard classifiers with 5-fold cross-validation and a per-participant temporal split, which is more informative than a purely aggregate assessment. However, the significance is conditional on the Section 7 result actually measuring answer quality rather than merely response latency; as written, the empirical support for the headline quality-improvement claim is incomplete.","major_comments":[{"comment":"The Section 7 title and text claim the ML component predicts 'answer quality,' and the opening defines quality as 'the number of correct answers.' But the label actually used is whether the participant answered within 30 minutes, with the threshold imported from reference [67] and no validation on this dataset that this threshold tracks correctness. The paper itself, in Section 3, notes that some answers (e.g., 'Where are you?') could be validated against GPS data, but no such validation is performed. Consequently, the 0.758 accuracy in Table 8 and the per-participant results in Fig. 13 currently demonstrate predictability of response latency only. The claim in Section 7 that 'we can effectively predict if the participant can answer questions within 30 minutes, thus predicting answer quality' is unsupported. The authors should either validate the 30-minute proxy against a ground-truth correctness measure or carefully reframe the entire section and all related conclusions as predicting response time, not answer quality.","section":"Section 7, Tables 5-8, Fig. 13"},{"comment":"The abstract states that giving researchers, participants, and the platform control enables 'a much improved quality and richness of the data collected,' and the conclusion credits the platform with 'enhancing the quality of the data collection.' The case study is a deployment at scale, but it contains no baseline, control condition, or before/after comparison: there is no evidence that the adaptive scheduling, dashboard monitoring, or run-time revision actually improves answer correctness, response quality, or data richness relative to a fixed-schedule collection without these features. Even if the ML result were a valid measure of answer quality, it would at most show predictability of good response opportunities, not that the platform's interventions improve outcomes. The authors should either add a comparative evaluation or downgrade the claim to one of feasibility and monitoring capability.","section":"Abstract and Section 8"},{"comment":"The text says the Random Forest was applied 'for each specific participant' using the first two weeks to train and the next two weeks to test, but Fig. 13 shows only 21 labeled user IDs and no information about how many participants were actually included, how they were selected, or how the accuracy values are distributed. Without this information, the statements about 'good average level of predictability' and 'does not decay much in the worst case' are not quantitatively supported. Please report the number of participants, the distribution of per-participant accuracy, and, ideally, confidence intervals or error bars.","section":"Section 7, per-participant experiment"}],"minor_comments":[{"comment":"The abstract says the case study involved 350 students, while Section 3 says the analysis uses a selection of 170 students; please clarify this discrepancy in both places.","section":"Abstract and Section 3"},{"comment":"The text contains typographical errors: 'Fig,4', 'Fig,5', and 'Fig,6' should be 'Fig. 4', 'Fig. 5', and 'Fig. 6', and 'Bachus-Naur Form' should be 'Backus-Naur Form'.","section":"Section 5, Figs. 4-6"},{"comment":"The sentence 'The BNF of the iLogCal sensor collection is reported in Fig,5' should refer to the question collection, not the sensor collection, since the accompanying BNF defines Question collection.","section":"Section 5.2"},{"comment":"The dashboard in Fig. 9 shows experiment dates 2022-04-18 to 2022-06-13, while the heatmap in Fig. 10 is described as having a missing date '2020-11-12'; please verify and align the experiment time frame.","section":"Section 6, Fig. 9 and Fig. 10"},{"comment":"The discussion under Table 5 refers to 'incorrect answers,' but the operational definition used in the analysis is the 30-minute threshold, not a correctness measure; please rephrase to avoid confusion until the proxy is validated.","section":"Section 7, Table 5"},{"comment":"Reference [66] has a garbled author list ('Y.R.Z.R.C.S.P.C.M.B.D.V.S.B.R.D.A. Folarin') and needs to be corrected.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main risk is Section 7's label mismatch: the paper's most concrete empirical sub-claim is that the ML component predicts answer quality, but the implemented target is only response latency within 30 minutes. If the authors cannot validate the 30-minute proxy with ground-truth correctness on a subset of the data, they should reframe the section and the abstract's quality-improvement claim accordingly. The lack of any baseline comparison is also a concern, but for a systems/methodology paper it is less fatal if the claims are presented as feasibility and monitoring rather than demonstrated quality improvement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the genuinely new engineering here—QA sensors, the iLogCal scheduling language, and runtime plan revision—is real and plausibly useful to anyone doing ESM or combined sensor/questionnaire studies. Second, the paper's central claim that this setup produces 'much improved quality and richness' of data is not actually tested. The only quantitative support is an ML section that predicts whether a participant answers within 30 minutes, which the authors equate with answer quality but do not validate as such on this population.\n\nWhat the paper does well: it gives a concrete, fairly detailed description of a platform that has been deployed at scale (350 students, with 170 used for analysis). The QA sensors are a sensible idea and, as far as related work shows, a genuine addition over AWARE, Beiwe, EARS, and similar. The iLogCal extension of iCal is clever—it gives researchers a standard way to express schedules and lets the dashboard reuse calendar views. The runtime plan revision mechanism, with the control hierarchy researcher/participant/platform, is a useful framing. The paper is also honest in places: Section 8 says validation in other domains remains future work.\n\nThe soft spots are real but localized. The ML section (Section 7) is the main one. The paper defines answer quality in Section 3 as an answer that is 'meaningfully provided and correct.' In Section 7, the target label becomes 'answered within 30 minutes,' with the threshold imported from reference [67] and no check that it tracks correctness in the Trento student data. The authors even mention a way to validate answers against sensors (e.g., comparing self-reported location with GPS) but do not use it here. So the 0.758 accuracy in Table 8 is about predicting response latency, not answer quality as defined. That overstatement should be fixed. The broader evaluation also lacks any control condition or comparison to a fixed schedule, so the improved-quality claim rests on plausibility rather than evidence. Those are fixable in revision—the platform and dataset are already there.\n\nMinor issues: the writing has scattered typos and the positioning against related work is sometimes hand-wavy, but those don't undercut the contribution.\n\nWho it's for: researchers building or choosing mobile sensing platforms for longitudinal personal data studies. It deserves a serious referee. I'd send it out, with the expectation that the authors either scale back the quality claim or add the missing evaluation: validate the 30-minute threshold on this population, add a baseline (e.g., fixed schedule or random timing), and report error analysis. I'd also ask them to make the platform artifacts and data-processing code public.","headline":"Solid systems paper with a genuinely useful platform, but the headline quality-improvement claim rests on an unvalidated proxy.","tokens_in":21226,"tokens_out":3509,"would_cite":true,"duration_ms":28132,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A platform that gives researchers, participants, and the app itself control over when and where data are collected can improve the quality and richness of combined sensor and questionnaire data.","keywords":["big-thick data","personal data collection","context-aware scheduling","situational context","temporal context","mobile sensing","answer quality prediction","experience sampling"],"falsifier":"Re-run the Section 7 analysis with a direct measure of answer correctness, for example validation questions with known right answers, and check whether high-quality predictions align with correct answers; if the 30-minute threshold fails to correlate with correctness in this dataset, the central empirical claim is refuted.","tokens_in":20189,"feed_emoji":"📱","tokens_out":7258,"duration_ms":59691,"temperature":0.7,"pith_summary":"The paper sets out to solve a known weakness in \"big-thick data\" collection: smartphones gather objective sensor streams easily, but the subjective questionnaire answers that give those streams meaning are often missing, rushed, or wrong. Its proposal is a methodology and platform, built on the existing iLog app, that gives the researcher, the participant, and the platform itself shared control over the data-collection process. Four mechanisms carry this: a five-dimension situational context model, a calendar-based scheduling language called iLogCal, a dashboard for real-time monitoring, and runtime plan revision. On a study of 350 university students, with 170 analyzed, the platform shows that contextual features predict whether an answer arrives within 30 minutes, the paper's proxy for answer quality, with up to 75.8 percent accuracy. A sympathetic reader would take the paper's central claim to be that placing these three actors in control of the collection context is what enables the step from raw sensor logs to high-quality personal data.","feed_headline":"Context-aware data collection predicts answer quality 76% of the time","feed_subtitle":"Giving researchers, participants, and the app control over when questions fire yields richer personal data.","key_machinery":"The load-bearing mechanism is the coupling of question-answering (QA) sensors with a scheduling language. QA sensors timestamp three events for every question, generation, delivery to the participant's device, and storage of the answer, which makes response time computable from platform data itself. iLogCal, an extension of the iCal/RFC5545 standard, expresses an experiment as calendars containing question and sensor collections whose activation is conditioned on situational context dimensions (WE, WA, WI, WO, WU) and on recurrence rules; this lets the researcher, participant, or platform decide when a question may fire. The situational context model represents each moment of a person's day as a tuple of five linguistic context components and connects them in a knowledge graph whose attributes and relations can be used as preconditions for questions or sensor tasks. The dashboard then makes this plan visible and editable in real time, closing the loop that the machine-learning component exploits.","core_discovery":"On the paper's own terms, the discovery is that the process of data collection can itself be treated as the object of data collection: by modeling the situational context (where, what, who with, internal state, tools) and the temporal context (scheduling constraints and execution annotations) of every question and sensor reading, the platform makes the conditions under which each answer was produced explicit and actionable. The empirical result is that these conditions predict answer quality, defined as answering within 30 minutes, better than chance: Random Forest reaches 0.758 accuracy, outperforming KNN, logistic regression, SVM, and Gaussian Naive Bayes, and per-participant accuracy ranges from about 65.9 to 88.6 percent. The authors conclude that the platform can learn when to ask a question so as to minimize reaction time and thereby improve the quality of the collected data.","pith_inferences":["The same QA-sensor pipeline could be reused outside research settings, e.g., in health or well-being apps, to time notifications or surveys when a user is most likely to respond carefully; this is an extension the paper does not develop.","Because per-participant prediction accuracy varies widely, from 65.9 to 88.6 percent, personalized models trained on an individual's own first weeks may outperform the global classifier; a direct comparison would be a natural next test.","The WU (tool/utensil) context dimension was not collected in the case study; adding it, or deriving it from sensors, could either sharpen the quality predictions or show that the five-dimension model is more than the four measured dimensions need.","If the 30-minute threshold is validated against true answer correctness on this population, the platform's quality metric could be used as a continuous monitoring signal, not just a post-hoc label."],"forward_implications":["An experiment plan written in iLogCal can be revised while it runs, by a participant within researcher-set bounds, by the researcher for one or many participants, or by the platform itself, so data collection can adapt to what is actually happening.","Because every answer carries a timestamp of when the question was generated, delivered, and answered, researchers can compute response and completion times for every question and use them as quality signals.","The dashboard gives researchers and participants live views of compliance, data quality, and answer patterns, allowing poor engagement to be caught and corrected during the experiment rather than after.","The machine-learning component, using temporal context, situational context, and demographics, can select moments likely to yield high-quality answers, with Random Forest reaching 0.758 accuracy.","Collected data become richer because objective sensor readings are joined with subjective answers and with metadata about the context in which each answer was given."],"supporting_citations":[{"why":"Earlier version of this methodology and system, supplying the baseline that the new platform extends.","marker":"[1]"},{"why":"Introduces the big-thick data notion that motivates combining sensor streams with qualitative answers.","marker":"[4]"},{"why":"Shows how context-aware personal data can be unified with environmental data to generate big-thick data, the goal the platform serves.","marker":"[11]"},{"why":"Describes the original iLog multi-device logging app on which the enhanced platform is built.","marker":"[20]"},{"why":"Documents the large-scale data-collection experiment whose methodology the authors fine-tune for the case study.","marker":"[23]"},{"why":"Provides the worldwide diversity pilot dataset and experiment that supply the 350-student case study.","marker":"[24]"},{"why":"Detailed description of the resulting dataset, including GDPR and ethics compliance, underpinning the empirical analyses.","marker":"[29]"},{"why":"Supplies the reaction-time/answer-quality relationship and the 30-minute threshold used as the quality-prediction label.","marker":"[67]"}],"fun_headline_variants":["Context-aware platform predicts answer speed with 76% accuracy","Modeling context of each question predicts answer quality at 76%","Predicting answer timeliness: context boosts accuracy to 76%","How context-aware timing improves data quality: 76% prediction","Context-driven scheduling predicts answer quality (76% accuracy)"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an answer given within 30 minutes is a high-quality answer, a threshold the paper adopts from earlier work rather than validating on this study's participants; if that proxy does not track actual answer correctness here, the quality-prediction claim loses its foundation even though the platform itself may work as described.","fun_headline_variants_meta":{"raw":{"variants":["Context-aware platform predicts answer speed with 76% accuracy","Modeling context of each question predicts answer quality at 76%","Predicting answer timeliness: context boosts accuracy to 76%","How context-aware timing improves data quality: 76% prediction","Context-driven scheduling predicts answer quality (76% accuracy)"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000456,"raw_usage":{"total_tokens":2321,"prompt_tokens":1005,"completion_tokens":1316,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":1231}},"tokens_in":621,"tokens_out":1316,"duration_ms":9446,"temperature":1.0,"reasoning_tokens":1231,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T10:04:26.717234+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Section 7 analysis with a direct measure of answer correctness, for example validation questions with known right answers, and check whether high-quality predictions align with correct answers; if the 30-minute threshold fails to correlate with correctness in this dataset, the central empirical claim is refuted.","supporting_citations":[{"cited_title":"A Methodology and System For Big-Thick Data Collection","cited_arxiv_id":"2404.17602","evidence_quote":"Earlier version of this methodology and system, supplying the baseline that the new platform extends."},{"cited_title":"Giunchiglia and X","cited_arxiv_id":null,"evidence_quote":"Shows how context-aware personal data can be unified with environmental data to generate big-thick data, the goal the platform serves."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the original iLog multi-device logging app on which the enhanced platform is built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the large-scale data-collection experiment whose methodology the authors fine-tune for the case study."},{"cited_title":"Giunchiglia, I","cited_arxiv_id":null,"evidence_quote":"Provides the worldwide diversity pilot dataset and experiment that supply the 350-student case study."},{"cited_title":"Busso, A","cited_arxiv_id":null,"evidence_quote":"Detailed description of the resulting dataset, including GDPR and ethics compliance, underpinning the empirical analyses."}],"review_version":1}