{"id":"ece35399-f94c-4321-9682-0acdb3f4dbcf","arxiv_id":"2501.13723","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An LLM pipeline using LLMOps tooling classified exercise posts, predicted durations, and estimated calories on a 133-member Facebook exercise community, reporting 95-96% classification accuracy and 72-89% accuracy for the other two tasks under a loosely defined tolerance.","lead":"This paper reports an LLM-based pipeline that automatically reads exercise posts in a small social fitness community, extracts exercise type, duration and calories, and writes personalized feedback. On 741 posts it reports over 95% exercise classification accuracy, with duration and calorie estimates between roughly 72% and 89% under a loosely defined tolerance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported duration and calorie accuracies are uninterpretable because the '1 unit' tolerance and target units are never defined, and the ground-truth counts conflict across tables.","rationale":"The reader's weakest assumption correctly identifies the ground-truth reliability and the one-unit tolerance as load-bearing. I agree that these are the central weaknesses, but I would sharpen the critique: the tolerance problem is not just a missing justification; the unit for calories is never defined at all, and Section 6.4.5 admits that total vs. active calorie confusion exists in the data. That makes the calorie accuracy figure especially ill-posed. Additionally, the data-count discrepancy between Table 7 (474/476) and Tables 10/11 (499/500) suggests an undocumented filtering step, which is a concrete, checkable inconsistency. The reader's recommendation of CONDITIONAL remains appropriate: the system's qualitative feasibility as an LLMOps pipeline is plausible, and the classification accuracy is likely reasonably high, but the quantitative claims need to be re-evaluated with defined units, inter-annotator agreement, and repeated runs. No change to the reader's verdict is needed, though the paper should be revised to address these issues before the numbers are taken at face value.","tokens_in":15204,"tokens_out":3541,"duration_ms":36406,"concrete_test":"Have a second annotator independently label a random 100-post subset from the 741 posts, compute Cohen's kappa for the is_exercise ground truth, and then re-run the full LLM evaluation pipeline with explicitly defined units and tolerances (e.g., duration in minutes with exact, ±5, ±10 minute tolerances; calories in kcal with exact, ±10, ±50 kcal tolerances) reporting MAE and RMSE in addition to binary accuracy. If the reported >95%, >86%, and >72% figures change by more than a few points, or if inter-annotator agreement is low, the conclusion's quantitative claims should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim in Section 7 is that the system achieved >95% classification, >86% duration, and >72% calorie accuracy. All three numbers depend on Section 6.3.1, which states that for duration and calories, predictions within 'a margin of error of 1 unit' are counted as correct. The unit is never defined. Section 4.3.1 says duration output should be 'in minutes', so the duration tolerance may be 1 minute, but the calorie unit is entirely unspecified. Section 6.4.5 even acknowledges confusion between total and active calories, so the ground-truth value for calories is not a well-defined quantity. With a 1-kcal tolerance, >72% accuracy would be surprisingly high; with a 1-unit tolerance of, say, 10 kcal, the number is much less meaningful. This makes the headline accuracies non-interpretable as stated. A second issue is data inconsistency: Table 7 reports 474 duration and 476 calorie ground-truth entries, while Tables 10 and 11 evaluate on 499 and 500 entries, with no explanation of how the subsets were chosen. Classification uses all 741 posts, but the duration and calorie evaluation subsets are uncharacterized, so selection bias cannot be ruled out. Finally, the manual ground truth is described as 'meticulously conducted' but no inter-annotator reliability is reported, and the LLM temperature is left at default 0.7, so stochastic sampling could change results run to run without repeated trials or confidence intervals. These issues do not disprove the system's usefulness, but they mean the specific accuracy claims in the conclusion are not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes an LLMOps-based system that collects exercise posts from the Ounwan Facebook community, analyzes them with LLMs to classify exercise activity, estimate duration, and estimate calorie expenditure, and generates personalized feedback. The implementation uses containerized modules, LangChain, CI/CD, and multiple LLM providers (OpenAI gpt-4o-mini, Azure OpenAI gpt-4o-mini, and Amazon Bedrock Claude 3.5 Sonnet). The central claim, stated in Section 7, is that the system achieves >95% exercise classification accuracy, >86% duration prediction accuracy, and >72% caloric expenditure estimation accuracy. The evaluation compares LLM outputs against manually labeled ground truth on 741 posts.","tokens_in":15452,"tokens_out":1984,"duration_ms":21190,"significance":"If the reported accuracies hold, the paper would demonstrate a practical use of LLMOps for automating community health moderation and personalized fitness feedback, with a useful multi-provider comparison and an operational architecture (containerization, CI/CD, monitoring) that goes beyond a single-model proof of concept. The paper also makes a concrete methodological contribution by applying LLMs to obfuscated date extraction and to multimodal exercise-post analysis. However, the headline quantitative claims are not yet interpretable because the evaluation metric for duration and calories uses an undefined tolerance and unit, and because the evaluation subsets are not reconciled with the reported ground-truth counts. The significance is therefore conditional on fixing these measurement and reporting issues.","major_comments":[{"comment":"The metric definition is load-bearing: the paper states that duration and calorie predictions within 'a margin of error of 1 unit' are counted as correct, but the unit is never defined. Section 4.3.1 says duration should be output 'in minutes,' which suggests a 1-minute tolerance, but no analogous statement exists for calories. Section 6.4.5 further acknowledges confusion between total calories and active calories, so the ground-truth calorie value is not a well-defined target. Without specifying the calorie unit and the tolerance, the >86% duration and >72% calorie accuracies in Section 7 are not interpretable. Please define the units, justify the tolerance (e.g., against expected device-level variation), and report sensitivity to the tolerance choice.","section":"§6.3.1"},{"comment":"The ground-truth counts are internally inconsistent: Table 7 reports 474 duration entries and 476 calorie entries, while Tables 10 and 11 evaluate on 499 and 500 entries, respectively, and Table 9 uses 739–741 posts for classification. The paper does not explain how the duration and calorie evaluation subsets were selected or why they differ from the ground-truth counts. Because the subsets are uncharacterized, selection bias cannot be ruled out. Please reconcile these numbers and describe the exact inclusion/exclusion criteria for each evaluation, ideally reporting results on the full ground-truth set.","section":"Table 7 vs. Tables 10–11"},{"comment":"The manual ground truth is described as 'meticulously conducted' but no inter-annotator reliability is reported. Since the accuracy numbers treat this ground truth as error-free, the absence of agreement statistics for exercise classification, duration, and calorie labels is a significant gap. Please report dual annotation or adjudication procedures and inter-annotator agreement (e.g., Cohen's kappa for classification and intraclass correlation for duration/calories).","section":"§6.1 and §6.2.3"},{"comment":"The LLM temperature is stated as the default value of 0.7, but the evaluation appears to consist of a single run per model. Because LLM outputs are stochastic at this temperature, the reported accuracies, precisions, and AUPRC values could vary across runs. Please report results over multiple runs with confidence intervals, or justify the use of a single run (e.g., by demonstrating low variance).","section":"§6.2.4"}],"minor_comments":[{"comment":"The sentence 'In the thesis, the data collection module was implemented...' at the end of the Dockerfile description is out of place for a journal article; consider rewording to refer to 'this study' or 'our implementation.'","section":"Section 3.3.1"},{"comment":"The table headers for the evaluation count columns are formatted awkwardly ('Number of data Evaluation Count TP TN FP FN'), making it hard to distinguish the total data count from the TP/TN/FP/FN columns. Please restructure the tables to have clear column labels.","section":"Tables 9–11"},{"comment":"The equations for accuracy, precision, recall, and AUPRC are referenced but not fully displayed in the provided text; please ensure the metric formulas are explicitly stated so that the thresholding for TP/TN/FP/FN in the duration and calorie tasks is unambiguous.","section":"§6.3.1"},{"comment":"For the duration and calorie tasks, precision and recall values are reported using TP/TN/FP/FN, but the precision-recall curves in Figure 11 appear to refer to the classification task only. Please clarify how AUPRC is defined for the regression-based duration and calorie tasks, since a single 1-unit tolerance not correspond to the full PR curve.","section":"§6.3.3–6.3.4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a genuine engineering deployment, not a scientific benchmark, and the paper's own limitations section is unusually honest. But the headline numbers in the conclusion (86% duration, 72% calorie) are not yet defensible because the evaluation metric is under-specified.\n\nWhat's actually new: using commercial LLM APIs through LangChain to extract exercise classification, duration, and calories from social posts, with OCR to handle obfuscated date fields, and comparing three providers. That's a legitimate applied contribution. The system is deployed in a real community of 133 users with 741 posts, and the authors report the challenges they hit (multi-exercise posts, temporal aggregation, total-vs-active calorie confusion). The >95% classification accuracy is plausible and robust across providers.\n\nThe soft spots are real and match the stress-test. Section 6.3.1 says predictions within 'a margin of error of 1 unit' count as correct, and the unit is never defined. For duration it's probably minutes given the prompt says 'output in minutes,' but calories are ambiguous, and Section 6.4.5 admits the ground truth itself mixes total and active calories. So the >72% calorie accuracy is not interpretable. Also, Table 7 lists 474 duration and 476 calorie ground truth entries, but Tables 10 and 11 evaluate on 499 and 500 entries. That's an internal inconsistency that needs explanation. No inter-annotator reliability is reported, and with temperature fixed at 0.7, single runs don't tell us variance.\n\nThese flaws don't sink the core feasibility claim, but they mean the specific accuracy numbers should be treated as preliminary, not conclusions. The authors should be asked to define the tolerance, fix the counts, report repeated runs and annotation stats, and soften the health-outcome language in the conclusion.\n\nI'd send this to peer review for an applied venue; it's exactly the kind of systems paper that referees can help fix. I would not cite it for its numbers, but it is a fair example of LLMOps evaluation pitfalls.","headline":"A plausible LLMOps engineering report whose headline accuracy numbers for duration and calories are not interpretable because the tolerance unit is undefined and the ground-truth counts don't match.","tokens_in":16019,"tokens_out":2135,"would_cite":false,"duration_ms":20561,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A production LLMOps pipeline can automate exercise analysis and feedback for a social health community, achieving above 95% classification, above 86% duration prediction, and above 72% calorie estimation accuracy.","keywords":["healthcare platforms","exercise analysis","personalized recommendations","automated feedback","social fitness communities","large language models","LLMOps","containerized infrastructure"],"falsifier":"Take the same 741 posts, have two or more independent annotators relabel them, define calorie error in kilocalories with a pre-specified tolerance, and recompute accuracy; if the new numbers fall below 95%, 86%, and 72%, the claimed robustness does not hold.","tokens_in":14965,"feed_emoji":"🏃","tokens_out":5981,"duration_ms":58334,"temperature":0.7,"pith_summary":"An LLMOps pipeline—operational practices for running large language models in production—can automate the analysis of free-form exercise posts and the delivery of personalized feedback in a social healthcare community. The paper reports on 741 posts from the Ounwan exercise community and claims that across three commercial LLM providers, the system classifies whether a post is exercise with over 95% accuracy, predicts exercise duration with over 86% accuracy, and estimates caloric expenditure with over 72% accuracy. The point of the system is to remove the manual analysis and generic advice bottleneck that limits current health platforms: collection, datafication, analysis, and feedback run automatically inside a containerized, CI/CD-managed loop. If these accuracy levels hold, community moderators can shift from reading every post to reviewing only edge cases, and users can receive timely, individualized fitness coaching.","feed_headline":"LLM system reads exercise posts with over 95% accuracy","feed_subtitle":"On 741 community posts, the same pipeline predicts workout duration and calorie burn, then posts personalized feedback.","key_machinery":"The load-bearing mechanism is a closed LLMOps loop built from containerized modules: a daily data-collection crawler, an exercise-datafication step that uses OCR plus an LLM to read obfuscated date fields, an analysis module that sends text and screenshot images in parallel to several LLMs through a common comparator interface, and a feedback-generation module that posts personalized replies. The comparator and batch-processing logic standardize prompts, JSON outputs, token and cost tracking, and error handling, while CI/CD and container orchestration let the same pipeline be redeployed and re-evaluated nightly against ground-truth labels.","core_discovery":"The paper's central claim is that a production-style LLMOps pipeline can take unstructured exercise posts from a social fitness community and automatically extract three standardized metrics—whether the post is exercise, how long the exercise lasted, and how many calories were burned—accurately enough to replace manual moderation. On 741 posts from the Ounwan community collected over about five months, the best performing model classified exercise posts with 96.5% accuracy, predicted duration within one unit with 89.0% accuracy, and estimated calories with 77.2% accuracy, with all three tested LLM providers staying above 95%, 86%, and 72% respectively. The paper further claims the containerized, CI/CD-driven architecture converts these predictions into personalized trainer-style feedback posted back to the community, closing a feedback loop that manual analysis cannot sustain at scale.","pith_inferences":["The unspecified calorie unit and the one-unit tolerance mean the absolute accuracy numbers are not directly comparable to published energy-expenditure benchmarks; a clearer error metric is needed before these figures generalize.","The same architecture likely transfers to other text-plus-screenshot communities, but the 95% classification result depends on the specific post format and prompt style, so a new community would need its own small labeled set for validation.","A deterministic post-processing rule for resolving active versus total calories could raise calorie accuracy more than additional prompt tuning, since the paper identifies that ambiguity as a main failure mode.","Replacing the single-label ground truth with multiple annotators would test whether the reported accuracy ceiling is the model's limit or the label noise floor."],"forward_implications":["If the reported accuracies hold, a health community can run automatic triage on exercise posts, so that more than 95% classification accuracy leaves only borderline posts for human review.","Duration prediction above 86% within one unit is enough to generate weekly summaries, streaks, and progress tracking without manual data entry.","Calorie estimation above 72% supports trend-level feedback and motivation, though not precise clinical prescription.","Because the analysis layer is provider-agnostic, the system can swap or ensemble LLMs as models improve without rebuilding the pipeline.","Personalized feedback can be generated and posted automatically, removing the main bottleneck that delays responses in volunteer-run communities."],"supporting_citations":[{"why":"Provides the LLMOps-driven personalized recommendation approach that this work extends to exercise posts.","marker":"[3]"},{"why":"Defines LLMOps practices and the enterprise rationale for managing LLM applications in production.","marker":"[4]"},{"why":"Supplies the MAPE-K feedback-control loop that the monitoring-analysis-feedback architecture mirrors.","marker":"[16]"},{"why":"Supports the use of LLM-based autonomic loops that justify replacing the classic plan-execute cycle.","marker":"[20]"},{"why":"Documents the LLM orchestration framework used to issue parallel model queries and track usage.","marker":"[26]"}],"fun_headline_variants":["LLMOps turns fitness posts into auto feedback with 96% accuracy","AI reads fitness posts, predicts duration and calories for auto replies","Automated exercise feedback from LLM with 96% classification accuracy","LLMOps pipeline gives social fitness communities accurate auto-feedback","Smart LLM system analyzes fitness posts and gives accurate feedback"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported percentages assume the manually created ground-truth labels are accurate and that counting any duration or calorie prediction within one unspecified unit of the label as correct is a fair measure, but the paper gives no inter-annotator agreement and no calorie unit.","fun_headline_variants_meta":{"raw":{"variants":["LLMOps turns fitness posts into auto feedback with 96% accuracy","AI reads fitness posts, predicts duration and calories for auto replies","Automated exercise feedback from LLM with 96% classification accuracy","LLMOps pipeline gives social fitness communities accurate auto-feedback","Smart LLM system analyzes fitness posts and gives accurate feedback"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001139,"raw_usage":{"total_tokens":4690,"prompt_tokens":865,"completion_tokens":3825,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":3737}},"tokens_in":481,"tokens_out":3825,"duration_ms":27924,"temperature":1.0,"reasoning_tokens":3737,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:48:49.513354+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 741 posts, have two or more independent annotators relabel them, define calorie error in kilocalories with a pre-specified tolerance, and recompute accuracy; if the new numbers fall below 95%, 86%, and 72%, the claimed robustness does not hold.","supporting_citations":[{"cited_title":"Maximizing User Experience with LLMOps-Driven Personalized Recommendation Systems","cited_arxiv_id":"2404.00903","evidence_quote":"Provides the LLMOps-driven personalized recommendation approach that this work extends to exercise posts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines LLMOps practices and the enterprise rationale for managing LLM applications in production."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the LLM orchestration framework used to issue parallel model queries and track usage."}],"review_version":1}