{"id":"ea561f8a-5f6d-47b8-8b62-7b2a70fa86cd","arxiv_id":"2511.23276","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An LLM-based two-agent system can forecast hand-foot-mouth disease cases with accuracy comparable to top numerical models while generating human-readable risk explanations.","lead":"This paper tests an AI system that uses one large language model to read news, weather, and school calendars into a single risk score, and a second model to turn that score plus past case counts into weekly hand-foot-mouth forecasts with explanations. On two data sets it reports accuracy close to the best numerical forecasters, though one headline accuracy claim is contradicted by its own table.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'robust 90% intervals' claim rests on LLM self-reported uncertainty u; without calibration or repeated runs, high coverage is just interval-widening, not validated probabilistic forecasting.","rationale":"I read the paper in good faith. The two-agent architecture is coherent, the point-accuracy results are plausible, and the Appendix B.4 failure case is an honest limitation. However, the central claim as stated in the abstract — 'robust 90% intervals (coverage approximately 0.85–1.00)' — has a load-bearing soft spot: the prediction interval is not a fitted or calibrated statistical object. It is a deterministic transformation of the LLM's self-reported uncertainty u, so the reported coverage numbers cannot by themselves validate the method. The fact that the highest-coverage configuration also has worse CRPS than both Moirai and Chronos suggests the intervals may be simply too wide. The proposed experiment (fixing u=0 or calibrating u on a holdout) would settle whether the coverage claim is an artifact. If it lands, the abstract should be softened from 'robust' to 'high coverage on two datasets under one uncalibrated configuration'; the point-accuracy and interpretability claims may remain. This does not change the reader's CONDITIONAL verdict.","tokens_in":19035,"tokens_out":7101,"duration_ms":68260,"concrete_test":"Re-run the full rolling-origin evaluation on both datasets with u_{t+k} fixed to 0 and, separately, with u_{t+k} calibrated on the first half of the evaluation window (e.g., choose a constant scaling so that empirical coverage on that half is 0.90), then report coverage and CRPS on the second half. If coverage stays near 0.90 with comparable or better CRPS, the intervals do not depend on unvalidated self-assessment; if coverage collapses or CRPS improves dramatically, the claimed coverage is an artifact of LLM width inflation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.3.3 builds the 90% interval from σ² = (μ·v·(1+u))², where u is the LLM's self-declared uncertainty score. Nothing in the pipeline calibrates u to observed coverage; coverage is then measured on the same data used to generate the forecasts. On Lishui, the Qwen variant reports coverage 1.000 with CRPS 2.319, while Moirai has CRPS 1.785 with coverage 0.794; on Hong Kong, NSF-LLM CRPS values (2.08–2.40) are worse than Chronos (1.71) and LSTM (1.79). This pattern is exactly what one would expect from intervals made wider by arbitrary u, not from a calibrated predictive distribution. The appendix's single-run outputs and API temperatures further mean that no repeated-run error bars support the MAE/coverage numbers. If u is merely a dial the LLM turns up when unsure, the headline 'robust 90% intervals' is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-agent neuro-symbolic framework for weekly HFMD forecasting. Agent 1, an LLM-based Event Interpreter, ingests heterogeneous context (school calendars, weather, government reports, retrieved guidelines) and outputs a scalar transmission-impact score I_t along with a confidence score and natural-language rationale. Agent 2, an LLM-based Forecast Generator, combines I_t with recent case history and an estimated volatility v to produce point forecasts, which are mapped to negative-binomial/90% prediction intervals via moment matching σ² = (μ·v·(1+u))², where u is the LLM's self-declared uncertainty. The model is evaluated on Lishui (33 weeks) and Hong Kong (90 weeks) against classical, ML, LLM, and time-series foundation-model baselines. The central claims are that the approach achieves competitive point accuracy while providing robust 90% intervals and human-readable rationales, demonstrating that LLM-based context interpretation can match strong numerical forecasters.","tokens_in":19350,"tokens_out":5191,"duration_ms":52631,"significance":"If the claims are fully supported, the paper offers a useful modular design for incorporating unstructured domain knowledge into epidemiological forecasting while retaining a probabilistic output and an auditable explanation layer. The manuscript has concrete strengths: public code, full prompt templates in the appendix, two real datasets with different characteristics, and ablations isolating school calendars, climate, and RAG retrieval. However, the probabilistic-calibration claim is currently not established: the 90% intervals are driven by an uncalibrated LLM self-reported uncertainty, and the point-accuracy claim is overstated in the text relative to the reported tables. The contribution is still valuable as a qualitative-context injection mechanism, but the headline robustness claim needs recalibration or reframing.","major_comments":[{"comment":"The text states that the Qwen3-based NSF-LLM 'achieves the best overall point-forecast accuracy, attaining the lowest MAE and RMSE among all models.' Table 2 directly contradicts this: TimesFM has MAE 3.972 vs. 4.124 for Qwen NSF-LLM, and Chronos/Moirai have lower CRPS (2.105/1.785 vs. 2.319). On Hong Kong (Table 3), Prophet/XGBoost/ARIMA/LSTM have MAE 3.49–3.53 vs. 3.49–4.32 for NSF-LLM variants. The 'lowest' claim should be replaced with 'competitive' and any remaining differences should be qualified with variance estimates given single-run evaluations.","section":"§3.3.1, Table 2"},{"comment":"The 90% interval width is a multiplicative function of the LLM's self-reported uncertainty u and a hand-clipped volatility v, with no recalibration step. Coverage is then measured on the same data used to generate the forecasts. The pattern in Table 2—Qwen NSF-LLM coverage 1.000 with CRPS 2.319, while Moirai has coverage 0.794 and CRPS 1.785—is exactly what one would expect from interval widening rather than calibrated predictive distributions. The abstract's 'robust 90% intervals (coverage approximately 0.85–1.00)' is therefore not supported. The authors should either calibrate u (e.g., conformal prediction, isotonic regression, or temperature scaling) or explicitly state that the intervals are conditional on LLM self-assessed uncertainty and not statistically calibrated.","section":"§2.3.3, Eq. (σ² = (μ·v·(1+u))²)"},{"comment":"All experiments appear to be single-run rolling forecasts with provider-specific temperatures (0.6, 1.0, 0.7) and no repeated sampling. The reported MAE differences are often small (e.g., 4.124 vs. 3.972 on Lishui), so without repeated runs or seed/temperature sensitivity analysis, the 'competitive' point-accuracy claim is not distinguishable from LLM stochasticity. The authors should report means/standard deviations over at least several repeated runs, or justify why single runs are sufficient for their API configuration.","section":"§2.6.3, Tables 2–3"},{"comment":"The baseline list includes context-free LLM forecasters (Qwen3, GPT-5.1, Gemini Pro, DeepSeek-V3), but none of these are reported in Tables 2 or 3. The 'No-Agent1' ablation is not equivalent because the Forecast Generator still receives the full history and a different prompt. Without the context-free LLM results, the manuscript cannot isolate whether Agent 1's context interpretation is what improves over a pure LLM time-series forecaster. These baselines should be added to the tables or explicitly reported in an appendix.","section":"§3.2.2, Tables 2–3"}],"minor_comments":[{"comment":"Typo: 'V olatility' should be 'Volatility'.","section":"§2.3.1"},{"comment":"The sentence 'Lower CRPS values from Moirai and the proposed Qwen-based framework' is misleading on Lishui: Qwen NSF-LLM CRPS 2.319 is higher than Chronos 2.105 and Moirai 1.785. Clarify that the Qwen variant has competitive but not lowest CRPS.","section":"§4.4"},{"comment":"Notation is inconsistent: §2.3.2 says the LLM outputs a 'median forecast' and 'uncertainty score u', while the prompt in A.4 asks for 'forecast_mean' and 'uncertainty_scale'. Reconcile the terminology.","section":"§2.3.2 vs Appendix A.4"},{"comment":"The temperature is stated as 0.6 in §2.2.4, but §2.6.3 gives provider-specific temperatures (0.6 for Qwen3, 1.0 for DeepSeek, 0.7 for GPT-5.1). Clarify which temperature applies to which agent and whether Agent 2 uses a different value (the setup text mentions 0.2 elsewhere).","section":"§2.2.4 vs §2.6.3"},{"comment":"The figures are referenced as trajectory plots but not discussed panel-by-panel; a short pointer to which panels illustrate peak-tracking or over-smoothing would help.","section":"Figures 2–3"},{"comment":"The repeated 'PRIME AI paper' header appears to be a template artifact and should be removed. The GitHub link should be verified before final submission.","section":"Various"}],"recommendation":"major_revision","confidential_remarks":"The interval-calibration issue is the main barrier to acceptance: the 'robust 90% intervals' claim is not established by the current evaluation. Point accuracy may be competitive, but the numerical evidence in Table 2 does not support the 'lowest MAE/RMSE' statement. I would ask for recalibration or an explicit weakening of the interval claim, plus repeated-run error bars and the missing context-free LLM baselines. These are fixable within the manuscript's scope, hence major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the two-agent pipeline is a reasonable design and the HFMD application is new, but the paper oversells its probabilistic claims and contains a direct factual contradiction in the results. It deserves a serious referee, not a desk reject — but the evaluation needs real work before the headline numbers can be trusted.\n\nThe architecture is worth a look. Agent 1 compressing heterogeneous context (school calendar, weather, government bulletins) into a scalar transmission-impact score, and Agent 2 turning that plus recent case counts into a negative-binomial forecast, is a clean modular decomposition. The Lishui and Hong Kong HFMD evaluations are new, the ablations sensibly separate context channels, and the appendix prompt templates plus the honest failure case show the authors care about reproducibility.\n\nBut there are two problems that need stating plainly. First, Section 3.3.1 claims the Qwen3-based framework achieves the lowest MAE and RMSE among all models, while Table 2 shows TimesFM at 3.972 vs. 4.124 for the Qwen3 system. That's not a subtle mismatch; it's a direct contradiction in the central results narrative. Second, the 90% intervals are built from sigma-squared = (mu * v * (1+u))^2, where u is the LLM's own self-declared uncertainty score, never calibrated against observed coverage. On Lishui, the Qwen variant gets 1.000 coverage while Moirai gets 0.794 — but Moirai's CRPS is 1.785 vs. 2.319. That pattern is exactly what you'd expect from a system that widens intervals rather than modeling the predictive distribution. So the \"robust 90% intervals\" claim is not established by the evidence as presented.\n\nOther soft spots: no repeated runs or error bars — every number comes from a single API pass at nonzero temperature — and the \"first LLM-agentic epidemic forecasting\" claim ignores a growing literature. The paper should have engaged with recent LLM-for-time-series work, not just cited Chronos and TimesFM. On the plus side, the point accuracy is genuinely competitive (4.12 vs. 3.97 on Lishui; 3.49–3.95 on Hong Kong, where the GPT variant ties Prophet and XGBoost), and the rationales do add something for public-health consumption.\n\nNone of this kills the core idea. The decoupling of context interpretation from numerical forecasting is worth pursuing. But the probabilistic claims need a calibration study and seeded/repeated runs before I'd trust them. My recommendation: send to peer review, with an explicit note that the overclaim in 3.3.1 and the uncalibrated u must be fixed. I'd bring this to a reading group as a case study in how LLM self-reported uncertainty can masquerade as statistical calibration.","headline":"Nice architectural idea, shaky uncertainty claims — the coverage is likely an artifact of LLM self-assessed width, and one key result contradicts its own table.","tokens_in":19810,"tokens_out":2719,"would_cite":false,"duration_ms":27615,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM that reads school calendars, weather, and official reports into a single transmission-impact number can match strong numerical forecasters on weekly HFMD point accuracy while adding calibrated prediction intervals and human-readable","keywords":["hand-foot-and-mouth disease","epidemic forecasting","large language models","neuro-symbolic agents","time-series forecasting","prediction intervals","negative binomial calibration","retrieval-augmented generation"],"falsifier":"Run the pipeline on the Lishui data twice with identical case counts, once with the real school calendar and once with the calendar shifted four weeks later; if point error and interval coverage do not materially change, the context channel is not carrying epidemiological information. A second check is a long held-out calibration: if empirical 90% coverage diverges sharply from the nominal level when the LLM's self-declared uncertainty is high, the variance formula is not a valid uncertainty model.","tokens_in":18930,"feed_emoji":"🦠","tokens_out":6711,"duration_ms":64285,"temperature":0.7,"pith_summary":"The paper is trying to establish that epidemic forecasting does not have to trade accuracy for interpretability. It proposes a two-agent system in which one large-language-model agent converts heterogeneous, mostly unstructured context — school schedules, weather summaries, government bulletins, and clinical guidelines — into a scalar transmission-impact score, and a second forecasting stage combines that score with historical case counts to issue probabilistic one-week-ahead forecasts. On hand-foot-and-mouth disease data from Hong Kong and a hospital in Lishui, the system reports point accuracy on par with strong classical and pretrained time-series baselines, 90% prediction-interval coverage in the 0.85–1.00 range, and concise narrative rationales for each forecast. The practical claim is that public-health teams can act on forecasts that say not just what will happen but why — and that new context can be folded in at inference time without retraining.","feed_headline":"LLM context agents match numeric forecasters on HFMD accuracy","feed_subtitle":"Weekly hand-foot-mouth forecasts pair competitive point error with 90% intervals and human-readable risk rationales.","key_machinery":"The load-bearing object is the Transmission Impact Score, a single number in [-1,1] produced by the LLM Event Interpreter that summarizes the net expected effect of all external context on HFMD transmission in the coming week, with a confidence score attached. The Forecast Generator then maps that score plus recent counts and an estimated volatility into a median forecast and an uncertainty score, and the pair is converted into a negative-binomial predictive distribution by moment matching, using variance constructed as the square of the product of mean, volatility, and one-plus-uncertainty. The 90% interval is read from the inverse CDF of that distribution. The score is what carries context","core_discovery":"On the paper's terms, the central discovery is that a neuro-symbolic division of labor works: let an LLM event interpreter read qualitative signals and compress them into a bounded transmission-impact score, and let a separate probabilistic forecaster consume that score alongside recent case counts. The authors show that this architecture matches or nearly matches the best numerical-only models on mean absolute error while producing substantially better-calibrated 90% intervals than classical statistical approaches, and it supplies an auditable rationale for every adjustment. The interpreter operates at inference time, so a new school closure or policy announcement can change the forecast im","pith_inferences":["Beyond the paper: the same two-agent template likely transfers to other environmentally sensitive infections such as influenza or dengue, since it only needs a disease-specific knowledge base and lag policy; the scalar interface keeps the forecasting core untouched.","Beyond the paper: the framework's advantage may be largest in short, irregular series like Lishui, where classical models under-cover and foundation models smooth peaks; a natural test is a multi-year rolling evaluation to see whether the coverage advantage persists once seasonal structure is well learned.","Beyond the paper: the 0.85–1.00 coverage hangs on the LLM's self-reported uncertainty being informative; an independent conformal calibration on held-out weeks would tell whether the intervals are statistically valid or partly an artifact of the moment-matching formula.","Beyond the paper: since each forecast costs several LLM calls, deployment would likely need a distilled, cheaper interpreter that approximates the transmission-impact score; the scalar target makes such distillation straightforward."],"forward_implications":["Weekly HFMD forecasts can come with calibrated 90% intervals and a short rationale naming the dominant driver (school term, weather, policy), which is directly useful for hospital-capacity and public-health planning.","New contextual information — a closure, a vaccination campaign, an extreme-weather event — can change the forecast immediately at inference time, with no retraining, because the interpreter runs at query time.","Ablations indicate school-calendar status is the strongest short-term driver in the Lishui data: removing it raises mean absolute error from about 4.1 to 6.7, implying systems that ignore school calendars will miss post-holiday surges.","Grounding the interpreter in retrieved official guidelines stabilizes its reasoning; without retrieval, it becomes cautious and underestimates peaks, so the retrieval step is part of the mechanism rather than a wrapper.","Because the two agents communicate through a single scalar, either side can be upgraded independently — a different language model or a different numerical forecaster — without rearchitecting the system."],"fun_headline_variants":["LLM context agents give HFMD forecasts better 90% intervals","Hybrid LLM forecaster improves calibration while matching accuracy","Auditable HFMD forecasts: LLM agents, numeric-grade accuracy","LLM event interpreters boost HFMD forecast trust and calibration"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the LLM's scalar transmission-impact score and its self-declared uncertainty faithfully encode real epidemiological drivers and calibrated uncertainty — if the score merely echoes hints in the prompt and the uncertainty is uncalibrated, the context channel adds no genuine forecasting value and the interval coverage numbers become artifacts.","fun_headline_variants_meta":{"raw":{"variants":["LLM context agents give HFMD forecasts better 90% intervals","Hybrid LLM forecaster improves calibration while matching accuracy","Auditable HFMD forecasts: LLM agents, numeric-grade accuracy","LLM event interpreters boost HFMD forecast trust and calibration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00153,"raw_usage":{"total_tokens":5992,"prompt_tokens":808,"completion_tokens":5184,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":5113}},"tokens_in":552,"tokens_out":5184,"duration_ms":34111,"temperature":1.0,"reasoning_tokens":5113,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T19:32:31.308818+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the pipeline on the Lishui data twice with identical case counts, once with the real school calendar and once with the calendar shifted four weeks later; if point error and interval coverage do not materially change, the context channel is not carrying epidemiological information. A second check is a long held-out calibration: if empirical 90% coverage diverges sharply from the nominal level when the LLM's self-declared uncertainty is high, the variance formula is not a valid uncertainty model.","supporting_citations":[],"review_version":1}