{"id":"897542e8-2237-48a5-a7d9-686cd1765cf7","arxiv_id":"2509.09683","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Adding RL-fine-tuned LLM summaries of campaign change logs to a numeric click forecaster improves MAE and RMSE on two held-out ad campaigns and produces interpretable reasoning.","lead":"An advertising forecasting system combines click numbers with text descriptions of campaign changes, using a reinforcement-learning-tuned language model to produce both predictions and explanations. The authors report better click forecasts and more readable reasoning than numeric-only and GPT-4o-based alternatives on their private ad campaign dataset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claim that change-log text drives forecasting gains is untested: the RL-finetuned LLM may just re-encode the same numeric series with target supervision; no numeric-only ablation is provided.","rationale":"I agree with the reader's weakest assumption. The most load-bearing condition for the paper's main claim is that the textual summaries contribute genuine semantic signal from change logs, not merely a target-predictive recoding of the numeric series the transformer already sees. The reward function's dependence on ground-truth trend makes this concern concrete: Eq. (1) directly trains the LLM to output the binary trend and sentiment aligned with the label, and that binary outcome is embedded into the text representation. The absence of an ablation where the LLM sees no change logs makes it impossible to attribute the Table 2 improvement to the text modality. The small test set (2 campaigns) further weakens statistical inference. I do not see internal inconsistency; the concern is about evidential support, not soundness. The reader's CONDITIONAL verdict is appropriate: add the numeric-only LLM ablation and ideally more held-out campaigns. I would not reject the paper because Table 2 does show a consistent (if small) improvement and the reasoning-quality evaluation is a separate contribution. The 'first to incorporate textual reasoning' novelty claim should also be softened.","tokens_in":6528,"tokens_out":5484,"duration_ms":63647,"concrete_test":"Retrain the LLM with the identical GRPO reward and pipeline but with the change-log field removed from the prompt (numeric rolling averages, ad type, bid strategy only). Generate summaries, embed, and run the same TSF fusion on the same two held-out campaigns. If the resulting MAE/RMSE are statistically indistinguishable from the reported 4.948/7.670 (e.g., within the 3-seed std), the multimodal claim fails: the text path is just a supervised re-encoding of numeric input. Also report per-campaign errors and a significance test across a larger held-out set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that combining RL-finetuned textual summaries with numeric clicks improves forecasts—rests on the assumption that the text path contributes information beyond the numeric series. This is not established. In Eq. (1), the reward includes I(yhat=y) plus a sentiment-alignment term I(s(r)=y)*c, so the LLM is explicitly optimized to produce summaries whose one-word prediction and sentiment match the ground-truth trend. At inference, the LLM prompt (Fig. 2) contains the same 14-day rolling-average click values fed to the numeric transformer, along with change logs. Therefore the XLM-R embedding of the summary (which includes '<Prediction> Increase/Decrease') is a learned, target-supervised function of the numeric input; the MLP projection can recover trend information from this embedding without relying on change-log text. The 'Multi + changelog' baseline, which uses raw change-log text, is worse than the numeric-only Uni baseline (MAE 5.379 vs 5.102), suggesting raw text is not informative on its own. The reported gain of Ours over Uni (MAE 4.948 vs 5.102) could be due to the extra supervised representation of the same numeric signal, not to multimodal fusion. The paper reports no ablation with the LLM prompted without change logs, and no text-free control. Test data is only two held-out campaigns (1,045 points), so effect size and generalizability are also fragile.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multimodal click forecasting pipeline for digital advertising. A Qwen2.5-3B LLM is fine-tuned with GRPO using a reward that combines format compliance, one-word trend accuracy, and sentiment alignment (Eq. 1); its outputs are embedded with XLM-R and fused via a projection MLP with a transformer numeric forecaster (alpha = 0.5). Experiments on a private 46-campaign dataset (44 train, 2 test) report lower MAE/RMSE for the multimodal model than unimodal and text baselines (Table 2), plus RL fine-tuning improves LLM trend accuracy by 18.38% and human evaluators rate the reasoning higher on alignment, coherence, and factual accuracy.","tokens_in":6893,"tokens_out":4464,"duration_ms":46518,"significance":"If the reported effect is real, the work demonstrates a practical way to obtain interpretable, text-grounded explanations from a forecasting system and suggests RL-tuned summaries can benefit numeric forecast error. Strengths include a real industry dataset, a transparent cost estimate, a qualitative comparison, and a small human evaluation. However, the manuscript provides no code or data, tests only two held-out campaigns, and lacks the central ablation needed to separate change-log information from label-conditioned re-encoding of the numeric input. The significance is therefore conditional on additional evidence.","major_comments":[{"comment":"The reward function in Eq. (1) includes I(yhat=y) and I(s(r)=y)*c, so the LLM is directly trained to encode the ground-truth trend into <Prediction> and sentiment. The prompt in Fig. 2 also contains the same 14-day rolling-average click values used by the numeric transformer. Thus the XLM-R embedding of the summary is a target-supervised function of the numeric series, and the gain of Ours over Uni (4.948 vs 5.102 in Table 2) may come from that re-encoding rather than from change-log text. This is amplified by Multi+Changelog being worse than Uni (5.379 vs 5.102). An ablation with the LLM prompted without change logs, or with the same numeric input embedded through the same MLP, is required to support the multimodal claim.","section":"Eq. (1), Fig. 2, Table 2"},{"comment":"The forecast evaluation is on two held-out campaigns (1,045 points). The per-seed std in Table 2 is over three runs, not over campaigns or random campaign splits. No code or data are released. Under these conditions the 3% MAE improvement over Uni (5.102 to 4.948) is not established as stable or generalizable. Additional campaign-level splits, leave-one-campaign-out results, or a public benchmark would be needed.","section":"Sec. 2.1, Table 2"},{"comment":"Fusion weight alpha is fixed at 0.5 with no sensitivity analysis or tuning protocol. Since the final prediction is Y = Y_tsf + alpha*Y_mlp, the reported advantage depends on alpha; a grid (e.g., 0, 0.25, 0.5, 0.75, 1) or learned alpha is needed to show that the fusion is robust.","section":"Sec. 2.3, Eq. (2)"},{"comment":"The LLM comparison is reported only as percentage improvements (18.38% prediction accuracy, 6.69% reward) with no raw scores, sample sizes, or intervals; Fig. 5a appears to show points without error bars. Human evaluation scores in Fig. 6 are means only, without variance, per-evaluator data, or inter-rater agreement. The claim of improved reasoning quality needs this statistical support.","section":"Sec. 3, Fig. 5a, Fig. 6"}],"minor_comments":[{"comment":"Typographical errors: 'illutrated' in the contribution list and 'Emperical Evaluations' in Section 3. Also, Figure 4's label 'LLM Summerizations' should be 'Summarizations'.","section":"Sec. 1 and Sec. 3"},{"comment":"The x-axis label 'Step' is unclear; it appears to list models such as GPT-4o, o3-mini, Qwen. Please relabel and include raw scores.","section":"Fig. 5a"},{"comment":"The caption says results are 'scaled by 10^2'. Please state explicitly what the reader should infer (e.g., original MAE 0.04948 or 494.8).","section":"Table 2"},{"comment":"The abstract says 46 campaigns; Section 2.1 says 44 training and 2 testing, which is consistent, but the discrepancy in phrasing should be resolved for clarity.","section":"Sec. 2.1"},{"comment":"The statement 'first work to incorporate textual reasoning into time series forecasting' is too strong given Time-MMD and other LLM-based forecasting works; consider tempering it.","section":"Sec. 4"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting, but with private data and no code, reproducibility is limited. The missing text-free ablation is fixable and should be required before acceptance. Given the small test set, even with the ablation, the statistical evidence is thin; the editor may wish to consider whether a two-campaign evaluation meets the journal's bar for generalizable claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper before assigning it: it builds a multimodal click forecaster by fusing a transformer on numeric click series with embeddings of LLM-generated summaries of campaign change logs, where the LLM is RL-finetuned with a reward that includes ground-truth trend and sentiment alignment. The full pipeline reports lower MAE/RMSE than baselines on a private industry dataset (two held-out campaigns). The writing is clear and the experiments are reproducible in spirit, except the data and code are not released.\n\nWhat is genuinely new: applying RL fine-tuning (GRPO) to an LLM that produces both a trend prediction and a concise reasoning for ad campaign changes, then feeding that summary embedding to a Time-MMD-style forecaster. The qualitative example in Table 1 is compelling, and the human evaluation, though small (five experts, ten queries), shows the fine-tuned model producing more aligned and coherent reasoning than GPT-4o or o3-mini. The reported cost of fine-tuning ($60 on two A100s) is a nice practical detail.\n\nThe soft spot is real and central: the claim that change-log text contributes information beyond the numeric series is not established. The RL reward (Eq. 1) explicitly optimizes the LLM to match the ground-truth trend, and the prompt already contains the same rolling-average click values fed to the transformer. So the summary embedding is a label-conditioned, target-supervised encoding of the numeric input. The \"Multi + changelog\" baseline, which uses raw text, is worse than the numeric-only Unimodal baseline, suggesting raw text is not informative on its own. The paper does not ablate the LLM prompted without change logs, nor a text-free control where the same numeric input goes through a supervised projection. Without that, the reported gain (MAE 4.948 vs 5.102) could come from the extra supervised representation of the same numeric signal, not from multimodal fusion. That is the load-bearing weakness.\n\nOther issues are minor relative to this: only 1,045 test points from two campaigns, LLM accuracy reported only as percentage gains without raw counts or intervals, and the claim of being \"the first work to incorporate textual reasoning into time series forecasting\" is too strong given existing LLM-based MMTSF work.\n\nOverall: the paper is a reasonable industrial application, clearly written and with an addressable hole. It deserves a serious referee, but the revision must include the missing ablation, and ideally more held-out data or a public dataset. I'd read a revised version; I wouldn't cite this version in my own work.\n\nRecommendation: send to peer review, but flag the ablation requirement clearly.","headline":"A plausible industrial MMTSF pipeline with RL-tuned LLM summaries, but the core claim that text adds signal beyond the numeric series is untested and the evidence base is two campaigns.","tokens_in":7340,"tokens_out":1275,"would_cite":false,"duration_ms":15910,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RL-tuned text summaries sharpen click forecasts in digital advertising","keywords":["multimodal time series forecasting","click forecasting","digital advertising","reinforcement learning","LLM fine-tuning","GRPO","interpretable explanations"],"falsifier":"Re-run Table 2 replacing the LLM-derived text embeddings with embeddings of a description generated from the numeric series alone (or from a text-format listing of the same rolling averages), keeping everything else identical; if MAE/RMSE match the reported 4.948/7.670, the text path is adding no independent information. A second check: train the same pipeline with RL summaries produced by a model that is never shown the ground-truth trend and compare.","tokens_in":6444,"feed_emoji":"📈","tokens_out":4022,"duration_ms":36372,"temperature":0.7,"pith_summary":"The paper tries to show that forecasting click volume in digital advertising is improved when the numerical click series is combined with textual change logs—campaign configuration updates such as keyword removals, bid changes, and ad copy edits—processed through a reinforcement-learning-fine-tuned language model. The forecaster outputs both the next five-day click trend and a short written reasoning. On two held-out campaigns, the multimodal model reports lower MAE and RMSE than a numeric-only transformer, a raw-changelog multimodal variant, and one using GPT-4o summaries. A separate human evaluation rates its explanations higher on alignment, factual accuracy, and coherence. If this holds, advertisers get forecasts that are both more accurate and easier to act on.","feed_headline":"RL-tuned text summaries sharpen ad-click forecasts","feed_subtitle":"Change-log summaries beat numeric-only and GPT-4o baselines on error and explanation quality.","key_machinery":"The pipeline's load-bearing piece is the reward function used to fine-tune the LLM: $R = S_{\\text{format}} + \\mathbb{I}(\\hat{y}=y) + \\mathbb{I}(s(r)=y)\\cdot c$, where $S_{\\text{format}}$ penalizes missing tags, $\\mathbb{I}(\\hat{y}=y)$ rewards matching the ground-truth trend direction, and $\\mathbb{I}(s(r)=y)\\cdot c$ rewards reasoning whose sentiment agrees with the actual trend scaled by the sentiment classifier's confidence. GRPO trains the LLM to produce formatted <Reasoning> and <Prediction> outputs. Those outputs are embedded with XLM-R, pooled, projected by a three-layer MLP, and combined with the transformer forecast as $Y = Y_{\\text{tsf}} + \\alpha Y_{\\text{mlp}}$ with $\\alpha=0.5$. Th","core_discovery":"The central claim is that a multimodal time-series forecaster, built by fine-tuning Qwen2.5-3B with GRPO under a reward that combines format compliance, one-word trend accuracy, and sentiment alignment, then feeding the resulting reasoning summaries through a frozen XLM-R embedding and a trainable MLP projection merged with a transformer encoder, outperforms all tested baselines on held-out campaigns. Table 2 reports MAE 4.948±0.02 and RMSE 7.670±0.001, both better than Copy, Uni, Multi with raw changelog, and Multi with GPT-4o. The paper also claims the RL-fine-tuned Qwen improves prediction accuracy 18.38% over o3-mini and receives higher human ratings on alignment, factual accuracy, and c","pith_inferences":["The 18.38% prediction-accuracy gain over o3-mini is measured on the LLM's one-word trend prediction, not on the final numeric forecast; the two gains should not be conflated.","Because the LLM is rewarded with ground-truth trend labels, its summaries may encode label information; without an ablation that feeds the same numeric features through the text path, part of the Table 2 gain could be label leakage rather than semantic value of change-logs.","A direct test would replace the LLM summaries with a deterministic text derived from numeric features, or summarize with a model never trained on the target labels, and check whether the advantage persists."],"forward_implications":["If the claim is correct, adding RL-tuned textual summaries to numeric click series is a practical way to reduce forecast error in digital-advertising settings.","The same pipeline can generate explanations that advertisers can read, not just numbers, with human-rated quality above GPT-4o and o3-mini summaries.","RL fine-tuning with a domain-specific reward can make a 3B open model outperform closed models on this forecasting-adjacent task at low compute cost (about USD 60 in reported GPU time).","The sparse-change-log problem (most days have no text events) is handled by compressing text into LLM reasoning, suggesting similar treatment for sparse event logs in other time-series domains."],"supporting_citations":[{"why":"Supplies the base multimodal time-series architecture and dataset construction the paper builds on.","marker":"[9]"},{"why":"Provides the transformer encoder backbone used for numeric time-series forecasting.","marker":"[20]"},{"why":"Frozen XLM-R embedding model that encodes LLM reasoning into fixed-length vectors.","marker":"[1]"},{"why":"Qwen2.5-3B Instruct is the LLM backbone chosen for its non-fine-tuned performance and low compute requirements.","marker":"[16]"},{"why":"Source of the Group Relative Policy Optimization (GRPO) method used for RL fine-tuning.","marker":"[5]"},{"why":"Also cited for GRPO, supporting the RL training approach.","marker":"[14]"},{"why":"GRPO implementation used for the fine-tuning experiments.","marker":"[12]"},{"why":"HybridFlow library underpins the RL fine-tuning framework.","marker":"[15]"},{"why":"BERT-based sentiment model provides the sentiment and confidence scores used in the reward function.","marker":"[3]"}],"fun_headline_variants":["Multimodal ad-click forecasts: RL-tuned text summaries beat GPT-4o","RL-optimized summaries improve click prediction accuracy and clarity","Text plus numbers: RL summaries sharpen ad-click forecasting","Interpretable ad-click forecasts via RL fine-tuned text reasoning"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The textual summaries must carry signal that is not already present in the rolling-average click series; the LLM is trained with ground-truth trend rewards, so the summaries could simply re-encode the numeric label.","fun_headline_variants_meta":{"raw":{"variants":["Multimodal ad-click forecasts: RL-tuned text summaries beat GPT-4o","RL-optimized summaries improve click prediction accuracy and clarity","Text plus numbers: RL summaries sharpen ad-click forecasting","Interpretable ad-click forecasts via RL fine-tuned text reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1152,"prompt_tokens":635,"completion_tokens":517,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":379,"completion_tokens_details":{"reasoning_tokens":444}},"tokens_in":379,"tokens_out":517,"duration_ms":6203,"temperature":1.0,"reasoning_tokens":444,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T19:57:56.056937+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run Table 2 replacing the LLM-derived text embeddings with embeddings of a description generated from the numeric series alone (or from a text-format listing of the same rolling averages), keeping everything else identical; if MAE/RMSE match the reported 4.948/7.670, the text path is adding no independent information. A second check: train the same pipeline with RL summaries produced by a model that is never shown the ground-truth trend and compare.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GRPO implementation used for the fine-tuning experiments."}],"review_version":1}