REVIEW 4 major objections 5 minor 22 references
Forecasting Clicks in Digital Advertising: Multimodal Inputs and Interpretable Outputs
T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read RL-tuned text summaries sharpen click forecasts in digital advertising
desk verdict A plausible industrial MMTSF pipeline with RL-tuned LLM summaries, but the core claim that text adds signal beyond the numeric series is untested and the evidence base is two campaigns. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The pipeline's load-bearing piece is the reward function used to fine-tune the LLM: $R = S_{\text{format}} + \mathbb{I}(\hat{y}=y) + \mathbb{I}(s(r)=y)\cdot c$, where $S_{\text{format}}$ penalizes missing tags, $\mathbb{I}(\hat{y}=y)$ rewards matching the ground-truth trend direction, and $\mathbb{I}(s(r)=y)\cdot c$ rewards reasoning whose sentiment agrees with the actual trend scaled by the sentiment classifier's confidence. GRPO trains the LLM to produce formatted <Reasoning> and <Prediction> outputs. Those outputs are embedded with XLM-R, pooled, projected by a three-layer MLP, and combined with the transformer forecast as $Y = Y_{\text{tsf}} + \alpha Y_{\text{mlp}}$ with $\alpha=0.5$. Th
What would settle it
Re-run Table 2 replacing the LLM-derived text embeddings with embeddings of a description generated from the numeric series alone (or from a text-format listing of the same rolling averages), keeping everything else identical; if MAE/RMSE match the reported 4.948/7.670, the text path is adding no independent information. A second check: train the same pipeline with RL summaries produced by a model that is never shown the ground-truth trend and compare.
Extended reading notes
Core claim
The central claim is that a multimodal time-series forecaster, built by fine-tuning Qwen2.5-3B with GRPO under a reward that combines format compliance, one-word trend accuracy, and sentiment alignment, then feeding the resulting reasoning summaries through a frozen XLM-R embedding and a trainable MLP projection merged with a transformer encoder, outperforms all tested baselines on held-out campaigns. Table 2 reports MAE 4.948±0.02 and RMSE 7.670±0.001, both better than Copy, Uni, Multi with raw changelog, and Multi with GPT-4o. The paper also claims the RL-fine-tuned Qwen improves prediction accuracy 18.38% over o3-mini and receives higher human ratings on alignment, factual accuracy, and c
Load-bearing premise
The textual summaries must carry signal that is not already present in the rolling-average click series; the LLM is trained with ground-truth trend rewards, so the summaries could simply re-encode the numeric label.
Editorial extensions
If this is right
- If the claim is correct, adding RL-tuned textual summaries to numeric click series is a practical way to reduce forecast error in digital-advertising settings.
- The same pipeline can generate explanations that advertisers can read, not just numbers, with human-rated quality above GPT-4o and o3-mini summaries.
- RL fine-tuning with a domain-specific reward can make a 3B open model outperform closed models on this forecasting-adjacent task at low compute cost (about USD 60 in reported GPU time).
- The sparse-change-log problem (most days have no text events) is handled by compressing text into LLM reasoning, suggesting similar treatment for sparse event logs in other time-series domains.
Reading between the lines
- The 18.38% prediction-accuracy gain over o3-mini is measured on the LLM's one-word trend prediction, not on the final numeric forecast; the two gains should not be conflated.
- Because the LLM is rewarded with ground-truth trend labels, its summaries may encode label information; without an ablation that feeds the same numeric features through the text path, part of the Table 2 gain could be label leakage rather than semantic value of change-logs.
- A direct test would replace the LLM summaries with a deterministic text derived from numeric features, or summarize with a model never trained on the target labels, and check whether the advantage persists.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a multimodal click forecasting pipeline for digital advertising. A Qwen2.5-3B LLM is fine-tuned with GRPO using a reward that combines format compliance, one-word trend accuracy, and sentiment alignment (Eq. 1); its outputs are embedded with XLM-R and fused via a projection MLP with a transformer numeric forecaster (alpha = 0.5). Experiments on a private 46-campaign dataset (44 train, 2 test) report lower MAE/RMSE for the multimodal model than unimodal and text baselines (Table 2), plus RL fine-tuning improves LLM trend accuracy by 18.38% and human evaluators rate the reasoning higher on alignment, coherence, and factual accuracy.
Significance. If the reported effect is real, the work demonstrates a practical way to obtain interpretable, text-grounded explanations from a forecasting system and suggests RL-tuned summaries can benefit numeric forecast error. Strengths include a real industry dataset, a transparent cost estimate, a qualitative comparison, and a small human evaluation. However, the manuscript provides no code or data, tests only two held-out campaigns, and lacks the central ablation needed to separate change-log information from label-conditioned re-encoding of the numeric input. The significance is therefore conditional on additional evidence.
major comments (4)
- [Eq. (1), Fig. 2, Table 2] The reward function in Eq. (1) includes I(yhat=y) and I(s(r)=y)*c, so the LLM is directly trained to encode the ground-truth trend into <Prediction> and sentiment. The prompt in Fig. 2 also contains the same 14-day rolling-average click values used by the numeric transformer. Thus the XLM-R embedding of the summary is a target-supervised function of the numeric series, and the gain of Ours over Uni (4.948 vs 5.102 in Table 2) may come from that re-encoding rather than from change-log text. This is amplified by Multi+Changelog being worse than Uni (5.379 vs 5.102). An ablation with the LLM prompted without change logs, or with the same numeric input embedded through the same MLP, is required to support the multimodal claim.
- [Sec. 2.1, Table 2] The forecast evaluation is on two held-out campaigns (1,045 points). The per-seed std in Table 2 is over three runs, not over campaigns or random campaign splits. No code or data are released. Under these conditions the 3% MAE improvement over Uni (5.102 to 4.948) is not established as stable or generalizable. Additional campaign-level splits, leave-one-campaign-out results, or a public benchmark would be needed.
- [Sec. 2.3, Eq. (2)] Fusion weight alpha is fixed at 0.5 with no sensitivity analysis or tuning protocol. Since the final prediction is Y = Y_tsf + alpha*Y_mlp, the reported advantage depends on alpha; a grid (e.g., 0, 0.25, 0.5, 0.75, 1) or learned alpha is needed to show that the fusion is robust.
- [Sec. 3, Fig. 5a, Fig. 6] The LLM comparison is reported only as percentage improvements (18.38% prediction accuracy, 6.69% reward) with no raw scores, sample sizes, or intervals; Fig. 5a appears to show points without error bars. Human evaluation scores in Fig. 6 are means only, without variance, per-evaluator data, or inter-rater agreement. The claim of improved reasoning quality needs this statistical support.
minor comments (5)
- [Sec. 1 and Sec. 3] Typographical errors: 'illutrated' in the contribution list and 'Emperical Evaluations' in Section 3. Also, Figure 4's label 'LLM Summerizations' should be 'Summarizations'.
- [Fig. 5a] The x-axis label 'Step' is unclear; it appears to list models such as GPT-4o, o3-mini, Qwen. Please relabel and include raw scores.
- [Table 2] The caption says results are 'scaled by 10^2'. Please state explicitly what the reader should infer (e.g., original MAE 0.04948 or 494.8).
- [Sec. 2.1] The abstract says 46 campaigns; Section 2.1 says 44 training and 2 testing, which is consistent, but the discrepancy in phrasing should be resolved for clarity.
- [Sec. 4] The statement 'first work to incorporate textual reasoning into time series forecasting' is too strong given Time-MMD and other LLM-based forecasting works; consider tempering it.
Circularity Check
No significant circularity: the RL reward uses ground-truth labels for standard supervised training, and the final forecast is an ensemble (Y = Y_tsf + alpha*Y_mlp), not a direct reuse of the LLM's binary prediction.
full rationale
The central derivation chain is: (1) a numeric transformer forecasts from click series; (2) an RL-fine-tuned LLM produces summaries from a prompt containing the same numeric series plus change logs; (3) the summary embedding is projected and added to the numeric forecast with weight alpha. The RL reward in Eq. (1) includes I(yhat=y) and sentiment alignment, so the LLM is explicitly trained to produce summaries whose one-word prediction matches the ground-truth trend. This is supervised/RL training, not a definitional circularity: at inference the LLM does not have access to the ground truth, and the final continuous forecast is not identical to the LLM's binary trend prediction. The architecture follows Time-MMD [9] and Informer [20], which are external references, not self-citations. No load-bearing step reduces, by the paper's own equations or by a self-citation chain, to its own inputs. The absence of an ablation removing change-log text while keeping a numeric embedding is a legitimate experimental limitation, but it does not constitute circularity under the strict requirement to exhibit an equation-level reduction. The held-out test evaluation and the comparison against baselines (Uni, Multi+changelog, Multi+GPT-4o) provide independent content, even if the source of the improvement is not fully disentangled. Therefore no circular step can be quoted, and the score is 0.
Assumptions & free parameters
free parameters (5)
- fusion weight alpha =
0.5
- lookback window l =
14
- forecast horizon h =
5
- format penalty in reward =
-0.5
- sentiment confidence multiplier c =
BERT model confidence
assumptions (5)
- domain assumption Change logs are correctly time-aligned with the click series and "no changes" days are accurately labeled
- domain assumption The two held-out campaigns are representative of the campaigns the model will encounter
- domain assumption BERT sentiment confidence c is a valid proxy for reasoning alignment with the actual trend
- domain assumption The frozen XLM-RoBERTa embeddings and the transformer backbone are treated as reliable black boxes
- domain assumption GRPO training with the custom reward converges to a policy that improves downstream forecasting
Cite this review
Pith. "Pith review of Forecasting Clicks in Digital Advertising: Multimodal Inputs and Interpretable Outputs." pith.science (2026). https://pith.science/paper/B6N52RHK
@misc{pith2026250909683,
author = {Pith},
title = {Pith review of: Forecasting Clicks in Digital Advertising: Multimodal Inputs and Interpretable Outputs},
year = {2026},
howpublished = {\url{https://pith.science/paper/B6N52RHK}},
note = {Machine review of arXiv:2509.09683}
}
read the original abstract
Forecasting click volume is a key task in digital advertising, influencing both revenue and campaign strategy. Traditional time series models rely solely on numerical data, often overlooking rich contextual information embedded in textual elements, such as keyword updates. We present a multimodal forecasting framework that combines click data with textual logs from real-world ad campaigns and generates human-interpretable explanations alongside numeric predictions. Reinforcement learning is used to improve comprehension of textual information and enhance fusion of modalities. Experiments on a large-scale industry dataset show that our method outperforms baselines in both accuracy and reasoning quality.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guil- laume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116 (2019)
arXiv 2019
-
[2]
Forbes Business Council. 2025. From Clicks to Conversions: Digital Marketing Trends Defining 2025. https://www.forbes.com/councils/forbesbusinesscouncil/ 2025/02/10/from-clicks-to-conversions-digital-marketing-trends-defining- 2025/ Accessed: 2025-03-24
work page 2025
-
[3]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 4171–4186
2019
-
[4]
Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew G Wilson. 2023. Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems 36 (2023), 19622–19635
work page 2023
-
[5]
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)
arXiv 2025
-
[6]
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: a factorization-machine based neural network for CTR prediction. arXiv preprint arXiv:1703.04247 (2017)
arXiv 2017
-
[7]
Zhaoxin Huan, Ke Ding, Ang Li, Xiaolu Zhang, Xu Min, Yong He, Liang Zhang, Jun Zhou, Linjian Mo, Jinjie Gu, et al . 2024. Exploring Multi-Scenario Multi- Modal CTR Prediction with a Large Scale Dataset. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1232–1241
work page 2024
-
[8]
Xiangzhen Li, Jiaxing Shen, Dezhi Wang, Wu Lu, and Yuanyi Chen. 2024. Multi- modal transform-based fusion model for new product sales forecasting. Engi- neering Applications of Artificial Intelligence 133 (2024), 108606
work page 2024
Show all 22 references
-
[9]
Haoxin Liu, Shangqing Xu, Zhiyuan Zhao, Lingkai Kong, Harshavardhan Prab- hakar Kamarthi, Aditya Sasanur, Megha Sharma, Jiaming Cui, Qingsong Wen, Chao Zhang, et al. 2024. Time-mmd: Multi-domain multimodal dataset for time series analysis. Advances in Neural Information Proces...
2024
-
[10]
Haoxin Liu, Zhiyuan Zhao, Jindong Wang, Harshavardhan Kamarthi, and B Aditya Prakash. 2024. Lstprompt: Large language models as zero-shot time series forecasters by long-short-term prompting. arXiv preprint arXiv:2402.16132 (2024)
2024 arXiv
-
[11]
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. 2023. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625 (2023)
2023 arXiv
-
[12]
Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr
-
[13]
Bill Sebald. 2020. Forecasting Search Marketing Success with a Click Curve. https://www.greenlanemarketing.com/resources/articles/forecasting- search-marketing-success-click-curve/ Accessed: 2025-03-24
2020
-
[14]
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al . 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024)
2024 arXiv
-
[15]
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv:2409.19256 (2024)
2024 arXiv
-
[16]
Qwen Team. 2024. Qwen2.5: A Party of Foundation Models. https://qwenlm. github.io/blog/qwen2.5/
2024
-
[17]
Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017. Deep & cross network for ad click predictions. In Proceedings of the ADKDD’17 . 1–7
2017
-
[18]
Zheng Yuan, Fajie Yuan, Yu Song, Youhua Li, Junchen Fu, Fei Yang, Yunzhu Pan, and Yongxin Ni. 2023. Where to go next for recommender systems? id- vs. modality-based recommender models revisited. In Proceedings of the 46th International ACM SIGIR Conference on Research and Deve...
2023
-
[19]
Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining ...
2018
-
[20]
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond efficient transformer for long se- quence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35. 11106–11115
2021
-
[21]
Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, Rong Jin, et al. 2022. Film: Frequency improved legendre memory model for long-term time series forecasting. Advances in neural information processing systems 35 (2022), 12677–12690
2022
-
[2025]
https://github.com/Jiayi-Pan/TinyZero
TinyZero. https://github.com/Jiayi-Pan/TinyZero. Accessed: 2025-01-24
2025
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.