{"id":"e3b54ce0-9c5e-4210-81a1-2d94b36b73f5","arxiv_id":"2412.10737","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":11,"one_line_summary":"A multimodal neural network with hashtag-guided attention and facial demographics reports large popularity-prediction gains, but the evaluation likely leaks the target via comment count.","lead":"This paper presents NARRATOR, a deep learning model that predicts social media post popularity using captions, images, hashtags, face-based demographics, and sentiment signals. The authors report large gains over earlier methods on two Flickr datasets, but the evaluation includes post-hoc features such as comment count that may leak the target.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Post-hoc features (Comment Count, Post Duration) leak the popularity outcome; the reported margin does not establish early popularity prediction.","rationale":"The reader's weakest_assumption identifies the same load-bearing point: the evaluation is not a valid test of early popularity prediction because post-hoc features are included. I agree. The concern is not about disagreement with consensus and not about the authors' intent; it is an internal evaluation-validity problem. The paper's ablations (Tables 2, 3, 5, 7) remove whole feature families but never isolate Comment Count or Post Duration, so no internal result addresses the leakage. Auxiliary issues, such as the absence of error bars, code, and data, and the selection of only posts with faces and hashtags, reinforce the difficulty of interpreting Table 1, but the decisive issue is outcome leakage. A reproducible rerun without the two leaked features would settle it. If the margin collapses, REJECT is appropriate; if it persists, the argument could be rehabilitated with a revised setup. Given the current manuscript, the central claim is not credible as stated.","tokens_in":26684,"tokens_out":4389,"duration_ms":44401,"concrete_test":"Re-run the full NARRATOR pipeline on TPIC and SMP with Comment Count and Post Duration excluded from f^s_i before PCA, keeping all other features, the same train/validation/test split, hyperparameters, and preprocessing. If test MSE/MAE remain close to Table 1 (e.g., TPIC MSE still below the best baseline's 1.711 by roughly the reported margin), the leak is not decisive; if MSE degrades materially or falls below the best baseline, the headline improvement is an artifact of post-hoc outcome features. A time-aligned variant that predicts views from features available at upload time would be the definitive check.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that NARRATOR outperforms state-of-the-art methods by a large margin (Table 1), and the paper explicitly frames the task as forecasting public attention 'early on' (Section 1). For that claim to be valid, every input feature must be observable before the popularity label is fixed. This fails in Section 4.1.6: the 'social feature' vector f^s_i includes Comment Count (Post Metadata item e) and Post Duration (Time item d), and this 85-dimensional vector is PCA-reduced (to 6) and fed through the CNN stream into the final predictor. The label is log-normalized view count. Comment count is an engagement outcome strongly concurrent with views, and post duration is the length of the observation window over which views accumulate; neither is available at upload time. Because Table 1's margin (TPIC MSE 1.196 vs 1.711 best baseline; SMP MSE 2.022 vs 4.211) is large, the burden is on the paper to show it survives removing these two leaked variables. No such ablation, temporal split, error bar, code, or dataset release is provided. As presented, the evaluation cannot distinguish predictive skill from outcome leakage, so the central claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NARRATOR, a multimodal deep neural network for predicting the popularity of social media posts, formulated as a regression to log-normalized view counts. The model combines BERT/LSTM text features, VGG19 visual features, DeepFace-derived demographic attributes, topic and graph-structure hashtag embeddings, CoreNLP sentiment features for captions and hashtags, user/post metadata, and a novel hashtag-guided attention mechanism that reweights text and image features using hashtag context. These representations are passed through per-modality CNNs, concatenated with the attention output, and fed to a 12-layer feedforward network. The empirical claim is that NARRATOR outperforms seven state-of-the-art baselines by a large margin on the TPIC and SMP datasets (Table 1). The paper also reports feature ablations, attention-mechanism comparisons, correlation analyses, feature ranking, and computational cost.","tokens_in":26952,"tokens_out":6861,"duration_ms":56771,"significance":"If the empirical claim were sound, the work would be a useful contribution to multimodal popularity prediction by demonstrating that hashtag-guided attention, visual demographics, and hashtag sentiment improve forecasting accuracy, with practical implications for recommendation, advertising, and trend analysis. The paper is honest about computational costs and limitations, and it provides ablations that isolate the proposed components. However, the evaluation is compromised by the inclusion of post-hoc outcome variables (comment count and post duration) as input features, by single-run metrics without variance estimates, and by comparisons on filtered subsets whose relationship to the baselines' original evaluation data is unspecified. As presented, the reported margins over baselines cannot be distinguished from outcome leakage, so the central claim is unsupported. The proposed mechanism is interesting, but its benefit is not demonstrated by the current experiments.","major_comments":[{"comment":"Comment Count (Post Metadata item e) and Post Duration (Time item d) are included in the 85-dimensional social feature vector f^s_i, which is PCA-reduced to 6 dimensions and fed through the CNN stream into the fused representation M_i and the final predictor. The label is the log-normalized view count, and the paper frames the task as forecasting popularity 'early on' (Section 1); comment count is a contemporaneous engagement outcome and post duration is the length of the observation window over which views accumulate, so neither is available at the time the prediction is supposed to be made. The large margins over the best baselines in Table 1 (TPIC MSE 1.196 vs. 1.711; SMP MSE 2.022 vs. 4.211) are therefore consistent with outcome leakage rather than predictive skill. No ablation without these two features, no temporal split, and no error bars are provided, so the central claim of the paper is unsupported as presented.","section":"§4.1.6, §4.3 (Eq. 14), Table 1"},{"comment":"The evaluation uses filtered subsets (21,000 SMP and 11,000 TPIC posts) restricted to posts that contain hashtags, titles, and faces. The paper does not state whether the baselines in Table 1 were re-run on the same filtered subsets with the same train/validation/test split; if their numbers are taken from the original papers, the comparison is against different data and the reported margins (e.g., 30-70% relative improvements) may reflect the changed evaluation set rather than model quality. The test sets are also small (about 2,100 and 1,100 posts, respectively) for a model with 34.9M parameters (Section 5.2.8), which makes the absence of variance estimates particularly concerning.","section":"§5.1.1 (Dataset Preprocessing) and §5.2.7"},{"comment":"All effectiveness results are single-run point estimates without standard deviations, confidence intervals, or significance tests. The statistical analysis reports SRCC values of 0.885 and 0.8773 and PCC values of 0.8355 and 0.8409 with p ≤ 0.05, but gives no test statistic, null hypothesis, sample size, or multiple-comparison correction. As a result, the claimed significant margin over baselines cannot be evaluated for statistical reliability, which is load-bearing for the empirical claim.","section":"§5.2.1 (Table 1) and §5.2.4"},{"comment":"Algorithm 1 has inconsistent notation: line 2 uses V[t] as a learnable parameter while lines 4-5 use V[i] as the image feature matrix from Eq. (6); the text defines U^t, V^t, U^i, V^i as learnable parameters. This makes the exact computation of the hashtag-guided attention mechanism ambiguous and impedes reproduction of the central methodological contribution. Additionally, Section 4.1.3 states the demographic feature vector has dimension 116, while Section 4.3 says it has dimension four, so the stated merged vector size of 27104 in Section 4.3 cannot be verified.","section":"§4.2 (Algorithm 1)"}],"minor_comments":[{"comment":"The text says the output layer 'directly predict[s] the continuous star count'; this should be the popularity score or view count. Equation (20) also says the objective is 'maximizing the Mean Squared Error (MSE) cost function' after defining MSE as the loss to be minimized.","section":"§4.4"},{"comment":"The entries in Table 5 are garbled (e.g., '1 .362(5)/0.867(6) 5.5'), making the feature ranking difficult to read; please reformat the table.","section":"§5.2.6 (Table 5)"},{"comment":"Parallel co-attention is cited as [25], but [25] (Zhang et al., hashtag recommendation) is not the source of parallel co-attention used earlier in the paper; the relevant citation appears to be [43] (Bansal et al.).","section":"§6.2"},{"comment":"It is stated that the embedding dimension D=768 is set uniformly for all comparative methods, but it is not explained how the baselines (e.g., TweetGage or HashPop) are adapted to this setting; please clarify whether all baselines were re-implemented and re-run under identical conditions.","section":"§5.2.7"},{"comment":"Equation (7) reuses the symbol v^k_i from Eq. (6) but now denotes the projected feature vector in R^D; using a different symbol would avoid confusion.","section":"§4.1.2"},{"comment":"The practical implication 'Combating Misinformation' claims that predicting popularity early can help identify harmful content, but popularity is not the same as harmfulness; this implication needs qualification.","section":"§6.4.2"}],"recommendation":"reject","confidential_remarks":"The leakage issue is decisive: the model has direct access to post-hoc outcomes (comment count and post duration) that are strongly related to the label, and the reported margins are large. The central empirical claim is unsupported as presented, and a proper fix would require redefining the feature set, re-running all baselines on leakage-free features with identical splits, and reporting uncertainty estimates. This is a substantial rework rather than a local revision, so I recommend rejection, though a resubmission with a clean temporal evaluation would be worth considering."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The genuinely new piece is visual demographics: pulling age, gender, race, and emotion from faces with DeepFace and feeding that into a popularity predictor. The hashtag sentiment feature and hashtag-guided attention are modest extensions of co-attention, and the paper more or less admits that. What the paper does well is present the architecture clearly, benchmark against seven existing methods on two standard datasets, and run ablations that isolate the new components. The limitations section is even honest about data bias and demographic accuracy. So this is not a sloppy or unserious submission.\n\nThe problem is the evaluation. Section 4.1.6 feeds Comment Count and Post Duration into the 85-dimensional social feature vector, which after PCA goes into the CNN stream and the final predictor. The label is log-normalized view count. Comment count is an engagement outcome, and post duration is the length of the observation window during which views accumulate. Neither is available \"early on,\" which is the framing in Section 1. The paper positions the task as forecasting public attention early, so including these two features is a target leak. The large margins in Table 1 (MSE 1.196 vs 1.711 and 2.022 vs 4.211) may be largely an artifact of this. There is no ablation that removes them, no temporal split, no code, no data, and no variance or significance testing. The preprocessing also filters to posts with faces, hashtags, and titles (21k and 11k samples), which makes direct comparison with published baselines on the original test sets unsafe.\n\nThe correlations and ablation tables are consistent with the story, but they do not fix the leak. This is the central claim, and as presented it is unsupported.\n\nWho is this for? A reader building a multimodal popularity model could mine it for feature ideas, especially the demographic angle. But as an empirical paper, it needs a re-run. I would not send it to full review in this state. The fix is mechanical: drop Comment Count and Post Duration, add a temporal split, report multiple seeds, and release code and data. If the margin survives that, it's a decent niche contribution worth revisiting.","headline":"Competent engineering with a genuinely new demographic feature, but the headline result is undermined by outcome-leaking features and no code/data.","tokens_in":27473,"tokens_out":3379,"would_cite":false,"duration_ms":31538,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A deep network that uses hashtags to guide attention over text and images, plus face-derived demographics and hashtag sentiment, predicts social media post popularity substantially better than seven existing methods on two real-world…","keywords":["multimodal popularity prediction","hashtag-guided attention","visual demographics","hashtag sentiment","social media analysis","deep neural network","multimodal fusion","post popularity prediction"],"falsifier":"Retrain NARRATOR on the same TPIC and SMP splits with comment count and post duration removed from the social feature vector, and compare MSE against the reported values; if the model still keeps its margin over the next-best baselines, the gain is not mainly leakage. A second check is to train a variant using only comment count and duration plus the cheapest available features, and see how much of the reported improvement it recovers.","tokens_in":26490,"feed_emoji":"📈","tokens_out":6411,"duration_ms":54994,"temperature":0.7,"pith_summary":"Social media posts carry text, images, and hashtags, and predicting which ones will attract attention is hard because existing models mostly look at content alone. This paper proposes NARRATOR, a deep network that adds three underused signals: demographic traits (age, gender, race, emotion) read from faces in the image, sentiment expressed through hashtags, and an attention mechanism that lets hashtags steer which parts of the text and image matter most. The authors claim that NARRATOR beats seven state-of-the-art baselines on two real-world Flickr datasets, with the largest reported gains in mean squared error (1.196 versus 1.711 on TPIC, and 2.022 versus 4.211 on SMP). If that holds, it would show that hashtags are not just topical labels but contextual guides that connect visual and textual content to audience interests, and that face-derived demographics carry predictive signal beyond user metadata. The paper also reports ablation studies indicating each added component contributes to the gain.","feed_headline":"Hashtag-guided attention network cuts popularity-prediction error","feed_subtitle":"Face demographics, hashtag sentiment, and hashtag guidance all contribute to the reported gains on two Flickr datasets.","key_machinery":"The load-bearing piece is the hashtag-guided attention mechanism: hashtags are embedded with BERT, then used to compute attention weights over the textual feature matrix (BERT embeddings passed through an LSTM) and the visual feature matrix (VGG19 regional features), so the attended content vector $\\tilde{c}$ is the sum of hashtag-attended text and image representations. Around this sit two other novel inputs: a demographic feature vector $\\{g,a,e,r\\}$ (gender, age, emotion, race predicted by a face-analysis model) and a 10-dimensional sentiment vector built by concatenating CoreNLP sentiment scores for the caption and for the hashtags. All feature vectors are passed through separate CNN towers and fused in a deep feed-forward network that outputs the popularity score.","core_discovery":"The paper's central claim is that the popularity of a multimodal post can be predicted more accurately by explicitly modeling what hashtags do to the interpretation of text and images, and by adding two previously neglected modalities: visual demographics and hashtag sentiment. On the TPIC2017 and SMP datasets, NARRATOR reports MSE of 1.196 and 2.022 respectively, outperforming all seven compared methods; the next-best baselines are VSCNN with 1.711 on TPIC and TweetGage with 4.211 on SMP. Ablation experiments show that removing hashtag sentiment and demographics together raises MSE by about 14 percent on both datasets, and replacing hashtag-guided attention with self-, cross-, or no attention increases error, supporting the claim that the new components are responsible for the improvement.","pith_inferences":["A clean test would retrain NARRATOR with comment count and post duration held out, because those features are only observable after a post has been online and may inflate the reported forecasting accuracy.","The hashtag-guided attention idea could transfer to other multimodal tasks where tags or labels provide context, such as content recommendation or content moderation, though that would need separate validation.","Face-derived demographics carry privacy and bias risks; if this approach moves toward deployment, consent, fairness, and generalization across platforms would need explicit examination."],"forward_implications":["On the two Flickr-derived datasets, NARRATOR reports MSE reductions of roughly 30 to 78 percent over seven baselines, implying the added modalities carry signal not captured by content-only or metadata-only models.","Ablation studies attribute part of the gain to visual demographics and hashtag sentiment, and the attention study attributes part to hashtag guidance over self-, cross-, and parallel co-attention.","The learned attention weights could be inspected to show which image regions and caption words a hashtag highlights, making the predictions more interpretable for users and content creators.","The model is computationally heavier at training than most baselines, but its inference time is low enough that the authors describe it as suitable for near-real-time predictions."],"supporting_citations":[{"why":"Baseline HashPop that predicts popularity from hashtag networks, captions, and images; NARRATOR compares against it.","marker":"[32]"},{"why":"Baseline VSCNN that uses social metadata and visual attributes; it is the strongest TPIC baseline that NARRATOR reports beating.","marker":"[26]"},{"why":"Baseline MMAtt that applies attention over categorical, visual, and textual features, providing the attention-based comparison point.","marker":"[16]"},{"why":"Baseline FuseDNN that fuses visual, textual, user, temporal, and location features for popularity prediction.","marker":"[61]"},{"why":"Baseline TweetGage that predicts engagement with graph neural networks over hashtag relationships.","marker":"[33]"},{"why":"SMP dataset, one of the two real-world datasets used for the evaluation.","marker":"[59]"},{"why":"TPIC2017 dataset, the other real-world dataset used for the evaluation.","marker":"[60]"}],"fun_headline_variants":["Hashtag-guided attention boosts post popularity prediction","NARRATOR uses hashtag sentiment and face demographics for popularity","Multimodal model with hashtag attention forecasts post reach","Face and hashtag cues improve social media popularity forecasts","Deep attention network reads hashtags and faces to predict post popularity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that all input features, including comment count and post duration, are known before the popularity label that the model is supposed to predict, so the reported gains may partly reflect these outcome-correlated variables rather than genuine forecasting skill.","fun_headline_variants_meta":{"raw":{"variants":["Hashtag-guided attention boosts post popularity prediction","NARRATOR uses hashtag sentiment and face demographics for popularity","Multimodal model with hashtag attention forecasts post reach","Face and hashtag cues improve social media popularity forecasts","Deep attention network reads hashtags and faces to predict post popularity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00033,"raw_usage":{"total_tokens":1861,"prompt_tokens":989,"completion_tokens":872,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":792}},"tokens_in":605,"tokens_out":872,"duration_ms":9151,"temperature":1.0,"reasoning_tokens":792,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:38:51.565326+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain NARRATOR on the same TPIC and SMP splits with comment count and post duration removed from the social feature vector, and compare MSE against the reported values; if the model still keeps its margin over the next-best baselines, the gain is not mainly leakage. A second check is to train a variant using only comment count and duration plus the cheapest available features, and see how much of the reported improvement it recovers.","supporting_citations":[{"cited_title":"In: Calzolari, N., B´ echet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Odijk, J., Piperidis, S","cited_arxiv_id":null,"evidence_quote":"Baseline HashPop that predicts popularity from hashtag networks, captions, and images; NARRATOR compares against it."},{"cited_title":"In: Proceedings of the 2023 ACM International Conference on Multimedia Retrieval, pp","cited_arxiv_id":null,"evidence_quote":"Baseline TweetGage that predicts engagement with graph neural networks over hashtag relationships."}],"review_version":1}