{"id":"379e63b2-f64c-4a82-ba89-c8a87c635e38","arxiv_id":"2501.01422","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A multi-modal ensemble of video backbones, LLM caption features, and XGBoost predicts short-video engagement and reportedly won the INFORMS 2024 Data Challenge.","lead":"This paper describes a competition pipeline that predicts how many views, likes, comments, and shares short videos receive, using video features, language-model-generated captions, and gradient-boosted trees. The authors report first place in the INFORMS 2024 Data Challenge, but they do not provide code, data, or leaderboard evidence, so the headline result cannot be independently checked.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"First-place claim rests on unreleased leaderboard evidence; reported MAPE may reflect validation or leaderboard selection rather than an unbiased test-set result. Without official leaderboard data, code, or a description of the evaluation protocol, Table 1's numbers cannot be independently verified.","rationale":"The reader's weakest_assumption correctly identifies the evaluation protocol as the load-bearing issue, and I agree with that assessment. My stress-test focuses on the same point: all reported metrics are self-reported, and the paper does not establish that they come from the official test set under the official protocol. The paper itself even contains a placeholder instead of a link to code, data, or leaderboard evidence, which is a self-asserted missing artifact. I considered whether an internal inconsistency could be more load-bearing, such as Section 2.4 citing Table 2 for the fusion result when Table 2 contains no fusion row, or the absence of a description of the fusion averaging formula. These are real but secondary; the fusion MAPE is not mathematically impossible, and the citation issue does not by itself invalidate the reported numbers. The dominant concern remains the unverifiable empirical claim: without the official leaderboard or reproducible code, Table 1's values are compatible with validation-set selection or leaderboard overfitting. Since the reader already assigned CONDITIONAL with low confidence, my read does not move the verdict; it reinforces it. This critique is about the evidence base, not the authors' integrity, and it would be fully settled if the official leaderboard record and released code were made available.","tokens_in":7936,"tokens_out":7940,"duration_ms":73009,"concrete_test":"Contact the INFORMS 2024 Data Challenge organizers to obtain the official leaderboard for Team 25 and the final submitted prediction file. If Team 25 is not listed first, or if a re-computation of MAPE on the official test set using the released code differs from Table 1 by more than 0.1 percentage points, the first-place claim is falsified. If the leaderboard is not public, require the authors to release code, feature vectors, and a nested cross-validation script; a held-out MAPE substantially above Table 1's values would indicate selection on the leaderboard.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Team 25 achieved first place on the INFORMS 2024 Data Challenge leaderboard with the MAPE values in Table 1 (62.56% comment, 65.28% heart, 63.38% play, 80.23% share). For this claim to be valid, those numbers must be computed on the competition's official test set under its official metric, and model/feature selection must not have used repeated leaderboard feedback. The paper does not support either condition. Section 2.3 says hyperparameters were tuned with Bayesian optimization and 10-fold cross-validation, but it never states which split produced Table 1's numbers, nor whether the feature-subset search in Table 3 or the decision to average neural-network and XGBoost outputs (Section 2.4) was made with knowledge of leaderboard scores. The sentence 'The code, video features extracted by our model and final result are Google Colab, Google Drive and Final Result' contains no links, so no artifact is available for inspection. Table 2 reports many models, but the 'Final Fusion' row appears only in Table 1, with no description of the fusion formula (e.g., weights, log-space vs. raw-space averaging). Without the official leaderboard record, the other teams' scores, and a reproducible evaluation pipeline, the reported MAPE values are self-reported numbers that could equally be validation-set performance or the result of leaderboard overfitting. The first-place claim is therefore unverifiable and the empirical central claim is not secure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a multi-modal pipeline for predicting four engagement metrics (comment, heart, play, and share counts) for short videos in the INFORMS 2024 Data Challenge. The authors extract video features with four pre-trained models (TimeSformer, ViViT, VideoMAE, X-CLIP), generate textual video descriptions with LLaVA-NeXT and InternVideo2, and encode these descriptions with BERT. They combine the resulting neural features with engineered tabular features (hashtag and mention frequencies, duration, frame statistics, time-related variables) in XGBoost, and average the XGBoost predictions with neural-network predictions. The paper claims first place on the leaderboard with final MAPE values of 62.56%, 65.28%, 63.38%, and 80.23% for comment, heart, play, and share counts.","tokens_in":8217,"tokens_out":7170,"duration_ms":64558,"significance":"If the reported leaderboard result is reliable, the paper demonstrates that a fusion of tabular features, pre-trained video backbones, and video-to-text descriptions can outperform other competition entries on a public benchmark. The observation that averaging a conservative neural predictor with a higher-variance tree ensemble reduces MAPE is plausible and worth reporting. The strengths are the internal consistency of the ablation tables (the 'Video' row in Table 1 matches the all-six-features row in Table 3) and the concrete feature-engineering ideas. The decisive weakness is that the paper provides no verifiable evaluation protocol, no leaderboard evidence, and no code or data artifacts, so the central empirical claim cannot be independently assessed.","major_comments":[{"comment":"The paper's central empirical claim—first place on the INFORMS 2024 leaderboard with MAPE 62.56/65.28/63.38/80.23—is not accompanied by a description of the evaluation protocol. The text does not state whether Table 1 was computed on the official test set, on a local validation split, or after leaderboard feedback, nor does it report the test-set size or the number of teams. Without this information, the reported numbers and the first-place claim cannot be independently verified; this is load-bearing because the entire contribution is the competition result.","section":"§2.4 and Table 1"},{"comment":"The IQR-based outlier removal is described without specifying whether it is applied only to the training set or also to the test set. If outliers are identified using test-set labels or test-set statistics, the MAPE values in Tables 1 and 2 are optimistically biased. The section also does not state whether the ten-fold cross-validation used for hyperparameter tuning is the same split that produced Table 2, or which split was used to select XGBoost among the 26 models in Table 2.","section":"§2.3"},{"comment":"The averaging procedure is underspecified. The paper says 'the predictions from the neural network and XGBoost models are averaged' without giving weights, a transformation (raw vs. log space), or a search range for the weights. For share, the final MAPE improves from 85.04% and 94.78% to 80.23%, which is possible with negatively correlated errors, but no residual analysis or error bars are provided. The 'Final Fusion' row in Table 1 is therefore non-reproducible as written.","section":"§2.4"},{"comment":"The ablation in Table 3 reports MAPE for many feature subsets, and the conclusion states that X-CLIP performs best and LLaVA-NeXT is best suited for text generation. Since no selection rule or validation procedure is described, the final feature subset and the choice of XGBoost may reflect overfitting to the validation set. A nested cross-validation or a pre-specified selection criterion is needed to support the generalization claim.","section":"Table 3 and §2.2"},{"comment":"The sentence 'The code, video features extracted by our model and final result are Google Colab, Google Drive and Final Result' contains no links, so no code, extracted features, or final result files are available. This prevents any reproducibility check of the reported MAPE values.","section":"Appendix and Conclusion"}],"minor_comments":[{"comment":"There is a typo in the affiliation: 'New Souuth Wales' should be 'New South Wales'.","section":"Title page"},{"comment":"The model name 'LLaV A-NeXT' should be written as 'LLaVA-NeXT', and the caption 'Feature Importanceof' in Figure 4 is missing a space.","section":"§2.2 and Figure 4"},{"comment":"Table 2 reports XGBoost MAPEs of 0.64/0.65/0.63/0.95, while Table 1's Tabular row gives 63.51/65.31/63.46/94.78; the discrepancy and the underlying evaluation split are not explained.","section":"Table 2 and Table 1"},{"comment":"Table 3 lists columns in the order Share, Heart, Comment, Play, whereas Table 1 uses Comment, Heart, Play, Share; reordering would improve comparability.","section":"Table 3"},{"comment":"Figures 3–6 are referenced but not described in the text; no information is given about what features are displayed or how feature importance is computed.","section":"Figures 3–6"},{"comment":"Several references are incomplete; for example, 'Liu H, Li C, Wu Q, Lee YJ (2023) Visual instruction tuning' has no venue, and the Kenton and Toutanova entry lacks page numbers.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper reads as a competition write-up rather than a self-contained research article. The first-place claim is not independently verifiable from the manuscript: no evaluation protocol, leaderboard record, or artifacts are provided. I would not recommend acceptance until the authors supply a documented train/test split, explicit outlier-removal policy, the fusion formula, and either code/data or the official leaderboard evidence. If the journal does not normally publish competition reports, the editors may also want to consider scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"John,\n\nQuick take: this is a decent INFORMS Data Challenge write-up with a useful empirical ablation of video backbones, but the headline claim — first place on the leaderboard — is not verifiable from the manuscript. The paper does not describe the test split, the fusion formula, or the selection procedure, and the line about code/video features/results contains no links. So treat the MAPE numbers as self-reported.\n\nWhat's good: Table 3 is a thorough ablation over six feature sets (four video backbones plus two video-to-text models), and the comparison across dozens of combinations is a real contribution — it tells you which backbones help for popularity prediction. The feature engineering on hashtags/mentions, video duration, and time online is sensible, and the IQR outlier removal is a reasonable practical choice. The observation that neural nets under-predict and XGBoost over-predicts, and that averaging reduces MAPE, is a nice empirical point, and the numbers in Table 2 are internally consistent.\n\nThe soft spots are the load-bearing ones. No code, data, or leaderboard exhibit. No statement about whether Table 1 numbers came from the official test set under the competition metric, or whether model/feature choices were made with leaderboard feedback. The final fusion formula isn't given. There are no error bars. So the first-place claim is a claim, not a result.\n\nThat said, this is a competition write-up, not a methods paper. The bar for verifiability should match the venue. If the authors release the artifacts and document the protocol, the empirical content is solid enough for a workshop or data-challenge report. As a journal submission, it would need a major revision.\n\nFor me: I'd cite it if I worked on short-video popularity prediction, because the backbone comparison is useful. I'd bring it to a reading group as an example of engineering practice. A serious referee could check the internal consistency and request artifacts — so I'd send it to review rather than desk reject, but with a strong request for reproducibility.\n\nBest.","headline":"A credible competition write-up whose first-place claim rests on missing artifacts and an undocumented evaluation protocol; the ablations are useful but the central number is unverifiable.","tokens_in":8794,"tokens_out":2265,"would_cite":true,"duration_ms":21846,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that video, caption-text, and tabular features can be combined and averaged to predict short-video engagement well enough to win the INFORMS 2024 Data Challenge popularity prediction competition.","keywords":["popularity prediction","short video","multimodal feature extraction","XGBoost","feature engineering","video-to-text generation","MAPE","neural network ensemble"],"falsifier":"Recomputing the four MAPE values on the official test set from the team's frozen submitted predictions would settle it: if the reproduced numbers match Table 1, the claim stands; if they drift substantially, the original numbers reflected selection on leaderboard feedback. A weaker but immediate check is whether retraining the same pipeline on random train/validation splits reproduces the reported 62–80% MAPE range without any test-set touches.","tokens_in":7715,"feed_emoji":"🎬","tokens_out":6901,"duration_ms":60393,"temperature":0.7,"pith_summary":"This paper claims that a carefully mixed diet of video pixels, generated caption text, and hand-built tabular statistics can predict how many comments, hearts, plays, and shares a short video will earn. The authors combine four pretrained video classifiers (TimeSformer, ViViT, VideoMAE, X-CLIP) with two video-to-text models (LLaVA-NeXT and InternVideo2) whose outputs are BERT-encoded, then fuse all six vector sets in a neural network. On the tabular side they engineer features such as hashtag and mention frequency, video duration, frame count, and time since posting, and feed them to XGBoost. The final prediction averages the neural network and XGBoost outputs, which the authors say lowers mean absolute percentage error because the two model families err in opposite directions. They report MAPEs of 62.56% for comments, 65.28% for hearts, 63.38% for plays, and 80.23% for shares, and state this won first place on the leaderboard.","feed_headline":"Winning video-popularity model blends pixels, captions, and stats","feed_subtitle":"The winning entry predicts comments, hearts, plays, and shares with 62–80% average error by averaging a neural net and XGBoost.","key_machinery":"The mechanism is a two-track fusion pipeline with a symmetry-breaking final average. Track one feeds six sets of feature vectors — four from video classification backbones (TimeSformer, ViViT, VideoMAE, X-CLIP) and two from BERT-encoded text descriptions produced by LLaVA-NeXT and InternVideo2 responding to an eight-question prompt about content, pacing, camera work, plot, presentation, and emotional tone — into a neural network with per-modality normalization. Track two builds tabular features, notably the summed corpus frequencies of hashtags and mentions, video duration, frame count, frame rate, resolution, a normalized posting timestamp, and log-transformed author engagement counts, and trains XGBoost with Bayesian-tuned hyperparameters. The final predictor averages the two tracks' outputs, exploiting the paper's observation that neural networks trained on mean squared error give conservative estimates on right-skewed counts while tree ensembles give expansive ones.","core_discovery":"The central claim is that no single modality suffices: useful popularity prediction emerges from averaging a conservative neural network and an expansive tree ensemble. The neural network is trained on video features from TimeSformer, ViViT, VideoMAE, and X-CLIP plus BERT-encoded text generated by LLaVA-NeXT and InternVideo2 from a prompt about content, pacing, camera movement, plot, and emotional tone. The tree ensemble is XGBoost trained on engineered tabular features, including aggregated hashtag and mention frequencies, video technical metadata, posting-time features, and log-transformed author statistics. The authors observe that the neural network underestimates and XGBoost overestimates on skewed interaction counts, so the averaged prediction is more accurate than either alone. Table 1 reports the final fusion MAPE values of 62.56%, 65.28%, 63.38%, and 80.23% for comment, heart, play, and share counts, and the paper states this result took first place on the leaderboard.","pith_inferences":["If the winning result transfers, the most portable lesson is not the specific backbones but the error-cancelling average: any pair of predictors with opposite biases on skewed count data can be combined the same way, with no extra data collection.","The engineered 'time since posting' feature encodes exposure time, which may dominate the popularity signal; on platforms where recommendation windows differ, the relative ranking of the video and tabular tracks could shift.","A direct testable extension would be to ablate the six video/text feature sets on a fresh platform and compare their marginal gains against the tabular track alone; Table 3 suggests the video features add less than the engineered statistics, but the paper does not isolate this comparison.","Because the competition test set is fixed, the first-place claim could be independently checked if the organizers release the submitted predictions; absent that release, the reported MAPEs are self-reported."],"forward_implications":["Averaging a conservative neural network with an expansive gradient-boosted tree model reduces MAPE on all four engagement metrics compared with either model alone.","Hashtag and mention frequency aggregation, video duration, and time-since-posting are the most useful engineered features for popularity prediction; the paper highlights duration and time-since-posting in the feature importance analysis.","Among the open-source video backbones tested, X-CLIP produces the best single video feature set for popularity prediction, while LLaVA-NeXT is the better text generator for short-video understanding.","Removing outliers with the interquartile range before training materially improves the regression performance of tree models on skewed interaction counts."],"supporting_citations":[{"why":"Supplies the TimeSformer video classification backbone that produces one of the four video feature sets.","marker":"Bertasius et al. 2021"},{"why":"Supplies the ViViT video transformer whose tubelet-embedded features enter the fusion network.","marker":"Arnab et al. 2021"},{"why":"Supplies VideoMAE, the masked-autoencoder video backbone used as a fourth feature extractor.","marker":"Tong et al. 2022"},{"why":"Supplies X-CLIP, the video-text alignment model reported as the strongest single video feature extractor.","marker":"Ni et al. 2022"},{"why":"Together with Liu et al. 2023, defines the LLaVA-NeXT video-to-text model that generates the prompt-driven caption descriptions.","marker":"Zhang et al. 2024"},{"why":"Provides the visual instruction tuning foundation of LLaVA-NeXT used for text generation.","marker":"Liu et al. 2023"},{"why":"Supplies InternVideo2, the second video-to-text generator whose outputs are BERT-encoded for the fusion network.","marker":"Wang et al. 2024"},{"why":"Defines XGBoost, the tree ensemble trained on engineered tabular features and averaged with the neural network.","marker":"Chen and Guestrin 2016"},{"why":"Provides BERT, the encoder that turns the generated video descriptions into the 768-dimensional text vectors.","marker":"Kenton and Toutanova 2019"}],"fun_headline_variants":["Neural net plus XGBoost fusion wins video popularity","Blending pixels, captions, and stats predicts virality","Averaging two models yields best short-video forecasts","Multi-modal features and model averaging top leaderboard"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported MAPE values and the first-place finish assume the fixed competition test set was never used to guide model or feature selection — if leaderboard feedback influenced the choice of XGBoost, the feature subsets, or the averaging weights, the numbers would be optimistic.","fun_headline_variants_meta":{"raw":{"variants":["Neural net plus XGBoost fusion wins video popularity","Blending pixels, captions, and stats predicts virality","Averaging two models yields best short-video forecasts","Multi-modal features and model averaging top leaderboard"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1389,"prompt_tokens":973,"completion_tokens":416,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":349}},"tokens_in":589,"tokens_out":416,"duration_ms":5084,"temperature":1.0,"reasoning_tokens":349,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:27:28.588840+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recomputing the four MAPE values on the official test set from the team's frozen submitted predictions would settle it: if the reproduced numbers match Table 1, the claim stands; if they drift substantially, the original numbers reflected selection on leaderboard feedback. A weaker but immediate check is whether retraining the same pipeline on random train/validation splits reproduces the reported 62–80% MAPE range without any test-set touches.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the TimeSformer video classification backbone that produces one of the four video feature sets."},{"cited_title":"Proceedings of the IEEE/CVF international conference on computer vision, 6836--6846","cited_arxiv_id":null,"evidence_quote":"Supplies the ViViT video transformer whose tubelet-embedded features enter the fusion network."},{"cited_title":"Advances in neural information processing systems 35:10078--10093","cited_arxiv_id":null,"evidence_quote":"Supplies VideoMAE, the masked-autoencoder video backbone used as a fourth feature extractor."},{"cited_title":"://llava-vl.github.io/blog/2024-04-30-llava-next-video/","cited_arxiv_id":null,"evidence_quote":"Together with Liu et al. 2023, defines the LLaVA-NeXT video-to-text model that generates the prompt-driven caption descriptions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the visual instruction tuning foundation of LLaVA-NeXT used for text generation."},{"cited_title":"Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 785--794","cited_arxiv_id":null,"evidence_quote":"Defines XGBoost, the tree ensemble trained on engineered tabular features and averaged with the neural network."},{"cited_title":"Proceedings of naacL-HLT, volume 1, 2 (Minneapolis, Minnesota)","cited_arxiv_id":null,"evidence_quote":"Provides BERT, the encoder that turns the generated video descriptions into the 768-dimensional text vectors."}],"review_version":1}