Multi-Modal Video Feature Extraction for Popularity Prediction

Haixu Liu; Haoxiang Zheng; Penghao Jiang; Qirui Wang; Qiuzhuang Sun; Ruiqing Yan; Wenning Wang

REVIEW 5 major objections 6 minor 1 cited by

Multi-Modal Video Feature Extraction for Popularity Prediction

T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that video, caption-text, and tabular features can be combined and averaged to predict short-video engagement well enough to win the INFORMS 2024 Data Challenge popularity prediction competition.

desk verdict A credible competition write-up whose first-place claim rests on missing artifacts and an undocumented evaluation protocol; the ablations are useful but the central number is unverifiable. read the letter →

arxiv 2501.01422 v1 pith:FP4JMK43 submitted 2025-01-02 cs.CV cs.AIcs.LG

Haixu Liu , Wenning Wang , Haoxiang Zheng , Penghao Jiang , Qirui Wang , Ruiqing Yan , Qiuzhuang Sun This is my paper · ORCID

classification cs.CVcs.AIcs.LG

keywords popularitypredictionshortvideomultimodalfeatureextractionXGBoostengineeringvideo-to-textgenerationMAPEneuralnetworkensemble

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a carefully mixed diet of video pixels, generated caption text, and hand-built tabular statistics can predict how many comments, hearts, plays, and shares a short video will earn. The authors combine four pretrained video classifiers (TimeSformer, ViViT, VideoMAE, X-CLIP) with two video-to-text models (LLaVA-NeXT and InternVideo2) whose outputs are BERT-encoded, then fuse all six vector sets in a neural network. On the tabular side they engineer features such as hashtag and mention frequency, video duration, frame count, and time since posting, and feed them to XGBoost. The final prediction averages the neural network and XGBoost outputs, which the authors say lowers mean absolute percentage error because the two model families err in opposite directions. They report MAPEs of 62.56% for comments, 65.28% for hearts, 63.38% for plays, and 80.23% for shares, and state this won first place on the leaderboard.

What carries the argument

The mechanism is a two-track fusion pipeline with a symmetry-breaking final average. Track one feeds six sets of feature vectors — four from video classification backbones (TimeSformer, ViViT, VideoMAE, X-CLIP) and two from BERT-encoded text descriptions produced by LLaVA-NeXT and InternVideo2 responding to an eight-question prompt about content, pacing, camera work, plot, presentation, and emotional tone — into a neural network with per-modality normalization. Track two builds tabular features, notably the summed corpus frequencies of hashtags and mentions, video duration, frame count, frame rate, resolution, a normalized posting timestamp, and log-transformed author engagement counts, and trains XGBoost with Bayesian-tuned hyperparameters. The final predictor averages the two tracks' outputs, exploiting the paper's observation that neural networks trained on mean squared error give conservative estimates on right-skewed counts while tree ensembles give expansive ones.

What would settle it

Recomputing the four MAPE values on the official test set from the team's frozen submitted predictions would settle it: if the reproduced numbers match Table 1, the claim stands; if they drift substantially, the original numbers reflected selection on leaderboard feedback. A weaker but immediate check is whether retraining the same pipeline on random train/validation splits reproduces the reported 62–80% MAPE range without any test-set touches.

Watch

Extended reading notes

Core claim

The central claim is that no single modality suffices: useful popularity prediction emerges from averaging a conservative neural network and an expansive tree ensemble. The neural network is trained on video features from TimeSformer, ViViT, VideoMAE, and X-CLIP plus BERT-encoded text generated by LLaVA-NeXT and InternVideo2 from a prompt about content, pacing, camera movement, plot, and emotional tone. The tree ensemble is XGBoost trained on engineered tabular features, including aggregated hashtag and mention frequencies, video technical metadata, posting-time features, and log-transformed author statistics. The authors observe that the neural network underestimates and XGBoost overestimates on skewed interaction counts, so the averaged prediction is more accurate than either alone. Table 1 reports the final fusion MAPE values of 62.56%, 65.28%, 63.38%, and 80.23% for comment, heart, play, and share counts, and the paper states this result took first place on the leaderboard.

Load-bearing premise

The reported MAPE values and the first-place finish assume the fixed competition test set was never used to guide model or feature selection — if leaderboard feedback influenced the choice of XGBoost, the feature subsets, or the averaging weights, the numbers would be optimistic.

Editorial extensions

If this is right

Averaging a conservative neural network with an expansive gradient-boosted tree model reduces MAPE on all four engagement metrics compared with either model alone.
Hashtag and mention frequency aggregation, video duration, and time-since-posting are the most useful engineered features for popularity prediction; the paper highlights duration and time-since-posting in the feature importance analysis.
Among the open-source video backbones tested, X-CLIP produces the best single video feature set for popularity prediction, while LLaVA-NeXT is the better text generator for short-video understanding.
Removing outliers with the interquartile range before training materially improves the regression performance of tree models on skewed interaction counts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the winning result transfers, the most portable lesson is not the specific backbones but the error-cancelling average: any pair of predictors with opposite biases on skewed count data can be combined the same way, with no extra data collection.
The engineered 'time since posting' feature encodes exposure time, which may dominate the popularity signal; on platforms where recommendation windows differ, the relative ranking of the video and tabular tracks could shift.
A direct testable extension would be to ablate the six video/text feature sets on a fresh platform and compare their marginal gains against the tabular track alone; Table 3 suggests the video features add less than the engineered statistics, but the paper does not isolate this comparison.
Because the competition test set is fixed, the first-place claim could be independently checked if the organizers release the submitted predictions; absent that release, the reported MAPEs are self-reported.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

John,

Quick take: this is a decent INFORMS Data Challenge write-up with a useful empirical ablation of video backbones, but the headline claim — first place on the leaderboard — is not verifiable from the manuscript. The paper does not describe the test split, the fusion formula, or the selection procedure, and the line about code/video features/results contains no links. So treat the MAPE numbers as self-reported.

What's good: Table 3 is a thorough ablation over six feature sets (four video backbones plus two video-to-text models), and the comparison across dozens of combinations is a real contribution — it tells you which backbones help for popularity prediction. The feature engineering on hashtags/mentions, video duration, and time online is sensible, and the IQR outlier removal is a reasonable practical choice. The observation that neural nets under-predict and XGBoost over-predicts, and that averaging reduces MAPE, is a nice empirical point, and the numbers in Table 2 are internally consistent.

The soft spots are the load-bearing ones. No code, data, or leaderboard exhibit. No statement about whether Table 1 numbers came from the official test set under the competition metric, or whether model/feature choices were made with leaderboard feedback. The final fusion formula isn't given. There are no error bars. So the first-place claim is a claim, not a result.

That said, this is a competition write-up, not a methods paper. The bar for verifiability should match the venue. If the authors release the artifacts and document the protocol, the empirical content is solid enough for a workshop or data-challenge report. As a journal submission, it would need a major revision.

For me: I'd cite it if I worked on short-video popularity prediction, because the backbone comparison is useful. I'd bring it to a reading group as an example of engineering practice. A serious referee could check the internal consistency and request artifacts — so I'd send it to review rather than desk reject, but with a strong request for reproducibility.

Best.

Referee Report

5 major / 6 minor

Summary. This paper reports a multi-modal pipeline for predicting four engagement metrics (comment, heart, play, and share counts) for short videos in the INFORMS 2024 Data Challenge. The authors extract video features with four pre-trained models (TimeSformer, ViViT, VideoMAE, X-CLIP), generate textual video descriptions with LLaVA-NeXT and InternVideo2, and encode these descriptions with BERT. They combine the resulting neural features with engineered tabular features (hashtag and mention frequencies, duration, frame statistics, time-related variables) in XGBoost, and average the XGBoost predictions with neural-network predictions. The paper claims first place on the leaderboard with final MAPE values of 62.56%, 65.28%, 63.38%, and 80.23% for comment, heart, play, and share counts.

Significance. If the reported leaderboard result is reliable, the paper demonstrates that a fusion of tabular features, pre-trained video backbones, and video-to-text descriptions can outperform other competition entries on a public benchmark. The observation that averaging a conservative neural predictor with a higher-variance tree ensemble reduces MAPE is plausible and worth reporting. The strengths are the internal consistency of the ablation tables (the 'Video' row in Table 1 matches the all-six-features row in Table 3) and the concrete feature-engineering ideas. The decisive weakness is that the paper provides no verifiable evaluation protocol, no leaderboard evidence, and no code or data artifacts, so the central empirical claim cannot be independently assessed.

major comments (5)

[§2.4 and Table 1] The paper's central empirical claim—first place on the INFORMS 2024 leaderboard with MAPE 62.56/65.28/63.38/80.23—is not accompanied by a description of the evaluation protocol. The text does not state whether Table 1 was computed on the official test set, on a local validation split, or after leaderboard feedback, nor does it report the test-set size or the number of teams. Without this information, the reported numbers and the first-place claim cannot be independently verified; this is load-bearing because the entire contribution is the competition result.
[§2.3] The IQR-based outlier removal is described without specifying whether it is applied only to the training set or also to the test set. If outliers are identified using test-set labels or test-set statistics, the MAPE values in Tables 1 and 2 are optimistically biased. The section also does not state whether the ten-fold cross-validation used for hyperparameter tuning is the same split that produced Table 2, or which split was used to select XGBoost among the 26 models in Table 2.
[§2.4] The averaging procedure is underspecified. The paper says 'the predictions from the neural network and XGBoost models are averaged' without giving weights, a transformation (raw vs. log space), or a search range for the weights. For share, the final MAPE improves from 85.04% and 94.78% to 80.23%, which is possible with negatively correlated errors, but no residual analysis or error bars are provided. The 'Final Fusion' row in Table 1 is therefore non-reproducible as written.
[Table 3 and §2.2] The ablation in Table 3 reports MAPE for many feature subsets, and the conclusion states that X-CLIP performs best and LLaVA-NeXT is best suited for text generation. Since no selection rule or validation procedure is described, the final feature subset and the choice of XGBoost may reflect overfitting to the validation set. A nested cross-validation or a pre-specified selection criterion is needed to support the generalization claim.
[Appendix and Conclusion] The sentence 'The code, video features extracted by our model and final result are Google Colab, Google Drive and Final Result' contains no links, so no code, extracted features, or final result files are available. This prevents any reproducibility check of the reported MAPE values.

minor comments (6)

[Title page] There is a typo in the affiliation: 'New Souuth Wales' should be 'New South Wales'.
[§2.2 and Figure 4] The model name 'LLaV A-NeXT' should be written as 'LLaVA-NeXT', and the caption 'Feature Importanceof' in Figure 4 is missing a space.
[Table 2 and Table 1] Table 2 reports XGBoost MAPEs of 0.64/0.65/0.63/0.95, while Table 1's Tabular row gives 63.51/65.31/63.46/94.78; the discrepancy and the underlying evaluation split are not explained.
[Table 3] Table 3 lists columns in the order Share, Heart, Comment, Play, whereas Table 1 uses Comment, Heart, Play, Share; reordering would improve comparability.
[Figures 3–6] Figures 3–6 are referenced but not described in the text; no information is given about what features are displayed or how feature importance is computed.
[References] Several references are incomplete; for example, 'Liu H, Li C, Wu Q, Lee YJ (2023) Visual instruction tuning' has no venue, and the Kenton and Toutanova entry lacks page numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: predictions are standard supervised-learning outputs evaluated against an external competition benchmark.

full rationale

The paper's derivation chain is a conventional supervised-learning pipeline: tabular features are engineered, video and text features are extracted with pre-trained models, and gradient-boosted trees and neural networks are trained to predict four engagement metrics. The reported MAPE values are evaluated against an external INFORMS 2024 Data Challenge benchmark, not derived from the paper's own assumptions or fitted parameters. The choice of XGBoost and the averaging of neural-network and XGBoost outputs are justified by validation-set comparisons reported in Table 2, which is standard model selection rather than circularity. No parameter is fitted to the test set and then renamed as a prediction, and no load-bearing claim rests on a self-citation: all cited backbones are independent published models. The unverifiability of the first-place leaderboard claim and the absence of code or official leaderboard data are reproducibility concerns, not instances of the prediction reducing to its inputs. Therefore no significant circularity is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new theoretical entities are introduced. The central numbers are empirical fits on a private dataset, and the model relies on standard supervised learning plus several domain assumptions about social-media engagement. Key hyperparameters and preprocessing choices are unspecified.

free parameters (4)

XGBoost hyperparameters = not reported
Tuned with Bayesian optimization and ten-fold cross-validation in Section 2.3, but the resulting values are not given, so the tabular baseline depends on unspecified tuning.
Neural network architecture hyperparameters = not reported
Layer sizes, dropout rates, learning rate, and early stopping patience are not specified in Section 2.3, so the fusion model cannot be reconstructed.
IQR outlier removal threshold = not reported
Section 2.3 removes outliers using the interquartile range, but the multiplier is not stated and the removal affects training labels and reported MAPE.
Prompt framework for video-to-text models = hand-designed
The eight-question prompt in Section 2.2 is manually adjusted without a quantitative comparison of prompt variants, yet it shapes the BERT text vectors.

assumptions (4)

domain assumption Video duration, frame rate, and resolution affect viewing and interaction patterns.
Invoked in Section 2.1 to justify engineering video-quality features; no evidence is provided that these properties causally affect engagement.
domain assumption Posting time, day, and holiday status influence user activity and engagement.
Invoked in Section 2.1 to justify time-based features; the paper does not validate this hypothesis beyond including the features.
domain assumption Pretrained video classification and video-to-text models transfer to popularity prediction on this short-video dataset.
Relied on throughout Section 2.2; the models were pretrained on generic video understanding tasks, not on engagement prediction.
domain assumption Hashtag and mention frequencies capture exposure and recommendation effects.
Invoked in Section 2.1 to justify the four hashtag and mention frequency features; no direct evidence from the platform is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Multi-Modal Video Feature Extraction for Popularity Prediction." pith.science (2026). https://pith.science/paper/FP4JMK43

@misc{pith2026250101422,
  author       = {Pith},
  title        = {Pith review of: Multi-Modal Video Feature Extraction for Popularity Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FP4JMK43}},
  note         = {Machine review of arXiv:2501.01422}
}

read the original abstract

This work aims to predict the popularity of short videos using the videos themselves and their related features. Popularity is measured by four key engagement metrics: view count, like count, comment count, and share count. This study employs video classification models with different architectures and training methods as backbone networks to extract video modality features. Meanwhile, the cleaned video captions are incorporated into a carefully designed prompt framework, along with the video, as input for video-to-text generation models, which generate detailed text-based video content understanding. These texts are then encoded into vectors using a pre-trained BERT model. Based on the six sets of vectors mentioned above, a neural network is trained for each of the four prediction metrics. Moreover, the study conducts data mining and feature engineering based on the video and tabular data, constructing practical features such as the total frequency of hashtag appearances, the total frequency of mention appearances, video duration, frame count, frame rate, and total time online. Multiple machine learning models are trained, and the most stable model, XGBoost, is selected. Finally, the predictions from the neural network and XGBoost models are averaged to obtain the final result.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

MVP: Winning Solution to SMP Challenge 2025 Video Track
cs.CV 2025-07 conditional novelty 3.0 of 10

MVP, a pipeline using XCLIP video features, user metadata, and a CatBoost regressor, won the SMP Challenge 2025 Video Track with a MAPE of 0.1754.

Reference graph

Works this paper leans on

15 extracted references · 9 canonical work pages · cited by 1 Pith paper

[1]

, " * write output.state after.block = add.period write newline

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence := #2 '...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in " " * FUNCTION format....

work page
[3]

, " * write output.state after.block = add.period write newline

ENTRY address author booktitle chapter doi edition editor eid howpublished institution isbn issn journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1...
[4]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in "" FUNCTION format.date year ...
[5]

Proceedings of the IEEE/CVF international conference on computer vision, 6836--6846

Arnab A, Dehghani M, Heigold G, Sun C, Lu c i \'c M, Schmid C (2021) Vivit: A video vision transformer. Proceedings of the IEEE/CVF international conference on computer vision, 6836--6846

work page 2021
[6]

Bertasius G, Wang H, Torresani L (2021) Is space-time attention all you need for video understanding? ICML, volume 2, 4

work page 2021
[7]

Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 785--794

Chen T, Guestrin C (2016) Xgboost: A scalable tree boosting system. Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 785--794

work page 2016
[8]

(2024) The llama 3 herd of models

Dubey A, Jauhri A, Pandey A, Kadian A, Al-Dahle A, Letman A, Mathur A, Schelten A, Yang A, Fan A, et al. (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783

arXiv 2024

Show all 15 references

[9]

(2023) Mistral 7b

Jiang AQ, Sablayrolles A, Mensch A, Bamford C, Chaplot DS, Casas Ddl, Bressand F, Lengyel G, Lample G, Saulnier L, et al. (2023) Mistral 7b. arXiv preprint arXiv:2310.06825

2023 arXiv
[10]

Proceedings of naacL-HLT, volume 1, 2 (Minneapolis, Minnesota)

Kenton JDMWC, Toutanova LK (2019) Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of naacL-HLT, volume 1, 2 (Minneapolis, Minnesota)

2019
[11]

Liu H, Li C, Wu Q, Lee YJ (2023) Visual instruction tuning

2023
[12]

European Conference on Computer Vision, 1--18 (Springer)

Ni B, Peng H, Chen M, Zhang S, Meng G, Fu J, Xiang S, Ling H (2022) Expanding language-image pretrained models for general video recognition. European Conference on Computer Vision, 1--18 (Springer)

2022
[13]

Advances in neural information processing systems 35:10078--10093

Tong Z, Song Y, Wang J, Wang L (2022) Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems 35:10078--10093

2022
[14]

(2024) Internvideo2: Scaling video foundation models for multimodal video understanding

Wang Y, Li K, Li X, Yu J, He Y, Chen G, Pei B, Zheng R, Xu J, Wang Z, et al. (2024) Internvideo2: Scaling video foundation models for multimodal video understanding. arXiv preprint arXiv:2403.15377

2024 arXiv
[15]

://llava-vl.github.io/blog/2024-04-30-llava-next-video/

Zhang Y, Li B, Liu h, Lee Yj, Gui L, Fu D, Feng J, Liu Z, Li C (2024) Llava-next: A strong zero-shot video understanding model. ://llava-vl.github.io/blog/2024-04-30-llava-next-video/

2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence := #2 '...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in " " * FUNCTION format....

work page

[3] [3]

, " * write output.state after.block = add.period write newline

ENTRY address author booktitle chapter doi edition editor eid howpublished institution isbn issn journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1...

[4] [4]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in "" FUNCTION format.date year ...

[5] [5]

Proceedings of the IEEE/CVF international conference on computer vision, 6836--6846

Arnab A, Dehghani M, Heigold G, Sun C, Lu c i \'c M, Schmid C (2021) Vivit: A video vision transformer. Proceedings of the IEEE/CVF international conference on computer vision, 6836--6846

work page 2021

[6] [6]

Bertasius G, Wang H, Torresani L (2021) Is space-time attention all you need for video understanding? ICML, volume 2, 4

work page 2021

[7] [7]

Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 785--794

Chen T, Guestrin C (2016) Xgboost: A scalable tree boosting system. Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 785--794

work page 2016

[8] [8]

(2024) The llama 3 herd of models

Dubey A, Jauhri A, Pandey A, Kadian A, Al-Dahle A, Letman A, Mathur A, Schelten A, Yang A, Fan A, et al. (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783

arXiv 2024