REVIEW 4 major objections 3 minor 30 references
Automated Let's Play Commentary
T0 review · 4 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A convolutional network trained on all frames predicts Minecraft Let's Play commentary sentence embeddings more accurately than a clustering-based baseline, but all models fall well short of human-quality text.
desk verdict A small, honest paper whose real value is the new Minecraft frame-commentary dataset; the empirical comparison is confounded and the metric is unvalidated, but the authors do not oversell what they have. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery that carries the argument is a convolutional network reading a single gameplay frame and outputting a 512-dimensional sentence embedding. The target representation is produced by the Universal Sentence Encoder, which turns each spoken commentary sentence into the vector the network is trained to approximate; the same embedding space also supplies the evaluation metric. The baseline variant replaces the one global model with K-medoids clustering over concatenated image and sentence embeddings—a clustering method whose centers are actual data points—followed by separate CNNs trained on the largest resulting clusters. The essential contrast the experiment tests is therefore a single model over all data versus several models over specialized subsets, with the sentence embedding as the fixed prediction target in both cases.
What would settle it
Train the same CNN on random subsets of sizes 684, 802, and 1,336 matched to the three largest baseline clusters and compare percentile error on the same test set; if error stays near 0.961, the advantage is data volume, not the unified-model idea, and if human raters cannot distinguish lower-error predictions from true commentary, the metric itself is not measuring commentary quality.
Extended reading notes
Core claim
The central claim is that a single convolutional network, trained on all 4,840 frame-commentary pairs, predicts human Let's Play commentary more accurately than the clustering-based baseline on this dataset. On the paper's evaluation measure—average percentile error between predicted and true sentence embeddings—the unified CNN achieves 0.961 ± 0.026, while the baseline's three largest per-cluster models achieve 0.977 ± 0.023, 0.975 ± 0.042, and 0.980 ± 0.024. The authors interpret this as showing that a CNN can produce artificial commentary directly from pixels, and that per-cluster specialization does not help at this data scale, while also acknowledging that all approaches fall well short of human-quality commentary.
Load-bearing premise
The central claim rests on assuming that a low average percentile error in the sentence-representation space used by the evaluation actually means better or more human-like commentary, and that it is fair to compare a model trained on all 4,840 instances with models trained on only 684 to 1,336 instances.
Editorial extensions
If this is right
- A single CNN mapping raw frames to sentence embeddings is a usable, simple baseline for automated Let's Play commentary, requiring no access to the game engine or action logs.
- The public Minecraft dataset of 4,840 frame-commentary pairs gives future work a common corpus for comparing video-to-commentary models.
- Per-cluster training, as in the prior approach, offers no advantage over a global model at this dataset size, so future comparisons can treat the unified CNN as the reference point.
- The gap between 0.961 and the ideal 0.0 percentile error shows that supervised frame-to-commentary prediction remains largely unsolved and likely needs larger datasets or additional context.
Reading between the lines
- The reported superiority may come from training-set size rather than the global-model idea, since the unified network sees 4,840 instances while each cluster model sees only 684 to 1,336; matching training sizes would settle this.
- The evaluation metric, which averages percentile error in a fixed embedding space, may reward generic predictions that sit near the center of the space; the repeated nearest-neighbor sentences shown in the paper's examples hint that specificity is not being rewarded.
- Because every frame within an utterance is paired with the same sentence, the dataset contains many duplicate targets; a temporal model that consumes short frame sequences, or that predicts once per utterance, is a natural untested extension.
- If the central claim generalizes, the same architecture should transfer to other visually simple games with large Let's Play corpora, and the reported accuracy would serve as a sanity-check baseline for those games.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a short, exploratory study of automated Let's Play commentary for Minecraft. The authors collect 4,840 frame-commentary pairs from three YouTube videos, represent each sentence with the Universal Sentence Encoder, and train a CNN to map gameplay frames to sentence embeddings. They compare this model against a baseline adapted from Guzdial et al. (2018) that first clusters the data and trains separate CNNs on the largest clusters. The paper reports that the proposed full-data CNN achieves lower average percentile error on a held-out test set than the per-cluster baseline models, and it shows example outputs consisting of nearest-neighbor sentences retrieved from the training set. The authors also acknowledge several limitations, including the weakness of the results and the lack of temporal context.
Significance. If the central comparison were sound, this would be a modest but useful contribution: a publicly released dataset of Minecraft frame-comment pairs and an initial demonstration that a full-data CNN can beat a per-cluster baseline on an embedding-space metric. The paper is honest about its limitations and the baseline is the authors' own prior work, so there is no circularity concern. However, the evaluation as written does not establish the central claim. The metric is underspecified and unvalidated, the comparison is confounded by training set size, and the displayed outputs are retrieved rather than generated. The dataset release is a concrete strength, but the evidence for the paper's main empirical claim needs substantial strengthening.
major comments (4)
- [Evaluation, Table 1] The central claim that 'our approach outperforms the baseline across all three of its largest clusters' rests entirely on the 'average percentile error' metric, but this metric is underspecified: the paper does not state whether the percentile is computed per dimension, per sentence, or relative to what reference distribution. More importantly, no validation is provided that lower percentile error corresponds to more human-like or more relevant commentary, and no chance-level baseline (e.g., random predictions, mean embedding, or majority utterance) is reported to calibrate the scale. Without such calibration, the numerical differences in Table 1 (0.961 vs. 0.975-0.980) do not establish that the proposed method produces better commentary.
- [Our Approach, Dataset] The paper states in the Dataset section that the dataset has 4,840 instances, with 3,600 used for training and the rest for testing, but later says 'we trained a convolutional neural network (CNN) with the 4840 training instances.' If the model was trained on all 4,840 instances including the test set, the evaluation in Table 1 is invalid; if this is a typo, it must be corrected to 3,600. This point directly affects the validity of every reported result and must be clarified.
- [Evaluation, Baseline] The comparison in Table 1 is confounded by training set size: the proposed CNN trains on all 4,840 instances, while each baseline cluster model trains on only 684-1,336 instances. The observed improvement may therefore be due to dataset size rather than to the modeling choice of training on all data instead of clustered subsets. A controlled comparison (e.g., training the same architecture on random subsets of comparable size, or including a non-clustered baseline trained on the full data) is needed to support the claim that the proposed approach is superior.
- [Example Output, Abstract] The example outputs in Figure 2 are not generated by the model; they are the five nearest-neighbor sentences retrieved from the training set, as the text states. The Abstract's claim that the paper 'demonstrate[s] the generation of automated, artificial commentary' is therefore not supported by the displayed output. The paper should either provide a decoder that maps the predicted embedding to novel text or explicitly reframe the contribution as retrieval-based selection rather than generation.
minor comments (3)
- [Clustering, Figure 2] The Figure 2 caption says the displayed sentences are the 'five closest nearest-neighbors of the model's actual predicted commentary,' while the body text says they are the 'closest neighbors to the actual predicted output from the baseline model.' Please make the caption and text consistent.
- [Our Approach] There is a typo in 'similarly sized datatsets'; it should be 'datasets.'
- [Clustering] There is a duplicated article in 'the the clusters with the most instances'; please correct it.
Circularity Check
No circular derivation; the empirical comparison has evaluation caveats but no circular reduction.
full rationale
This paper contains no mathematical derivation that could collapse into its inputs; it is an empirical CNN training-and-evaluation study. The proposed model is trained on frame-commentary pairs and evaluated on a held-out test set, with reported percent error comparing predicted sentence embeddings against true test utterances, so the evaluation is not simply the training objective renamed. The baseline is the authors' own prior work (Guzdial, Shah, and Riedl 2018), and the paper reimplements and compares against it; this is a standard empirical comparison rather than a load-bearing self-citation. The displayed output is retrieved from the training set, but the authors state this explicitly: "We include the closest sentences from our training set to the predicted sentence encoding as novel commentary due to the limitations of the Universal Sentence Encoder (Cer et al. 2018), but with another sentence embedding we could directly output novel commentary." The quantitative claim does not depend on that retrieval, and the authors also acknowledge the weakness of the results and that "the commentary doesn't correlate well to the images shown." Concerns about the unvalidated percentile-error metric and the training-set-size confound between the full-data CNN and the smaller cluster models are legitimate threats to the strength of the empirical claim, but they are not circularity as defined here. The score of 1 reflects only the minor self-referential element that the comparison baseline is the authors' own earlier system, which is not itself a circular argument.
Assumptions & free parameters
free parameters (4)
- Number of K-medoids clusters K =
estimated via distortion ratio
- CNN learning rate =
0.001
- CNN dropout rate =
0.9
- CNN architecture sizes =
32/64/64 filters, 1024 and 512 fully connected units
assumptions (4)
- domain assumption Universal Sentence Encoder embeddings preserve semantic similarity for short commentary sentences
- domain assumption ImageNet-trained ResNet features are informative for clustering Minecraft frames
- ad hoc to paper Pairing every 1 FPS frame with the currently spoken sentence creates a valid supervised task
- domain assumption A CNN architecture borrowed from image captioning can map frames to sentence embedding vectors
Cite this review
Pith. "Pith review of Automated Let's Play Commentary." pith.science (2026). https://pith.science/paper/K3PWDBTI
@misc{pith2026190902195,
author = {Pith},
title = {Pith review of: Automated Let's Play Commentary},
year = {2026},
howpublished = {\url{https://pith.science/paper/K3PWDBTI}},
note = {Machine review of arXiv:1909.02195}
}
read the original abstract
Let's Plays of video games represent a relatively unexplored area for experimental AI in games. In this short paper, we discuss an approach to generate automated commentary for Let's Play videos, drawing on convolutional deep neural networks. We focus on Let's Plays of the popular game Minecraft. We compare our approach and a prior approach and demonstrate the generation of automated, artificial commentary.
Figures
Reference graph
Works this paper leans on
-
[5]
Dodge, J.; Penney, S.; Hilderbrand, C.; Anderson, A.; and Burnett, M. 2018. How the experts do it: Assessing and explaining agent behaviors in real-time strategy games. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 562. ACM
work page 2018
-
[6]
Ehsan, U.; Harrison, B.; Chan, L.; and Riedl, M. O
-
[7]
In Pro- ceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’18, 81–87
Rationalization: A neural machine translation ap- proach to generating natural language explanations. In Pro- ceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’18, 81–87. New York, NY , USA: ACM
work page 2018
-
[9]
Graefe, A. 2016. Guide to automated journalism
work page 2016
-
[10]
Guzdial, M., and Riedl, M. 2016. Game level genera- tion from gameplay videos. In Twelfth Artificial Intelligence and Interactive Digital Entertainment Conference
work page 2016
-
[11]
Guzdial, M.; Li, B.; and Riedl, M. O. 2017. Game engine learning from video. In IJCAI, 3707–3713
work page 2017
-
[12]
Guzdial, M.; Shah, S.; and Riedl, M. 2018. To- wards automated let’s play commentary. arXiv preprint arXiv:1809.09424
work page Pith review arXiv 2018
-
[13]
Guzdial, M. J.; Sturtevant, N.; and Li, B. 2016. Deep static and dynamic level analysis: A study on infinite mario. In Twelfth Artificial Intelligence and Interactive Digital En- tertainment Conference
work page 2016
Show all 30 references
-
[16]
P., and Ba, J
Kingma, D. P., and Ba, J. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980
2014 arXiv
-
[18]
Lee, G.; Bulitko, V .; and Ludvig, E. A. 2014. Auto- mated story selection for color commentary in sports. IEEE transactions on computational intelligence and ai in games 6(2):144–155
2014
-
[19]
Li, C.; Gandhi, S.; and Harrison, B. 2019. End-to-end let’s play commentary generation using multi-modal video representations. In Proceedings of the 14th International Conference on the Foundations of Digital Games, 76. ACM
2019
-
[20]
Liao, N.; Guzdial, M.; and Riedl, M. 2017. Deep convolutional player modeling on log and level data. InPro- ceedings of the 12th International Conference on the Foun- dations of Digital Games, 41. ACM
2017
-
[21]
Liapis, A.; Karavolos, D.; Makantasis, K.; Sfikas, K.; and Yannakakis, G. N. 2019. Fusing level and ruleset fea- tures for multimodal learning of gameplay outcomes. In Proceedings of the IEEE Conference on Games
2019
-
[22]
Lin, D.; Bezemer, C.-P.; and Hassan, A. E. 2019. Identifying gameplay videos that exhibit bugs in computer games. Empirical Software Engineering 1–28
2019
-
[23]
Sj ¨oblom, M., and Hamari, J. 2017. Why do people watch others play video games? an empirical study on the motivations of twitch users. Computers in Human Behavior 75:985–996
2017
-
[24]
Sj ¨oblom, M.; T ¨orh¨onen, M.; Hamari, J.; and Macey, J. 2017. Content structure is king: An empirical study on gratifications, game genres and content type on twitch.Com- puters in Human Behavior 73:161–171
2017
-
[25]
A.; Hess, A.; and Buro, M
Stanescu, M.; Barriga, N. A.; Hess, A.; and Buro, M
-
[26]
In 2016 IEEE Conference on Computational Intelligence and Games (CIG), 1–7
Evaluating real-time strategy game states using con- volutional neural networks. In 2016 IEEE Conference on Computational Intelligence and Games (CIG), 1–7. IEEE
2016
-
[27]
Summerville, A.; Guzdial, M.; Mateas, M.; and Riedl, M. O. 2016. Learning player tailored content from obser- vation: Platformer level generation from video traces using lstms. In Twelfth Artificial Intelligence and Interactive Dig- ital Entertainment Conference
2016
-
[28]
K.; Isaksen, A.; Nealen, A.; and Togelius, J
Summerville, A.; Snodgrass, S.; Guzdial, M.; Holmg˚ard, C.; Hoover, A. K.; Isaksen, A.; Nealen, A.; and Togelius, J. 2017. Procedural content generation via ma- chine learning (pcgml). arXiv preprint arXiv:1702.00539
2017 arXiv
-
[29]
Targ, S.; Almeida, D.; and Lyman, K. 2016. Resnet in resnet: Generalizing residual architectures. arXiv preprint arXiv:1603.08029
2016 arXiv
-
[30]
Xu, B.; Wang, N.; Chen, T.; and Li, M. 2015. Empir- ical evaluation of rectified activations in convolutional net- work. arXiv preprint arXiv:1505.00853
2015 arXiv
-
[2006]
H., and Sengupta, S
Kolekar, M. H., and Sengupta, S. 2006. Event- importance based customized and automatic cricket high- light generation. In Multimedia and Expo, 2006 IEEE Inter- national Conference on, 1617–1620. IEEE
2006
-
[2009]
Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 248–255. Ieee
2009
-
[2014]
A.; Garretson, O.; and Kerne, A
Hamilton, W. A.; Garretson, O.; and Kerne, A. 2014. Streaming on twitch: fostering participatory communities of play within live mixed media. In Proceedings of the SIGCHI Conference on Human Factors in Computing Sys- tems, 1315–1324. ACM
2014
-
[2015]
Hosang, J.; Omran, M.; Benenson, R.; and Schiele, B. 2015. Taking a deeper look at pedestrians. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4073–4082
2015
-
[2016]
Abadi, M.; Barham, P.; Chen, J.; Chen, Z.; Davis, A.; Dean, J.; Devin, M.; Ghemawat, S.; Irving, G.; Isard, M.; et al. 2016. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} Symposium on Operating Sys- tems Design and Implementation ({OSDI} 16), 265–283
2016
-
[2017]
E.; Eger, M.; Glatz, M.; Green, N.; Mattice, J.; Potts, C
Barot, C.; Branon, M.; Cardona-Rivera, R. E.; Eger, M.; Glatz, M.; Green, N.; Mattice, J.; Potts, C. M.; Robert- son, J.; Shukonobe, M.; et al. 2017. Bardic: Generating multimedia narrative reports for game logs
2017
-
[2018]
S.; Constant, N.; Guajardo-Cespedes, M.; Yuan, S.; Tar, C.; et al
Cer, D.; Yang, Y .; Kong, S.-y.; Hua, N.; Limtiaco, N.; John, R. S.; Constant, N.; Guajardo-Cespedes, M.; Yuan, S.; Tar, C.; et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175
2018 arXiv
-
[2019]
Ehsan, U.; Tambwekar, P.; Chan, L.; Harrison, B.; and Riedl, M. O. 2019. Automated rationale generation: A tech- nique for explainable ai and its effects on human percep- tions. In Proceedings of the 24th International Conference on Intelligent User Interfaces, IUI ’19, 263–27...
2019
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.