Pith. sign in

REVIEW 4 major objections 3 minor 30 references

Automated Let's Play Commentary

T0 review · 4 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read A convolutional network trained on all frames predicts Minecraft Let's Play commentary sentence embeddings more accurately than a clustering-based baseline, but all models fall well short of human-quality text.

desk verdict A small, honest paper whose real value is the new Minecraft frame-commentary dataset; the empirical comparison is confounded and the metric is unvalidated, but the authors do not oversell what they have. read the letter →

arxiv 1909.02195 v2 pith:K3PWDBTI submitted 2019-09-05 cs.CL cs.AI

classification cs.CLcs.AI
keywords Let'sPlaycommentaryMinecraftconvolutionalneuralnetworksentenceembeddingsUniversalEncodervideogameAIframe-to-textgenerationK-medoidsclustering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Let's Play videos show a player narrating gameplay in improvised spoken commentary, and this paper asks whether that commentary can be generated from raw video frames alone. It trains a convolutional network to map each Minecraft gameplay frame to the sentence embedding of the commentary uttered at that moment, and compares it with a prior approach that clusters frame-commentary pairs and trains a separate model for each cluster. Using average percentile error in the Universal Sentence Encoder's 512-dimensional space, the unified network scores 0.961 and beats every one of the baseline's three largest clusters (0.975 to 0.980), although the authors note that none of the approaches do particularly well. The paper's contribution is therefore both a public dataset of 4,840 frame-commentary pairs and evidence that on this scale a single global model is the better starting point.

What carries the argument

The machinery that carries the argument is a convolutional network reading a single gameplay frame and outputting a 512-dimensional sentence embedding. The target representation is produced by the Universal Sentence Encoder, which turns each spoken commentary sentence into the vector the network is trained to approximate; the same embedding space also supplies the evaluation metric. The baseline variant replaces the one global model with K-medoids clustering over concatenated image and sentence embeddings—a clustering method whose centers are actual data points—followed by separate CNNs trained on the largest resulting clusters. The essential contrast the experiment tests is therefore a single model over all data versus several models over specialized subsets, with the sentence embedding as the fixed prediction target in both cases.

What would settle it

Train the same CNN on random subsets of sizes 684, 802, and 1,336 matched to the three largest baseline clusters and compare percentile error on the same test set; if error stays near 0.961, the advantage is data volume, not the unified-model idea, and if human raters cannot distinguish lower-error predictions from true commentary, the metric itself is not measuring commentary quality.

Watch

Extended reading notes

Core claim

The central claim is that a single convolutional network, trained on all 4,840 frame-commentary pairs, predicts human Let's Play commentary more accurately than the clustering-based baseline on this dataset. On the paper's evaluation measure—average percentile error between predicted and true sentence embeddings—the unified CNN achieves 0.961 ± 0.026, while the baseline's three largest per-cluster models achieve 0.977 ± 0.023, 0.975 ± 0.042, and 0.980 ± 0.024. The authors interpret this as showing that a CNN can produce artificial commentary directly from pixels, and that per-cluster specialization does not help at this data scale, while also acknowledging that all approaches fall well short of human-quality commentary.

Load-bearing premise

The central claim rests on assuming that a low average percentile error in the sentence-representation space used by the evaluation actually means better or more human-like commentary, and that it is fair to compare a model trained on all 4,840 instances with models trained on only 684 to 1,336 instances.

Editorial extensions

If this is right

  • A single CNN mapping raw frames to sentence embeddings is a usable, simple baseline for automated Let's Play commentary, requiring no access to the game engine or action logs.
  • The public Minecraft dataset of 4,840 frame-commentary pairs gives future work a common corpus for comparing video-to-commentary models.
  • Per-cluster training, as in the prior approach, offers no advantage over a global model at this dataset size, so future comparisons can treat the unified CNN as the reference point.
  • The gap between 0.961 and the ideal 0.0 percentile error shows that supervised frame-to-commentary prediction remains largely unsolved and likely needs larger datasets or additional context.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported superiority may come from training-set size rather than the global-model idea, since the unified network sees 4,840 instances while each cluster model sees only 684 to 1,336; matching training sizes would settle this.
  • The evaluation metric, which averages percentile error in a fixed embedding space, may reward generic predictions that sit near the center of the space; the repeated nearest-neighbor sentences shown in the paper's examples hint that specificity is not being rewarded.
  • Because every frame within an utterance is paired with the same sentence, the dataset contains many duplicate targets; a temporal model that consumes short frame sequences, or that predicts once per utterance, is a natural untested extension.
  • If the central claim generalizes, the same architecture should transfer to other visually simple games with large Let's Play corpora, and the reported accuracy would serve as a sanity-check baseline for those games.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper presents a short, exploratory study of automated Let's Play commentary for Minecraft. The authors collect 4,840 frame-commentary pairs from three YouTube videos, represent each sentence with the Universal Sentence Encoder, and train a CNN to map gameplay frames to sentence embeddings. They compare this model against a baseline adapted from Guzdial et al. (2018) that first clusters the data and trains separate CNNs on the largest clusters. The paper reports that the proposed full-data CNN achieves lower average percentile error on a held-out test set than the per-cluster baseline models, and it shows example outputs consisting of nearest-neighbor sentences retrieved from the training set. The authors also acknowledge several limitations, including the weakness of the results and the lack of temporal context.

Significance. If the central comparison were sound, this would be a modest but useful contribution: a publicly released dataset of Minecraft frame-comment pairs and an initial demonstration that a full-data CNN can beat a per-cluster baseline on an embedding-space metric. The paper is honest about its limitations and the baseline is the authors' own prior work, so there is no circularity concern. However, the evaluation as written does not establish the central claim. The metric is underspecified and unvalidated, the comparison is confounded by training set size, and the displayed outputs are retrieved rather than generated. The dataset release is a concrete strength, but the evidence for the paper's main empirical claim needs substantial strengthening.

major comments (4)
  1. [Evaluation, Table 1] The central claim that 'our approach outperforms the baseline across all three of its largest clusters' rests entirely on the 'average percentile error' metric, but this metric is underspecified: the paper does not state whether the percentile is computed per dimension, per sentence, or relative to what reference distribution. More importantly, no validation is provided that lower percentile error corresponds to more human-like or more relevant commentary, and no chance-level baseline (e.g., random predictions, mean embedding, or majority utterance) is reported to calibrate the scale. Without such calibration, the numerical differences in Table 1 (0.961 vs. 0.975-0.980) do not establish that the proposed method produces better commentary.
  2. [Our Approach, Dataset] The paper states in the Dataset section that the dataset has 4,840 instances, with 3,600 used for training and the rest for testing, but later says 'we trained a convolutional neural network (CNN) with the 4840 training instances.' If the model was trained on all 4,840 instances including the test set, the evaluation in Table 1 is invalid; if this is a typo, it must be corrected to 3,600. This point directly affects the validity of every reported result and must be clarified.
  3. [Evaluation, Baseline] The comparison in Table 1 is confounded by training set size: the proposed CNN trains on all 4,840 instances, while each baseline cluster model trains on only 684-1,336 instances. The observed improvement may therefore be due to dataset size rather than to the modeling choice of training on all data instead of clustered subsets. A controlled comparison (e.g., training the same architecture on random subsets of comparable size, or including a non-clustered baseline trained on the full data) is needed to support the claim that the proposed approach is superior.
  4. [Example Output, Abstract] The example outputs in Figure 2 are not generated by the model; they are the five nearest-neighbor sentences retrieved from the training set, as the text states. The Abstract's claim that the paper 'demonstrate[s] the generation of automated, artificial commentary' is therefore not supported by the displayed output. The paper should either provide a decoder that maps the predicted embedding to novel text or explicitly reframe the contribution as retrieval-based selection rather than generation.
minor comments (3)
  1. [Clustering, Figure 2] The Figure 2 caption says the displayed sentences are the 'five closest nearest-neighbors of the model's actual predicted commentary,' while the body text says they are the 'closest neighbors to the actual predicted output from the baseline model.' Please make the caption and text consistent.
  2. [Our Approach] There is a typo in 'similarly sized datatsets'; it should be 'datasets.'
  3. [Clustering] There is a duplicated article in 'the the clusters with the most instances'; please correct it.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation; the empirical comparison has evaluation caveats but no circular reduction.

full rationale

This paper contains no mathematical derivation that could collapse into its inputs; it is an empirical CNN training-and-evaluation study. The proposed model is trained on frame-commentary pairs and evaluated on a held-out test set, with reported percent error comparing predicted sentence embeddings against true test utterances, so the evaluation is not simply the training objective renamed. The baseline is the authors' own prior work (Guzdial, Shah, and Riedl 2018), and the paper reimplements and compares against it; this is a standard empirical comparison rather than a load-bearing self-citation. The displayed output is retrieved from the training set, but the authors state this explicitly: "We include the closest sentences from our training set to the predicted sentence encoding as novel commentary due to the limitations of the Universal Sentence Encoder (Cer et al. 2018), but with another sentence embedding we could directly output novel commentary." The quantitative claim does not depend on that retrieval, and the authors also acknowledge the weakness of the results and that "the commentary doesn't correlate well to the images shown." Concerns about the unvalidated percentile-error metric and the training-set-size confound between the full-data CNN and the smaller cluster models are legitimate threats to the strength of the empirical claim, but they are not circularity as defined here. The score of 1 reflects only the minor self-referential element that the comparison baseline is the authors' own earlier system, which is not itself a circular argument.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's central comparison rests on several unvalidated domain assumptions: that embedding-space distance reflects commentary quality, that ImageNet features represent Minecraft visuals, and that frame-sentence pairing is a sensible supervision signal. The only data-derived parameter is the clustering K, estimated via distortion ratio.

free parameters (4)
  • Number of K-medoids clusters K = estimated via distortion ratio
    This is a data-driven parameter for the baseline clustering. The paper reports that K was estimated via the distortion ratio but does not give the final value.
  • CNN learning rate = 0.001
    A hand-chosen optimization hyperparameter, stated in the Our Approach section.
  • CNN dropout rate = 0.9
    A hand-chosen hyperparameter, stated in the Our Approach section. The value 0.9 is unusually high and may be intended as a keep-probability of 0.9 or a typo for 0.1.
  • CNN architecture sizes = 32/64/64 filters, 1024 and 512 fully connected units
    Selected by analogy to CifarNet and image captioning models rather than by a systematic search, as stated in the Our Approach section.
assumptions (4)
  • domain assumption Universal Sentence Encoder embeddings preserve semantic similarity for short commentary sentences
    The evaluation and the retrieval of nearest training sentences both depend on cosine distance in USE space being meaningful. The paper cites Cer et al. 2018 but does not validate this assumption on Minecraft commentary text.
  • domain assumption ImageNet-trained ResNet features are informative for clustering Minecraft frames
    The baseline uses ResNet penultimate-layer activations as image embeddings, following prior work. The paper provides no evidence that these features capture the visual content relevant to commentary.
  • ad hoc to paper Pairing every 1 FPS frame with the currently spoken sentence creates a valid supervised task
    The authors state they paired each frame with a sentence 'for simplicity's sake', even though each sentence usually spans several frames. This produces repeated labels and may bias the model toward frequent sentences.
  • domain assumption A CNN architecture borrowed from image captioning can map frames to sentence embedding vectors
    The architecture is transferred from CifarNet and related models without task-specific adaptation, assuming that frame-to-embedding prediction is similar to image captioning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automated Let's Play Commentary." pith.science (2026). https://pith.science/paper/K3PWDBTI

@misc{pith2026190902195,
  author       = {Pith},
  title        = {Pith review of: Automated Let's Play Commentary},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K3PWDBTI}},
  note         = {Machine review of arXiv:1909.02195}
}
read the original abstract

Let's Plays of video games represent a relatively unexplored area for experimental AI in games. In this short paper, we discuss an approach to generate automated commentary for Let's Play videos, drawing on convolutional deep neural networks. We focus on Let's Plays of the popular game Minecraft. We compare our approach and a prior approach and demonstrate the generation of automated, artificial commentary.

Figures

Figures reproduced from arXiv: 1909.02195 by the authors.

Figure 1
Figure 1. The medoids of each of the clusters found by the K-Medoids clustering algorithm. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Each frame is paired with the five closest nearest-neighbors of the model’s actual predicted commentary. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 25 canonical work pages

  1. [5]

    Dodge, J.; Penney, S.; Hilderbrand, C.; Anderson, A.; and Burnett, M. 2018. How the experts do it: Assessing and explaining agent behaviors in real-time strategy games. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 562. ACM

  2. [6]

    Ehsan, U.; Harrison, B.; Chan, L.; and Riedl, M. O

  3. [7]

    In Pro- ceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’18, 81–87

    Rationalization: A neural machine translation ap- proach to generating natural language explanations. In Pro- ceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’18, 81–87. New York, NY , USA: ACM

  4. [9]

    Graefe, A. 2016. Guide to automated journalism

  5. [10]

    Guzdial, M., and Riedl, M. 2016. Game level genera- tion from gameplay videos. In Twelfth Artificial Intelligence and Interactive Digital Entertainment Conference

  6. [11]

    Guzdial, M.; Li, B.; and Riedl, M. O. 2017. Game engine learning from video. In IJCAI, 3707–3713

  7. [12]

    Guzdial, M.; Shah, S.; and Riedl, M. 2018. To- wards automated let’s play commentary. arXiv preprint arXiv:1809.09424

  8. [13]

    J.; Sturtevant, N.; and Li, B

    Guzdial, M. J.; Sturtevant, N.; and Li, B. 2016. Deep static and dynamic level analysis: A study on infinite mario. In Twelfth Artificial Intelligence and Interactive Digital En- tertainment Conference

Show all 30 references
  1. [16]

    P., and Ba, J

    Kingma, D. P., and Ba, J. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

  2. [18]

    Lee, G.; Bulitko, V .; and Ludvig, E. A. 2014. Auto- mated story selection for color commentary in sports. IEEE transactions on computational intelligence and ai in games 6(2):144–155

  3. [19]

    Li, C.; Gandhi, S.; and Harrison, B. 2019. End-to-end let’s play commentary generation using multi-modal video representations. In Proceedings of the 14th International Conference on the Foundations of Digital Games, 76. ACM

  4. [20]

    Liao, N.; Guzdial, M.; and Riedl, M. 2017. Deep convolutional player modeling on log and level data. InPro- ceedings of the 12th International Conference on the Foun- dations of Digital Games, 41. ACM

  5. [21]

    Liapis, A.; Karavolos, D.; Makantasis, K.; Sfikas, K.; and Yannakakis, G. N. 2019. Fusing level and ruleset fea- tures for multimodal learning of gameplay outcomes. In Proceedings of the IEEE Conference on Games

  6. [22]

    Lin, D.; Bezemer, C.-P.; and Hassan, A. E. 2019. Identifying gameplay videos that exhibit bugs in computer games. Empirical Software Engineering 1–28

  7. [23]

    Sj ¨oblom, M., and Hamari, J. 2017. Why do people watch others play video games? an empirical study on the motivations of twitch users. Computers in Human Behavior 75:985–996

  8. [24]

    Sj ¨oblom, M.; T ¨orh¨onen, M.; Hamari, J.; and Macey, J. 2017. Content structure is king: An empirical study on gratifications, game genres and content type on twitch.Com- puters in Human Behavior 73:161–171

  9. [25]

    A.; Hess, A.; and Buro, M

    Stanescu, M.; Barriga, N. A.; Hess, A.; and Buro, M

  10. [26]

    In 2016 IEEE Conference on Computational Intelligence and Games (CIG), 1–7

    Evaluating real-time strategy game states using con- volutional neural networks. In 2016 IEEE Conference on Computational Intelligence and Games (CIG), 1–7. IEEE

  11. [27]

    Summerville, A.; Guzdial, M.; Mateas, M.; and Riedl, M. O. 2016. Learning player tailored content from obser- vation: Platformer level generation from video traces using lstms. In Twelfth Artificial Intelligence and Interactive Dig- ital Entertainment Conference

  12. [28]

    K.; Isaksen, A.; Nealen, A.; and Togelius, J

    Summerville, A.; Snodgrass, S.; Guzdial, M.; Holmg˚ard, C.; Hoover, A. K.; Isaksen, A.; Nealen, A.; and Togelius, J. 2017. Procedural content generation via ma- chine learning (pcgml). arXiv preprint arXiv:1702.00539

  13. [29]

    Targ, S.; Almeida, D.; and Lyman, K. 2016. Resnet in resnet: Generalizing residual architectures. arXiv preprint arXiv:1603.08029

  14. [30]

    Xu, B.; Wang, N.; Chen, T.; and Li, M. 2015. Empir- ical evaluation of rectified activations in convolutional net- work. arXiv preprint arXiv:1505.00853

  15. [2006]

    H., and Sengupta, S

    Kolekar, M. H., and Sengupta, S. 2006. Event- importance based customized and automatic cricket high- light generation. In Multimedia and Expo, 2006 IEEE Inter- national Conference on, 1617–1620. IEEE

  16. [2009]

    Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 248–255. Ieee

  17. [2014]

    A.; Garretson, O.; and Kerne, A

    Hamilton, W. A.; Garretson, O.; and Kerne, A. 2014. Streaming on twitch: fostering participatory communities of play within live mixed media. In Proceedings of the SIGCHI Conference on Human Factors in Computing Sys- tems, 1315–1324. ACM

  18. [2015]

    Hosang, J.; Omran, M.; Benenson, R.; and Schiele, B. 2015. Taking a deeper look at pedestrians. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4073–4082

  19. [2016]

    Abadi, M.; Barham, P.; Chen, J.; Chen, Z.; Davis, A.; Dean, J.; Devin, M.; Ghemawat, S.; Irving, G.; Isard, M.; et al. 2016. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} Symposium on Operating Sys- tems Design and Implementation ({OSDI} 16), 265–283

  20. [2017]

    E.; Eger, M.; Glatz, M.; Green, N.; Mattice, J.; Potts, C

    Barot, C.; Branon, M.; Cardona-Rivera, R. E.; Eger, M.; Glatz, M.; Green, N.; Mattice, J.; Potts, C. M.; Robert- son, J.; Shukonobe, M.; et al. 2017. Bardic: Generating multimedia narrative reports for game logs

  21. [2018]

    S.; Constant, N.; Guajardo-Cespedes, M.; Yuan, S.; Tar, C.; et al

    Cer, D.; Yang, Y .; Kong, S.-y.; Hua, N.; Limtiaco, N.; John, R. S.; Constant, N.; Guajardo-Cespedes, M.; Yuan, S.; Tar, C.; et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175

  22. [2019]

    Ehsan, U.; Tambwekar, P.; Chan, L.; Harrison, B.; and Riedl, M. O. 2019. Automated rationale generation: A tech- nique for explainable ai and its effects on human percep- tions. In Proceedings of the 24th International Conference on Intelligent User Interfaces, IUI ’19, 263–27...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.