Pith. sign in

REVIEW 4 major objections 5 minor 19 references

Voice Communication Analysis in Esports

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Transcribed voice chat from esports matches can be scored per sentence by maximum cosine similarity against a player's own recent utterances and against coach-defined parasite phrasings, yielding per-player repetition and interference…

desk verdict A genuinely new esports-voice application with a cleart pipeline, but the absolute-value cosine similarity in Eq. 2 makes contradictory statements look identical, undermining the core metric before thresholds are even considered. read the letter →

arxiv 2411.19793 v1 pith:JEHC7KAC submitted 2024-11-21 cs.SD cs.AIcs.CLeess.AS

Aymeric Vinot , Nicolas Perez This is my paper · ORCID
classification cs.SDcs.AIcs.CLeess.AS
keywords voicecommunicationanalysisesportsLeagueofLegendssentenceembeddingssemanticsimilarityduplicateparasiteNLP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to turn raw voice chat from a League of Legends team into two measurable communication faults: repeated sentences (duplicates) and unclear, hesitant, or off-target phrasings (parasites). The proposed method transcribes and diarizes a game's audio, embeds each sentence with a sentence-transformer model, and compares each sentence against the same player's previous sentences within a $W=15$ second window, taking the maximum cosine similarity as a repetition score. A second comparison runs each sentence against a coach-curated list of parasite phrasings and flags any sentence whose top similarity passes 0.6. The paper reports that this pipeline gives usable per-player rates, such as a 39.1% parasite rate for one speaker, and that embedding-model choice changes accuracy, precision, and recall substantially. A sympathetic reading is that the authors are establishing a proof-of-concept that off-the-shelf NLP embeddings can produce interpretable communication analytics for esports coaching without any sport-specific training data.

What carries the argument

The load-bearing mechanism is the sentence transformer plus cosine similarity. Each utterance is mapped to a high-dimensional embedding of roughly 1024 dimensions by a BERT-based sentence encoder, and two utterances are judged close when the absolute cosine of the angle between their embedding vectors is near 1. Duplicate scoring takes the maximum of that similarity over the player's own prior utterances inside a sliding $W=15$ second window; parasite scoring takes the maximum over a coach-defined list of phrasings and applies a fixed 0.6 decision threshold. A refinement step recomputes single-word sentence embeddings by pooling token embeddings after feeding the preceding conversational context through the transformer, which the paper shows lowers spurious similarity to "Maybe" and similar parasite phrasings.

What would settle it

Run the published duplicate and parasite classifiers on a held-out set of League of Legends voice logs with fresh human labels, sweep the decision threshold from 0.4 to 0.8 and the window from 5 to 30 seconds, and check whether the F1 scores in Table 1 are reproduced or whether the optimal operating point moves. A large drop outside the narrow tested conditions would show the 0.6 threshold is dataset-specific rather than a stable property of esports communication.

Watch

Extended reading notes

Core claim

The paper's central claim is that a purely semantic similarity score can operationalize communication quality in esports. For duplicates, the score for each sentence is the maximum cosine similarity between its embedding and all of that player's sentence embeddings in the previous $W=15$ seconds; a score above 0.6 marks the sentence as repetitive. For parasites, the same 0.6 cutoff is applied to the maximum cosine similarity between the sentence embedding and a fixed set of parasite phrasings such as "I think", "Maybe", and "Can we?" The paper argues this separates uncertain, non-directive calls from concise ones, and shows that recomputing embeddings of single-word sentences with surrounding conversation context removes false parasite flags. Reported results on a 359-second, three-speaker, 129-sentence labeled sample give accuracy up to 86.8% for duplicates and 84.5% for parasites with some models, and the paper identifies one embedding model as the most balanced across tasks.

Load-bearing premise

The paper assumes that the hand-picked 0.6 cosine similarity threshold and the 15-second lookback window, chosen on the same small sample used for evaluation, will keep working for other games, players, and communication styles; no sensitivity analysis or held-out validation is reported to support that.

Editorial extensions

If this is right

  • Coaches can automatically derive per-player repetition rates and parasite rates from voice logs, such as the 39.1% parasite rate computed for one speaker, without manually labeling every utterance.
  • The same sentence-similarity pipeline extends to other team esports, since the method uses no League-of-Legends-specific training data.
  • Model choice materially affects detection: the paper's comparison shows accuracy, precision, and recall vary widely across sentence-embedding architectures, so deployment needs a model-selection step.
  • A context-refinement step for short sentences prevents common words like "Okay" from being mislabeled as parasite communications.
  • The reported metrics can be correlated in future work with in-game performance indicators, which the paper identifies as the natural next step for validating communication quality's impact on outcomes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 0.6 threshold proves stable outside this dataset, the same max-over-window scoring could also detect a player ignoring a teammate's call by repeating a similar phrase after a failed acknowledgement.
  • Because only transcribed text is used, tone and confidence are invisible; a testable extension is to fuse audio embeddings from the speech recognizer with sentence embeddings and check whether parasite detection improves.
  • The context-recompute trick for single-word sentences suggests a broader principle: in esports, short utterances derive meaning from context, so embedding refinement may matter more for short calls than for full sentences.
  • A concrete validation experiment would be to run the published pipeline on the same audio with different sentence-boundary segmentations and report how much duplicate scores move, since the metric depends on how the transcription is split into sentences.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a pipeline for analyzing in-game voice communication in esports (League of Legends) by transcribing team voice logs, diarizing speakers, and then computing two semantic-similarity scores: a duplicate score that compares each sentence embedding to the speaker's previous sentences within a 15-second window, and a parasite score that compares each sentence embedding to a fixed list of 12 coach-defined "parasite" phrasings. The authors evaluate three embedding models on a single 359-second recording containing 129 sentences, reporting accuracy, precision, recall, and F1 for duplicate and parasite detection, and conclude that mxbai-embed-large-v1 gives the most balanced performance. They also propose a context-based embedding refinement for single-word sentences.

Significance. The paper addresses a genuinely important and under-studied problem: quantitative analysis of voice communication quality in esports. The practical motivation is clear, and the pipeline (WhisperX transcription, diarization, sentence embeddings) is sensible and reproducible, with the exact embedding models and the full list of parasite phrasings provided. If the methodological concerns raised below are resolved, the metrics could be useful for coaches and analysts. However, the current evaluation is too small to support the reported quantitative conclusions, and the absolute-value cosine similarity is internally inconsistent with the stated semantics of the task.

major comments (4)
  1. [Section 3.1.2, Eq. (2)] The definition of cosine_sim takes the absolute value of the cosine. This is not a harmless convention: for sentence embeddings, a pair of contradictory statements (e.g., 'we can go drake' and 'we cannot go drake') will have cosine near -1, which Eq. (2) maps to +1, making them appear maximally similar. This directly contradicts the intended meaning of 'closest in meaning' used in Eq. (3) and propagated to the parasite score in Eqs. (5)-(6). The paper does not report any signed-cosine statistics, nor does it show that the embedding space is restricted to a single orthant for this domain. The metric must be corrected (likely by removing the absolute value) and the experiments rerun; the current results characterize a metric that does not match the stated claim.
  2. [Section 7.1 / Section 5.2 (Table 1)] The threshold of 0.6 is described in Section 7.1 as 'one of the most reliable', which indicates it was selected using the same evaluation data. No held-out set, cross-validation, or sensitivity analysis is provided. Moreover, Table 1 is based on a single audio segment (359 seconds, 129 sentences) with only 17 duplicate-positive and 21 parasite-positive instances. With such small counts, the reported F1 differences between models (e.g., 50.91% versus 37.04% for duplicates) are not statistically reliable. A rigorous evaluation with held-out data, confidence intervals, and a threshold sweep is necessary to support the central performance claims.
  3. [Sections 4.1.1 and 5.1 (Appendix 8.2)] The ground-truth definition of 'parasite' appears to overlap with the detector's reference list. The same coach-involved process that produced the 12 parasite phrasings in Appendix 8.2 is used to define the concept, and the human labeling described in Section 5.1 also involved professional coaches. If the annotators were given these same phrases (explicitly or implicitly) when labeling, the parasite detection is partially circular: the detector would be rewarded for reproducing the list rather than for detecting an independently defined communication deficiency. Please clarify the labeling protocol and state clearly whether the human labels are independent of the 12 phrases; if they are not, the reported parasite F1 must be interpreted as a measure of agreement with a fixed reference set, not as detection of a broader construct.
  4. [Section 4.2 (Embedding refining)] The context-based embedding refinement for single-word sentences is only illustrated on one example (sentence 018 in Figure 8). It is not evaluated on the full test set, so it is unclear whether this proposed enhancement actually improves the quantitative results in Table 1. If this refinement is intended as a contribution, it needs a systematic evaluation; otherwise, the paper should state explicitly that Table 1 reflects only the unrefined method.
minor comments (5)
  1. [Section 3.1.2] The standard cosine similarity is signed; the use of the absolute value should be justified or removed, and its effect on the reported results (if any) should be discussed.
  2. [Section 4.1.1, Eq. (5)] The index set in Eq. (5) uses n_i (the number of sentences spoken by SPEAKER_i) as the upper bound for the parasite phrasing index j. This should be the number of parasite phrasings (12 in Appendix 8.2), not n_i.
  3. [Section 5.1] No inter-annotator agreement is reported for the human labels. Given the subjectivity of the 'parasite' concept, agreement statistics (e.g., Cohen's kappa) would help establish label reliability.
  4. [Throughout] There are numerous typographical and grammatical errors, including 'direclty', 'apropach', 'labelized', 'interference' used for 'parasite' in some figure captions, and inconsistent spacing around equations. A thorough copyedit is needed.
  5. [Section 5.2] The three models are compared only via overall accuracy/F1; no statistical tests (e.g., permutation tests or confidence intervals for F1) are provided, so the claim that mxbai-embed-large-v1 is superior is not substantiated.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the detection pipeline compares embeddings to external inputs and labels, and none of the reported results is equivalent to its own inputs by construction.

full rationale

The derivation chain is self-contained with respect to its inputs. Duplicate scores are computed from external sentence embeddings via Eqs. (2)-(3), and parasite scores from the same embeddings against a coach-built phrase list via Eqs. (4)-(6). Table 1 then compares those scores to human/coach labels, which are not derived from the detector equations. The coach-built phrase list is an input, not a fit to the labels, so the parasite detector is not equivalent to the ground truth by construction. The 0.6 threshold is acknowledged in Section 7.1 as chosen for reliability, which is an in-sample validation weakness rather than a definitional identity: the detector outputs are not computed from the labels. There are no load-bearing self-citations, imported uniqueness theorems, or ansatz-smuggling citations. Equation (2)'s use of absolute-value cosine is a metric-correctness concern—opposite-meaning sentences can receive high similarity—but that is not circularity. Accordingly, no circular step is identified.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central method rests on hand-picked threshold 0.6, the 15-second window, the coach-defined phrase list, and general-purpose embeddings. The ledger shows the method is a thin application layer over existing components, with no independent validation of the threshold or phrase-list choices.

free parameters (2)
  • similarity threshold tau = 0.6
    Used to flag both duplicate and parasite sentences; paper says in Section 7.1 that 0.6 was chosen as one of the most reliable thresholds, implying data-dependent selection.
  • lookback window W = 15 seconds
    Set for both experiments; no sensitivity analysis is reported, so the chosen window is an untested hand-picked number.
assumptions (4)
  • domain assumption Pretrained sentence embeddings encode enough semantic meaning for short esports utterances to make cosine similarity a valid proxy for communicative redundancy and vagueness.
    The entire detection pipeline relies on this proxy; the paper itself notes the model is not trained on League of Legends jargon and single-word utterances behave poorly (Sections 3.2.2 and 4.1.2).
  • ad hoc to paper The 12 parasite phrasings in Appendix 8.2 are representative of unwanted communication and are sufficient to define parasitic speech.
    This list was assembled with professional coaches but is not derived from any external corpus or validated against independent annotation.
  • domain assumption Transcription and speaker diarization errors are small enough not to change sentence-level similarity results.
    Section 2 admits some speech was wrongly attributed to speakers; errors in speaker identity directly affect the per-player 15-second windows used for scoring.
  • domain assumption A 15-second lookback window captures the relevant conversational context for judging whether a sentence is repetitive.
    W is fixed at 15 seconds in Sections 3.2.1 and 5.1 with no sensitivity analysis or justification from communication science.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Voice Communication Analysis in Esports." pith.science (2026). https://pith.science/paper/JEHC7KAC

@misc{pith2026241119793,
  author       = {Pith},
  title        = {Pith review of: Voice Communication Analysis in Esports},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JEHC7KAC}},
  note         = {Machine review of arXiv:2411.19793}
}
read the original abstract

In most team-based esports, voice communications are prominent in the team efficiency and synergy. In fact it has been observed that not only the skill aspect of the team but also the team effective voice communication comes into play when trying to have good performance in official matches. With the recent emergence of LLM (Large Language Models) tools regarding NLP (Natural Language Processing) (Vaswani et. al.), we decided to try applying them in order to have a better understanding on how to improve the effectiveness of the voice communications. In this paper the study has been made through the prism of League of Legends esport. However the main concepts and ideas can be easily applicable in any other team related esports.

Figures

Figures reproduced from arXiv: 2411.19793 by the authors.

Figure 1
Figure 1. Pipeline of audio transcription from [2] [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Duplicate communication scores on each sentence for each speaker. Each sentence is assigned a [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Duplicate communication scores on each sentence for each speaker on another game [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Bottom : Overall interference score of the speaker’s communication. Above : interference percentage [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Similarity score of each sentence spoken with eash parasite phrasing [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Similarity score of each sentence spoken with eash parasite phrasing [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Recomputing the embedding by taking the preceding context and pooling the individual token [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Similarity score sentence embedding, by recomputing embedding of single-word sentences, with [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Interference scores of another games On [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Interference Heatmap of SPEAKER_00 16 [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Interference Heatmap of SPEAKER_00 without embedding refinment [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 12 canonical work pages

  1. [1]

    Robust speech recognition via large-scale weak supervision.arXiv preprint arXiv:2212.04356 , 2022

    Tao Xu Greg Brockman Christine McLeavey Ilya Sutskever Alec Radford, Jong Wook Kim. Robust speech recognition via large-scale weak supervision.arXiv preprint arXiv:2212.04356 , 2022

    arXiv 2022

  2. [2]

    Whisperx: Time-accurate speech transcription of long-form audio.INTERSPEECH 2023, 2023

    Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. Whisperx: Time-accurate speech transcription of long-form audio.INTERSPEECH 2023, 2023

    2023

  3. [3]

    Bridging the lexical chasm: Statistical approaches to answer-finding.SIGIR Forum (ACM Special Interest Group on Information Retrieval), pages 192–199, 12 2002

    Adam Berger, Rich Caruana, David Cohn, Dayne Freitag, and Vibhu Mittal. Bridging the lexical chasm: Statistical approaches to answer-finding.SIGIR Forum (ACM Special Interest Group on Information Retrieval), pages 192–199, 12 2002

    work page 2002

  4. [4]

    pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe

    Hervé Bredin. pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe. In Proc. INTERSPEECH 2023, 2023

    work page 2023

  5. [5]

    Reading wikipedia to answer open-domain questions

    Jason Weston Antoine Bordes Danqi Chen, Adam Fisch. Reading wikipedia to answer open-domain questions. arXiv preprint arXiv:1704.00051 , 2017

    arXiv 2017

  6. [6]

    Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805 , 2018

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805 , 2018

    arXiv 2018

  7. [7]

    DraftRec: Personalized Draft Recommendation for Winning in Multi-Player Online Battle Arena Games

    Hyunseung Kim Byungkun Lee Jaegul Choo Hojoon Lee, Dongyoon Hwang. Draftrec: Personalized draft recommendation for winning in multi-player online battle arena games.arXiv preprint arXiv:2204.12750, 2022

    work page Pith review arXiv 2022

  8. [8]

    Claudio E. C. Campelo Jailson B. S. Junior. League of legends: Real-time result prediction.arXiv preprint arXiv:2309.02449, 2023

    work page Pith review arXiv 2023

Show all 19 references
  1. [9]

    Jalovaara

    P. Jalovaara. Win probability estimation for strategic decision-making in esports. Master’s thesis, Aalto University, 2024

    2024

  2. [10]

    Individualized context-aware tensor factorization for online games predictions

    Emilio Ferrara Julie Jiang, Kristina Lerman. Individualized context-aware tensor factorization for online games predictions. arXiv preprint arXiv:2102.11352 , 2021

    2021 arXiv

  3. [11]

    PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordings

    Joonas Kalda, Clément Pagés, Ricard Marxer, Tanel Alumäe, and Hervé Bredin. PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordings. InProc. Odyssey 2024, 2024

    2024

  4. [12]

    T. G. Kolda and B. W. Bader.Tensor decompositions and applications, volume 51. SIAM Rev, 2009

    2009

  5. [13]

    Latent retrieval for weakly supervised open do- main question answering

    Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open do- main question answering. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, 2019. Association for Computational Linguistics

    2019

  6. [14]

    Open source strikes bread - new fluffy embeddings model, 2024

    Sean Lee, Aamir Shakir, Darius Koenig, and Julius Lipp. Open source strikes bread - new fluffy embeddings model, 2024. Blog post, Mixedbread AI

    2024

  7. [15]

    Mteb: Massive text embedding benchmark

    Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316 , 2022

    2022 arXiv

  8. [16]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, 11 2019

    2019

  9. [17]

    The probabilistic relevance framework: Bm25 and beyond

    Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval , 3(4):333–389, 2009

    2009

  10. [18]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.arXiv preprint arXiv:1706.03762 , 2017

    2017 arXiv

  11. [19]

    C-pack: Packaged resources to advance general chinese embedding, 2023

    Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023. 14 Voice Communication Analysis in esports 8 Appendix 8.1 Communication Logs 000 - [00.942:02.163] SPEAKER_01 You need to push out, XXXX. 001 -...

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.