{"id":"b20db0b2-c232-4acb-b13f-f431a64f59f5","arxiv_id":"2411.19793","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Semantic cosine-similarity scoring against previous sentences and against coach-defined phrase templates can flag duplicate and parasite communications in transcribed esports voice chat, with best F1 around 51 to 53 percent on a 129-sentence sample.","lead":"This paper applies AI sentence embedding models to League of Legends team voice chat to automatically flag repeated and unclear statements. It offers coaches a first quantitative look at communication quality, though the evidence is preliminary.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 2's absolute-value cosine can rank contradictory statements as duplicates, so the central similarity metric may not measure what the paper claims.","rationale":"The reader's CONDITIONAL verdict is reasonable, but the most load-bearing weakness is not only the threshold transferability. The paper defines the core similarity with an absolute value in Eq. 2, which can invert the intended ordering between similarity and opposition. This is an internal correctness risk that affects both duplicate and parasite detection before any threshold is chosen. The concrete test would settle whether the flaw has practical impact: if signed-cosine scores produce materially different Table 1 results, the central claim needs revision; if the embedding space happens to yield no negative cosines, the concern is moot. Either way, the existing verdict remains CONDITIONAL: the method must be re-evaluated with the corrected similarity definition and held-out data before its reported rates, such as the 39.1% parasite rate in Figure 9, can be trusted. I did not find a basis to move to REJECT because the proposed pipeline is coherent and the fix is straightforward, and I did not find grounds to ACCEPT because no released artifacts, error bars, or sensitivity analysis accompany the claims. The reader and I converge on the need for validation, though I locate the primary hazard earlier in the pipeline.","tokens_in":10958,"tokens_out":5076,"duration_ms":58809,"concrete_test":"Recompute the duplicate and parasite scores in Table 1 using signed cosine similarity (drop the absolute value in Eq. 2) on the same 129 labeled sentences with the same 0.6 threshold and the same mxbai-embed-large-v1 model. If the F1 scores change materially, or if antonymic pairs such as \"we can\" versus \"we can't\" are ranked above synonymous pairs, the absolute value is not an innocent convention and the central claim fails as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central construction in Eq. 2 defines cosine_sim as |<E_St|E_St'>| / (||E_St|| * ||E_St'||), i.e., absolute cosine similarity. This is not a harmless notational choice: it makes a pair of sentences with opposite meanings score as maximally similar whenever their embeddings point in opposite directions. For example, \"we can go drake\" and \"we cannot go drake\" would receive a similarity near 1, and Eq. 3 would flag the second as a duplicate of the first — exactly the opposite of the intended \"same idea\" semantics. The same absolute-value definition is carried into the parasite scoring in Section 4.1.1, so a sentence semantically opposed to a parasite phrasing could also be flagged as parasite. The paper reports no signed-cosine statistics and does not establish that the embedding space is confined to a single orthant, so the F1 scores in Table 1 characterize an absolute-value variant whose behavior can diverge from the stated claim. This internal issue is more load-bearing than the threshold-transferability concern because it undermines the metric before any threshold is applied. If the embedding model produces negative cosine values for antonymic pairs, the reported results do not support the central claim as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a pipeline for analyzing in-game voice communication in esports (League of Legends) by transcribing team voice logs, diarizing speakers, and then computing two semantic-similarity scores: a duplicate score that compares each sentence embedding to the speaker's previous sentences within a 15-second window, and a parasite score that compares each sentence embedding to a fixed list of 12 coach-defined \"parasite\" phrasings. The authors evaluate three embedding models on a single 359-second recording containing 129 sentences, reporting accuracy, precision, recall, and F1 for duplicate and parasite detection, and conclude that mxbai-embed-large-v1 gives the most balanced performance. They also propose a context-based embedding refinement for single-word sentences.","tokens_in":11205,"tokens_out":4767,"duration_ms":45493,"significance":"The paper addresses a genuinely important and under-studied problem: quantitative analysis of voice communication quality in esports. The practical motivation is clear, and the pipeline (WhisperX transcription, diarization, sentence embeddings) is sensible and reproducible, with the exact embedding models and the full list of parasite phrasings provided. If the methodological concerns raised below are resolved, the metrics could be useful for coaches and analysts. However, the current evaluation is too small to support the reported quantitative conclusions, and the absolute-value cosine similarity is internally inconsistent with the stated semantics of the task.","major_comments":[{"comment":"The definition of cosine_sim takes the absolute value of the cosine. This is not a harmless convention: for sentence embeddings, a pair of contradictory statements (e.g., 'we can go drake' and 'we cannot go drake') will have cosine near -1, which Eq. (2) maps to +1, making them appear maximally similar. This directly contradicts the intended meaning of 'closest in meaning' used in Eq. (3) and propagated to the parasite score in Eqs. (5)-(6). The paper does not report any signed-cosine statistics, nor does it show that the embedding space is restricted to a single orthant for this domain. The metric must be corrected (likely by removing the absolute value) and the experiments rerun; the current results characterize a metric that does not match the stated claim.","section":"Section 3.1.2, Eq. (2)"},{"comment":"The threshold of 0.6 is described in Section 7.1 as 'one of the most reliable', which indicates it was selected using the same evaluation data. No held-out set, cross-validation, or sensitivity analysis is provided. Moreover, Table 1 is based on a single audio segment (359 seconds, 129 sentences) with only 17 duplicate-positive and 21 parasite-positive instances. With such small counts, the reported F1 differences between models (e.g., 50.91% versus 37.04% for duplicates) are not statistically reliable. A rigorous evaluation with held-out data, confidence intervals, and a threshold sweep is necessary to support the central performance claims.","section":"Section 7.1 / Section 5.2 (Table 1)"},{"comment":"The ground-truth definition of 'parasite' appears to overlap with the detector's reference list. The same coach-involved process that produced the 12 parasite phrasings in Appendix 8.2 is used to define the concept, and the human labeling described in Section 5.1 also involved professional coaches. If the annotators were given these same phrases (explicitly or implicitly) when labeling, the parasite detection is partially circular: the detector would be rewarded for reproducing the list rather than for detecting an independently defined communication deficiency. Please clarify the labeling protocol and state clearly whether the human labels are independent of the 12 phrases; if they are not, the reported parasite F1 must be interpreted as a measure of agreement with a fixed reference set, not as detection of a broader construct.","section":"Sections 4.1.1 and 5.1 (Appendix 8.2)"},{"comment":"The context-based embedding refinement for single-word sentences is only illustrated on one example (sentence 018 in Figure 8). It is not evaluated on the full test set, so it is unclear whether this proposed enhancement actually improves the quantitative results in Table 1. If this refinement is intended as a contribution, it needs a systematic evaluation; otherwise, the paper should state explicitly that Table 1 reflects only the unrefined method.","section":"Section 4.2 (Embedding refining)"}],"minor_comments":[{"comment":"The standard cosine similarity is signed; the use of the absolute value should be justified or removed, and its effect on the reported results (if any) should be discussed.","section":"Section 3.1.2"},{"comment":"The index set in Eq. (5) uses n_i (the number of sentences spoken by SPEAKER_i) as the upper bound for the parasite phrasing index j. This should be the number of parasite phrasings (12 in Appendix 8.2), not n_i.","section":"Section 4.1.1, Eq. (5)"},{"comment":"No inter-annotator agreement is reported for the human labels. Given the subjectivity of the 'parasite' concept, agreement statistics (e.g., Cohen's kappa) would help establish label reliability.","section":"Section 5.1"},{"comment":"There are numerous typographical and grammatical errors, including 'direclty', 'apropach', 'labelized', 'interference' used for 'parasite' in some figure captions, and inconsistent spacing around equations. A thorough copyedit is needed.","section":"Throughout"},{"comment":"The three models are compared only via overall accuracy/F1; no statistical tests (e.g., permutation tests or confidence intervals for F1) are provided, so the claim that mxbai-embed-large-v1 is superior is not substantiated.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"This is a promising preliminary study, but the current evaluation is too small and the technical issues (especially the absolute-value cosine) are load-bearing for the central claims. The paper would benefit from a more rigorous evaluation protocol, perhaps on multiple games and with held-out validation, before it is suitable for a journal venue. The authors should also clarify the labeling process to rule out circularity in the parasite detection evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this one. First, the application is real: nobody else has tried sentence-transformers and cosine similarity on esports voice chat to flag duplicate and 'parasite' comms, and the coach-authored parasite phrasing list plus context-pooling for single-word utterances are thoughtful domain touches. Second, the central metric has a load-bearing flaw that the paper's own definition invites: Eq. 2 defines cosine_sim as the absolute value of the cosine. That means 'we can go drake' and 'we cannot go drake' get a similarity near 1, and Eq. 3 would flag the second as a duplicate of the first. The same absolute value is carried into parasite scoring. Since sentence embeddings are not confined to a single orthant, antonyms will often produce negative cosines that the absolute value flips to high similarity. The paper never reports signed-cosine statistics, so the F1 scores in Table 1 characterize a variant that can systematically reward contradiction. That is not a threshold-tuning problem; it breaks the metric's semantic interpretation before any threshold is applied.\n\nWhat the paper does well: the pipeline is reproducible in principle (WhisperX transcription, pyannote diarization, mxbai embeddings, lookback windows), the authors are transparent about the small evaluation set (129 sentences, 17 and 21 positive examples) and about threshold choice, and they explicitly list limitations and future work. The context-pooling trick for single-word sentences is a sensible fix and appears to work on the one example shown.\n\nThe soft spots beyond the absolute value: the 0.6 threshold and 15s window appear to be tuned on the same data with no held-out set or sensitivity analysis; there are no baselines (e.g., TF-IDF or BM25) to justify the semantic approach; and the reported F1s have no error bars. The paper also does not connect communication scores to match outcomes, so the practical value stays speculative. These are the usual problems with a small applied study, and the authors acknowledge most of them.\n\nWho is this for? Esports analysts or hobbyist data scientists looking for a starting point. It is not publishable as is because of the absolute-value issue, but the idea and the pipeline deserve a serious referee. I would send this to peer review with the expectation of major revision: fix the metric to signed cosine, re-run the evaluation, and at minimum release the labeled data and code so the numbers can be checked. If that is done, the paper could become a useful niche contribution.\n\nRecommendation: engage, but only conditionally. The absolute-value cosine is a genuine flaw that the authors and reviewers must catch before any of the performance claims are taken seriously.","headline":"A genuinely new esports-voice application with a cleart pipeline, but the absolute-value cosine similarity in Eq. 2 makes contradictory statements look identical, undermining the core metric before thresholds are even considered.","tokens_in":11706,"tokens_out":1783,"would_cite":false,"duration_ms":19584,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Transcribed voice chat from esports matches can be scored per sentence by maximum cosine similarity against a player's own recent utterances and against coach-defined parasite phrasings, yielding per-player repetition and interference…","keywords":["voice communication analysis","esports","League of Legends","sentence embeddings","semantic similarity","duplicate communication","parasite communication","NLP"],"falsifier":"Run the published duplicate and parasite classifiers on a held-out set of League of Legends voice logs with fresh human labels, sweep the decision threshold from 0.4 to 0.8 and the window from 5 to 30 seconds, and check whether the F1 scores in Table 1 are reproduced or whether the optimal operating point moves. A large drop outside the narrow tested conditions would show the 0.6 threshold is dataset-specific rather than a stable property of esports communication.","tokens_in":10731,"feed_emoji":"🎙️","tokens_out":7626,"duration_ms":62017,"temperature":0.7,"pith_summary":"This paper tries to turn raw voice chat from a League of Legends team into two measurable communication faults: repeated sentences (duplicates) and unclear, hesitant, or off-target phrasings (parasites). The proposed method transcribes and diarizes a game's audio, embeds each sentence with a sentence-transformer model, and compares each sentence against the same player's previous sentences within a $W=15$ second window, taking the maximum cosine similarity as a repetition score. A second comparison runs each sentence against a coach-curated list of parasite phrasings and flags any sentence whose top similarity passes 0.6. The paper reports that this pipeline gives usable per-player rates, such as a 39.1% parasite rate for one speaker, and that embedding-model choice changes accuracy, precision, and recall substantially. A sympathetic reading is that the authors are establishing a proof-of-concept that off-the-shelf NLP embeddings can produce interpretable communication analytics for esports coaching without any sport-specific training data.","feed_headline":"Cosine similarity exposes repeated and hesitant calls in esports","feed_subtitle":"Voice logs become per-player repetition and parasite rates via sentence-similarity scoring.","key_machinery":"The load-bearing mechanism is the sentence transformer plus cosine similarity. Each utterance is mapped to a high-dimensional embedding of roughly 1024 dimensions by a BERT-based sentence encoder, and two utterances are judged close when the absolute cosine of the angle between their embedding vectors is near 1. Duplicate scoring takes the maximum of that similarity over the player's own prior utterances inside a sliding $W=15$ second window; parasite scoring takes the maximum over a coach-defined list of phrasings and applies a fixed 0.6 decision threshold. A refinement step recomputes single-word sentence embeddings by pooling token embeddings after feeding the preceding conversational context through the transformer, which the paper shows lowers spurious similarity to \"Maybe\" and similar parasite phrasings.","core_discovery":"The paper's central claim is that a purely semantic similarity score can operationalize communication quality in esports. For duplicates, the score for each sentence is the maximum cosine similarity between its embedding and all of that player's sentence embeddings in the previous $W=15$ seconds; a score above 0.6 marks the sentence as repetitive. For parasites, the same 0.6 cutoff is applied to the maximum cosine similarity between the sentence embedding and a fixed set of parasite phrasings such as \"I think\", \"Maybe\", and \"Can we?\" The paper argues this separates uncertain, non-directive calls from concise ones, and shows that recomputing embeddings of single-word sentences with surrounding conversation context removes false parasite flags. Reported results on a 359-second, three-speaker, 129-sentence labeled sample give accuracy up to 86.8% for duplicates and 84.5% for parasites with some models, and the paper identifies one embedding model as the most balanced across tasks.","pith_inferences":["If the 0.6 threshold proves stable outside this dataset, the same max-over-window scoring could also detect a player ignoring a teammate's call by repeating a similar phrase after a failed acknowledgement.","Because only transcribed text is used, tone and confidence are invisible; a testable extension is to fuse audio embeddings from the speech recognizer with sentence embeddings and check whether parasite detection improves.","The context-recompute trick for single-word sentences suggests a broader principle: in esports, short utterances derive meaning from context, so embedding refinement may matter more for short calls than for full sentences.","A concrete validation experiment would be to run the published pipeline on the same audio with different sentence-boundary segmentations and report how much duplicate scores move, since the metric depends on how the transcription is split into sentences."],"forward_implications":["Coaches can automatically derive per-player repetition rates and parasite rates from voice logs, such as the 39.1% parasite rate computed for one speaker, without manually labeling every utterance.","The same sentence-similarity pipeline extends to other team esports, since the method uses no League-of-Legends-specific training data.","Model choice materially affects detection: the paper's comparison shows accuracy, precision, and recall vary widely across sentence-embedding architectures, so deployment needs a model-selection step.","A context-refinement step for short sentences prevents common words like \"Okay\" from being mislabeled as parasite communications.","The reported metrics can be correlated in future work with in-game performance indicators, which the paper identifies as the natural next step for validating communication quality's impact on outcomes."],"supporting_citations":[{"why":"Supplies the sentence-embedding model used to compute both duplicate and parasite similarity scores.","marker":"[14]"},{"why":"Provides the sentence-transformer method that encodes whole sentences into contextual vectors and is cited for why single-word inputs are unreliable.","marker":"[16]"},{"why":"Gives the transcription-and-alignment pipeline that turns raw game audio into timestamped text.","marker":"[2]"},{"why":"Provides the speech-recognition model used in the audio-processing pipeline.","marker":"[1]"},{"why":"Supplies the speaker-diarization model that attributes each utterance to a specific player.","marker":"[11]"},{"why":"Implements the diarization pipeline used with the speech-separation model.","marker":"[4]"},{"why":"Defines the BERT transformer blocks that the sentence transformer is built from.","marker":"[6]"},{"why":"Presents the benchmark used to rank candidate embedding models by general performance.","marker":"[15]"},{"why":"Supplies the retrieval analogy that motivates comparing a sentence against prior context by similarity.","marker":"[5]"},{"why":"Provides the lexical BM25 baseline that the paper contrasts with semantic similarity.","marker":"[17]"}],"fun_headline_variants":["Cosine similarity exposes repeated and hesitant calls","Semantic scoring detects repetition and parasite phrases","AI embedding model flags uncertain and repetitive comms","Esports voice analysis: quantifying hesitation via embeddings","Threshold-based similarity score identifies weak calls"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the hand-picked 0.6 cosine similarity threshold and the 15-second lookback window, chosen on the same small sample used for evaluation, will keep working for other games, players, and communication styles; no sensitivity analysis or held-out validation is reported to support that.","fun_headline_variants_meta":{"raw":{"variants":["Cosine similarity exposes repeated and hesitant calls","Semantic scoring detects repetition and parasite phrases","AI embedding model flags uncertain and repetitive comms","Esports voice analysis: quantifying hesitation via embeddings","Threshold-based similarity score identifies weak calls"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000615,"raw_usage":{"total_tokens":2805,"prompt_tokens":840,"completion_tokens":1965,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":456,"completion_tokens_details":{"reasoning_tokens":1899}},"tokens_in":456,"tokens_out":1965,"duration_ms":15119,"temperature":1.0,"reasoning_tokens":1899,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:34:19.230234+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the published duplicate and parasite classifiers on a held-out set of League of Legends voice logs with fresh human labels, sweep the decision threshold from 0.4 to 0.8 and the window from 5 to 30 seconds, and check whether the F1 scores in Table 1 are reproduced or whether the optimal operating point moves. A large drop outside the narrow tested conditions would show the 0.6 threshold is dataset-specific rather than a stable property of esports communication.","supporting_citations":[{"cited_title":"Open source strikes bread - new fluffy embeddings model, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the sentence-embedding model used to compute both duplicate and parasite similarity scores."},{"cited_title":"PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordings","cited_arxiv_id":null,"evidence_quote":"Supplies the speaker-diarization model that attributes each utterance to a specific player."},{"cited_title":"pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe","cited_arxiv_id":null,"evidence_quote":"Implements the diarization pipeline used with the speech-separation model."},{"cited_title":"The probabilistic relevance framework: Bm25 and beyond","cited_arxiv_id":null,"evidence_quote":"Provides the lexical BM25 baseline that the paper contrasts with semantic similarity."}],"review_version":1}