{"id":"504a70e4-a36c-4241-beb7-35cef688bef6","arxiv_id":"2506.00924","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":9,"one_line_summary":"A comment-scoring and per-provider aggregation pipeline for QoE is proposed, but its outage-detection test injects the low scores by hand and its ISP labels are random, so the claimed detection capability is not demonstrated.","lead":"This paper builds a two-layer system for monitoring video quality: an AI model predicts quality scores from network statistics, and a pipeline turns live-stream viewer comments into quality scores grouped by internet provider. It is a concrete attempt to use chat sentiment as a network-quality sensor, but the outage-detection test writes the outage into the scores by hand, so the headline capability remains unvalidated.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The outage-detection experiment is circular: Eq. (4) overwrites the outcome variable, so the test would pass even if comments contained no signal about network degradation; no evidence links comment text or volume to real outages.","rationale":"The reader's weakest_assumption and my own reading converge on the same load-bearing defect: §6.4's controlled outage simulation is circular. Eq. (4) directly writes the outcome variable, so the resulting ΔMOS drop is a tautology of the aggregation and thresholding code, not evidence that subjective platform feedback can sense network degradations. Because the abstract's headline claim is precisely that the framework can identify disruptions 'through comment-based trends alone,' the central empirical demonstration does not support the central claim. The random ISP assignment (§4.5, §5.3) and the absence of any human or ITU validation of the LLM MOS scale are compounding gaps, but the Eq. (4) circularity is decisive on its own. I also note mechanical inconsistencies (33,770 vs. 40,000 valid records; Fig. 9(c) caption/title mismatch; unfinished sentence in §6.5) and the lack of released code/data, though those are secondary. The paper is not without merit: the aggregation equations, filtering rules, and windowing logic are clearly specified and could be re-implemented, and the idea of benchmarking an operator's comment-based sentiment against a platform baseline is a reasonable dashboard primitive. But a re-implementable pipeline is not the same as a validated sensing capability. The verdict should remain REJECT: the paper would need either a non-circular outage experiment on real or realistically generated comment text, or a validation of the LLM MOS against human labels, before its central claims are supportable.","tokens_in":20108,"tokens_out":5610,"duration_ms":55804,"concrete_test":"Re-run the §6.4 experiment without Eq. (4). Use a time window with a known ISP3 degradation and raw comments from that window (e.g., from TwitchChat plus network telemetry) and run the full pipeline—semantic filtering, LLM MOS scoring from the original text, 5-minute aggregation, ΔMOS—without modifying any MOS values. The central claim is supported only if ΔMOS crosses the -0.4 threshold in this text-only run. As a secondary control, re-score a held-out sample with each comment's MOS overwritten by the dataset's mean; detection should not occur, confirming the threshold is sensitive to comment-derived scores, not to the injection.","verdict_should_be":"REJECT","load_bearing_attack":"The load-bearing step is §6.4's simulated outage (Eq. 4). The abstract claims the framework identifies service disruptions 'through comment-based trends alone,' but Eq. (4) does not simulate a change in comment content, volume, or timing: it forcibly sets MOS_i = 1.0 for every ISP3 comment inside [15:00, 16:30]. Re-aggregating and comparing to the global mean then guarantees a negative ΔMOS that crosses the -0.4 threshold. The experiment tests only the arithmetic of the averaging/thresholding pipeline; it would pass even if the comments were entirely unrelated to network quality, if the LLM scored randomly, or if the pipeline ignored comment text. The second load-bearing premise is equally untested: §4.5 and §5.3 assign ISP labels by uniform random draw, so per-ISP structure in the base data is noise; the only deterministic signal is the hand-injected overwrite. A related gap is that the LLM-assigned MOS values are never validated against human ratings or ITU-standard subjective scores, so even a non-circular comment-based experiment would not establish that the scores carry the meaning the framework assigns them. Positive side: Eqs. (1)-(3), the filtering stages, and the 5-minute aggregation are transparent and re-implementable, so the concern is specifically in the demonstration, not the mechanics.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a dual-layer QoE monitoring framework: an objective layer that maps network KPIs (latency, jitter, loss, throughput) to MOS via a Random Forest model trained on ITU-T P.1203 reference scores, and a subjective layer that filters live-stream comments for QoE relevance, assigns scalar MOS values with a decoder-only LLM at temperature 0, and aggregates these scores per simulated ISP in 5-minute windows. A new metric, delta MOS (the per-ISP average minus the platform-wide average per window), is introduced, and a 'controlled outage simulation' in Section 6.4 is presented as evidence that comment-based trends alone can detect provider-specific degradations. The dataset consists of 47,894 comments, of which 33,770 receive valid MOS scores after filtering, with synthetic timestamps at 3-second spacing and uniform random ISP labels.","tokens_in":20271,"tokens_out":2390,"duration_ms":25430,"significance":"The framework's mechanics are transparent and re-implementable: Eqs. (1)-(3) correctly define per-provider averaging, platform-wide averaging, and per-provider deviation, and the filtering stages are described in sufficient detail to reproduce. The dataset, with roughly 34,000 QoE-relevant comments, is a potentially useful artifact for future work on comment-based QoE. However, the central validation claim is not supported. The outage experiment in Section 6.4 directly overwrites the outcome variable (MOS_i = 1.0 for all ISP3 comments in the window), so the resulting delta-MOS drop is guaranteed by construction and does not test whether comment text, volume, or timing responds to network degradation. In addition, the random assignment of ISP labels in Sections 4.5 and 5.3 means that all per-ISP structure in the base data is noise, and the LLM-assigned MOS values are never calibrated against human subjective ratings or an ITU-standard subjective dataset.","major_comments":[{"comment":"The simulated outage is circular and therefore does not support the abstract's claim that service disruptions are identified 'through comment-based trends alone.' Eq. (4) forcibly sets MOS_i = 1.0 for every ISP3 comment in [15:00, 16:30], and the subsequent re-aggregation then subtracts the global mean from this depressed ISP3 average, guaranteeing a negative delta MOS that crosses the -0.4 threshold. The experiment would pass even if the comments were unrelated to network quality, if the LLM scored randomly, or if the pipeline ignored comment text entirely; it tests only the arithmetic of Eqs. (1)-(3). A valid simulation would need to inject a change in comment content, volume, or timing plausibly associated with an outage and then verify that the pipeline recovers the event, but no such test is provided.","section":"Section 6.4, Eq. (4)"},{"comment":"Uniform random assignment of comments to ISP1-ISP3 does not model operator-level attribution and undermines the per-ISP analysis. With random labels, any ISP-specific structure in the base data is statistical noise, and the only deterministic per-provider signal in the entire paper is the hand-injected overwrite of Eq. (4). Consequently, Figs. 6-9 cannot demonstrate that the framework isolates provider-specific phenomena; at most they show the aggregation formulas applied to synthetic metadata. The paper needs either real IP-range-based attribution or a realistic generative model of user-to-ISP association before claims of operator-level detection are justified.","section":"Sections 4.5 and 5.3"},{"comment":"The LLM-assigned scalar MOS values are never validated against human ratings or standardized subjective QoE data. The prompt defines a 5-point scale, and Fig. 5 shows a heavily skewed distribution, but no evidence is given that the scores correspond to user-perceived quality in any measurable sense. Since the entire subjective side, and the later comparison with objective P.1203-based MOS, depends on the semantic validity of these scores, the absence of any calibration study is a load-bearing gap. The paper should compare LLM scores against at least a small human-annotated set, or against established subjective MOS datasets, before claiming that comment-derived MOS can serve as a 'high-resolution QoE monitoring signal.'","section":"Sections 4.4 and 5.2"},{"comment":"The modeling assumptions of equal user bases and uniform comment volume over time are acknowledged, but the synthetic 3-second timestamp spacing makes Figs. 6-10 a property of simulated metadata rather than of observed comment dynamics. Real comment streams exhibit bursts, pauses, and diurnal variation, and the paper's own stated future work concedes that temporal normalization would be needed in practice. As a result, the time-series visualizations cannot be read as evidence about real-world feedback behavior; they are demonstrations of the aggregation code on an artificial schedule.","section":"Section 6.1 and Section 5.3"}],"minor_comments":[{"comment":"The caption of Fig. 9(c) reads 'ISP2 MOS Over Time' even though the panel is described as the ISP3 time series, and the caption text repeats the same phrasing as the ISP2 panel; the figure and caption should be aligned to avoid mislabeling.","section":"Figure 9(c) caption"},{"comment":"The final sentence of Section 6.5 ends with 'It also supports real-time' and is incomplete; the sentence should be finished or removed.","section":"Section 6.5"},{"comment":"Table 1 has a typo in the header ('Refrence'), and Section 5.1 refers to the field 'commen mos' instead of 'comment mos'; these small errors should be corrected.","section":"Table 1 and Section 5.1"},{"comment":"The description of the LLM is intentionally model-agnostic ('a multilingual, multimodal, instruction-tuned transformer released in 2024'), but the reproducibility claim in Section 2 would be stronger if the exact model identifier, version, and prompt template were provided in the text or in an appendix.","section":"Section 4.4"},{"comment":"The phrase 'controlled outage simulation' is misleading because Eq. (4) is a manual overwrite of the output variable rather than a simulation of comment-level effects; rewording to 'synthetic outcome injection' would more accurately describe the experiment.","section":"Abstract and Section 6.4"}],"recommendation":"reject","confidential_remarks":"The manuscript's core validation is circular in a way that cannot be fixed by local edits: the outage-detection result in Section 6.4 is entailed by the injection in Eq. (4), and the random ISP assignment removes any operator-level signal from the real comment data. The objective-side contribution, inherited from prior work [9], is not the focus of this submission, and the subjective-side claims would require substantial new experimental work, including a realistic outage simulation and LLM-score validation against human ratings. There is also a concern about fit with the journal's scope, since the main evaluation is on synthetic metadata rather than network or platform measurements. If the authors later add real attribution and a genuine content-based outage test, the aggregation framework itself could be resubmitted as a methodological note or dataset paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper assembles something genuinely new—a 33,770-comment LLM-scored dataset with synthetic ISP/time metadata and a per-operator delta-MOS aggregation pipeline—but its headline result is circular. The outage detection in §6.4 doesn't test the framework; it overwrites the outcome.\n\nWhat's good: Eqs. (1)–(3) are correct, the filtering and scoring pipeline is described concretely enough to re-implement, and the delta-from-platform-average metric is a reasonable dashboard primitive. The authors are honest about the equal-ISP and uniform-volume assumptions. That's real infrastructure work.\n\nThe soft spot is load-bearing. In §6.4, Eq. (4) sets every ISP3 comment's MOS to 1.0 inside the window, then the re-aggregated delta MOS naturally crosses the –0.4 threshold. That experiment would pass even if the comments had nothing to do with network quality, or if the LLM scored randomly. It tests the arithmetic of averaging, nothing else. The paper's abstract claim that the framework identifies 'service disruptions through comment-based trends alone' is not supported by this demonstration.\n\nTwo further gaps compound it. ISP labels are assigned uniformly at random (§4.5, §5.3), so the per-ISP structure in the base data is noise; the only deterministic signal is the hand-injected overwrite. And the LLM-assigned MOS values are never validated against human ratings or ITU-standard subjective scores, so even a non-circular experiment wouldn't establish that the scores mean what the framework assumes. Minor issues: the 40,000 vs 33,770 record count, the mismatched Fig. 9(c) caption, and the unfinished sentence in §6.5, plus no released code or data despite the open-source claim.\n\nOverall: the framework idea is worth discussing, but this version doesn't demonstrate it. The fix is a real validation—e.g., using actual outage events with comment data collected during them, or human-annotated MOS for a comment sample. I'd send it to peer review because the idea is useful and the flaw is clear and correctable; an editor shouldn't desk-reject it, but any referee should require major revision before acceptance.","headline":"A genuinely new dataset and aggregation pipeline, but the outage-detection validation is circular and the paper's central claim doesn't survive contact with its own methods.","tokens_in":20999,"tokens_out":2289,"would_cite":false,"duration_ms":19352,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that LLM-scored live-stream comments, filtered for performance complaints and aggregated per provider in five-minute windows, can detect service degradations through comment trends alone.","keywords":["quality of experience","QoE monitoring","live-stream comments","LLM scoring","mean opinion score","delta MOS","ITU-T P.1203","semantic filtering"],"falsifier":"Correlate real, per-provider comment-derived MOS against a known network incident, such as an announced regional ISP outage, using true ISP attribution rather than random assignment; the central claim fails if comment-derived MOS does not drop significantly during the incident for the affected provider while staying stable for unaffected providers.","tokens_in":19706,"feed_emoji":"📡","tokens_out":6172,"duration_ms":57803,"temperature":0.7,"pith_summary":"This paper tries to establish that unstructured viewer comments on live-streaming platforms can be turned into a high-resolution, operator-level quality-of-experience (QoE) monitoring signal. It builds a two-layer framework: an objective layer that predicts video MOS from network key performance indicators using a machine-learning model trained on ITU-T P.1203 reference scores, and a subjective layer that filters comments for performance complaints and uses a large language model to assign each a scalar 1\\textendash 5 MOS score. The scores are aggregated by simulated Internet service provider and into five-minute time windows, and a delta MOS metric measures each provider's deviation from the platform-wide average. A controlled simulation in which all of one provider's comment scores are forcibly lowered inside a 90-minute window shows the delta signal dropping below a threshold, which the authors present as evidence that comment-derived trends alone can localize a disruption. If the framework holds, operators would receive privacy-preserving, per-provider subjective MOS reports every five minutes without any client-side instrumentation or video content analysis.","feed_headline":"Detect ISP outages from viewer comments alone","feed_subtitle":"A comment-scoring pipeline turns live-stream chat into per-provider quality ratings every five minutes.","key_machinery":"The central object is the delta MOS metric, defined as $\\Delta\\text{MOS}_{i,t} = \\mu_{i,t} - \\text{GlobalMOS}_t$, where $\\mu_{i,t}$ is the average LLM-derived MOS for provider $i$ in time window $t$ and $\\text{GlobalMOS}_t$ is the equal-weight platform-wide average over providers in the same window. This subtraction is what separates provider-specific shifts from platform-wide trends, so a negative deviation flags a localized problem while a value near zero means the provider tracks the platform baseline. The supporting machinery is the semantic filter (problem-plus-context keyword matching, meme rejection, and embedding similarity against anchors with a higher threshold for short comments), the deterministic LLM scoring at temperature zero, and the aggregation of comments into five-minute windows.","core_discovery":"The central discovery is that a two-stage pipeline can convert noisy live-stream chat into a per-operator, time-resolved subjective quality metric that is directly comparable to network-derived MOS. The filtering stage keeps only comments that mention both a problem (such as lag or buffering) and a streaming or network context, rejects meme phrases, and applies embedding-based cosine similarity to anchor phrases with length-sensitive thresholds. The scoring stage prompts an instruction-tuned multilingual LLM in deterministic mode to return a MOS value from 1 to 5, or \\textendash 1 for irrelevant text, yielding reproducible scalar scores. Once each comment is randomly assigned to one of three simulated ISPs and given a synthetic timestamp, per-window averages are computed, and subtracting the platform-wide mean from each provider's average defines the delta MOS. The paper argues that a sustained negative delta identifies provider-specific degradation even without network telemetry, demonstrating this with an injected outage in which every one of ISP3's comments inside the window is overwritten to the minimum score.","pith_inferences":["The outage simulation validates the aggregation arithmetic but not the sensing chain: because the simulated event overwrites the outcome variable (every comment score becomes 1.0) rather than perturbing network state and letting real comments respond, the experiment cannot by itself show that genuine outages would move comment-derived MOS.","A field test would close that gap: real ISP attribution and known outage records could be correlated against comment-derived per-ISP MOS, or LLM scores from a measured degraded window compared against a control window.","The uniform random ISP assignment means any baseline per-provider structure in the data is noise; once real attribution is available, the paper's equal-user-base and uniform-comment-volume assumptions would need to be replaced by volume- and engagement-weighted aggregation.","The same delta MOS machinery transfers to any time-stamped user text with inferable provider or regional attribution, including support forums, app-store reviews, and multiplayer game chat."],"forward_implications":["Each operator would receive, every five minutes, its own subjective MOS and the platform-wide average, letting it distinguish a quality drop confined to its infrastructure from an issue affecting all providers.","A per-ISP delta MOS below an operator-configured threshold (the paper illustrates \\textendash 0.4) can act as an outage alert driven purely by comment-derived sentiment.","Operators can triangulate the comment-based MOS against objective MOS from their own telemetry or from the P.1203-based model, surfacing cases where network KPIs look healthy but users report poor experiences.","The pipeline scales because filtering is keyword-plus-embedding based and each comment is scored by a single-turn LLM call, with no per-user data exposure since only windowed aggregates are reported."],"supporting_citations":[{"why":"Supplies the objective side of the framework: the prior random-forest model trained on ITU-T P.1203 MOS that produces the network-based scores the subjective signal is compared against.","marker":"[9]"},{"why":"The TwitchChat dataset, the stated source of the live-stream comments processed by the filtering and scoring pipeline.","marker":"[30]"},{"why":"The timestamped Twitch chat log data used for records and ISP-level aggregation support.","marker":"[31]"},{"why":"The sentence-transformer encoder used to embed comments for semantic similarity filtering against anchor phrases.","marker":"[34]"},{"why":"Supplies the evaluation of pretrained sentence-transformer models that underlies the anchor-based similarity thresholding.","marker":"[35]"},{"why":"Demonstrates that LLMs can predict MOS for speech content, motivating the use of an LLM to score subjective quality from text.","marker":"[36]"},{"why":"Evidence that instruction-tuned models can capture subjective assessments, supporting the scalar MOS prompting strategy.","marker":"[37]"},{"why":"Survey of LLM uncertainty that motivates the simplified verbalization-based scalar scoring design used in the pipeline.","marker":"[39]"},{"why":"Documents the instruction-tuned multilingual LLM that performs the deterministic, temperature-zero MOS scoring of comments.","marker":"[41]"}],"fun_headline_variants":["LLM scoring of chat comments flags ISP outages","Per-provider QoE from live-stream chat analysis","Delta MOS turns viewer comments into outage alerts","AI-driven comment filters track operator quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The outage-detection result depends on the untested premise that real viewer complaints, and the LLM scores they receive, actually move when genuine network quality degrades, since the simulation sidesteps that link by directly overwriting all of one provider's scores inside the outage window.","fun_headline_variants_meta":{"raw":{"variants":["LLM scoring of chat comments flags ISP outages","Per-provider QoE from live-stream chat analysis","Delta MOS turns viewer comments into outage alerts","AI-driven comment filters track operator quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000231,"raw_usage":{"total_tokens":1539,"prompt_tokens":1049,"completion_tokens":490,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":665,"completion_tokens_details":{"reasoning_tokens":432}},"tokens_in":665,"tokens_out":490,"duration_ms":5074,"temperature":1.0,"reasoning_tokens":432,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:57:15.412534+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Correlate real, per-provider comment-derived MOS against a known network incident, such as an announced regional ISP outage, using true ISP attribution rather than random assignment; the central claim fails if comment-derived MOS does not drop significantly during the incident for the affected provider while staying stable for unaffected providers.","supporting_citations":[{"cited_title":"An efficient network-based qoe assess- ment framework for multimedia networks using a machine learning approach,","cited_arxiv_id":null,"evidence_quote":"Supplies the objective side of the framework: the prior random-forest model trained on ITU-T P.1203 MOS that produces the network-based scores the subjective signal is compared against."},{"cited_title":"Twitchchat: A dataset for exploring livestream chat,","cited_arxiv_id":null,"evidence_quote":"The TwitchChat dataset, the stated source of the live-stream comments processed by the filtering and scoring pipeline."},{"cited_title":"Twitch.tv Chat Log Data,","cited_arxiv_id":null,"evidence_quote":"The timestamped Twitch chat log data used for records and ISP-level aggregation support."},{"cited_title":"Document-level text simplification,","cited_arxiv_id":null,"evidence_quote":"The sentence-transformer encoder used to embed comments for semantic similarity filtering against anchor phrases."},{"cited_title":"Performance of 4 pre-trained sentence transformer models in the semantic query of a systematic review dataset on peri-implantitis,","cited_arxiv_id":null,"evidence_quote":"Supplies the evaluation of pretrained sentence-transformer models that underlies the anchor-based similarity thresholding."},{"cited_title":"Enabling auditory large language models for automatic speech quality evaluation,","cited_arxiv_id":null,"evidence_quote":"Demonstrates that LLMs can predict MOS for speech content, motivating the use of an LLM to score subjective quality from text."},{"cited_title":"Extraction of subjective information from large language models,","cited_arxiv_id":null,"evidence_quote":"Evidence that instruction-tuned models can capture subjective assessments, supporting the scalar MOS prompting strategy."},{"cited_title":"Gpt-4o overview and capabilities,","cited_arxiv_id":null,"evidence_quote":"Documents the instruction-tuned multilingual LLM that performs the deterministic, temperature-zero MOS scoring of comments."}],"review_version":1}