{"id":"ac2c0e5e-cbc0-417a-a121-aa0b474c7511","arxiv_id":"2508.00593","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"An analysis of 50 million user feedback items shows most feedback is not about system issues, severe issues are hard to detect from feedback text, and feedback topics remain stable over time.","lead":"This paper examines over 50 million user feedback messages from six services inside a large online system. It finds that much feedback is irrelevant, serious issues are hard to identify from feedback alone, and topic patterns stay stable over time.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract infers ML viability from stability of marginal topic distributions, but stable marginals do not imply stable topic-to-severity mappings; ground-truth label completeness is also unverified.","rationale":"The reader's weakest assumption was ground-truth label correctness and completeness, which is indeed load-bearing for the negative finding about severe issues. My concern extends this to the positive viability claim: the same label-quality issue and the additional inferential leap from marginal topic stability to supervised model generalization. The reader's focus was label noise; I agree that is critical, but the paper's main confirmatory claim depends equally on the stability of the conditional distribution P(severe | topic), which is not established by the abstract's reported result. Since this review is based only on the abstract, I cannot assess whether the full paper addresses this with temporal holdout experiments or manual label validation. The current UNVERDICTED verdict is appropriate and should remain. I am not raising an internal inconsistency, only an external validity concern about the strength of the inference. The proposed temporal generalization test is concrete and would settle whether stable topic marginals imply practical ML viability for issue detection, while the manual verification of severe-issue labels would settle the ground-truth concern.","tokens_in":775,"tokens_out":2360,"duration_ms":24246,"concrete_test":"Ask the authors to conduct a temporal generalization experiment: train their proposed topic-based classifier on feedback items from one time interval, using the ground-truth severe-issue labels, and evaluate on a later interval. Report precision/recall against a manually verified sample of severe issues in the test interval. If the held-out-interval performance is comparable to within-interval cross-validation and manual verification confirms label completeness, the concern is resolved; if performance collapses or label verification reveals many missing/escalated items, the viability conclusion overreaches. A secondary check is to compare topic distributions using only the severe-issue-labeled feedback subset across intervals, rather than all feedback, to test whether the conditional signal is stable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that 'the distributions of the feedback topics in different time intervals are similar,' which is taken to 'confirm that designing machine learning-based approaches is a viable direction.' The load-bearing step is the inference from similarity of marginal topic distributions to the practical usability of a supervised model for issue detection. Even if P(topic | time) is stationary, the conditional distribution P(severe issue | topic, time) can drift, especially for rare severe issues that may not be well represented in the dominant topic clusters. The abstract itself reports that severe issues cannot be easily detected from feedback characteristics, which suggests the topic model used for the distribution comparison might be driven by frequent, issue-irrelevant content and could miss the rare signal needed for detection. A second load-bearing premise, as the reader noted, is that ground-truth labels of severe issues from incident reports or operational alerts are complete and correct. If many severe issues expressed in feedback are never escalated, or if escalation criteria change over time, the observed inability to detect them from feedback characteristics may be an artifact of label noise or missing labels. Both premises are required for the empirical evidence to directly support the viability claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical study on 50,378,766 user feedback items collected from six services in a large online service system with over one billion users. It analyzes what users write in feedback, examines whether feedback features can identify severe issues, and investigates whether machine-learning-based analysis of feedback is viable by comparing topic distributions across time intervals. The main findings are that a large proportion of feedback is issue-irrelevant, that severe issues are not easily detectable from feedback characteristics alone, and that feedback topic distributions are similar across time, which the authors interpret as supporting the viability of ML-based feedback analysis.","tokens_in":1135,"tokens_out":3315,"duration_ms":31692,"significance":"If the empirical results hold, this would be a valuable large-scale descriptive foundation for feedback-based issue detection, with practical implications for industrial systems. The dataset size is a clear strength, and the finding that much feedback is issue-irrelevant is practically important. However, the abstract alone does not provide enough methodological detail to assess the validity of the central claims, particularly the labeling of severe issues, the definition of relevance, and the logical link between stable topic distributions and the viability of supervised ML approaches. The study is observational and descriptive, so its value depends critically on data quality and the correctness of the inference chain.","major_comments":[{"comment":"The inference from 'the distributions of the feedback topics in different time intervals are similar' to 'designing machine learning-based approaches is a viable direction' is not justified by the abstract alone. Stable marginal topic distributions do not imply stable conditional distributions such as P(severe issue | topic, time), which is what a supervised model would need to generalize. Given the abstract also reports that severe issues are not easily detected from feedback characteristics, the dominant topics may be dominated by issue-irrelevant content, and the rare severe-issue signal may drift even if the overall topic distribution is stable. Please provide a direct demonstration of ML transferability, such as training on one time period and evaluating on later periods, or explicitly qualify the claim as applying only to topic-discovery tasks rather than issue-detection tasks.","section":"Abstract, Results (topic stability)"},{"comment":"The finding that severe issues cannot be easily detected from feedback features is only meaningful if the ground-truth labels of severe issues are complete and correct. The abstract does not state how severe issues are identified (e.g., incident reports, operational alerts), how those labels are matched to feedback items, or how label noise and missing labels are handled. If many severe issues expressed in feedback are never escalated, or if escalation criteria change over time, the observed inability to detect them may be an artifact of the labeling process rather than a property of the feedback. Please report the labeling procedure, any validation of label quality (such as inter-rater agreement or comparison with multiple data sources), and a robustness analysis under assumed label noise.","section":"Abstract, Method (severe issue detection)"},{"comment":"The claim that 'a large proportion of user feedback provides irrelevant information about system issues' requires a clear operational definition of 'irrelevant' and evidence that the relevance labels are reliable. The abstract does not specify the annotation scheme, whether relevance is binary or graded, or the inter-annotator agreement. Without this information, the proportion of irrelevant feedback is not interpretable, and the downstream recommendation to filter issue-irrelevant content is not well supported. Please provide the annotation criteria, the number of annotators, agreement measures, and a breakdown of the categories used to label relevance.","section":"Abstract, Method (issue-irrelevant feedback)"}],"minor_comments":[{"comment":"Please state the time span over which the 50,378,766 feedback items were collected and the distribution of items across the six services, as this affects the generalizability of the temporal-stability analysis.","section":"Abstract, Method"},{"comment":"The abstract does not report the number or proportion of severe issues in the dataset; including this would help readers calibrate the difficulty of the detection task.","section":"Abstract, Results"},{"comment":"A table summarizing the six services, their types, and feedback volumes would improve the paper's readability and allow readers to assess the diversity of the dataset.","section":"General"},{"comment":"Consider adding a limitations paragraph that explicitly discusses the potential impact of label noise, missing incident reports, and drift in labeling criteria on the conclusions.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"This review is based on the abstract only, as the full text was not available. The paper's central claims are plausible and the dataset scale is a strength, but the abstract as written contains load-bearing gaps: the inference from topic stability to ML viability is not logically sufficient, and the ground-truth labeling of severe issues is not described. If the full text provides the missing validations and directly addresses the time-split generalization concern, the paper could be acceptable after revision. I recommend major revision rather than rejection because these issues are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is an abstract-only read, but what's on the page is a useful, large-scale descriptive study. The 50 million feedback items from six production services is genuinely new, and the two negative results—lots of irrelevant feedback, severe issues not detectable from feedback features—are worth having on record. The paper earns a referee.\n\nThe main claim I can actually check is the one about topic distributions being similar across time intervals, which the authors read as confirmation that ML-based analysis is a viable direction. That inference is softer than it looks. Stable marginal topic distributions don't imply stable conditional mappings between topics and severity, and since the paper itself reports that severe issues are hard to detect from feedback characteristics, the topic clusters may be dominated by frequent, issue-irrelevant content. Also, the whole severe-issue analysis depends on ground-truth labels being complete and correctly matched to feedback items; the abstract doesn't show how that was done or validated. These are real limits, but they're limits on interpretation, not obvious flaws in the measurements.\n\nWhat the paper does well is report concrete numbers at production scale. That alone is useful to anyone building feedback pipelines. The framing is modest—'empirical foundation'—and the conclusions stay close to the data. I don't see circularity in the sense of fitting a model to a dataset and then claiming prediction; the topic-stability measurement is descriptive.\n\nIf I had to advise a referee: check how severe-issue labels were obtained and matched, and ask for robustness checks on the topic-stability result (e.g., different preprocessing, different time granularity, per-service breakdown). The core data is valuable and the paper is honest about what it can and cannot conclude.\n\nI'd send this to peer review. It's not a paradigm changer, but it's solid empirical grounding for a practical problem.","headline":"A large, production-scale empirical study that is worth a referee: the dataset is new and the measurements are honest, but the ML-viability claim rests on a softer inference than the abstract suggests.","tokens_in":1474,"tokens_out":1284,"would_cite":true,"duration_ms":12213,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Feedback topic distributions stay stable over time, so machine-learning-based issue detection is viable despite noisy feedback.","keywords":["user feedback","issue detection","empirical study","online service system","machine learning","topic distribution","severe issue detection"],"falsifier":"Recompute the topic distributions at finer weekly granularity and during documented incident windows: if the distributions shift significantly around outages or feature rollouts, the stability claim fails. Alternatively, build a classifier that combines feedback text, metadata, and temporal features and evaluate it on held-out severe-issue labels; if it clearly outperforms a severity-based baseline, the claim that severe issues cannot be detected from feedback characteristics would be contradicted.","tokens_in":633,"feed_emoji":"📊","tokens_out":3284,"duration_ms":28810,"temperature":0.7,"pith_summary":"The paper studies 50,378,766 user feedback items from six services in a one-billion-user online service system to learn what user feedback can and cannot reveal about system issues. It finds that a large proportion of feedback is irrelevant to actual issues, that severe issues cannot be easily detected from feedback features alone, and that feedback topic distributions are similar across different time intervals. The paper takes the topic-distribution stability as the key empirical justification for building machine-learning models that analyze feedback: if topics repeat over time, a model trained on one period should generalize to later periods. For practitioners, the message is that feedback-based issue detection is viable, but only if issue-irrelevant feedback is filtered out first and severe issues are not expected to emerge from text features alone.","feed_headline":"Feedback topics stay stable, making ML issue detection viable","feed_subtitle":"Study of 50 million user-feedback items across six services shows topics repeat over time, despite much irrelevant chatter.","key_machinery":"The carrying object is an empirical dataset of 50,378,766 user feedback items drawn from six real-world services in a one-billion-user online service system, with user feedback written in natural language and severe issues identified through a ground-truth labeling process such as incident reports or operational alerts. The analysis works by examining what users write in feedback, testing whether feedback features can indicate severe issues, and comparing the distributions of feedback topics across different time intervals to see whether they are similar. The topic-distribution comparison is the mechanism that carries the paper's main conclusion: similarity across time intervals is what makes a machine-learning model trained on labeled feedback transferable to future feedback.","core_discovery":"The central claim is that, in a large production online service system, the distributions of feedback topics in different time intervals are similar, and this stability makes machine-learning-based analysis of user feedback a viable direction. The paper also reports that a large proportion of user feedback provides issue-irrelevant information, so filtering is a necessary preprocessing step, and that severe issues cannot be easily detected based solely on user feedback characteristics. Read sympathetically, the paper is establishing an empirical foundation: the noise level of feedback is high, the signal for severe issues is weak in the text itself, but the thematic structure of feedback is stable enough for learned models to exploit.","pith_inferences":["If topic distributions stay stable only in normal operation, then during outages or large feature rollouts the assumption may break; a testable extension is to measure topic drift around incident windows.","The stability result would make feedback analysis most valuable for triage and prioritization of recurring issues, while critical-incident detection would still depend on infrastructure signals rather than user text.","The scale of the system studied means the conclusions may not transfer to small services where feedback volume is low and user populations are less diverse."],"forward_implications":["Feedback-based issue detection pipelines should include an explicit filtering step to remove issue-irrelevant feedback before any analysis.","Severe-issue detection cannot rely on feedback content alone; it needs to combine feedback with operational signals such as alerts or incident records.","A model trained on feedback topics from one time period can plausibly be reused on later periods without frequent retraining, because the topic distributions are similar.","Empirical evaluations of feedback-based issue detection should report the proportion of irrelevant feedback and the stability of topic distributions as context for accuracy results."],"supporting_citations":[],"fun_headline_variants":["Feedback topics stable enough for ML issue detection","50M feedback items show stable topics, aiding ML","Stable feedback topics make ML detection feasible","User feedback noise high, but topics stable for ML","ML issue detection viable: feedback topics hold steady"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion about severe issues depends on ground-truth labels of severe issues being correct and complete, and on those labels being matched accurately to the right user feedback items; if incident labels are noisy or missing, the weak text signal for severe issues may be an artifact of labeling rather than a property of the feedback.","fun_headline_variants_meta":{"raw":{"variants":["Feedback topics stable enough for ML issue detection","50M feedback items show stable topics, aiding ML","Stable feedback topics make ML detection feasible","User feedback noise high, but topics stable for ML","ML issue detection viable: feedback topics hold steady"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1389,"prompt_tokens":941,"completion_tokens":448,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":377}},"tokens_in":557,"tokens_out":448,"duration_ms":3973,"temperature":1.0,"reasoning_tokens":377,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:02:05.224583+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the topic distributions at finer weekly granularity and during documented incident windows: if the distributions shift significantly around outages or feature rollouts, the stability claim fails. Alternatively, build a classifier that combines feedback text, metadata, and temporal features and evaluate it on held-out severe-issue labels; if it clearly outperforms a severity-based baseline, the claim that severe issues cannot be detected from feedback characteristics would be contradicted.","supporting_citations":[],"review_version":1}