REVIEW 3 major objections 4 minor
Can User Feedback Help Issue Detection? An Empirical Study on a One-billion-user Online Service System
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Feedback topic distributions stay stable over time, so machine-learning-based issue detection is viable despite noisy feedback.
desk verdict A large, production-scale empirical study that is worth a referee: the dataset is new and the measurements are honest, but the ML-viability claim rests on a softer inference than the abstract suggests. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is an empirical dataset of 50,378,766 user feedback items drawn from six real-world services in a one-billion-user online service system, with user feedback written in natural language and severe issues identified through a ground-truth labeling process such as incident reports or operational alerts. The analysis works by examining what users write in feedback, testing whether feedback features can indicate severe issues, and comparing the distributions of feedback topics across different time intervals to see whether they are similar. The topic-distribution comparison is the mechanism that carries the paper's main conclusion: similarity across time intervals is what makes a machine-learning model trained on labeled feedback transferable to future feedback.
What would settle it
Recompute the topic distributions at finer weekly granularity and during documented incident windows: if the distributions shift significantly around outages or feature rollouts, the stability claim fails. Alternatively, build a classifier that combines feedback text, metadata, and temporal features and evaluate it on held-out severe-issue labels; if it clearly outperforms a severity-based baseline, the claim that severe issues cannot be detected from feedback characteristics would be contradicted.
Extended reading notes
Core claim
The central claim is that, in a large production online service system, the distributions of feedback topics in different time intervals are similar, and this stability makes machine-learning-based analysis of user feedback a viable direction. The paper also reports that a large proportion of user feedback provides issue-irrelevant information, so filtering is a necessary preprocessing step, and that severe issues cannot be easily detected based solely on user feedback characteristics. Read sympathetically, the paper is establishing an empirical foundation: the noise level of feedback is high, the signal for severe issues is weak in the text itself, but the thematic structure of feedback is stable enough for learned models to exploit.
Load-bearing premise
The conclusion about severe issues depends on ground-truth labels of severe issues being correct and complete, and on those labels being matched accurately to the right user feedback items; if incident labels are noisy or missing, the weak text signal for severe issues may be an artifact of labeling rather than a property of the feedback.
Editorial extensions
If this is right
- Feedback-based issue detection pipelines should include an explicit filtering step to remove issue-irrelevant feedback before any analysis.
- Severe-issue detection cannot rely on feedback content alone; it needs to combine feedback with operational signals such as alerts or incident records.
- A model trained on feedback topics from one time period can plausibly be reused on later periods without frequent retraining, because the topic distributions are similar.
- Empirical evaluations of feedback-based issue detection should report the proportion of irrelevant feedback and the stability of topic distributions as context for accuracy results.
Reading between the lines
- If topic distributions stay stable only in normal operation, then during outages or large feature rollouts the assumption may break; a testable extension is to measure topic drift around incident windows.
- The stability result would make feedback analysis most valuable for triage and prioritization of recurring issues, while critical-incident detection would still depend on infrastructure signals rather than user text.
- The scale of the system studied means the conclusions may not transfer to small services where feedback volume is low and user populations are less diverse.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports an empirical study on 50,378,766 user feedback items collected from six services in a large online service system with over one billion users. It analyzes what users write in feedback, examines whether feedback features can identify severe issues, and investigates whether machine-learning-based analysis of feedback is viable by comparing topic distributions across time intervals. The main findings are that a large proportion of feedback is issue-irrelevant, that severe issues are not easily detectable from feedback characteristics alone, and that feedback topic distributions are similar across time, which the authors interpret as supporting the viability of ML-based feedback analysis.
Significance. If the empirical results hold, this would be a valuable large-scale descriptive foundation for feedback-based issue detection, with practical implications for industrial systems. The dataset size is a clear strength, and the finding that much feedback is issue-irrelevant is practically important. However, the abstract alone does not provide enough methodological detail to assess the validity of the central claims, particularly the labeling of severe issues, the definition of relevance, and the logical link between stable topic distributions and the viability of supervised ML approaches. The study is observational and descriptive, so its value depends critically on data quality and the correctness of the inference chain.
major comments (3)
- [Abstract, Results (topic stability)] The inference from 'the distributions of the feedback topics in different time intervals are similar' to 'designing machine learning-based approaches is a viable direction' is not justified by the abstract alone. Stable marginal topic distributions do not imply stable conditional distributions such as P(severe issue | topic, time), which is what a supervised model would need to generalize. Given the abstract also reports that severe issues are not easily detected from feedback characteristics, the dominant topics may be dominated by issue-irrelevant content, and the rare severe-issue signal may drift even if the overall topic distribution is stable. Please provide a direct demonstration of ML transferability, such as training on one time period and evaluating on later periods, or explicitly qualify the claim as applying only to topic-discovery tasks rather than issue-detection tasks.
- [Abstract, Method (severe issue detection)] The finding that severe issues cannot be easily detected from feedback features is only meaningful if the ground-truth labels of severe issues are complete and correct. The abstract does not state how severe issues are identified (e.g., incident reports, operational alerts), how those labels are matched to feedback items, or how label noise and missing labels are handled. If many severe issues expressed in feedback are never escalated, or if escalation criteria change over time, the observed inability to detect them may be an artifact of the labeling process rather than a property of the feedback. Please report the labeling procedure, any validation of label quality (such as inter-rater agreement or comparison with multiple data sources), and a robustness analysis under assumed label noise.
- [Abstract, Method (issue-irrelevant feedback)] The claim that 'a large proportion of user feedback provides irrelevant information about system issues' requires a clear operational definition of 'irrelevant' and evidence that the relevance labels are reliable. The abstract does not specify the annotation scheme, whether relevance is binary or graded, or the inter-annotator agreement. Without this information, the proportion of irrelevant feedback is not interpretable, and the downstream recommendation to filter issue-irrelevant content is not well supported. Please provide the annotation criteria, the number of annotators, agreement measures, and a breakdown of the categories used to label relevance.
minor comments (4)
- [Abstract, Method] Please state the time span over which the 50,378,766 feedback items were collected and the distribution of items across the six services, as this affects the generalizability of the temporal-stability analysis.
- [Abstract, Results] The abstract does not report the number or proportion of severe issues in the dataset; including this would help readers calibrate the difficulty of the detection task.
- [General] A table summarizing the six services, their types, and feedback volumes would improve the paper's readability and allow readers to assess the diversity of the dataset.
- [General] Consider adding a limitations paragraph that explicitly discusses the potential impact of label noise, missing incident reports, and drift in labeling criteria on the conclusions.
Circularity Check
No circular reasoning: the study is observational and its conclusion is an empirical inference, not an input by construction.
full rationale
The paper reports an empirical study on user feedback data and draws three findings: (1) much feedback is issue-irrelevant, (2) severe issues cannot be easily detected from feedback characteristics alone, and (3) feedback topic distributions are similar across time intervals, which the authors interpret as supporting the viability of machine-learning-based analysis. None of these claims is derived by defining a quantity in terms of the conclusion, fitting a parameter to a subset and then predicting it, or invoking a self-citation as the load-bearing justification. The topic-stability result is an empirical observation about marginal distributions, and the inference to ML viability is a reasonable inductive step, even if one might question whether marginal stability implies stability of topic-to-severity mappings. That concern is a threat to external validity or construct validity, not circularity. There is no equation in the abstract that reduces to an input, and no fitted value is relabeled as a prediction. Therefore, the paper exhibits no significant circularity.
Assumptions & free parameters
assumptions (3)
- domain assumption Severe-issue ground truth, from operational incident records or similar sources, is accurate and complete.
- domain assumption The six services and 50,378,766 feedback items are representative of the one-billion-user system.
- standard math The statistical and topic-modeling methods used to compute topic stability are valid.
Cite this review
Pith. "Pith review of Can User Feedback Help Issue Detection? An Empirical Study on a One-billion-user Online Service System." pith.science (2026). https://pith.science/paper/7HPV4KXK
@misc{pith2026250800593,
author = {Pith},
title = {Pith review of: Can User Feedback Help Issue Detection? An Empirical Study on a One-billion-user Online Service System},
year = {2026},
howpublished = {\url{https://pith.science/paper/7HPV4KXK}},
note = {Machine review of arXiv:2508.00593}
}
read the original abstract
Background: It has long been suggested that user feedback, typically written in natural language by end-users, can help issue detection. However, for large-scale online service systems that receive a tremendous amount of feedback, it remains a challenging task to identify severe issues from user feedback. Aims: To develop a better feedback-based issue detection approach, it is crucial first to gain a comprehensive understanding of the characteristics of user feedback in real production systems. Method: In this paper, we conduct an empirical study on 50,378,766 user feedback items from six real-world services in a one-billion-user online service system. We first study what users provide in their feedback. We then examine whether certain features of feedback items can be good indicators of severe issues. Finally, we investigate whether adopting machine learning techniques to analyze user feedback is reasonable. Results: Our results show that a large proportion of user feedback provides irrelevant information about system issues. As a result, it is crucial to filter out issue-irrelevant information when processing user feedback. Moreover, we find severe issues that cannot be easily detected based solely on user feedback characteristics. Finally, we find that the distributions of the feedback topics in different time intervals are similar. This confirms that designing machine learning-based approaches is a viable direction for better analyzing user feedback. Conclusions: We consider that our findings can serve as an empirical foundation for feedback-based issue detection in large-scale service systems, which sheds light on the design and implementation of practical issue detection approaches.
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.