REVIEW 4 major objections 6 minor 22 references
Exploring React Library Related Questions on Stack Overflow: Answered vs. Unanswered
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper claims that Stack Overflow React questions are more likely to be answered when they have more views, include code, and come from reputable askers, while comments, long text, and images lower the odds, and that high-reputation…
desk verdict RQ1 is a solid incremental replication of known answerability factors on React questions; RQ2's 'experienced users ask harder questions' is not supported because the difficulty proxy is unvalidated and the regression is under-reported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument runs on two statistical instruments. For RQ1, a logistic regression model predicts a binary answered/unanswered outcome from seven transformed attributes (code snippet flag, image flag, log-transformed reputation, comment count, view count, question line count, and code line count), using square-root transformations to handle multicollinearity and SMOTE on the training set to balance the classes; the regression coefficients serve as the evidence for which attributes help or hurt answerability. For RQ2, the Performance Difficulty Score (PD Score) is computed as the average answer count divided by the average view count, multiplied by 100, and treated as an inverse proxy for question difficulty, with lower scores supposedly indicating harder questions. A simple linear regression of reputation on PD Score yields the -0.092 slope that supports the claim that experienced users ask harder questions. The PD Score is the load-bearing piece: it converts platform activity into a difficulty measure, and the entire RQ2 conclusion depends on that conversion being valid.
What would settle it
Have a panel of experienced React developers blindly rate the difficulty of a random sample of questions, then compute each question's PD Score; if the rank correlation between human-rated difficulty and PD Score is close to zero, the claim that lower PD Scores mean harder questions, and hence the RQ2 conclusion, fails.
Extended reading notes
Core claim
The central claim is that answerability of React-related Stack Overflow questions is systematically associated with a small set of observable post attributes: ViewCount has the largest positive coefficient (15.40), followed by code snippet presence (0.38), code line count (0.31), and asker reputation (0.10), while CommentCount (-0.40), question line count (-0.35), and image presence (-0.06) are negative. The paper further claims that the Performance Difficulty Score (PD Score), defined as average answer count divided by average view count times 100, correlates negatively with user reputation (slope -0.092, intercept 0.561), with low-reputation askers averaging a PD Score of 0.62 versus 0.46 for high-reputation askers, which is interpreted as evidence that more experienced users submit more technically difficult questions. The authors present these as statistical associations from cross-sectional data, not proven causal effects, and they flag explicitly that the direction of the view–answer relationship is uncertain.
Load-bearing premise
The conclusion that high-reputation users ask harder questions depends entirely on the assumption that a question's PD Score, average answers divided by average views times 100, truly measures how difficult or complex the question is.
Editorial extensions
If this is right
- Developers writing React questions can raise the model-predicted odds of getting an answer by including a focused code snippet, keeping the text short, and avoiding images.
- Questions that accumulate many comments are less likely to be answered, which the paper interprets as excessive discussion reducing focus and productive interaction.
- The strongest positive factor, view count (coefficient 15.40), is also the most ambiguous, because the paper acknowledges that good answers may attract views rather than views causing answers.
- Platform managers could use these coefficients to build real-time feedback tools that warn users when a draft question has the characteristics of an unanswered post.
Reading between the lines
- If the PD Score is a valid difficulty proxy, the same ratio-based metric could be applied to other JavaScript frameworks such as Vue and Angular to test whether the reputation–difficulty gradient is a general property of Q&A platforms rather than a React-specific pattern.
- The negative coefficient on comments (-0.40) may reflect a selection effect: unclear questions tend to attract clarifying comments before going unanswered, so a survival analysis treating comments as time-varying covariates could disentangle that from the paper's interpretation that comments themselves reduce answerability.
- A natural testable extension is to build a predictive model on the seven attributes and measure its out-of-sample accuracy for answerability, which would show whether the reported associations have practical classification value.
- Because the PD Score uses views in the denominator, the same reverse-causality concern the paper raises for RQ1 (answers attract views) could also distort RQ2, since an unanswered but difficult question might have few views and thus a misleadingly high PD Score.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates factors associated with whether React-related questions on Stack Overflow receive answers, and whether user reputation correlates with question difficulty. Using 534,820 questions filtered by 23 React tags, the authors apply logistic regression to seven attributes (code snippet presence, image presence, reputation, comment count, view count, question length, code length) and report that views, code snippets, code length, and reputation positively associate with answerability, while comments, question length, and images negatively associate. For RQ2, they define a Performance Difficulty Score (PD Score) as (average answer count / average view count) × 100, categorize users into low/mid/high reputation, and report a simple linear regression slope of -0.092, concluding that higher-reputation users ask more complex questions. The paper includes a replication package and is positioned as an empirical study of answerability and difficulty in React-related Stack Overflow posts.
Significance. If the findings are valid, the RQ1 results would provide a useful, large-scale descriptive baseline for answerability of React questions on Stack Overflow, complementing prior work on Python and other topics. The dataset size, explicit tag-selection procedure, and public replication package are strengths that support reproducibility. However, the paper's second contribution depends entirely on an unvalidated PD Score construct and on a regression estimated from only three aggregate reputation categories, so the reputation-complexity conclusion is not currently established. The RQ1 analysis is more plausible but is weakened by applying SMOTE before inferential hypothesis testing and by the acknowledged endogeneity of view counts. The paper's practical implications for question crafting are therefore only as strong as these statistical caveats allow.
major comments (4)
- [§II-B, Eq. (3)] The PD Score is defined as (Average Answer Count / Average View Count) × 100, and lower scores are interpreted as more complex questions, but the paper provides no construct validation for this interpretation. The ratio conflates low visibility, narrow topic appeal, and lack of answers with technical difficulty; a niche but trivial question with few views can receive a low PD Score, while a popular yet conceptually simple question can receive a high score. The only cited basis is prior work [6] by the same group, which is not independent validation. Without an external criterion, human rating, or robustness check, the central RQ2 conclusion that higher-reputation users ask more complex questions is unsupported.
- [§III-B, Table VII and Fig. 9] The simple linear regression for RQ2 is estimated from only three aggregate reputation-category averages (Low, Mid, High in Table VI), and Table VII reports only the slope (-0.092) and intercept (0.561), with no standard errors, confidence intervals, p-values, R², or sample size. With three data points, the regression has one residual degree of freedom and cannot by itself establish a statistically significant correlation. The authors should report a regression on the underlying question-level data or, at minimum, provide full inference statistics for the category-level fit, along with a scatterplot showing the individual category means and uncertainty.
- [§III-A, Table V; §II-B] SMOTE is applied to the training dataset before computing the logistic regression coefficients and p-values reported in Table V. SMOTE is appropriate for improving predictive classification on imbalanced data, but hypothesis tests and confidence intervals based on synthetic resampled observations are not valid inferential statistics for the population of Stack Overflow questions; the reported p-values of 0.00 are therefore not trustworthy as evidence for the stated hypotheses. The paper should either estimate the regression on the original data and use SMOTE only for a separate predictive performance evaluation, or use an inference-preserving method such as weighted regression or exact tests. No model diagnostics (e.g., multicollinearity, separation, goodness-of-fit) are reported for the logistic model.
- [§II-B, Eq. (3); §IV] The Threats to Validity section acknowledges that view count is endogenous for RQ1: higher view counts may result from answered questions rather than cause answers. The same concern directly applies to the PD Score, which divides by average view count. If high-reputation users' questions attract more views because they are more visible or because they receive more attention, their PD Scores will be lower regardless of technical difficulty. The paper does not address this circularity for RQ2, and the negative reputation-PD Score correlation could therefore reflect visibility dynamics rather than question complexity. The authors need to explicitly discuss this and provide evidence that PD Score tracks difficulty independently of visibility.
minor comments (6)
- [§II-A, Table III] The preprocessing step from 583,419 to 534,820 questions is labeled only as 'Duplications removal' in Table III, but the text also mentions removing records with missing essential fields; the table should reflect all preprocessing steps or the text should be reconciled with the table.
- [§II-B, Eq. (1)] The logistic regression formula is written as Logit(1/(1+e^-(α+Σβ_i X_i))) = ln(P(X)/(1-P(X))); this mixes the sigmoid and logit forms. The standard logit form is ln(P/(1-P)) = α + Σβ_i X_i, and the equation should be corrected to avoid confusion.
- [§II-A, Table II] The tag 'konvajs-reactjso' appears likely to be a typo for 'konvajs-reactjs'; please verify the tag string and its frequency count.
- [§III-A, Fig. 5] The text states that code snippet inclusion is 'not significantly different' between answered and unanswered questions, but the subsequent logistic regression reports a significant positive coefficient for code snippet; please clarify whether Fig. 5 refers to a univariate descriptive difference and reconcile the wording with the multivariate result.
- [§III-B, Fig. 8 and Fig. 9] Figure 8 uses 'Average PD Score' on the y-axis, but the caption and text do not specify whether this is the PD Score from Eq. (3) computed per category with answer counts and view counts aggregated at category level; please define exactly how the category-level PD Score was computed.
- [§II-C] The replication package link is mentioned in the text, but the paper does not describe what files or scripts the package contains; a brief description of the repository contents would help reproducibility.
Circularity Check
RQ1 is self-contained; RQ2 relies on a self-cited PD Score proxy, but no equation reduces to its inputs by construction.
full rationale
RQ1 is a standard data-driven logistic regression on 534,820 Stack Overflow posts; coefficients are fitted to extracted attributes and reported as associations, not as predictions derived from first principles. No fitted parameter is renamed as an independent prediction. RQ2's PD Score (Eq. 3) is a transparent ratio of average answer count to average view count, attributed to the authors' prior work [6]; the assertion that lower scores mean more complex questions is an explicit operationalization, not a theorem derived in this paper. The negative reputation-PD Score slope is data-dependent, so the conclusion does not reduce to the definition by construction. The 23-tag selection from [5] is a direct reuse with no hidden equivalence. The main limitations, such as view-count endogeneity acknowledged in Section IV and the lack of external validation for PD Score as a difficulty construct, are validity threats rather than circularity. There is no self-referential derivation chain: no equation is equivalent to its inputs by construction, and no self-citation is used to forbid alternatives. Score 2 reflects the single minor self-citation for the PD Score measure, which is not load-bearing in a circular sense.
Assumptions & free parameters
free parameters (12)
- Logistic regression coefficient, ViewCount =
15.40
- Logistic regression coefficient, CommentCount =
-0.40
- Logistic regression coefficient, question line count =
-0.35
- Logistic regression coefficient, code line count =
0.31
- Logistic regression coefficient, code snippet =
0.38
- Logistic regression coefficient, Reputation =
0.10
- Logistic regression coefficient, image =
-0.06
- Linear regression slope, reputation vs PD Score =
-0.092
- Linear regression intercept =
0.561
- PD Score scaling factor (×100) =
100
- Reputation category thresholds =
2400, 400
- SMOTE oversampling ratio
assumptions (6)
- domain assumption Questions tagged with the selected 23 React tags represent the population of React-related Stack Overflow questions.
- domain assumption A question with at least one answer is 'answered'; answerability is binary.
- ad hoc to paper PD Score = (average answer count / average view count) × 100 is a valid measure of question difficulty, with lower scores indicating more complex questions.
- standard math Logistic regression and linear regression assumptions (e.g., relevant predictors, no severe multicollinearity, residual properties) hold after transformations.
- domain assumption SMOTE-augmented training data preserves the statistical validity of coefficient estimates and p-values.
- domain assumption User reputation scores at data extraction time reflect reputation at time of question posting.
invented entities (1)
-
Performance Difficulty Score (PD Score)
Cite this review
Pith. "Pith review of Exploring React Library Related Questions on Stack Overflow: Answered vs. Unanswered." pith.science (2026). https://pith.science/paper/G6CZGZ3D
@misc{pith2026250704390,
author = {Pith},
title = {Pith review of: Exploring React Library Related Questions on Stack Overflow: Answered vs. Unanswered},
year = {2026},
howpublished = {\url{https://pith.science/paper/G6CZGZ3D}},
note = {Machine review of arXiv:2507.04390}
}
read the original abstract
React is a popular JavaScript framework in modern web application development. Due to its high performance and efficiency, many developers use this framework. Although React library offers many advantages, it is not without its challenges. When using React library, developers often face problems where they often seek solutions through question-and-answer forums, such as Stack Overflow (SO). However, despite its high popularity, many React-related questions on SO remain unanswered. Thus, this study aims to analyze the factors associated with question answerability and difficulty levels of React-related questions on SO. To facilitate our study, Exploratory Data Analysis was applied to 534,820 questions, where they are filtered based on 23 React-related tags. We implemented a quantitative approach through text mining and statistical analysis. A logistic regression model was used to identify attributes associated with question answerability, while a simple linear regression model was employed to examine the correlation between user reputations and performance difficulty scores (PD Score). The results show that some attributes, such as number of views, code snippet inclusion, number of lines of code, and user reputation, positively affect the likelihood of question answerability. In contrast, the number of comments, question lengths, and presence of images in React-related questions reduce the probability of a question receiving responses from users. Further investigation indicates a negative correlation between user reputations and PD Score, where reputation increase corresponds to -0.092 reduction in PD score, signaling experienced users tend to propose more complex technical inquiries. This study provides insights into the characteristics of technical question-and-answer platforms, such as SO, that users need to consider the answerability factors when posting questions related to React.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[6]
What network simulator questions do users ask? a large-scale study of stack overflow posts,
S. Islam, Y . S. Nugroho, and M. J. Hossain, “What network simulator questions do users ask? a large-scale study of stack overflow posts,” Indonesian Journal of Electrical Engineering and Computer Science , vol. 21, no. 3, pp. 1622–1633, 2021
2021
-
[1]
Performance optimization techniques for reactjs,
A. Javeed, “Performance optimization techniques for reactjs,” in 2019 IEEE International Conference on Electrical, Computer and Communi- cation Technologies (ICECCT) . IEEE, 2019, pp. 1–5
work page 2019
-
[2]
S. Chen, U. R. Thaduri, and V . K. R. Ballamudi, ”Front-end development in react: an overview,” Engineering International , vol. 7, no. 2, pp. 117–126, 2019
work page 2019
-
[3]
D. I. Samudio and T. D. LaToza, ”Barriers in front-end web development,” in 2022 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) . IEEE, 2022, pp. 1–11
work page 2022
-
[4]
What makes geeks tick? a study of stack overflow careers,
L. Xu, T. Nian, and L. Cabral, “What makes geeks tick? a study of stack overflow careers,” Management Science, vol. 66, no. 2, pp. 587–604, 2020
work page 2020
-
[5]
A preliminary empirical study of react library related questions shared on stack overflow,
G. T. Kurniaji, Y . S. Nugroho, and S. Islam, “A preliminary empirical study of react library related questions shared on stack overflow,” Computer Science and Information Technologies , vol. 4, no. 1, pp. 14–23, 2023
work page 2023
-
[7]
Y . S. Nugroho, S. A. A. Halim, S. Islam, Y . I. Kurniawan, and T. Erlina, ”An empirical study of unanswered Python-related questions on Stack Overflow,” in 2024 International Conference on Information Technology Research and Innovation (ICITRI) . IEEE, 2024, pp. 230–235
work page 2024
-
[8]
Text-to-SQL in the wild: A naturally-occurring dataset based on Stack Exchange data,
M. Hazoom, V . Malik, and B. Bogin, “Text-to-SQL in the wild: A naturally-occurring dataset based on Stack Exchange data,” arXiv preprint arXiv:2106.05006, 2021
arXiv 2021
Show all 22 references
-
[9]
Solving string constraints with regex-dependent functions through transducers with priorities and variables,
T. Chen, A. Flores-Lamas, M. Hague, Z. Han, D. Hu, S. Kan, A. W. Lin, P. Rummer, and Z. Wu, “Solving string constraints with regex-dependent functions through transducers with priorities and variables,” Proceedings of the ACM on Programming Languages , vol. 6, no. POPL, pp. 1–31, 2022
2022
-
[10]
Answers or no answers: Studying question answerability in Stack Overflow,
A. Y . Chua and S. Banerjee, “Answers or no answers: Studying question answerability in Stack Overflow,” Journal of Information Science , vol. 41, no. 5, pp. 720–731, 2015
2015
-
[11]
Logistic regression was as good as machine learning for predicting major chronic diseases,
S. Nusinovici, Y . C. Tham, M. Y . C. Yan, D. S. W. Ting, J. Li, C. Sabanayagam, T. Y . Wong, and C.-Y . Cheng, “Logistic regression was as good as machine learning for predicting major chronic diseases,” Journal of Clinical Epidemiology , vol. 122, pp. 56–69, 2020
2020
-
[12]
Predictive models in software engineering: Challenges and opportunities,
Y . Yang, X. Xia, D. Lo, T. Bi, J. Grundy, and X. Yang, “Predictive models in software engineering: Challenges and opportunities,” ACM Transactions on Software Engineering and Methodology (TOSEM) , vol. 31, no. 3, pp. 1–72, 2022
2022
-
[13]
The implementation of machine learning for software effort estimation: A literature review,
E. Hariyanti, M. A. Paradista, M. L. J. Goyayi, A. Arthalia, D. A. Shabirina, E. Nurjanah, O. I. Husna, and F. A. S. Yahrani, “The implementation of machine learning for software effort estimation: A literature review,” Khazanah Informatika: Jurnal Ilmu Komputer dan Informatik...
2024
-
[14]
D. G. Kleinbaum, K. Dietz, M. Gail, M. Klein, and M. Klein, Logistic regression. Springer, 2002
2002
-
[15]
Is reputation on Stack Overflow always a good indicator for users’ expertise? no!
S. Wang, D. M. German, T.-H. Chen, Y . Tian, and A. E. Hassan, “Is reputation on Stack Overflow always a good indicator for users’ expertise? no!” in 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 2021, pp. 614–618
2021
-
[16]
Analysis of the reputation system and user contributions on a question answering website: Stackoverflow,
D. Movshovitz-Attias, Y . Movshovitz-Attias, P. Steenkiste, and C. Faloutsos, “Analysis of the reputation system and user contributions on a question answering website: Stackoverflow,” in Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks ...
2013
-
[17]
Linear regression,
T. M. Hope, “Linear regression,” in Machine learning . Elsevier, 2020, pp. 67–81
2020
-
[18]
James, D
G. James, D. Witten, T. Hastie, R. Tibshirani, and J. Taylor, An introduction to statistical learning: With applications in Python . Springer Nature, 2023
2023
-
[19]
A review on linear regression comprehensive in machine learning,
D. Maulud and A. M. Abdulazeez, “A review on linear regression comprehensive in machine learning,” Journal of Applied Science and Technology Trends, vol. 1, no. 2, pp. 140–147, 2020
2020
-
[20]
Combining usability testing with in-depth interview for online credit hour website evaluation,
P. Sukmasetya, E. R. Arumi, and A. Setiawan, “Combining usability testing with in-depth interview for online credit hour website evaluation,” Khazanah Informatika: Jurnal Ilmu Komputer dan Informatika , vol. 6, no. 2, 2020
2020
-
[21]
A comprehensive review of bias reduction methods for logistic regression,
M. Stolte, S. Herbrandt, and U. Ligges, “A comprehensive review of bias reduction methods for logistic regression,” Statistic Surveys, vol. 18, pp. 139–162, 2024
2024
-
[22]
Answering questions about unanswered questions of Stack Overflow,
M. Asaduzzaman, A. S. Mashiyat, C. K. Roy, and K. A. Schneider, “Answering questions about unanswered questions of Stack Overflow,” in 2013 10th Working Conference on Mining Software Repositories (MSR) . IEEE, 2013, pp. 97–100
2013
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.