{"id":"fd76f7e0-78af-4c3d-8e04-03eb5cd8f9ea","arxiv_id":"2507.04390","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":12,"one_line_summary":"React questions on Stack Overflow are more likely to be answered when they have more views, code snippets, more code lines, and higher-reputation askers, while comment count, length, and images reduce answerability.","lead":"This paper analyzed 534,820 React-related questions on Stack Overflow and used regression models to find which question features make an answer more likely. It reports that views, code snippets, and user reputation help, while long questions and images hurt, and that higher-reputation users ask harder questions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ2's finding rests entirely on an unvalidated PD Score (Eq. 3) that conflates low visibility with high difficulty; without construct validation or view-count controls, the reputation-complexity conclusion is unsupported.","rationale":"The paper has genuine strengths: a large 534,820-question sample, an available replication package, and explicit discussion of several threats to validity. RQ1's answerability associations are broadly consistent with prior Stack Overflow mining work, so the direction of most RQ1 effects is plausible even if the SMOTE-based p-values are not inferentially clean. The RQ2 difficulty analysis is different: it introduces an unvalidated metric, treats it as measuring cognitive complexity rather than attention or visibility, and derives a headline claim about expertise. This is the load-bearing weakness because if PD Score is not measuring difficulty, the entire RQ2 contribution disappears. The reader's weakest-assumption analysis correctly identifies the same construct-validity problem, and the present stress test agrees. The CONDITIONAL verdict remains appropriate: the paper should validate or replace PD Score, report the RQ2 regression with uncertainty and appropriate controls, and clarify whether RQ1 statements are associational rather than causal. No change to the reader's verdict is needed.","tokens_in":7953,"tokens_out":5970,"duration_ms":72485,"concrete_test":"Using the replication package, draw a stratified random sample of 300 React Stack Overflow questions (100 per reputation category) and have at least three annotators, blind to reputation and view counts, rate question complexity on a pre-registered five-point rubric; compute Spearman rho between PD Score and mean complexity, and a partial correlation of reputation with PD Score controlling for log(ViewCount) and tag. If rho is weak (below 0.2) or the partial reputation correlation is no longer negative, the RQ2 conclusion is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RQ2's conclusion that higher-reputation users post more complex questions rests on PD Score, defined in Section II-B Eq. (3) as (Average Answer Count / Average View Count) × 100, with the assertion that lower scores mean more complex questions. The paper offers no construct validation: no independent rubric, human ratings, or criterion measure is provided, and the cited basis [6] is prior work by the same group. Structurally, this ratio rewards popular questions and penalizes low-traffic ones, so niche or poorly discoverable questions can receive low PD Scores regardless of technical depth. The paper itself acknowledges in Threats that view counts are contaminated by endogeneity for RQ1, and that same contamination applies here. The regression reporting is also thin: Table VII gives a slope of -0.092 with no standard error, confidence interval, or sample size, and it appears to summarize only three aggregate reputation-category averages, so the claimed correlation is not statistically established. If PD Score tracks visibility or topic niche rather than difficulty, the RQ2 claim collapses; the RQ1 answerability findings may still hold, but the 'experienced users ask harder questions' contribution becomes unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates factors associated with whether React-related questions on Stack Overflow receive answers, and whether user reputation correlates with question difficulty. Using 534,820 questions filtered by 23 React tags, the authors apply logistic regression to seven attributes (code snippet presence, image presence, reputation, comment count, view count, question length, code length) and report that views, code snippets, code length, and reputation positively associate with answerability, while comments, question length, and images negatively associate. For RQ2, they define a Performance Difficulty Score (PD Score) as (average answer count / average view count) × 100, categorize users into low/mid/high reputation, and report a simple linear regression slope of -0.092, concluding that higher-reputation users ask more complex questions. The paper includes a replication package and is positioned as an empirical study of answerability and difficulty in React-related Stack Overflow posts.","tokens_in":8248,"tokens_out":2961,"duration_ms":33036,"significance":"If the findings are valid, the RQ1 results would provide a useful, large-scale descriptive baseline for answerability of React questions on Stack Overflow, complementing prior work on Python and other topics. The dataset size, explicit tag-selection procedure, and public replication package are strengths that support reproducibility. However, the paper's second contribution depends entirely on an unvalidated PD Score construct and on a regression estimated from only three aggregate reputation categories, so the reputation-complexity conclusion is not currently established. The RQ1 analysis is more plausible but is weakened by applying SMOTE before inferential hypothesis testing and by the acknowledged endogeneity of view counts. The paper's practical implications for question crafting are therefore only as strong as these statistical caveats allow.","major_comments":[{"comment":"The PD Score is defined as (Average Answer Count / Average View Count) × 100, and lower scores are interpreted as more complex questions, but the paper provides no construct validation for this interpretation. The ratio conflates low visibility, narrow topic appeal, and lack of answers with technical difficulty; a niche but trivial question with few views can receive a low PD Score, while a popular yet conceptually simple question can receive a high score. The only cited basis is prior work [6] by the same group, which is not independent validation. Without an external criterion, human rating, or robustness check, the central RQ2 conclusion that higher-reputation users ask more complex questions is unsupported.","section":"§II-B, Eq. (3)"},{"comment":"The simple linear regression for RQ2 is estimated from only three aggregate reputation-category averages (Low, Mid, High in Table VI), and Table VII reports only the slope (-0.092) and intercept (0.561), with no standard errors, confidence intervals, p-values, R², or sample size. With three data points, the regression has one residual degree of freedom and cannot by itself establish a statistically significant correlation. The authors should report a regression on the underlying question-level data or, at minimum, provide full inference statistics for the category-level fit, along with a scatterplot showing the individual category means and uncertainty.","section":"§III-B, Table VII and Fig. 9"},{"comment":"SMOTE is applied to the training dataset before computing the logistic regression coefficients and p-values reported in Table V. SMOTE is appropriate for improving predictive classification on imbalanced data, but hypothesis tests and confidence intervals based on synthetic resampled observations are not valid inferential statistics for the population of Stack Overflow questions; the reported p-values of 0.00 are therefore not trustworthy as evidence for the stated hypotheses. The paper should either estimate the regression on the original data and use SMOTE only for a separate predictive performance evaluation, or use an inference-preserving method such as weighted regression or exact tests. No model diagnostics (e.g., multicollinearity, separation, goodness-of-fit) are reported for the logistic model.","section":"§III-A, Table V; §II-B"},{"comment":"The Threats to Validity section acknowledges that view count is endogenous for RQ1: higher view counts may result from answered questions rather than cause answers. The same concern directly applies to the PD Score, which divides by average view count. If high-reputation users' questions attract more views because they are more visible or because they receive more attention, their PD Scores will be lower regardless of technical difficulty. The paper does not address this circularity for RQ2, and the negative reputation-PD Score correlation could therefore reflect visibility dynamics rather than question complexity. The authors need to explicitly discuss this and provide evidence that PD Score tracks difficulty independently of visibility.","section":"§II-B, Eq. (3); §IV"}],"minor_comments":[{"comment":"The preprocessing step from 583,419 to 534,820 questions is labeled only as 'Duplications removal' in Table III, but the text also mentions removing records with missing essential fields; the table should reflect all preprocessing steps or the text should be reconciled with the table.","section":"§II-A, Table III"},{"comment":"The logistic regression formula is written as Logit(1/(1+e^-(α+Σβ_i X_i))) = ln(P(X)/(1-P(X))); this mixes the sigmoid and logit forms. The standard logit form is ln(P/(1-P)) = α + Σβ_i X_i, and the equation should be corrected to avoid confusion.","section":"§II-B, Eq. (1)"},{"comment":"The tag 'konvajs-reactjso' appears likely to be a typo for 'konvajs-reactjs'; please verify the tag string and its frequency count.","section":"§II-A, Table II"},{"comment":"The text states that code snippet inclusion is 'not significantly different' between answered and unanswered questions, but the subsequent logistic regression reports a significant positive coefficient for code snippet; please clarify whether Fig. 5 refers to a univariate descriptive difference and reconcile the wording with the multivariate result.","section":"§III-A, Fig. 5"},{"comment":"Figure 8 uses 'Average PD Score' on the y-axis, but the caption and text do not specify whether this is the PD Score from Eq. (3) computed per category with answer counts and view counts aggregated at category level; please define exactly how the category-level PD Score was computed.","section":"§III-B, Fig. 8 and Fig. 9"},{"comment":"The replication package link is mentioned in the text, but the paper does not describe what files or scripts the package contains; a brief description of the repository contents would help reproducibility.","section":"§II-C"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a relevant empirical question and provides a useful large dataset, but the RQ2 analysis rests on an unvalidated PD Score and an under-reported three-point regression. If the authors can provide construct validation, full inference statistics, and a clear discussion of the view-count endogeneity, the contribution could become publishable; otherwise RQ2 should be reframed as a descriptive claim about visibility-related performance rather than question difficulty. The RQ1 analysis is more defensible but needs inferentially sound handling of SMOTE and model diagnostics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the React/SO paper. It does what it says: 534k React questions, logistic regression for answerability, linear regression for reputation vs. PD Score. The RQ1 results are in line with previous SO mining work—views, code snippets, code length, and reputation help; comments, long text, and images hurt. The coefficients are plausible and the paper compares them with Python and general SO taxonomies. The replication package is a plus, and the threats section is honest about view-count endogeneity and using current reputation rather than reputation at posting time. That part is a reasonable incremental contribution.\n\nThe soft spot is RQ2. The PD Score is defined as average answer count over average view count, times 100, with low scores meaning 'complex'. That is an unvalidated proxy. A niche or poorly visible question can score low regardless of technical difficulty, and the paper's own view-count endogeneity caveat cuts against it. More importantly, the regression evidence is thin: Table VII gives a slope and intercept, no standard error, no CI, no sample size, no p-value. Figure 9 appears to summarize three reputation categories, so the -0.092 slope is not statistically established. The stress-test note is right to call this out. I don't think the RQ2 conclusion is supportable as stated.\n\nAlso, SMOTE is applied before training; if the p-values in Table V come from the augmented data, they're not valid for inference. The authors should report the model on the original data or use appropriate corrections. The ViewCount coefficient of 15.40 also needs explanation—probably a scaling issue.\n\nWho is this for? Researchers working on Stack Overflow answerability will find the RQ1 replication useful. Practitioners can get generic advice they already know. The paper deserves a serious referee, but it needs revisions: replace or validate the difficulty proxy, report RQ2 properly, and clarify the SMOTE/inference pipeline. I'd lean to conditional accept or major revision, not desk reject.","headline":"RQ1 is a solid incremental replication of known answerability factors on React questions; RQ2's 'experienced users ask harder questions' is not supported because the difficulty proxy is unvalidated and the regression is under-reported.","tokens_in":8802,"tokens_out":2805,"would_cite":false,"duration_ms":29038,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that Stack Overflow React questions are more likely to be answered when they have more views, include code, and come from reputable askers, while comments, long text, and images lower the odds, and that high-reputation…","keywords":["answerability","stack overflow","react","logistic regression","linear regression","user reputation","question difficulty","PD Score"],"falsifier":"Have a panel of experienced React developers blindly rate the difficulty of a random sample of questions, then compute each question's PD Score; if the rank correlation between human-rated difficulty and PD Score is close to zero, the claim that lower PD Scores mean harder questions, and hence the RQ2 conclusion, fails.","tokens_in":7699,"feed_emoji":"📊","tokens_out":10567,"duration_ms":90289,"temperature":0.7,"pith_summary":"This paper analyzes 534,820 React-related questions from Stack Overflow to identify which post attributes are associated with receiving an answer. Using logistic regression, it finds that more views, the presence of code snippets, more lines of code, and higher asker reputation are positively associated with answerability, while more comments, longer question text, and images are negatively associated. The paper then introduces a Performance Difficulty Score (PD Score), the ratio of average answer count to average view count times 100, and reports a negative correlation with user reputation, interpreting it as evidence that high-reputation users ask more complex questions. If these associations survive scrutiny, they give developers practical guidance for writing answerable questions and give the platform a basis for nudging users toward better question formulation.","feed_headline":"Views, code, and reputation predict React answers","feed_subtitle":"A 534,820-question study also links high user reputation to harder, more complex questions.","key_machinery":"The argument runs on two statistical instruments. For RQ1, a logistic regression model predicts a binary answered/unanswered outcome from seven transformed attributes (code snippet flag, image flag, log-transformed reputation, comment count, view count, question line count, and code line count), using square-root transformations to handle multicollinearity and SMOTE on the training set to balance the classes; the regression coefficients serve as the evidence for which attributes help or hurt answerability. For RQ2, the Performance Difficulty Score (PD Score) is computed as the average answer count divided by the average view count, multiplied by 100, and treated as an inverse proxy for question difficulty, with lower scores supposedly indicating harder questions. A simple linear regression of reputation on PD Score yields the -0.092 slope that supports the claim that experienced users ask harder questions. The PD Score is the load-bearing piece: it converts platform activity into a difficulty measure, and the entire RQ2 conclusion depends on that conversion being valid.","core_discovery":"The central claim is that answerability of React-related Stack Overflow questions is systematically associated with a small set of observable post attributes: ViewCount has the largest positive coefficient (15.40), followed by code snippet presence (0.38), code line count (0.31), and asker reputation (0.10), while CommentCount (-0.40), question line count (-0.35), and image presence (-0.06) are negative. The paper further claims that the Performance Difficulty Score (PD Score), defined as average answer count divided by average view count times 100, correlates negatively with user reputation (slope -0.092, intercept 0.561), with low-reputation askers averaging a PD Score of 0.62 versus 0.46 for high-reputation askers, which is interpreted as evidence that more experienced users submit more technically difficult questions. The authors present these as statistical associations from cross-sectional data, not proven causal effects, and they flag explicitly that the direction of the view–answer relationship is uncertain.","pith_inferences":["If the PD Score is a valid difficulty proxy, the same ratio-based metric could be applied to other JavaScript frameworks such as Vue and Angular to test whether the reputation–difficulty gradient is a general property of Q&A platforms rather than a React-specific pattern.","The negative coefficient on comments (-0.40) may reflect a selection effect: unclear questions tend to attract clarifying comments before going unanswered, so a survival analysis treating comments as time-varying covariates could disentangle that from the paper's interpretation that comments themselves reduce answerability.","A natural testable extension is to build a predictive model on the seven attributes and measure its out-of-sample accuracy for answerability, which would show whether the reported associations have practical classification value.","Because the PD Score uses views in the denominator, the same reverse-causality concern the paper raises for RQ1 (answers attract views) could also distort RQ2, since an unanswered but difficult question might have few views and thus a misleadingly high PD Score."],"forward_implications":["Developers writing React questions can raise the model-predicted odds of getting an answer by including a focused code snippet, keeping the text short, and avoiding images.","Questions that accumulate many comments are less likely to be answered, which the paper interprets as excessive discussion reducing focus and productive interaction.","The strongest positive factor, view count (coefficient 15.40), is also the most ambiguous, because the paper acknowledges that good answers may attract views rather than views causing answers.","Platform managers could use these coefficients to build real-time feedback tools that warn users when a draft question has the characteristics of an unanswered post."],"supporting_citations":[{"why":"Supplies the 23-tag filtration and prior React question trend analysis used to extract the dataset.","marker":"[5]"},{"why":"Provides the PD Score formula (average answer count divided by average view count, times 100) used as the difficulty proxy.","marker":"[6]"},{"why":"Supplies the logistic regression equation used to model answerability in RQ1.","marker":"[14]"},{"why":"Provides the reputation score thresholds for low, mid, and high reputation categories in RQ2.","marker":"[16]"},{"why":"Supplies the simple linear regression model used to test the reputation–PD Score correlation.","marker":"[21]"}],"fun_headline_variants":["Views and code predict React answer likelihood","High reputation React users ask harder questions","Comments and images suppress React answers","534k React questions reveal answerability clues","Reputation, views, code: key React answer factors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that high-reputation users ask harder questions depends entirely on the assumption that a question's PD Score, average answers divided by average views times 100, truly measures how difficult or complex the question is.","fun_headline_variants_meta":{"raw":{"variants":["Views and code predict React answer likelihood","High reputation React users ask harder questions","Comments and images suppress React answers","534k React questions reveal answerability clues","Reputation, views, code: key React answer factors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000331,"raw_usage":{"total_tokens":1893,"prompt_tokens":1048,"completion_tokens":845,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":664,"completion_tokens_details":{"reasoning_tokens":780}},"tokens_in":664,"tokens_out":845,"duration_ms":9776,"temperature":1.0,"reasoning_tokens":780,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:48:28.509753+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a panel of experienced React developers blindly rate the difficulty of a random sample of questions, then compute each question's PD Score; if the rank correlation between human-rated difficulty and PD Score is close to zero, the claim that lower PD Scores mean harder questions, and hence the RQ2 conclusion, fails.","supporting_citations":[{"cited_title":"A preliminary empirical study of react library related questions shared on stack overflow,","cited_arxiv_id":null,"evidence_quote":"Supplies the 23-tag filtration and prior React question trend analysis used to extract the dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the logistic regression equation used to model answerability in RQ1."},{"cited_title":"Analysis of the reputation system and user contributions on a question answering website: Stackoverflow,","cited_arxiv_id":null,"evidence_quote":"Provides the reputation score thresholds for low, mid, and high reputation categories in RQ2."},{"cited_title":"A comprehensive review of bias reduction methods for logistic regression,","cited_arxiv_id":null,"evidence_quote":"Supplies the simple linear regression model used to test the reputation–PD Score correlation."}],"review_version":1}