{"id":"dfade4db-25b3-4f3f-8949-4e7219b5e8df","arxiv_id":"2509.05879","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"After ChatGPT's launch, Stack Overflow posts became longer and shifted toward medium-difficulty questions, suggesting users turn to the crowd for more complex problems.","lead":"Analyzing roughly six million Stack Overflow posts from 2021 to 2023, this paper finds that after ChatGPT's launch, questions, answers, and code examples became longer and questions shifted toward medium difficulty. The authors argue users now go to ChatGPT for simple problems and to the Stack Overflow crowd for harder ones, so the platform is evolving rather than dying.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Causal claim rests on an unmatched one-year-lagged control; user-composition drift from selective attrition could produce the observed increases in length and difficulty even with no within-user behavior change.","rationale":"The reader's weakest assumption is also the most load-bearing one, so I agree. The paper's descriptive contribution is real: the weekly plots show a post-launch upward tilt in length and code length that is visible across tag groups, and the DiD coefficients are consistently positive. The parallel-trend check in Eq. 2 and the sliding-window analysis add some support. However, the control group is a calendar-year-lagged quasi-control, not a matched set of users. Since the treated period lost about 570,000 posts relative to control, composition is not a minor detail: the entire volume decline could be concentrated among short/easy posts, and the paper's outcome means would shift upward without any user changing behavior. The paper itself flags this in Limitations ('no significant distributional drifts in the users, their behavior, and the content they post'), so the concern is not manufactured. A clean panel or matching test using the username field would settle whether the effect is within-user or compositional; the paper does not report such a test. A secondary concern is that the difficulty outcome uses a classifier trained on LeetCode and applied to Stack Overflow without target-domain validation, which could make the difficulty result partly a proxy for length; this strengthens the case for tempering the causal language. The paper deserves a CONDITIONAL verdict, not rejection, because the length findings are visually robust and the main causal limitation is explicitly acknowledged; but the abstract's causal phrasing goes beyond what the design can establish. No code or data are released, so independent replication is not currently possible.","tokens_in":26228,"tokens_out":9360,"duration_ms":96638,"concrete_test":"Re-estimate the DiD using only a balanced panel of users who posted in both the six months before and the six months after ChatGPT, with user fixed effects (and, if feasible, a matched control cohort from 21/22 using the username field in the dump). If the positive coefficients for question length, code length, and medium-difficulty probability shrink to zero or change sign, the headline effect is a composition artifact of selective attrition; if they remain positive and significant, the within-user behavioral interpretation is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim—that ChatGPT caused longer questions, longer code, and more difficult questions—is identified only through a difference-in-differences regression (Eq. 1) that uses the same calendar months one year earlier as the control group. The identification therefore requires the assumption, acknowledged in the Limitations, that there were no distributional drifts in users, their behavior, or posted content between 21/22 and 22/23 apart from the ChatGPT intervention. That assumption is load-bearing because the treated period contains roughly half a million fewer posts than the control period. If users who posted short, easy questions disproportionately left the platform—whether because of ChatGPT or for other reasons—the mean length and the predicted medium-difficulty probability of the remaining posts would rise mechanically, even if every individual user's posting behavior were unchanged. The parallel-trend check (Eq. 2) compares pre-treatment slopes of aggregate weekly means and cannot detect a composition shift that accelerates exactly at the launch. No user-level matching, panel restriction, or placebo test is reported. Consequently, the data support the descriptive pattern but do not establish that ChatGPT changed what individual users ask; the causal statement in the abstract is stronger than the design supports.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes two years of Stack Overflow posts (May 2021 through May 2023) to estimate the effect of the November 2022 ChatGPT launch on the content of user contributions. Using the 21/22 period as a quasi-control, the authors fit difference-in-differences regressions (Eq. 1) on question length, answer length, code length extracted from <code> elements, and a predicted question-difficulty score. The difficulty score is produced by an XGBoost classifier trained on LeetCode tasks using CodeT5 embeddings. The authors report significant positive DiD coefficients for question and answer length, code length, and the probability of medium difficulty, accompanied by a decline in post volume, and interpret the results as evidence that ChatGPT raised the bar for Stack Overflow questions and that users now reserve the crowd for more advanced problems.","tokens_in":26342,"tokens_out":3965,"duration_ms":39267,"significance":"If the causal interpretation holds, the paper makes an important contribution by showing that ChatGPT's impact on Stack Overflow is not merely a decline in quantity but also a qualitative transformation in content. The study's strengths include its large public dataset, transparent regression specifications, confidence intervals for the DiD estimates, parallel-trend checks, and disaggregation by tag group. The main limitation is that the central causal claim rests on a one-year-lagged, unmatched control group, so the evidence is most convincing as a description of platform-level shifts coinciding with ChatGPT. The difficulty measure is also a model output whose validation is performed on a different distribution, which weakens the difficulty-specific conclusion.","major_comments":[{"comment":"The causal claim that ChatGPT caused longer and more difficult questions depends on the assumption, acknowledged in Limitations, that there were no distributional drifts in users or content beyond the intervention. This assumption is load-bearing because Table 1 shows the 22/23 period contains about 570,000 fewer posts than 21/22. If users who posted short, easy questions disproportionately left the platform, average length and the predicted medium-difficulty probability would rise mechanically even if every individual user's behavior were unchanged. The parallel-trend check in Eq. (2) compares aggregate weekly slopes before treatment and cannot detect a composition shift that accelerates exactly at the launch. Please add a user-level panel or fixed-effects specification restricted to users active in both periods, or reweight the control group to match pre-treatment covariates, and report whether the DiD estimates survive. Without such a check, the abstract's 'significant positive effect' is established only at the platform level, not as a within-user behavioral change.","section":"Section 4, Eq. (1); Limitations"},{"comment":"The difficulty outcome is a predicted probability from an XGBoost classifier trained on LeetCode and applied to Stack Overflow questions, with CodeT5 embeddings computed from the title, body, tags, and code examples. Because the classifier input includes exactly the content whose length increases in the treated period, the rise in predicted medium difficulty may partly reflect the same length signal rather than an independent change in difficulty. The reported ROC-AUC of 0.99 and F1 of 0.95 are on a held-out LeetCode test set, not on Stack Overflow data. Please validate the classifier on a human-annotated sample of Stack Overflow questions, or include post length as a control in the difficulty regressions; without this, the difficulty result is not separable from the length result.","section":"Section 3, 'Question and code difficulty'"},{"comment":"The abstract states that ChatGPT had a significant positive effect 'across programming languages,' but Figures 7-9 report results only for the web, python, and java tag groups, and the text reports that smaller tag groups show weaker effects and, for difficulty, non-significant coefficients for prolonged periods. This is an overstatement relative to the evidence presented. Please either report the full per-tag results for all groups or qualify the claim to the largest tag groups.","section":"Abstract and Section 1"}],"minor_comments":[{"comment":"There is a typo: 'Novemeber' should be 'November.'","section":"Section 4"},{"comment":"The in-text citation '(Gelman and and, 2008)' should be '(Gelman and Huang, 2008).'","section":"References"},{"comment":"The vertical axis labels appear to render the coefficient symbol incorrectly (e.g., a bare '3' next to numeric values); please check the LaTeX/vector output.","section":"Figures 6-9"},{"comment":"Typo: 'Retrival Augmented Generation' should be 'Retrieval Augmented Generation.'","section":"Section 5"},{"comment":"There are minor typos: 'statically significant' should be 'statistically significant' and 'coefficent' should be 'coefficient.'","section":"Section 4.1"},{"comment":"The reference 'Santos et al., 2019a' contains 'q8a' in the title; this should be 'Q&A.'","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for cs.CY and the descriptive findings are likely of interest. The key revision needed is to bring the causal language in line with what the design can support, either by adding a composition-robust specification or by softening the causal claim. I do not see a need for new data collection, but the difficulty-classifier validation and tag-level reporting should also be addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline: this is a solid descriptive study with a causal frame that overreaches. The patterns—longer questions, longer code, a bump in predicted medium difficulty—are probably real and worth knowing. But \"ChatGPT caused the shift\" is not established by a one-year-lagged control that is vulnerable to composition drift, and the paper's own Limitations concede the underlying assumption without testing it.\n\nWhat's actually new: prior work (Burtch et al.; del Rio-Chanona et al.) documented volume decline and word-length increases. This paper adds code-length analysis on about six million posts, a LeetCode-trained difficulty classifier, and a BERTopic drift analysis. The code-length result is a genuine addition, and the tag-level analysis (python, java, web) is careful; the sliding-window DiD, taken from Gelman and Huang's \"secret weapon\" approach, gives a nice view of effect dynamics. The parallel-trend check in Eq. 2 is reasonable as far as it goes.\n\nWhere it gets soft: the control group is the same calendar period one year prior, not a matched set of posts or users. The treated period has roughly half a million fewer posts. If the users who left were disproportionately asking short, easy questions, mean length and predicted difficulty rise mechanically. The paper acknowledges this assumption in Limitations but offers no user-level panel, no matching, and no placebo test. The aggregate parallel-trend check cannot detect a composition shift that accelerates at the launch. So read the abstract's causal language as \"we observe an association\" rather than \"ChatGPT caused\". The difficulty outcome also rests on a classifier fit on LeetCode and applied to Stack Overflow without target-domain validation; the 0.99 AUC on LeetCode test data does not tell us how it calibrates on SO. That said, the length findings do not depend on the classifier, so the core empirical contribution survives. Minor: no code release, a couple of citation typos, and some loose phrasing around \"disruptive innovation\".\n\nRecommendation: send it to a serious referee. The data work is substantial and the code-length finding is a real step beyond prior work. The right outcome is probably major revision: soften the causal framing, add robustness checks for composition (user-level panel or at least a placebo using a different pre-period), and validate the difficulty classifier on a sample of human-annotated SO questions. If that happens, this becomes a citeable contribution to the AI-and-online-communities literature.","headline":"The descriptive patterns are real and useful, but the causal claim that ChatGPT caused them is oversold by an unmatched control group.","tokens_in":26984,"tokens_out":1936,"would_cite":false,"duration_ms":18495,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that ChatGPT did not just shrink Stack Overflow; it pushed the platform toward longer, more difficult questions, with the crowd increasingly reserved for advanced programming problems.","keywords":["Stack Overflow","ChatGPT","question difficulty","difference-in-differences","code length","online Q&A communities","help-seeking in programming","generative AI impact"],"falsifier":"A user-level panel study that follows the same active Stack Overflow users through the ChatGPT launch and estimates the same difference-in-differences with user fixed effects would settle it: if question length and difficulty no longer rise once each user serves as their own control, the observed shift is a change in who posts rather than a ChatGPT effect.","tokens_in":25903,"feed_emoji":"💬","tokens_out":8240,"duration_ms":67369,"temperature":0.7,"pith_summary":"This paper tries to establish that ChatGPT did not simply accelerate Stack Overflow's decline; it changed the type of content users post there. Using two years of posts—the treated year containing ChatGPT's launch and the same calendar period one year earlier as control—the authors estimate that ChatGPT increased question length, answer length, code length, and the share of medium-difficulty questions while overall post volume fell. They interpret this as users reserving the crowd for advanced problems and relying on ChatGPT for simpler ones. If right, the platform is transforming into a higher-difficulty forum rather than dying.","feed_headline":"ChatGPT made Stack Overflow questions longer and harder","feed_subtitle":"Fewer posts, but the ones left carry more code and harder problems—the crowd now handles advanced work.","key_machinery":"The argument is carried by a difference-in-differences regression comparing the treated 22/23 year with the same calendar period one year earlier, interacting an after-launch indicator with a treatment indicator while controlling for week of year; a sliding one-month window over the post-launch period produces a daily series of effect estimates rather than a single point estimate. Outcome variables are log-transformed and standardized so coefficients read as percentage-of-standard-deviation changes. Question difficulty is measured by a classifier trained on labeled programming tasks: questions and code are embedded with a pretrained code/text encoder, and a gradient-boosted tree predicts easy, medium, and hard probabilities. A topic model over the same embeddings supplies the content-drift analysis that connects longer, harder posts to shifts toward advanced topics such as object-oriented programming.","core_discovery":"The paper claims that the arrival of ChatGPT caused a measurable qualitative shift in Stack Overflow content, not just a drop in quantity. Six months after launch, question length had increased by about 6 percent of a standard deviation, answer length by about 5 percent, and tag-level effects were larger: python code examples grew by about 21 percent and java question difficulty by about 11 percent. Across languages, the probability of medium-difficulty questions rose at the expense of easy questions, while hard-question probability stayed roughly flat. The authors read this as a division of labor: ChatGPT absorbs simpler, well-known programming questions, while the crowd remains the destination for more advanced and context-heavy problems.","pith_inferences":["If the composition of active users changed after the launch, the observed shift could be an expert-retention effect rather than a behavioral change; a user-fixed-effects panel would separate the two.","A measure of actual problem-solving difficulty may diverge from the classifier's difficulty labels, because the classifier was trained on interview-style tasks; human rating of Stack Overflow posts would test whether the medium-difficulty rise reflects cognitive difficulty or topical drift toward object-oriented programming.","Should the trend continue, Stack Overflow may become a small expert forum whose archives are increasingly valuable training data for hard, long-context programming problems, while beginner content becomes scarce—a feedback loop that could shape future model training."],"forward_implications":["Stack Overflow's remaining role is shifting to advanced problem-solving: fewer posts, but each one longer and more difficult.","Simple and well-known programming questions increasingly go to ChatGPT, so the crowd's comparative advantage lies in complex, context-heavy problems.","Platform operators should expect the difficulty distribution to keep climbing and adapt moderation, search, and expert-retention features accordingly.","As language models improve, the threshold for what users bother to ask the crowd will keep rising, making Stack Overflow a higher-stakes, lower-volume knowledge archive."],"supporting_citations":[{"why":"Supplies the prior finding of an accelerated post-ChatGPT decline in posts that this paper extends to content length and difficulty.","marker":"Burtch et al., 2024"},{"why":"Documents that large language models reduce public knowledge sharing on Q&A platforms, the baseline the paper corroborates and refines.","marker":"del Rio-Chanona et al., 2024"},{"why":"Supplies the pretrained text/code encoder used to embed questions and code for the difficulty classifier.","marker":"Wang et al., 2021"},{"why":"Supplies the gradient-boosted tree algorithm used to train the difficulty classifier.","marker":"Chen and Guestrin, 2016"},{"why":"Supplies the sliding-window regression approach that yields the short- and long-term effect estimates.","marker":"Gelman and Huang, 2008"},{"why":"Shows ChatGPT answers to Stack Overflow questions are frequently incorrect or incomplete, supporting the claim that the crowd is still needed for harder questions.","marker":"Kabir et al., 2024"},{"why":"Shows language models can outperform Stack Overflow on compiler-error questions, supporting the claim that simpler questions migrate away.","marker":"Widjojo and Treude, 2023"},{"why":"Supplies the topic-modeling procedure used to show the content drift toward advanced topics.","marker":"Grootendorst, 2022"}],"fun_headline_variants":["Crowd picks up harder questions as ChatGPT takes easy ones","AI shifts Stack Overflow to advanced questions","Stack Overflow questions get harder after ChatGPT","Fewer posts, harder questions on Stack Overflow","ChatGPT raises the bar for Stack Overflow questions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The causal claim stands on the assumption that, apart from ChatGPT's arrival, the people posting on Stack Overflow and the way they post stayed essentially the same between the treatment year and the control year, so that any difference can be attributed to the launch.","fun_headline_variants_meta":{"raw":{"variants":["Crowd picks up harder questions as ChatGPT takes easy ones","AI shifts Stack Overflow to advanced questions","Stack Overflow questions get harder after ChatGPT","Fewer posts, harder questions on Stack Overflow","ChatGPT raises the bar for Stack Overflow questions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000789,"raw_usage":{"total_tokens":3463,"prompt_tokens":911,"completion_tokens":2552,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":2482}},"tokens_in":527,"tokens_out":2552,"duration_ms":17630,"temperature":1.0,"reasoning_tokens":2482,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:20:30.940822+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A user-level panel study that follows the same active Stack Overflow users through the ChatGPT launch and estimates the same difference-in-differences with user fixed effects would settle it: if question length and difficulty no longer rise once each user serves as their own control, the observed shift is a change in who posts rather than a ChatGPT effect.","supporting_citations":[],"review_version":2}