Pith. sign in

REVIEW 3 major objections 6 minor 65 references

Stack Overflow Is Not Dead Yet: Crowd Answers Still Matter

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that ChatGPT did not just shrink Stack Overflow; it pushed the platform toward longer, more difficult questions, with the crowd increasingly reserved for advanced programming problems.

desk verdict The descriptive patterns are real and useful, but the causal claim that ChatGPT caused them is oversold by an unmatched control group. read the letter →

arxiv 2509.05879 v2 pith:D5VXRPWR submitted 2025-09-07 cs.CY cs.SI

Denis Helic , Tiago Santos This is my paper · ORCID
classification cs.CYcs.SI
keywords StackOverflowChatGPTquestiondifficultydifference-in-differencescodelengthonlineQ&Acommunitieshelp-seekinginprogramminggenerativeAIimpact
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that ChatGPT did not simply accelerate Stack Overflow's decline; it changed the type of content users post there. Using two years of posts—the treated year containing ChatGPT's launch and the same calendar period one year earlier as control—the authors estimate that ChatGPT increased question length, answer length, code length, and the share of medium-difficulty questions while overall post volume fell. They interpret this as users reserving the crowd for advanced problems and relying on ChatGPT for simpler ones. If right, the platform is transforming into a higher-difficulty forum rather than dying.

What carries the argument

The argument is carried by a difference-in-differences regression comparing the treated 22/23 year with the same calendar period one year earlier, interacting an after-launch indicator with a treatment indicator while controlling for week of year; a sliding one-month window over the post-launch period produces a daily series of effect estimates rather than a single point estimate. Outcome variables are log-transformed and standardized so coefficients read as percentage-of-standard-deviation changes. Question difficulty is measured by a classifier trained on labeled programming tasks: questions and code are embedded with a pretrained code/text encoder, and a gradient-boosted tree predicts easy, medium, and hard probabilities. A topic model over the same embeddings supplies the content-drift analysis that connects longer, harder posts to shifts toward advanced topics such as object-oriented programming.

What would settle it

A user-level panel study that follows the same active Stack Overflow users through the ChatGPT launch and estimates the same difference-in-differences with user fixed effects would settle it: if question length and difficulty no longer rise once each user serves as their own control, the observed shift is a change in who posts rather than a ChatGPT effect.

Watch

Extended reading notes

Core claim

The paper claims that the arrival of ChatGPT caused a measurable qualitative shift in Stack Overflow content, not just a drop in quantity. Six months after launch, question length had increased by about 6 percent of a standard deviation, answer length by about 5 percent, and tag-level effects were larger: python code examples grew by about 21 percent and java question difficulty by about 11 percent. Across languages, the probability of medium-difficulty questions rose at the expense of easy questions, while hard-question probability stayed roughly flat. The authors read this as a division of labor: ChatGPT absorbs simpler, well-known programming questions, while the crowd remains the destination for more advanced and context-heavy problems.

Load-bearing premise

The causal claim stands on the assumption that, apart from ChatGPT's arrival, the people posting on Stack Overflow and the way they post stayed essentially the same between the treatment year and the control year, so that any difference can be attributed to the launch.

Editorial extensions

If this is right

  • Stack Overflow's remaining role is shifting to advanced problem-solving: fewer posts, but each one longer and more difficult.
  • Simple and well-known programming questions increasingly go to ChatGPT, so the crowd's comparative advantage lies in complex, context-heavy problems.
  • Platform operators should expect the difficulty distribution to keep climbing and adapt moderation, search, and expert-retention features accordingly.
  • As language models improve, the threshold for what users bother to ask the crowd will keep rising, making Stack Overflow a higher-stakes, lower-volume knowledge archive.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the composition of active users changed after the launch, the observed shift could be an expert-retention effect rather than a behavioral change; a user-fixed-effects panel would separate the two.
  • A measure of actual problem-solving difficulty may diverge from the classifier's difficulty labels, because the classifier was trained on interview-style tasks; human rating of Stack Overflow posts would test whether the medium-difficulty rise reflects cognitive difficulty or topical drift toward object-oriented programming.
  • Should the trend continue, Stack Overflow may become a small expert forum whose archives are increasingly valuable training data for hard, long-context programming problems, while beginner content becomes scarce—a feedback loop that could shape future model training.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper analyzes two years of Stack Overflow posts (May 2021 through May 2023) to estimate the effect of the November 2022 ChatGPT launch on the content of user contributions. Using the 21/22 period as a quasi-control, the authors fit difference-in-differences regressions (Eq. 1) on question length, answer length, code length extracted from <code> elements, and a predicted question-difficulty score. The difficulty score is produced by an XGBoost classifier trained on LeetCode tasks using CodeT5 embeddings. The authors report significant positive DiD coefficients for question and answer length, code length, and the probability of medium difficulty, accompanied by a decline in post volume, and interpret the results as evidence that ChatGPT raised the bar for Stack Overflow questions and that users now reserve the crowd for more advanced problems.

Significance. If the causal interpretation holds, the paper makes an important contribution by showing that ChatGPT's impact on Stack Overflow is not merely a decline in quantity but also a qualitative transformation in content. The study's strengths include its large public dataset, transparent regression specifications, confidence intervals for the DiD estimates, parallel-trend checks, and disaggregation by tag group. The main limitation is that the central causal claim rests on a one-year-lagged, unmatched control group, so the evidence is most convincing as a description of platform-level shifts coinciding with ChatGPT. The difficulty measure is also a model output whose validation is performed on a different distribution, which weakens the difficulty-specific conclusion.

major comments (3)
  1. [Section 4, Eq. (1); Limitations] The causal claim that ChatGPT caused longer and more difficult questions depends on the assumption, acknowledged in Limitations, that there were no distributional drifts in users or content beyond the intervention. This assumption is load-bearing because Table 1 shows the 22/23 period contains about 570,000 fewer posts than 21/22. If users who posted short, easy questions disproportionately left the platform, average length and the predicted medium-difficulty probability would rise mechanically even if every individual user's behavior were unchanged. The parallel-trend check in Eq. (2) compares aggregate weekly slopes before treatment and cannot detect a composition shift that accelerates exactly at the launch. Please add a user-level panel or fixed-effects specification restricted to users active in both periods, or reweight the control group to match pre-treatment covariates, and report whether the DiD estimates survive. Without such a check, the abstract's 'significant positive effect' is established only at the platform level, not as a within-user behavioral change.
  2. [Section 3, 'Question and code difficulty'] The difficulty outcome is a predicted probability from an XGBoost classifier trained on LeetCode and applied to Stack Overflow questions, with CodeT5 embeddings computed from the title, body, tags, and code examples. Because the classifier input includes exactly the content whose length increases in the treated period, the rise in predicted medium difficulty may partly reflect the same length signal rather than an independent change in difficulty. The reported ROC-AUC of 0.99 and F1 of 0.95 are on a held-out LeetCode test set, not on Stack Overflow data. Please validate the classifier on a human-annotated sample of Stack Overflow questions, or include post length as a control in the difficulty regressions; without this, the difficulty result is not separable from the length result.
  3. [Abstract and Section 1] The abstract states that ChatGPT had a significant positive effect 'across programming languages,' but Figures 7-9 report results only for the web, python, and java tag groups, and the text reports that smaller tag groups show weaker effects and, for difficulty, non-significant coefficients for prolonged periods. This is an overstatement relative to the evidence presented. Please either report the full per-tag results for all groups or qualify the claim to the largest tag groups.
minor comments (6)
  1. [Section 4] There is a typo: 'Novemeber' should be 'November.'
  2. [References] The in-text citation '(Gelman and and, 2008)' should be '(Gelman and Huang, 2008).'
  3. [Figures 6-9] The vertical axis labels appear to render the coefficient symbol incorrectly (e.g., a bare '3' next to numeric values); please check the LaTeX/vector output.
  4. [Section 5] Typo: 'Retrival Augmented Generation' should be 'Retrieval Augmented Generation.'
  5. [Section 4.1] There are minor typos: 'statically significant' should be 'statistically significant' and 'coefficent' should be 'coefficient.'
  6. [References] The reference 'Santos et al., 2019a' contains 'q8a' in the title; this should be 'Q&A.'

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central DiD estimates are free parameters fit directly to raw Stack Overflow post data, and the difficulty classifier is trained on external LeetCode labels before being applied to the platform data.

full rationale

The paper's causal claims are empirical difference-in-differences estimates, not derived quantities that collapse into their own inputs. Question length, answer length, and code length are computed directly from parsed post HTML and entered as outcomes in Equation 1; the interaction coefficient beta_3 is estimated from the data and is not constrained to equal any fitted parameter. The difficulty measure is produced by a CodeT5/XGBoost classifier trained on an external LeetCode dataset with easy/medium/hard labels, achieving ROC-AUC 0.99 on a held-out test set, and only then applied to Stack Overflow questions; the classifier output is therefore not fit to the conclusion that medium difficulty increased. The paper's own Limitations explicitly concede the quasi-experimental control group assumes 'no significant distributional drifts in the users, their behavior, and the content they post (apart from the drift caused by the ChatGPT intervention)', which is a genuine identification threat rather than a circularity, because no equation or fitted value forces the outcome to match the treatment indicator. The authors' self-citations (Santos et al. 2019a, 2019b, 2020) concern general community health and badge effects and are background material, not load-bearing premises for the ChatGPT effect. No uniqueness theorem, ansatz-smuggling citation, or renamed-known-result pattern is present. The substantive weakness is causal identification quality, not circular derivation.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim relies on the difference-in-differences identification strategy, the transferability of a LeetCode-trained difficulty classifier, and several measurement choices. The free parameters listed are hand-chosen analysis settings rather than constants fitted to force a result; they affect the magnitude and significance of the reported effects. The axioms are the structural assumptions needed for the causal interpretation. No invented entities are introduced.

free parameters (5)
  • Post-period sliding window length = 1 month
    Each DiD model uses all pre-launch data plus one month of post-launch data; the reported long-term effects come from the sliding window ending in April/May 2023. This choice affects which coefficient is highlighted.
  • Pre-period length = 6 months
    The six months before the launch define the baseline for both groups; a different pre-period could change the parallel-trend check and the DiD estimates.
  • Number of BERTopic topics per period = 8
    The topic drift analysis in Section 4.2 extracts eight topics per period; this number is arbitrary and shapes the comparison of topic shares.
  • XGBoost hyperparameters = library defaults
    The difficulty classifier uses default XGBoost parameters without tuning; other hyperparameters could change predicted difficulty probabilities.
  • CodeT5-base embedding model = 768-dimensional vectors
    The choice of pretrained encoder determines the feature space for difficulty classification; a different encoder would yield different difficulty estimates.
assumptions (6)
  • domain assumption Parallel trends between the 21/22 control and 22/23 treatment periods before ChatGPT
    DiD identification requires that outcomes in the control and treatment groups would have followed parallel trends absent treatment. The paper checks this visually and via Eq. 2, finding small slope differences, but the test covers only the six months before the launch and cannot rule out diverging trends from unmeasured confounders.
  • domain assumption No composition drift in users or content between the two periods beyond the ChatGPT effect
    Explicitly stated in Limitations: 'we make an implicit assumption that there are no significant distributional drifts in the users, their behavior, and the content they post'. If the active user base changed between 2021-22 and 2022-23, length and difficulty shifts could occur without ChatGPT.
  • domain assumption LeetCode difficulty labels transfer to Stack Overflow questions
    The XGBoost classifier is trained on LeetCode easy/medium/hard labels and applied to Stack Overflow (Section 3). No validation on Stack Overflow questions with known difficulty is reported; the 0.99 ROC-AUC is measured on a held-out LeetCode split in the same distribution.
  • domain assumption CodeT5 embeddings represent difficulty-relevant features
    CodeT5-base is pretrained on CodeSearchNet for code understanding, not for difficulty estimation; using its embeddings as the feature space assumes difficulty is reflected in these vectors.
  • domain assumption A linear week-of-year term controls for seasonality
    Equation 1 includes W as a single linear control for the week of the year. If seasonal patterns are nonlinear or group-specific, this could leave residual confounding in the DiD coefficient.
  • domain assumption Line counts from HTML bodies are valid measures of post length and code length
    Question and answer length are computed as the number of lines in the raw HTML body; code length is the number of lines inside <code> elements. Formatting choices affect these counts, and the metric may not equal substantive content length.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Stack Overflow Is Not Dead Yet: Crowd Answers Still Matter." pith.science (2026). https://pith.science/paper/D5VXRPWR

@misc{pith2026250905879,
  author       = {Pith},
  title        = {Pith review of: Stack Overflow Is Not Dead Yet: Crowd Answers Still Matter},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D5VXRPWR}},
  note         = {Machine review of arXiv:2509.05879}
}
read the original abstract

Millions of users visit Stack Overflow regularly to ask community for answers to their programming questions. However, like many other platforms, Stack Overflow consistently struggles with low user retention and declining levels of user contributions to the platform. With the introduction of ChatGPT in November 2022, these ongoing difficulties on Stack Overflow were further magnified, as many users moved toward ChatGPT for programming help. In this paper, we build upon recent research on this phenomenon by analyzing the transformation of user-generated content on Stack Overflow during the post-ChatGPT period. Specifically, we analyze two years of Stack Overflow data and fit multiple causal regression models to estimate the effect of ChatGPT on the length and difficulty of user questions and code examples. We confirm an acceleration of decline in user contributions but find that ChatGPT had a significant positive effect on question and answer length, code length, and question difficulty on Stack Overflow across programming languages. Our results suggest that ChatGPT has effectively raised the bar for questions on Stack Overflow, as users increasingly turn to crowdsourced platforms for help with more complex and challenging problems. With our work we contribute to the ongoing discussion on the impact of tools such as ChatGPT on help-seeking in programming and, more broadly, on collaborative knowledge creation. Our results provide actionable insights for platform operators to support information management and user retention in the aftermath of ChatGPT's launch.

Figures

Figures reproduced from arXiv: 2509.05879 by the authors.

Figure 1
Figure 1. Questions on Stack Overflow. We compare the volume (a), views (b), score (c), and number of lines (d) in questions on Stack Overflow in 22/23 (May 30, 2022 through May 28, 2023) and 21/22 periods (May 31, 2021 through May 29, 2022). The launch of ChatGPT is in the middle of the 22/23 period (November 30, 2022). Stack Overflow already experienced an ongoing downwards trend in the number of questions, question views, … view at source ↗
Figure 2
Figure 2. Answers on Stack Overflow. We compare the volume (a), score (b), and number of lines (c) in answers on Stack Overflow between 22/23 period and 21/22 periods. We observe a similar temporal evolution as in questions (cf [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Tags on Stack Overflow. In (a) we show top 10 tags on Stack Overflow in our two years observation period and compare their volume for periods 22/23 and 21/22. We observe a substantially lower tag volume in the period 22/23 as compared to 21/22, similar to the decreasing question and answer volume. In (b) and (c) we show top 10 tags for both periods individually. While the top seven tags including e.g., python, javas… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Question tags on Stack Overflow. We categorize questions by their tags (python and java) and tag groups (web group includes javascript, reactjs, html, node.js, and css tags) and show top three categories by question volume. In particular, we show web tags in the first,…
Figure 5
Figure 5. Figure 5: Question and code complexity on Stack Overflow. We classify questions by their difficulty as easy, medium, and hard. We first compute embeddings from questions including the code examples by using a pretrained question/code embedding model CodeT5 (Wang et al., 2021). U…
Figure 6
Figure 6. Figure 6: Effect of ChatGPT on the question and answer length over time. We fit multiple DiD regressions using periods 22/23 as the treated group and 21/22 as the control group. We fit our models using all the data prior to the ChatGPT launch (or prior to November 30, 2021 for t…
Figure 7
Figure 7. Figure 7: Effect of ChatGPT on the question length over tags and time. Using the same regression setup as before, we depict the effect of ChatGPT on question length across tag groups. We observe strong positive and lasting effects on the question length for web tags in (a), pyth…
Figure 8
Figure 8. Figure 8: Effect of ChatGPT on the code length over programming language and time. Again, utilizing our DiD setup we estimate the effect of ChatGPT on the code length users post in their code examples on Stack Overflow. Similar to the question length, we observe a strong and con…
Figure 9
Figure 9. Figure 9: Effect of ChatGPT on the question difficulty over programming language and time. With our DiD setup, we estimate the effect of ChatGPT on the question difficulty (here the probability of the medium question difficulty) on Stack Overflow. Similar to the question length …
Figure 10
Figure 10. Figure 10: Content drift on Stack Overflow. We extract eight topics from python questions before the ChatGPT launch and another eight topics after the launch. While the topics remain relatively stable, their relative sizes change. Before ChatGPT launch, the largest category of q…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 18 canonical work pages

  1. [2]

    23 Santos, T., Walk, S., Kern, R., Strohmaier, M., Helic, D., 2019b

    doi:10.1145/3301612. 23 Santos, T., Walk, S., Kern, R., Strohmaier, M., Helic, D., 2019b. Self- and cross-excitation in stack exchange question & answer communities, in: The World Wide Web Conference, ACM. p. 1634–1645. doi:10.1145/3308558. 3313440. Shen, X., Chen, Z., Backes, M., Zhang, Y .,

    work page doi:10.1145/3301612

  2. [3]

    Answering questions about unanswered ques- tions of stack overflow, in: 2013 10th Working Conference on Mining Software Repositories (MSR), pp. 97–100. doi:10.1109/MSR.2013.6624015. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., Sutton, C.,

    arXiv 2013

  3. [9]

    Chen, T., Guestrin, C.,

    Evaluating large language models trained on code.arXiv:2107.03374. Chen, T., Guestrin, C.,

    arXiv

  4. [12]

    Evaluating privacy questions from stack overflow: Can chatgpt compete?, in: 2023 IEEE 31st International Requirements Engineering Conference Workshops (REW), IEEE. pp. 239–244. doi:10.1109/REW57809.2023.00048. Dell’Acqua, F., McFowland III, E., Mollick, E.R., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., Krayer, L., Can- delon, F., Lakhani, K.R.,

    arXiv 2023

  5. [13]

    Stack overflow considered harmful? the impact of copy&paste on android application security, in: 2017 IEEE Symposium on Security and Privacy (SP), IEEE. pp. 121–136. doi:10.1109/SP.2017.31. Gao, C.A., Howard, F.M., Markov, N.S., Dyer, E.C., Ramesh, S., Luo, Y ., Pearson, A.T.,

    doi:10.1109/sp.2017.31 2017

  6. [15]

    doi:10.1145/3641289. Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H.P., Kaplan, J., Edwards, H., Burda, Y ., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F.P., Cummings, ...

    doi:10.1145/3641289

  7. [16]

    García-Peñalvo, F.J.,

    Retrieval-augmented generation for large language models: A survey.arXiv:2312.10997. García-Peñalvo, F.J.,

    arXiv

  8. [17]

    doi:10.14201/eks.31279

    The perception of artificial intelligence in educational contexts after the launch of chatgpt: Disruption or panic? Education in the Knowledge Society 24, e31279. doi:10.14201/eks.31279. 21 Gelman, A., and, Z.H.,

    work page doi:10.14201/eks.31279

Show all 65 references
  1. [19]

    Hang, C.C., Garnsey, E., Ruan, Y .,

    How close is chatgpt to human experts? comparison corpus, evaluation, and detection.arXiv:2301.07597. Hang, C.C., Garnsey, E., Ruan, Y .,

    arXiv

  2. [21]

    Heng, J.J.Y ., Teo, D.B., Tan, L.F.,

    I think this is the most disruptive technology: Exploring sentiments of chatgpt early adopters using twitter data.arXiv:2212.05856. Heng, J.J.Y ., Teo, D.B., Tan, L.F.,

    arXiv

  3. [22]

    Postgraduate Medical Journal 99, 1125–1127

    The impact of chat generative pre-trained transformer (chatgpt) on medical education. Postgraduate Medical Journal 99, 1125–1127. doi:10.1093/postmj/qgad058. Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., Liu, T.,

    doi

  4. [25]

    (Eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), Association for Computational Linguistics, Berlin, Germany

    Summarizing source code using a neural attention model, in: Erk, K., Smith, N.A. (Eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), Association for Computational Linguistics, Berlin, Germany. pp. 2073–2083....

    doi

  5. [26]

    Large language models and simple, stupid bugs, in: 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), IEEE. pp. 563–575. doi:10. 1109/MSR59073.2023.00082. Jimenez, C.E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., Narasimhan, K.,

    2023

  6. [27]

    Kabir, S., Udo-Imeh, D.N., Kou, B., Zhang, T.,

    Swe-bench: Can language models resolve real-world github issues?arXiv:2310.06770. Kabir, S., Udo-Imeh, D.N., Kou, B., Zhang, T.,

    arXiv

  7. [28]

    Is stack overflow obsolete? an empirical study of the charac- teristics of chatgpt answers to stack overflow questions, in: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, ACM. pp. 1–17. doi:10.1145/3613904.3642596. Kou, B., Chen, M., Zhang, T.,

    2024

  8. [29]

    1853–1865

    Automated summarization of stack overflow posts, in: 2023 IEEE/ACM 45th In- ternational Conference on Software Engineering (ICSE), pp. 1853–1865. doi:10.1109/ICSE48619.2023.00158. Kumar, A.N., Raj, R.K., Aly, S.G., Anderson, M.D., Becker, B.A., Blumenthal, R.L., Eaton, E., Eps...

    2023

  9. [30]

    doi:10.1145/3664191

    ACM. doi:10.1145/3664191. Liu, J., Xia, C.S., Wang, Y ., ZHANG, L.,

    doi

  10. [31]

    (Eds.), Advances in Neural Information Processing Systems, Curran Asso- ciates, Inc

    Is your code generated by chatgpt really correct? rig- orous evaluation of large language models for code generation, in: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (Eds.), Advances in Neural Information Processing Systems, Curran Asso- ciates, Inc.....

    2023

  11. [32]

    22 Mamykina, L., Manoim, B., Mittal, M., Hripcsak, G., Hartmann, B.,

    Wizardcoder: Empowering code large language models with evol-instruct.arXiv:2306.08568. 22 Mamykina, L., Manoim, B., Mittal, M., Hripcsak, G., Hartmann, B.,

    arXiv

  12. [34]

    cited by:

    doi:10.1111/j.1540-5885.2005.00177.x. cited by:

    2005

  13. [36]

    doi:https://doi.org/ 10.1016/j.jss.2023.111734

    Github copilot ai pair programmer: Asset or liability? Journal of Systems and Software 203, 111734. doi:https://doi.org/ 10.1016/j.jss.2023.111734. Nagy, D., Schuessler, J., Dubinsky, A.,

    2023

  14. [37]

    Industrial Marketing Management 57, 119–126

    Defining and identifying disruptive innovations. Industrial Marketing Management 57, 119–126. doi:https://doi.org/10.1016/j.indmarman.2015.11.017. Naveed, H., Khan, A.U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., Mian, A.,

    2015 doi

  15. [38]

    Noy, S., Zhang, W.,

    doi:10.1145/3744746. Noy, S., Zhang, W.,

    doi

  16. [39]

    Science 381, 187–192

    Experimental evidence on the productivity effects of generative artificial intelligence. Science 381, 187–192. doi:10.1126/science.adh2586. Peng, S., Kalliamvakou, E., Cihon, P., Demirer, M.,

    doi

  17. [40]

    Perry, N., Srivastava, M., Kumar, D., Boneh, D.,

    The impact of ai on developer productivity: Evidence from github copilot.arXiv:2302.06590. Perry, N., Srivastava, M., Kumar, D., Boneh, D.,

    arXiv

  18. [41]

    Do users write more insecure code with ai assistants?, in: Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, ACM. p. 2785–2799. doi:10.1145/3576915.3623157. Rahman, M.M., Barson, J., Paul, S., Kayani, J., Lois, F.A., Quezada, S.F., Parnin, ...

    2023

  19. [43]

    Husain, H., Wu, H.H., Gazit, T., Allamanis, M., Brockschmidt, M.,

    doi:10.1145/3703155. Husain, H., Wu, H.H., Gazit, T., Allamanis, M., Brockschmidt, M.,

    doi

  20. [44]

    arXiv:2405.10205

    Exploring the impact of chatgpt on wikipedia engagement. arXiv:2405.10205. del Rio-Chanona, R.M., Laurentsyeva, N., Wachs, J.,

    arXiv

  21. [45]

    PNAS Nexus 3, pgae400

    Large language models reduce public knowledge sharing on online q&a platforms. PNAS Nexus 3, pgae400. doi:10.1093/pnasnexus/pgae400. Ross, S.I., Martinez, F., Houde, S., Muller, M., Weisz, J.D.,

    doi

  22. [46]

    The programmer’s assistant: Conversational inter- action with a large language model for software development, in: Proceedings of the 28th International Conference on Intelligent User Interfaces, ACM. p. 491–514. doi:10.1145/3581641.3584037. Sandoval, G., Pearce, H., Nys, T., ...

  23. [47]

    doi:10.1609/icwsm

    Can badges foster a more welcoming culture on q&a boards? Proceedings of the International AAAI Conference on Web and Social Media 14, 969–973. doi:10.1609/icwsm. v14i1.7368. Santos, T., Walk, S., Kern, R., Strohmaier, M., Helic, D., 2019a. Activity archetypes in question-and-...

    doi

  24. [48]

    Shojaee, P., Mirzadeh, I., Alizadeh, K., Horton, M., Bengio, S., Farajtabar, M.,

    In chatgpt we trust? measuring and characterizing the reliability of chatgpt.arXiv:2304.08979. Shojaee, P., Mirzadeh, I., Alizadeh, K., Horton, M., Bengio, S., Farajtabar, M.,

    arXiv

  25. [50]

    Journal of Engineering and Technology Management 56, 101568

    A literature review of disruptive innovation: What it is, how it works and where it goes. Journal of Engineering and Technology Management 56, 101568. doi:https://doi.org/10.1016/j.jengtecman. 2020.101568. Siddiq, M.L., Majumder, S.H., Mim, M.R., Jajodia, S., Santos, J.C.S.,

    2020

  26. [51]

    An empirical study of code smells in transformer-based code generation techniques, in: 2022 IEEE 22nd International Working Conference on Source Code Analysis and Manipulation (SCAM), IEEE. pp. 71–82. doi:10.1109/SCAM55253.2022.00014. de Souza, L.B.L., Campos, E.C., Maia, M.d.A.,

    2022

  27. [54]

    Moradi Dakhel, A., Majdinasab, V ., Nikanjam, A., Khomh, F., Desmarais, M.C., Jiang, Z.M.J.,

    doi:10.1145/3457607. Moradi Dakhel, A., Majdinasab, V ., Nikanjam, A., Khomh, F., Desmarais, M.C., Jiang, Z.M.J.,

    doi

  28. [55]

    How do programmers ask and answer questions on the web? (nier track), in: Proceedings of the 33rd International Conference on Software Engineering, ACM. p. 804–807. doi:10. 1145/1985793.1985907. Vaithilingam, P., Zhang, T., Glassman, E.L.,

  29. [56]

    Expectation vs experience: Evaluating the usability of code generation tools powered by large language models, in: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, ACM. pp. 1–7. doi:10.1145/3491101.3519665. Valmeekam, K., Olmo, A., Sreedhara...

    2022

  30. [57]

    URL:https://openreview.net/forum?id=wUU-7XTL5XO

    Large language models still can’t plan (a bench- mark for llms on planning and reasoning about change), in: NeurIPS 2022 Foundation Models for Decision Making Workshop. URL:https://openreview.net/forum?id=wUU-7XTL5XO. Varghese, J., Chapiro, J.,

    2022

  31. [58]

    Journal of Hepatology 80, 977–980

    Chatgpt: The transformative influence of generative ai on science and healthcare. Journal of Hepatology 80, 977–980. doi:https://doi.org/10.1016/j.jhep.2023.07.028. Vasilescu, B., Filkov, V ., Serebrenik, A.,

    2023 doi

  32. [59]

    Stackoverflow and github: Associations between software development and crowdsourced knowledge, in: 2013 International Conference on Social Computing, pp. 188–195. doi:10. 1109/SocialCom.2013.35. Wang, Y ., Wang, W., Joty, S., Hoi, S.C.,

    2013

  33. [60]

    CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation, in: Moens, M.F., Huang, X., Specia, L., Yih, S.W.t. (Eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Association for Comput...

    2021 doi

  34. [61]

    (Eds.), Advances in Neural Information Processing Systems, Curran Asso- ciates, Inc

    Chain- of-thought prompting elicits reasoning in large language models, in: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A. (Eds.), Advances in Neural Information Processing Systems, Curran Asso- ciates, Inc.. pp. 24824–24837. URL:https://proceedings.neurip...

    2022

  35. [62]

    Xu, F.F., Alon, U., Neubig, G., Hellendoorn, V .J.,

    Addressing compiler errors: Stack overflow or large language models? arXiv:2307.10793. Xu, F.F., Alon, U., Neubig, G., Hellendoorn, V .J.,

    arXiv

  36. [63]

    A systematic evaluation of large language models of code, in: Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, ACM. p. 1–10. doi:10.1145/3520312.3534862. Yang, D., Hussain, A., Lopes, C.V .,

  37. [64]

    From query to usable code: an analysis of stack overflow code snippets, in: Proceedings of the 13th International Conference on Mining Software Repositories, ACM. p. 391–402. doi:10. 1145/2901739.2901767. Ye, J., Chen, X., Xu, N., Zu, C., Shao, Z., Liu, S., Cui, Y ., Zhou, Z.,...

  38. [65]

    arXiv:2303.10420

    A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv:2303.10420. Yetistiren, B., Ozsoy, I., Tuzun, E.,

    arXiv

  39. [66]

    Assessing the quality of github copilot’s code generation, in: Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering, ACM. p. 62–71. doi:10.1145/3558489.3559072. Yin, P., Deng, B., Chen, E., Vasilescu, B., Neubig, G.,

  40. [67]

    Learning to mine aligned code and natural language pairs from stack overflow, in: Proceedings of the 15th International Conference on Mining Software Repositories, ACM. p. 476–486. doi:10.1145/3196398.3196408. Zhong, L., Wang, Z.,

  41. [75]

    Gao, Y ., Xiong, Y ., Gao, X., Jia, K., Pan, J., Bi, Y ., Dai, Y ., Sun, J., Wang, M., Wang, H.,

    doi:10.1038/s41746-023-00819-6. Gao, Y ., Xiong, Y ., Gao, X., Jia, K., Pan, J., Bi, Y ., Dai, Y ., Sun, J., Wang, M., Wang, H.,

    doi

  42. [93]

    Haque, M.U., Dharmadasa, I., Sworna, Z.T., Rajapakse, R.N., Ahmad, H.,

    doi:10.1016/ j.technovation.2014.11.005. Haque, M.U., Dharmadasa, I., Sworna, Z.T., Rajapakse, R.N., Ahmad, H.,

    2014

  43. [1972]

    Journal of the American Statistical Association 67, 364–366

    On simpson’s paradox and the sure-thing principle. Journal of the American Statistical Association 67, 364–366. doi:10.1080/01621459.1972.10482387. Burtch, G., Lee, D., Chen, Z.,

    1972

  44. [2008]

    Journal of the American Statistical Association 103, 437–446

    Estimating incumbency advantage and its variation, as an example of a before–after study. Journal of the American Statistical Association 103, 437–446. doi:10.1198/016214507000000626. Grootendorst, M.,

    doi

  45. [2010]

    The impact of social media on software engineering practices and tools, in: Proceedings of the FSE/SDP Workshop on Future of Software Engineering Research, ACM. p. 359–364. doi:10.1145/1882362.1882435. Teubner, T., Flath, C.M., Weinhardt, C., Van Der Aalst, W., Hinz, O.,

  46. [2011]

    Design lessons from the fastest q&a site in the west, in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM. p. 2857–2866. doi:10.1145/1978942.1979366. Markides, C.,

  47. [2013]

    Why, when, and what: Analyzing stack overflow questions by topic, type, and code, in: 2013 10th Working Conference on Mining Software Repositories (MSR), pp. 53–56. doi:10.1109/MSR.2013. 6624004. Asaduzzaman, M., Mashiyat, A.S., Roy, C.K., Schneider, K.A.,

    2013 doi

  48. [2014]

    Ranking crowd knowledge to assist software development, in: Proceedings of the 22nd International Conference on Program Comprehension, ACM. p. 72–82. doi:10.1145/ 2597008.2597146. Storey, M.A., Treude, C., van Deursen, A., Cheng, L.T.,

  49. [2016]

    Xgboost: A scalable tree boosting system, in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM. p. 785–794. doi:10.1145/2939672. 2939785. Chow, J.C., Sanders, L., Li, K.,

    doi

  50. [2017]

    IEEE Software 34, 53–60

    What do developers use the crowd for? a study using stack overflow. IEEE Software 34, 53–60. doi:10.1109/MS.2017.31. Allamanis, M., Sutton, C.,

    2017 doi

  51. [2018]

    Information and Software Technology 94, 186–207

    How to ask for technical help? evidence-based guidelines for writing questions on stack overflow. Information and Software Technology 94, 186–207. doi:10.1016/j.infsof.2017. 10.009. Chang, Y ., Wang, X., Wang, J., Wu, Y ., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y...

    2017 doi

  52. [2020]

    Iyer, S., Konstas, I., Cheung, A., Zettlemoyer, L.,

    Codesearchnet challenge: Evaluating the state of semantic code search.arXiv:1909.09436. Iyer, S., Konstas, I., Cheung, A., Zettlemoyer, L.,

    1909 arXiv

  53. [2021]

    20 Blyth, C.R.,

    Program synthesis with large language models.arXiv:2108.07732. 20 Blyth, C.R.,

    arXiv

  54. [2022]

    Guo, B., Zhang, X., Wang, Z., Jiang, M., Nie, J., Ding, Y ., Yue, J., Wu, Y .,

    Bertopic: Neural topic modeling with a class-based tf-idf procedure.arXiv:2203.05794. Guo, B., Zhang, X., Wang, Z., Jiang, M., Nie, J., Ding, Y ., Yue, J., Wu, Y .,

    arXiv

  55. [2023]

    Frontiers in Artificial Intelligence 6, 1166014

    Impact of chatgpt on medical chatbots as a disruptive technology. Frontiers in Artificial Intelligence 6, 1166014. doi:10.3389/frai.2023.1166014. Christensen, C., Raynor, M.,

    2023

  56. [2024]

    Scientific Reports 14, 10413

    The consequences of generative ai for online knowledge communities. Scientific Reports 14, 10413. doi:10.1038/s41598-024-61221-0. Calefato, F., Lanubile, F., Novielli, N.,

    doi

  57. [2025]

    Si, S., Chen, H.,

    The illusion of thinking: Under- standing the strengths and limitations of reasoning models via the lens of problem complexity.arXiv:2506.06941. Si, S., Chen, H.,

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.