{"id":"6ed6d8b3-99e1-4557-bc21-2f4b368f7b74","arxiv_id":"2411.10873","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A BERTopic analysis of 8,593 Stack Overflow posts and 26,474 OpenAI Developer Forum posts yields 9 and 17 LLM developer challenge topics, with API usage dominant and high unresolved rates.","lead":"This paper maps the challenges that software developers face when building with large language models by analyzing questions on Stack Overflow and the OpenAI Developer Forum. It finds that API issues are the dominant topic, that many LLM questions remain unanswered, and that emerging areas such as agents and GPU utilization are the hardest to get help with.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Corpus contamination from the unthresholded 'openai' tag admits OpenAI Gym posts, biasing the topic taxonomy and the 79.03% difficulty figure.","rationale":"The reader's weakest assumption is also the most load-bearing threat. The paper's central claim is a taxonomy of LLM developer challenges plus difficulty metrics, and both are computed on a corpus defined by tags. Section 3.1 Step 2 adds 'openai' unconditionally after setting TRT/TST thresholds for other tags. The paper's own results show what this lets in: Table 2's 'Environment Management' topic is built from keywords such as gym, environment, reward, game, and reinforcement, and the discussion explicitly cites OpenAI Gym posts. These are reinforcement-learning questions, not LLM questions. This is not an external disagreement about threshold choice; it is visible in the paper's output and directly contradicts the claim that the nine topics are LLM-related challenges. The contamination affects the denominator for the headline 79.03% unresolved figure, the topic count, and the cross-field comparison. The proposed test is straightforward: remove or strictly filter the 'openai' tag and recompute. I agree with the reader's conditional verdict: the methodology is broadly reasonable but needs this cleanup and robustness check before the quantitative claims are reliable. I do not see grounds to reject the paper outright, because most topics and the overall framing are still likely salvageable, but the current numbers should not be taken at face value.","tokens_in":39375,"tokens_out":3430,"duration_ms":36846,"concrete_test":"Re-run the Section 3.1 pipeline on the Stack Overflow data through 2024-09-14 with a single modification: remove 'openai' from the tag set in Step 2, keeping the TRT/TST-derived tags and all BERTopic settings identical. Then compare (a) the resulting topic list and sizes, (b) the overall percentage without accepted answers (currently 79.03%), and (c) Table 6 topic-level values. As a cross-check, keep 'openai' but filter out posts whose title/body contains 'gym', 'reward', 'reinforcement', or 'openai gym'. If the Environment Management topic disappears or shrinks substantially, or if the 79.03% figure changes by more than 1-2 percentage points, the central claims need to be re-scoped.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3.1 Step 2, the authors add 'openai' to the tag set without applying the TRT/TST thresholds used for every other tag. On Stack Overflow, 'openai' is also used for OpenAI Gym and reinforcement-learning content, and the paper's own Table 2 confirms this: topic 3, 'Environment Management', has keywords 'gym', 'environment', 'reward', 'game', 'reinforcement', and the text explicitly discusses OpenAI Gym posts. These posts are about reinforcement-learning environments, not LLM development. Because the corpus is the input to every downstream analysis, the contamination affects the claimed nine-topic taxonomy (one topic appears to be non-LLM), the overall 79.03% unresolved rate, topic-level difficulty metrics in Table 6, and the cross-field comparison in Section 5.2. The reader's weakest assumption correctly identifies this as the load-bearing threat: if the corpus mixes non-LLM posts, the central descriptive claims lose precision and the difficulty comparisons are not trustworthy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an empirical study of the challenges faced by developers working with large language models (LLMs). The authors collect 8,593 Stack Overflow posts via an LLM-related tag set and 26,474 posts from the OpenAI Developer Forum, apply BERTopic to extract topics (9 on Stack Overflow, 17 on the forum), classify question types, and measure topic difficulty using percentage of posts without accepted answers and median response time (Stack Overflow) and reply counts (forum). They report that LLM questions are often difficult, with 79.03% of Stack Overflow posts unresolved, and compare these metrics with other software engineering fields. The central contribution is the topic taxonomy and the difficulty ranking.","tokens_in":39560,"tokens_out":6073,"duration_ms":54140,"significance":"The study addresses a timely and relevant question and combines two complementary data sources, which is a strength. It builds on established methods from prior MSR studies (TRT/TST tag selection, BERTopic, Cohen's Kappa for the Stack Overflow classification), and the data tables are detailed. If the corpus were clean and the classification validated, the resulting taxonomy would be a useful resource for researchers and practitioners. However, the trustworthiness of the central claims depends on the corpus construction, and the analysis currently has a contamination problem in the Stack Overflow data that directly affects the taxonomy and difficulty figures.","major_comments":[{"comment":"The 'openai' tag is added to the extraction tag set without applying the TRT and TST thresholds used for every other tag. On Stack Overflow, 'openai' also covers OpenAI Gym, a reinforcement-learning environment library, and the paper's own Table 2 topic 3 ('Environment Management') confirms this: its keywords include 'gym', 'reward', 'game', 'reinforcement', and the accompanying text discusses examples such as 'OpenAI Gym' and 'mujoco-py'. These 606 posts are not LLM development posts, yet they are included in the 8,593-post corpus that generates the nine-topic taxonomy, the difficulty metrics in Table 6, the 79.03% unresolved figure in Table 9, and the cross-field comparison in Section 5.2. Because the corpus is the input to every downstream analysis, the authors must either apply the same thresholds to 'openai' or explicitly filter out known non-LLM subtags such as 'openai-gym', and then re-run the analysis to show that the substantive conclusions are unchanged.","section":"3.1 Step 2, Table 2 (topic 3)"},{"comment":"The eight-type classification of OpenAI Developer Forum posts is performed with GPT-4o, but the manuscript reports no validation statistics for this step, in contrast to the Cohen's Kappa of 0.87 reported for the Stack Overflow question-type classification. The text only says that a manual validation process was used. Because Table 5's percentages support the claim that Troubleshooting dominates the OpenAI forum and are compared with the Stack Overflow question types, the authors should report the size of the validation sample and the agreement or accuracy of the GPT-4o labels against human annotation.","section":"4.2, Table 5"},{"comment":"The cross-field comparison relies on 'reconstructed' datasets from prior studies, but the reconstruction procedure is not described (e.g., which tags, date ranges, and filtering were used for each field), and the reported post counts appear inconsistent with the original studies' descriptions (for example, WebApp is listed as 14,596 posts while Mobile is 2,502,811, without explanation). The conclusion that LLM challenges are harder than mobile and security depends on this comparison, so the authors should make the reconstruction explicit and transparent, or soften the comparative claims.","section":"5.2, Table 9"}],"minor_comments":[{"comment":"The tag name is inconsistently written as 'large-language-model' (Section 3.1) and 'large-language-models' (Section 6); the authors should use one consistent name.","section":"3.1 Step 1 and Section 6"},{"comment":"The sentence 'our main topic begins with the label -1' is confusing; the authors should clarify whether the BERTopic outlier topic (-1) is excluded from the nine reported topics and how many posts it contains.","section":"3.1 Step 5"},{"comment":"The paragraph titled 'Function Calling' appears after the 'RQ1 Summary' and seems to be a misplaced portion of the OpenAI forum topic descriptions; it should be moved into the earlier topic list or integrated with Table 3.","section":"4.1"},{"comment":"The sample size for the OpenAI forum classification is not reported; the authors give the total of 1,918 posts for Stack Overflow but only say a representative sample was drawn for the forum, so the reader cannot assess the statistical reliability of Table 5.","section":"4.2"},{"comment":"The keyword list for topic 1 ('LLM Ecosystem and Challenges') contains very generic terms ('use', 'model', 'try', 'error'), and the topic might be a catch-all; the authors should discuss the coherence of this large topic (4,344 posts) in more detail.","section":"Table 2"},{"comment":"The abstract contains a typo: 'Topic Modeling,, Developer Challenges' with a double comma.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"This is a timely study with a clear contribution if the corpus contamination is fixed. The major comments require re-running the analysis, so I recommend major revision. The comparison table (Table 9) also needs more careful reporting to be convincing. The paper fits the scope of a software engineering empirical studies venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent but somewhat rushed empirical study. The genuinely new thing is the combined look at both Stack Overflow and the OpenAI Developer Forum with the same topic-modeling pipeline, and the difficulty comparison across platforms. If the data were clean, that would be a useful map for anyone building LLM developer tooling or documentation.\n\nThe paper does several things right. The methodology follows the established tag-co-occurrence recipe (TRT/TST) from prior MSR work. The manual labeling of topics with three annotators and Cohen's kappa is fine. The comparison to adjacent fields (chatbot, mobile, security) is a good way to contextualize the 79.03% unresolved figure.\n\nThe soft spot is not minor. In Section 3.1, Step 2, the authors add the 'openai' tag to the tag set without applying the TRT/TST thresholds they used for every other tag. On Stack Overflow, 'openai' is also the tag for OpenAI Gym, a reinforcement-learning library. The paper's own Table 2 shows the result: topic 3, 'Environment Management', has keywords 'gym', 'reward', 'action', 'reinforcement', and the text admits it is about OpenAI Gym posts. Those posts are not about LLM development. Because the corpus feeds every downstream analysis, the nine-topic taxonomy, the 79.03% unresolved rate, the topic-level difficulty metrics, and the cross-field comparison in Section 5.2 are all potentially skewed. The paper flags the 'openai' inclusion in its internal validity section, but treats it as a strength rather than a threat, and does not measure how many gym posts got in.\n\nOther soft spots: the TRT/TST thresholds are chosen after manual inspection, which is standard but subjective; the GPT-4o classification of OpenAI forum posts is reported with no validation against a human-labeled gold set; and there are no replication artifacts, which is a missed opportunity for a purely descriptive study.\n\nWho is this for? Researchers and practitioners wanting a rough map of where LLM developers struggle. The rough map is probably mostly right, but the headline numbers are not trustworthy until the contamination is measured and fixed.\n\nRecommendation: this deserves a serious referee, but the verdict should be major revision. Ask for a re-run with the 'openai' tag filtered to LLM-relevant posts, or a sensitivity analysis showing the figures are stable after removing gym posts. Also ask for validation of the GPT-4o labels and a replication package.","headline":"Useful first combined taxonomy of LLM developer challenges, but the unthresholded 'openai' tag lets in OpenAI Gym posts and contaminates the headline numbers; needs a data-cleaning pass before the claims can be trusted.","tokens_in":40032,"tokens_out":2038,"would_cite":false,"duration_ms":21101,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims to derive a nine-topic taxonomy for Stack Overflow and a 17-topic taxonomy for the OpenAI Developer Forum, with LLM questions unresolved 79.03% of the time on Stack Overflow.","keywords":["Large Language Models","Stack Overflow","OpenAI Developer Forum","BERTopic","topic modeling","developer challenges","question difficulty","Q&A communities"],"falsifier":"Take a random sample of the 606 posts in the 'Environment Management' topic (keywords: gym, reward, game) and check how many actually discuss large language models rather than OpenAI Gym reinforcement-learning environments. If a substantial share are non-LLM posts, then the tag-expansion step contaminated the dataset, and the claimed topic list and 79.03% unresolved figure would not be a valid measure of LLM developer challenges.","tokens_in":39215,"feed_emoji":"💬","tokens_out":4655,"duration_ms":43010,"temperature":0.7,"pith_summary":"The paper sets out to map the recurring practical problems developers hit when building with large language models, by mining what they actually ask on Stack Overflow and the OpenAI Developer Forum. It claims the questions cluster into nine Stack Overflow topics and 17 forum topics, from API usage and fine-tuning to agent-tool interactions and GPU indexing. It further claims these questions are measurably harder than established software fields: 79.03% of the Stack Overflow posts have no accepted answer, and the median time to a first answer is 19.88 hours overall. A sympathetic reader would care because the resulting topic list and difficulty ranking suggest where documentation, tools, and community support are missing.","feed_headline":"LLM questions go unresolved 79% of the time on Stack Overflow","feed_subtitle":"Nine Stack Overflow and 17 OpenAI forum topics reveal where LLM developers struggle, from API errors to GPU indexing.","key_machinery":"The load-bearing machinery is a pipeline that turns community posts into labeled challenge topics. The paper expands a seed 'large-language-model' tag using co-occurrence thresholds (TRT and TST), then applies BERTopic with HDBSCAN clustering and a Multi-QA sentence transformer to embed and cluster post texts, then has three annotators label each cluster. Difficulty is measured by the percentage of posts without accepted answers and median response time on Stack Overflow, and by reply-count distribution on the OpenAI forum; the same tag-selection and difficulty recipe is reused across comparison fields.","core_discovery":"The paper's central claim is that LLM development has a recognizable, recurring challenge landscape that can be recovered from community Q&A data. Using BERTopic on 8,593 Stack Overflow posts and 26,474 OpenAI Developer Forum posts, it identifies nine topics on Stack Overflow and 17 on the forum; the most prominent are LLM Ecosystem and Challenges and API Usage and Error Handling, respectively. The paper also claims difficulty is high and uneven: Agents and Tool Interactions has 90.63% of posts without accepted answers, Llama Indexing and GPU Utilization has the longest median response time at 55.76 hours, and overall LLM questions lag mature fields such as mobile and security on resolution speed and acceptance rate.","pith_inferences":["If the dataset were cleaned to exclude OpenAI Gym posts, the Environment Management topic could shrink or disappear, and the remaining percentages might shift.","The forum's question-type distribution was produced with GPT-4o classification plus manual validation; re-running with a different classifier or annotation scheme could yield a different breakdown, especially for Ethical and Best Practices and Maintenance categories.","Because the two platforms use different difficulty metrics (accepted answers and timings versus reply counts), the paper's cross-platform difficulty comparison is indirect; a direct comparison would require applying the same metric to both.","A testable extension would be to track the same topic labels over the next year to see whether resolution rates improve as LLM tooling matures, which would test the claim that difficulty reflects the field's newness."],"forward_implications":["API usage and error handling is the dominant challenge on both platforms, so better API documentation and debugging guides would address the largest share of developer pain.","Specialized topics, especially Agents and Tool Interactions and Llama Indexing and GPU Utilization, will need targeted support because community answers are slowest and rarest there.","LLM questions are harder to resolve than mobile, security, web, and big-data questions, implying the field still lacks the accumulated expertise mature ecosystems have.","The nine and 17 topic taxonomies give educators and tool builders a concrete checklist of where LLM developers get stuck.","The strong negative correlation between a topic's average views and its unresolved rate suggests that drawing attention to a topic may help it get answered, so community visibility tools could reduce difficulty."],"supporting_citations":[{"why":"Supplies the BERTopic neural topic-modeling method used to derive the nine and 17 topic lists.","marker":"[97]"},{"why":"Supplies the TRT/TST tag-selection procedure, question-type classification, and difficulty metrics for chatbot posts that the paper adapts.","marker":"[23]"},{"why":"Supplies the tag-expansion thresholds and popularity/difficulty metrics for mobile developer questions that the paper follows.","marker":"[22]"},{"why":"Supplies difficulty metrics (unresolved posts, median time) for big-data questions used in the cross-field comparison.","marker":"[25]"},{"why":"Supplies the prior taxonomy of LLM developer challenges from OpenAI data that this study extends to two platforms.","marker":"[29]"},{"why":"Supplies the general Stack Overflow topic and trend analysis approach for developer discussions.","marker":"[21]"},{"why":"Supplies security-domain comparison metrics for the cross-field difficulty table.","marker":"[26]"}],"fun_headline_variants":["79% of LLM questions on Stack Overflow stay unanswered","Llama indexing posts take 55 hours median to answer","LLM developers get slower help than mobile or security","API errors top OpenAI forum, many posts unresolved","Only 21% of LLM questions on Stack Overflow get accepted answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The taxonomy and difficulty ranking collapse if the tag-expansion step mixes non-LLM posts into the dataset, as the added 'openai' tag may do by capturing OpenAI Gym reinforcement-learning discussions.","fun_headline_variants_meta":{"raw":{"variants":["79% of LLM questions on Stack Overflow stay unanswered","Llama indexing posts take 55 hours median to answer","LLM developers get slower help than mobile or security","API errors top OpenAI forum, many posts unresolved","Only 21% of LLM questions on Stack Overflow get accepted answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001061,"raw_usage":{"total_tokens":4471,"prompt_tokens":988,"completion_tokens":3483,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":3402}},"tokens_in":604,"tokens_out":3483,"duration_ms":23963,"temperature":1.0,"reasoning_tokens":3402,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:11:48.028960+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 606 posts in the 'Environment Management' topic (keywords: gym, reward, game) and check how many actually discuss large language models rather than OpenAI Gym reinforcement-learning environments. If a substantial share are non-LLM posts, then the tag-expansion step contaminated the dataset, and the claimed topic list and 79.03% unresolved figure would not be a valid measure of LLM developer challenges.","supporting_citations":[],"review_version":1}