{"id":"becd6b59-bc2a-401a-8cb7-d6a3b3e6dd34","arxiv_id":"2505.04084","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A taxonomy of developer challenges for OpenAI APIs derived from 2,874 Stack Overflow discussions, with popularity trends, difficulty metrics, and implications for vendors and researchers.","lead":"This paper analyzed 2,874 Stack Overflow posts about OpenAI APIs and manually sorted them into nine API categories with topic modeling. It maps the main developer struggles, such as prompt design, token costs, and API deprecation, into a checklist for developers, vendors, and researchers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The most load-bearing concern is that RQ3's challenge taxonomy is derived from post titles only, so the topics in Table 2 may reflect title phrasing rather than the actual challenges in post bodies.","rationale":"The reader's weakest_assumption combined two pipeline assumptions: tag-based sampling plus title-only topic modeling. I agree with the second half and treat it as the most load-bearing because RQ3, which produces the per-category challenge topics and feeds the implications, rests directly on it. The tag-based sampling concern is real but less decisive: the paper restricts its scope to Stack Overflow, acknowledges generalizability limits in Section 7.2, and the manual inspection step with Kappa = 0.832 provides some mitigation. The difficulty-ranking concern is also real, especially for categories with about 40 to 70 posts, but the central claim is broader than the difficulty ranking. The novelty concern about reference [36] is checkable and could require rewording the 'first comprehensive' claim, but it does not invalidate the empirical findings. The title-only choice is the condition least supported by evidence specific to this corpus: the cited representativeness studies predate OpenAI API questions, and no internal validation compares title-derived topics to body-derived topics. A body-based re-run would settle whether the taxonomy is stable or an artifact of title wording. This keeps the reader's CONDITIONAL verdict unchanged: the paper is a plausible and useful descriptive study, but the central challenge taxonomy should be verified against full post content before the findings are taken as comprehensive.","tokens_in":25107,"tokens_out":8624,"duration_ms":90334,"concrete_test":"Re-run the Section 3.5 pipeline on the same 2,874 posts using full post bodies, and separately title+body, with identical preprocessing, K sweep, and coherence selection. Quantify topic stability against Table 2 using top-word Jaccard overlap, topic alignment, and selected K values. If body-based topics add new challenges or materially shift topic labels or coherence, the title-only restriction in Step 1 is not innocuous and RQ3 needs re-analysis; if topics are stable, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.5 Step 1 excludes post bodies and runs topic modeling on titles only, citing [14], [21] for noise reduction and [29], [30] for title representativeness. The transfer of that representativeness claim to OpenAI API posts is not validated: [29] and [30] predate the OpenAI API era, and the paper provides no internal check on whether titles align with bodies for these 2,874 posts. Many Stack Overflow titles for API problems are generic, e.g., 'OpenAI API error' or 'ChatGPT API not working', while the actual challenge (abandoned streaming, token-limit failures, SDK deprecation, CORS, RAG chunking) appears in the body. Since RQ3 and Findings 3 through 11 build the per-category challenge lists and the Section 7 implications on topics derived from titles, a title/body mismatch would change the taxonomy even if the manual labels and Cohen's Kappa values are sound. This is the least secure condition for the central claim that the paper provides a comprehensive map of developer challenges.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents what it claims is the first comprehensive empirical study of developer challenges with OpenAI APIs, based on 2,874 Stack Overflow posts. The authors collect posts using a tag-based filter plus manual inspection, categorize them into nine OpenAI API categories, analyze popularity trends and difficulty metrics, and apply topic modeling to post titles to identify per-category challenges. They then derive implications for developers, LLM vendors, and researchers, and state that the dataset and scripts are available on GitHub. The central contribution is a reproducible map of common OpenAI API difficulties, including prompt engineering, cost management, deprecation, streaming, and third-party integration.","tokens_in":25256,"tokens_out":6178,"duration_ms":60398,"significance":"If the findings hold, this is a useful and timely contribution for the software engineering community. The strengths of the manuscript are the manual labeling effort with high inter-rater agreement (Cohen's Kappa 0.832 for filtering and 0.822 for categorization), the use of concrete Stack Overflow posts to support each topic, and the stated intention to share the dataset and analysis scripts for reproducibility. The paper also connects its taxonomy to actionable implications for different stakeholders. However, the significance of the central 'comprehensive map' claim depends on two testable assumptions that are not yet validated: the representativeness of the tag-based sample and the validity of title-only topic modeling as a proxy for post-body challenges. Because these assumptions are load-bearing for RQ3 and the overall novelty claim, the paper needs additional validation before it can be accepted.","major_comments":[{"comment":"The challenge taxonomy for RQ3 is built from topic modeling of post titles only, with body content excluded during preprocessing. The paper justifies this by citing [14], [21] for noise reduction and [29], [30] for title representativeness, but those studies are not about OpenAI API posts, and the manuscript provides no internal check that the titles of the 2,874 collected posts align with their bodies. Since Findings 3–11 and the Section 7 implications are derived from these topics, a title/body mismatch would change the central map of developer challenges. I recommend adding a validation step, for example manually comparing a random sample of titles with their bodies, or running the same topic modeling on body text as a robustness check, and reporting agreement between title-derived and body-derived topic assignments.","section":"§3.5 Step 1"},{"comment":"The difficulty ranking in Table 1 is presented without confidence intervals or significance tests. The category sizes differ greatly: GPT Actions API has roughly 40 posts (1.4% of 2,874) while Chat API has roughly 1,270 posts, so the difference between 94.9% and 83.0% unresolved questions may not be statistically meaningful. I recommend adding bootstrap confidence intervals for the percentages and median times, and appropriate significance tests such as a chi-square or Fisher exact test for proportions and a permutation test for medians. In addition, the definition of 'median time to receive an accepted answer' needs clarification: the text states that the time is the creation time of the answer rather than the time the answer was marked accepted, which is ambiguous.","section":"§5, Table 1"},{"comment":"The abstract and introduction claim that this is 'the first comprehensive empirical study' of OpenAI API challenges, yet reference [36] (Alam et al., 'Developer challenges on large language models: A study of stack overflow and openai developer forum posts') is a closely related study that is cited in Section 6 but not discussed in Related Work. Reference [10] (Chen et al., 'An empirical study on challenges for LLM application developers') is also directly relevant. The paper should position its contribution explicitly against these works and qualify the 'first' claim to its specific scope, or demonstrate concretely what is novel beyond them.","section":"§1 and §8"},{"comment":"The representativeness of the sample depends on the six selected tags plus manual filtering, but no recall check is reported for OpenAI API posts that do not carry any of these tags, such as posts tagged only gpt, chatgpt, dall-e, or whisper. Since the paper's central claim is a 'comprehensive' map of developer challenges, I recommend quantifying tag coverage, for example by searching a random sample of posts containing known OpenAI-related terms and measuring how many are missed by the chosen tag set.","section":"§3.1 Step 2"}],"minor_comments":[{"comment":"The threshold ranges contain formatting errors, such as '0.15 0.2 0.25', '0 .015', '0 .25', and '0 .25 0 .03'; please fix the notation and ensure all numeric values are properly comma-separated.","section":"§3.1 Step 2"},{"comment":"The 2025 data point is based on 42 posts through January 22, 2025; describing this as 'relatively stable' is too strong for a partial year of data.","section":"§4"},{"comment":"The phrase 'median time to receive an accepted answer' should state explicitly whether the interval is question creation to answer creation, question creation to answer acceptance, or another interval; the current wording is confusing.","section":"§5"},{"comment":"The keywords listed for topic A2 ('token, exceed, llm, modul, attributeerror, quota, valid, paramet') do not appear to match the narrative about model migration and SDK deprecation; please verify the keyword assignment.","section":"§6.1, Table 2"},{"comment":"The labeling process is described as following 'descriptions and definitions of OpenAI APIs,' but the manuscript does not provide the labeling instructions or a codebook; adding the codebook to the online appendix would improve reproducibility.","section":"§3.2"},{"comment":"The phrase 'issues related to threat management' appears to be a typo; the surrounding text discusses permission, configuration, and tool-integration issues rather than threat management.","section":"§6.4, Finding 6"}],"recommendation":"major_revision","confidential_remarks":"The main technical risk is the title-only topic modeling pipeline; the authors should be asked to supply the body-validation check before the RQ3 findings can be considered fully supported. The 'first comprehensive' claim also needs repositioning relative to the closely related works [10] and [36], which are currently cited but not discussed in the related-work comparison. The dataset, manual labeling, and concrete post-level evidence are solid enough that a revision addressing these points could plausibly lead to acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — this is a solid but not path-breaking empirical study. The main contribution is a reproducible map of OpenAI API developer discussions on Stack Overflow: 2,874 posts manually categorized into nine API families with good inter-rater agreement (Kappa 0.83 and 0.82), and a topic model per category with concrete example posts. That is real work and the taxonomy is checkable. If you work on LLM/API developer pain points, this is useful evidence.\n\nThe paper's biggest weakness is its own framing. It claims 'the first comprehensive empirical study' of OpenAI API challenges, yet cites [36], an arXiv study of LLM developer challenges on Stack Overflow and the OpenAI developer forum. The authors never compare their taxonomy or findings with [36] or explain what the new paper adds beyond a narrower vendor-specific lens. That needs fixing.\n\nTwo other soft spots, both moderate. First, the difficulty ranking in Table 1 has no confidence intervals or significance tests, and category sizes range from roughly 1,270 posts (Chat) to about 40 (GPT Actions). Calling GPT Actions 'the most challenging' based on 94.9% unanswered is suggestive but not statistically supported. Second, RQ3 topic models run on post titles only. The stress-test worry that titles may not match bodies is not fully resolved by citing pre-OpenAI papers; however, the authors do manually validate topics by sampling and reading 30 posts per topic, and the challenge descriptions are drawn from post bodies. So the taxonomy is not purely title-driven, but the topic structure could still miss body-only issues. An internal title-body agreement check on this dataset would settle it.\n\nMinor: the GitHub artifact lacks a commit hash and a clear reproduction script; the 'first' claim appears in the abstract and related work without hedging. The implications section is generic (better docs, cost management, deprecation tools) but acceptable for this type of paper.\n\nBottom line: this deserves peer review, not desk reject. A serious editor should send it out with a request for a direct comparison with [36], confidence intervals around Table 1, and an honest positioning of the novelty. I'd bring it to a reading group only if the group cares about SO mining methodology; otherwise it's a useful reference for anyone studying LLM API usability.","headline":"A solid descriptive map of OpenAI API pain points that overclaims novelty against prior work and needs statistical and reproducibility polish before it can be called comprehensive.","tokens_in":25832,"tokens_out":2624,"would_cite":true,"duration_ms":27143,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Analyzing 2,874 Stack Overflow posts, this paper maps the distinctive challenges of OpenAI API development — prompt design, token cost, deprecation, third-party integration — and ranks GPT Actions as the hardest category.","keywords":["Stack Overflow mining","OpenAI API","empirical study","topic modeling","developer challenges","large language models","API usability","community Q&A"],"falsifier":"Re-run the pipeline on the same date range with an expanded tag set (for example, adding `gpt-4`, `whisper`, `dalle`, `function-calling`, or `chatgpt`) and compare the nine-category distribution and the per-category topic lists; a material shift would show that tag selection under-samples the true space of OpenAI API questions. A second check: sample posts whose titles mention OpenAI but carry none of the six tags and see whether they contain challenges the current taxonomy misses.","tokens_in":24863,"feed_emoji":"💬","tokens_out":8488,"duration_ms":76826,"temperature":0.7,"pith_summary":"This paper asks what actually goes wrong when developers build on OpenAI's APIs, and answers with a systematic read of 2,874 Stack Overflow discussions collected through January 2025. It claims to be the first comprehensive empirical study of OpenAI API challenges, classifying posts into nine API categories and using topic modeling to surface the most common problems. The result is a concrete map: prompt engineering, token-based cost control, model and SDK deprecation, context management, and third-party integration dominate developer struggles, with GPT Actions API questions proving the hardest to get answered. If the findings hold, they give LLM vendors a prioritized list of documentation and compatibility gaps and give researchers a starting point for API-misuse and deprecation tools.","feed_headline":"Study maps OpenAI API pain points from 2,874 Stack Overflow posts","feed_subtitle":"Prompt design, token costs, deprecation, and third-party integration dominate the questions developers ask.","key_machinery":"The load-bearing mechanism is a five-step Stack Overflow mining pipeline: tag-based filtering that expands from the seed tag `openai-api` to a six-tag set using significance ($\\alpha$) and relevance ($\\beta$) thresholds; manual inspection of every candidate post with a chance-corrected inter-rater agreement statistic; manual labeling into the nine API categories; and Latent Dirichlet Allocation (a statistical topic model) run on post titles only, with coherence scores to choose the number of topics and open card sorting to name them. The pipeline is what turns 2,874 unstructured forum posts into a reproducible taxonomy of developer challenges. Its two most consequential design choices — title-only preprocessing and the specific tag thresholds $\\alpha=0.1$ and $\\beta=0.01$ — do the work of keeping the taxonomy focused, and they are also the assumptions that the findings rest on.","core_discovery":"The central claim is that the difficulties of working with OpenAI's APIs are distinct from those of traditional APIs and can be systematically catalogued from developer questions. Concretely, the paper identifies nine API categories — Chat, Image Generation, Fine-tuning, Embeddings, Audio, Code Generation, Assistants, GPT Actions, and Others — and shows that the Chat API alone accounts for 44.2% of all posts. Difficulty metrics (the percentage of questions without accepted answers and the median time to receive an accepted answer) rank GPT Actions as the hardest category, with 94.9% of its questions unanswered, while general-purpose APIs such as Assistants, Fine-tuning, and Embeddings turn out harder than specialized ones. Topic modeling yields per-category challenge topics — nine for Chat, seven for Embeddings, six for Audio, and so on — that repeatedly point to prompt design, cost optimization, deprecation and compatibility, and third-party tool integration. The paper concludes with actionable implications for developers, LLM vendors, and researchers.","pith_inferences":["The same pipeline could be applied to other LLM API vendors to test whether this challenge taxonomy is specific to OpenAI or general to LLM APIs; the paper's external-validity discussion names vendor expansion as future work.","Title-only topic modeling is testable: re-running the analysis on post bodies or on title-plus-body and comparing the resulting topics would show whether titles compress developer concerns or distort them.","The difficulty ranking could become a live benchmark: if GPT Actions remains the most-unanswered category after documentation improvements, that would suggest its difficulty is intrinsic to third-party integration, not just a documentation gap.","The attribution of the 2024 decline to community tensions and AI tools could be checked by measuring deleted-post rates and answer-acceptance activity around the OpenAI–Stack Overflow partnership announcement."],"forward_implications":["If the taxonomy is right, documentation and tutorial efforts should target prompt design, context management, and token-cost control before more exotic API features.","LLM vendors need explicit deprecation policies and version pinning, because SDK and model upgrades are a recurring source of developer failures.","GPT Actions API deserves dedicated debugging support: 94.9% of its questions receive no accepted answer, and the median wait for an answer is 583 hours.","Researchers can build API-misuse detectors, deprecated-API detectors, and API recommendation tools that target the patterns in this corpus.","The 2024 decline in posts suggests that AI-assisted coding and community dynamics are shifting where developers get help, which any future Stack Overflow mining study should account for."],"supporting_citations":[{"why":"Supplies the prior study of LLM application developer challenges that the nine-category classification builds on.","marker":"[10]"},{"why":"Provides the difficulty metrics (unanswered-question percentage, median time to answer) and topic-modeling workflow for Stack Overflow mining.","marker":"[14]"},{"why":"Earlier study of machine-learning app development challenges that this paper adapts for popularity and difficulty analysis.","marker":"[18]"},{"why":"Source of the tag-based filtering, manual inspection, and topic modeling methodology for chatbot development questions.","marker":"[21]"},{"why":"Establishes the significance and relevance thresholds used to prune the OpenAI API tag set.","marker":"[22]"},{"why":"Empirical support for the claim that post titles are representative of body content, justifying title-only topic modeling.","marker":"[29]"},{"why":"Provides the coherence score used to choose the number of topics in each category.","marker":"[31]"},{"why":"Open card sort method used to name and merge topics.","marker":"[34]"}],"fun_headline_variants":["OpenAI API pain points mapped from 2,874 Stack Overflow posts","Chat API dominates 44% of OpenAI questions; GPT Actions hardest","Study: OpenAI API challenges differ from traditional APIs","OpenAI API questions on Stack Overflow: 2,874 posts analyzed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire taxonomy and difficulty ranking rest on the assumption that the six selected Stack Overflow tags, plus manual filtering, capture a representative sample of OpenAI API questions, and that topic modeling on post titles alone accurately reflects what developers are actually asking.","fun_headline_variants_meta":{"raw":{"variants":["OpenAI API pain points mapped from 2,874 Stack Overflow posts","Chat API dominates 44% of OpenAI questions; GPT Actions hardest","Study: OpenAI API challenges differ from traditional APIs","OpenAI API questions on Stack Overflow: 2,874 posts analyzed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000954,"raw_usage":{"total_tokens":4051,"prompt_tokens":909,"completion_tokens":3142,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":3069}},"tokens_in":525,"tokens_out":3142,"duration_ms":21774,"temperature":1.0,"reasoning_tokens":3069,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:37:55.303752+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the pipeline on the same date range with an expanded tag set (for example, adding `gpt-4`, `whisper`, `dalle`, `function-calling`, or `chatgpt`) and compare the nine-category distribution and the per-category topic lists; a material shift would show that tag selection under-samples the true space of OpenAI API questions. A second check: sample posts whose titles mention OpenAI but carry none of the six tags and see whether they contain challenges the current taxonomy misses.","supporting_citations":[{"cited_title":"An empirical study on challenges for llm application developers,","cited_arxiv_id":null,"evidence_quote":"Supplies the prior study of LLM application developer challenges that the nine-category classification builds on."},{"cited_title":"Challenges in devel- oping desktop web apps: a study of stack overflow and github,","cited_arxiv_id":null,"evidence_quote":"Provides the difficulty metrics (unanswered-question percentage, median time to answer) and topic-modeling workflow for Stack Overflow mining."},{"cited_title":"Why is developing machine learning applications challenging? a study on stack overflow posts,","cited_arxiv_id":null,"evidence_quote":"Earlier study of machine-learning app development challenges that this paper adapts for popularity and difficulty analysis."},{"cited_title":"Challenges in chatbot development: A study of stack overflow posts,","cited_arxiv_id":null,"evidence_quote":"Source of the tag-based filtering, manual inspection, and topic modeling methodology for chatbot development questions."},{"cited_title":"What do concurrency developers ask about? a large-scale study using stack overflow,","cited_arxiv_id":null,"evidence_quote":"Establishes the significance and relevance thresholds used to prune the OpenAI API tag set."},{"cited_title":"Learning a dual-language vector space for domain-specific cross-lingual question retrieval,","cited_arxiv_id":null,"evidence_quote":"Empirical support for the claim that post titles are representative of body content, justifying title-only topic modeling."},{"cited_title":"Exploring the space of topic coherence measures,","cited_arxiv_id":null,"evidence_quote":"Provides the coherence score used to choose the number of topics in each category."},{"cited_title":"Making sense of card sorting data,","cited_arxiv_id":null,"evidence_quote":"Open card sort method used to name and merge topics."}],"review_version":1}