{"id":"ca04fbc4-429e-4166-bf7a-b9f8d44fad3b","arxiv_id":"2506.13114","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An empirical study of 38,742 issue reports and 19 interviews produces a 20-theme, 75-sub-theme taxonomy of LLM-centric framework challenges and five recommendations.","lead":"Researchers analyzed 38,742 issue reports from PyTorch, TensorFlow, MindSpore, and eight LLM toolkits, plus interviews with practitioners, to map where deep learning frameworks fail LLM users. The result is a taxonomy of bugs, questions, and feature requests that highlights fragile setup, silent training failures, and weak observability as the biggest pain points.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Absence of a non-LLM baseline makes the 'LLM-centric' qualifier unsupported; generic keywords and toolkit-dominated data could yield the same taxonomy for general DL issues.","rationale":"Reader's weakest assumption is about representativeness of keyword filtering; I sharpen this to a construct-validity problem: 'LLM-centric' is an attribute of the sample, not just of the filter. The paper's own novelty argument (Section 8) depends on a qualitative difference from small-model issues, yet no non-LLM control group is studied. This matters because the headline findings, e.g., Finding 1's installation confusion at 27.31% and Finding 3's compatibility bugs at 24.38%, are presented as LLM-specific pain points. If identical distributions hold for general DL issues, the taxonomy may be valid as a DL-framework taxonomy but not as an LLM-centric one, which would undercut the paper's main contribution. I did not choose the 39.29% arithmetic error (C.6-C.10 sums to 34.25%) as the load-bearing issue because correcting it does not change the qualitative claim that observability/maintainability bugs are the largest combined cluster. The interview-related naming inconsistencies are also real but secondary. A controlled baseline comparison is cheap and would settle the concern. The reader's CONDITIONAL verdict remains appropriate; I would keep it, with the baseline comparison as an explicit condition for acceptance.","tokens_in":42143,"tokens_out":9291,"duration_ms":89842,"concrete_test":"Use the released data at [80] to partition the 38,742 reports into (a) issues matched only by generic terms (e.g., oom, quantization, bfloat16, cuda out of memory, torch.distributed) and (b) issues matched by LLM-specific terms (e.g., llama, gpt, qwen, megatron, vllm), and also draw a random non-LLM issue sample from the same repositories. Recompute the Figure 3 theme percentages for each subset and test whether the distributions are statistically distinguishable (e.g., chi-square or bootstrap CI on top themes). If the distributions match within confidence intervals, the 'LLM-centric' claim fails; if they differ, the taxonomy is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support the central claim that the taxonomy captures LLM-centric challenges, Section 4.1 must isolate issues that are specific to LLM workloads. The keyword filter includes generic terms such as 'oom', 'cuda out of memory', 'quantization', 'bfloat16', 'tf.function', and 'graph mode', which also match non-LLM workflows. Furthermore, the dataset is dominated by LLM toolkits: vLLM alone contributes 13,546 of 38,742 reports (~35%), and all eight toolkits together contribute ~75%, while the three core frameworks contribute only ~25%. Section 8 (Table 3) asserts that LLM issues differ 'qualitatively in frequency, impact, and diagnosability' from small-model issues, but no non-LLM issue sample is analyzed. The reported 92.9% precision (Section 4.1) only means that a spot-checked 10% of retained issues were judged LLM-related by annotators; it does not show that these issues are LLM-specific or that the observed theme frequencies (e.g., 27.31% installation questions, 24.38% compatibility bugs) would not appear in a matched sample of general DL issues. Without a baseline, the five key challenges and the 'LLM-centric' framing are not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an empirical study of challenges that DL frameworks face in supporting large language models. The authors collect and analyze 38,742 issue reports from PyTorch, TensorFlow, MindSpore, and eight associated LLM toolkits (e.g., vLLM, DeepSpeed, Megatron), manually label them, and construct a three-level taxonomy with three issue types (questions, requirements, bugs), 20 themes, and 75 sub-themes. They then interview 11 LLM users and 8 DL framework developers to validate and refine the taxonomy, and derive five key findings and five optimization recommendations. The paper claims that the taxonomy captures a comprehensive set of LLM-centric challenges and that five key challenges, especially fragile environment setup and silent execution instability, limit DL framework support for LLMs.","tokens_in":42371,"tokens_out":10010,"duration_ms":84558,"significance":"If the central claims hold, the paper provides a useful, large-scale empirical map of LLM-related issues in DL frameworks, with practical implications for framework developers, testing researchers, and tool designers. The study's strengths include a large annotated dataset, explicit inter-annotator agreement (Fleiss' kappa 0.717-0.757), a described two-stage precision-checking procedure, and a publicly promised data repository. However, the 'LLM-centric' qualifier is not yet established because no non-LLM baseline is analyzed, and several reported summary percentages are miscomputed. These issues affect the paper's main quantitative claims but are addressable within the manuscript's scope.","major_comments":[{"comment":"The central qualifier 'LLM-centric' is not supported by the evidence because the analysis lacks a non-LLM baseline. The keyword filter includes generic terms such as 'oom', 'cuda out of memory', 'quantization', 'bfloat16', 'tf.function', and 'graph mode', which also match small-scale or non-LLM deep learning workflows, and the dataset is dominated by LLM-specific toolkits: vLLM alone contributes roughly 35% of the 38,742 reports, and the eight toolkits together contribute about 75%, while the three core frameworks contribute only about 25%. The reported 92.9% precision in Section 4.1 only indicates that a spot-checked 10% of retained issues were judged LLM-related by annotators; it does not show that these issues are LLM-specific or that the observed theme frequencies would differ in a matched sample of general DL issues. The assertion in Section 8 (Table 3) that LLM issues differ 'qualitatively in frequency, impact, and diagnosability' from small-model issues is asserted without a comparative sample. Without a non-LLM baseline, the taxonomy's theme frequencies and the five key challenges cannot be uniquely attributed to LLM workloads.","section":"Section 4.1 and Section 8 (Table 3)"},{"comment":"Several quantitative claims about bug theme percentages are incorrect. The summary states that 'observability and maintainability weaknesses (39.29%, C.6–C.10) dominate at scale', but the per-theme percentages in the same section sum to 34.25% (C.6 4.01% + C.7 6.47% + C.8 1.14% + C.9 19.72% + C.10 2.91%). This error also appears in Finding 3 and in Section 6.1's reference to '39.29% of bugs (C.6–C.10)'. Additionally, C.1 is reported as 18.21% of bug-related issues in the theme description in Section 4.4.3 and in the Bug Summary, but Figure 3 reports 6,268 reports for C.1, which is 24.38% of the 25,709 bugs; the 18.21% figure is actually the percentage for requirement theme B.5. These quantitative errors are load-bearing for the ranking claims in Finding 3 and for the derivation of the observability/maintainability challenge.","section":"Section 4.4.3, Bug Summary and Finding 3"},{"comment":"The derivation of the 'Environment Setup Fragility' challenge misstates Finding 1. The text says that 'Finding 1 identifies installation and runtime errors (27.31%, A.1) as the most frequently reported user questions', but Section 4.4.1 reports that A.3 (Unexpected Behavior and Unintuitive Design) is the largest question theme at 30.07%, with A.1 tied with A.4 at 27.31%. This contradiction undermines the evidential basis for the paper's first key challenge and should be corrected for internal consistency.","section":"Section 6.1, first key challenge"}],"minor_comments":[{"comment":"The toolkit name is misspelled as 'DeepSeed'; it should be 'DeepSpeed'.","section":"Table 1"},{"comment":"The sentence 'This group includes eight LLM users from leading Internet companies' should read 'eight DL framework developers', because the paragraph is describing the developer interviewee group.","section":"Section 5.1"},{"comment":"The word 'captures' appears as a stray token in the B.5 sub-theme list, and should be removed.","section":"Figure 2"},{"comment":"The text references 'Fig. ??' when discussing detailed differences between questions, requirements, and bugs; this unresolved reference should be fixed.","section":"Section 4.4.4"},{"comment":"The interview section describes the newly added sub-themes as B.1.iv 'Multi-platform Interoperability' and B.4.iii 'Example Drift and Documentation Mismatch', but the final taxonomy in Figure 2 and Section 4.4.2 names them 'Local Development and Lightweight Setup Support' and 'Minimal Working Examples'; the paper should reconcile these names or explicitly describe the renaming.","section":"Section 5.3.1 vs Section 4.4.2/Figure 2"},{"comment":"The A.1 sub-themes are initially described with their pre-rename names, 'System Support and Dependency Management' and 'Configuration and Parameter Usage', even though the paper later renames them to 'System Environment and Dependency Setup' and 'Runtime Configuration and Launch Parameters'; using final names consistently would improve clarity.","section":"Section 4.4.1"},{"comment":"The paper reports Fleiss' kappa values but does not state the number of reports or the sampling method used for the agreement computation, which limits reproducibility.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's scope and the dataset is a potentially valuable community resource. The main risk is the unsubstantiated 'LLM-centric' framing given the absence of a non-LLM baseline and the generic keywords. The numerical errors in the bug summary are easily fixed but currently weaken the key findings. I would encourage the editor to require a substantive revision that either adds a comparative non-LLM sample or reframes the claims to 'LLM-related' rather than 'LLM-centric'."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something real: it builds a three-part taxonomy (questions, requirements, bugs) for LLM-related issues in DL frameworks and their toolkits, with 20 themes and 75 sub-themes, grounded in 38,742 issue reports and then refined through interviews with 11 users and 8 developers. That taxonomy is not present in the prior work cited, and the sub-themes around hybrid parallelism misconfiguration, checkpoint precision compatibility, and documentation drift look genuinely new. The inter-annotator agreement (Fleiss kappa 0.717–0.757) and the two-stage filtering with a spot-checked precision of 92.9% are reasonable evidence that the labeling is not arbitrary. The five findings and five recommendations are actionable and match what practitioners would expect. You could hand this to a framework team and they would find useful signals.\n\nThe soft spots are real but not fatal. The most important one is the baseline problem: the paper claims LLM issues differ 'qualitatively in frequency, impact, and diagnosability' from non-LLM issues, but it never analyzes a matched sample of non-LLM issues. The keyword filter includes generic terms like 'oom', 'quantization', and 'bfloat16', and the dataset is dominated by LLM toolkits (vLLM alone is ~35%). That means the observed frequencies (e.g., 27.31% installation questions) might be similar for general DL issues. The 'LLM-centric' qualifier is plausible but not yet supported by the data. This is a matter of framing, not a fatal flaw: the taxonomy itself is built from external reports and is useful regardless of whether every category is exclusive to LLMs. The paper should add a non-LLM control corpus or soften the claim.\n\nThere are also two smaller issues the reader caught, and they check out. First, the summary percentage for C.6–C.10 bugs is stated as 39.29%, but the individual percentages sum to roughly 34.26%. That arithmetic error should be fixed. Second, the interview-derived sub-theme names (e.g., 'Multi-platform Interoperability' vs. B.1.iv 'Local Development and Lightweight Setup Support', and 'Example Drift and Documentation Mismatch' vs. B.4.iii 'Minimal Working Examples') do not match the final taxonomy figure; the text describes changes that the figure does not reflect. The interview pool is also small and skewed toward academic users, which the authors acknowledge in the threats section—so it is a minor concern, not a hidden flaw.\n\nThe citation pattern looks fair, and the self-citation is limited to the authors' own testing work. The paper ships its taxonomy and interview data on a project page, which helps reproducibility.\n\nWho is this for? Researchers and developers working on DL framework testing, issue triage, and LLM infrastructure. A serious editor should send this to peer review; with a baseline and the arithmetic and naming fixes, it would be a solid contribution to TOSEM or similar.","headline":"A solid, genuinely useful empirical taxonomy of LLM-centric DL framework issues, but the 'LLM-centric' claim needs a non-LLM baseline and a couple of arithmetic fixes before it is fully convincing.","tokens_in":42924,"tokens_out":851,"would_cite":true,"duration_ms":10609,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An analysis of 38,742 issue reports and 19 interviews claims that deep-learning frameworks fail to support large language models in five specific ways, with fragile environment setup and silent execution instability topping the list.","keywords":["deep learning frameworks","large language models","issue report analysis","taxonomy construction","empirical software engineering","LLM-centric bugs","framework usability","distributed training stability"],"falsifier":"A replication that applies the published keyword lists to a set of issues independently known to be LLM-related, such as reports maintainers have explicitly tagged as LLM-specific, and finds that a large fraction are missed, or a re-annotation of a random sample by a fresh team that yields substantially different type or theme assignments, would settle whether the taxonomy is representative and stable.","tokens_in":41938,"feed_emoji":"🐛","tokens_out":7154,"duration_ms":70180,"temperature":0.7,"pith_summary":"This paper tries to establish that the difficulties deep-learning (DL) frameworks face in supporting large language models (LLMs) are systematic enough to be captured in a single map. Drawing on 38,742 issue reports from PyTorch, MindSpore, TensorFlow, and eight LLM toolkits, plus interviews with 11 LLM users and eight framework developers, the authors construct a taxonomy of LLM-centric bugs, user questions, and requirements. They then claim that five key challenges, above all fragile environment setup and silent execution instability, are what most limit DL framework support for LLM workflows. If the claim holds, the result is a prioritized, lifecycle-aware list of framework weaknesses that developers and testing researchers can act on directly.","feed_headline":"Deep-learning frameworks fail LLMs in five predictable ways","feed_subtitle":"Analysis of 38,742 issue reports and 19 interviews maps where framework support breaks.","key_machinery":"The central object is the LLM-centric Taxonomy of Challenges in DL Frameworks. It is produced by a two-stage screening pipeline: first a validated keyword filter (72 to 84 keywords per framework) selects candidate issue reports, then manual inspection removes false positives and labels each report as a question, requirement, or bug; a bottom-up clustering step groups the labeled reports into themes and sub-themes. The taxonomy carries the argument because every downstream finding, interview protocol, and recommendation is anchored in its categories and their frequencies. The interview guideline is the second mechanism: it takes the taxonomy to practitioners, validates and extends it, and attaches priority rankings that the issue reports alone cannot provide.","core_discovery":"The paper's central claim is that a taxonomy with three issue types (questions, requirements, bugs), 20 themes, and 75 sub-themes captures a comprehensive set of LLM-centric challenges in DL frameworks, and that five key challenges, especially fragile environment setup and silent execution instability, limit framework support for LLMs. The taxonomy is built bottom-up from manually labeled issue reports and then refined and enriched through interviews, which added categories such as training-strategy confusion (A.5), memory fragmentation and leakage (C.2.iv), and hybrid parallelism misconfiguration (C.3.iv). The paper further argues that users and developers perceive these challenges differently: users prioritize configuration clarity, observability, and reproducibility, while developers prioritize operator portability, memory alignment, and runtime robustness. These divergences are presented as evidence that framework support for LLMs is not just a scaling problem but a distinct systems-engineering problem.","pith_inferences":["An implication the authors leave implicit is that the taxonomy's sub-theme frequencies can serve as a triage oracle for issue trackers: a new report matching a high-frequency LLM-centric sub-theme is more likely to be a genuine signal amid the rising volume of AI-generated noise.","The 19-person interview sample is small relative to the 38,742-report dataset, so a natural extension is a large-scale survey or instrumented telemetry collection to test whether the five key challenges remain on top outside the studied repositories.","The taxonomy has a shelf life tied to the rapid evolution of LLM tooling; as new model families, hardware, and parallelism schemes appear, the keyword list and sub-themes will need periodic refresh, and the paper's pipeline provides a reusable template for that refresh."],"forward_implications":["Framework developers get a prioritized defect list: installation and configuration confusion is 27.31% of all user questions (A.1), and unexpected behavior without diagnostic cues is 30.07% (A.3).","Observability and maintainability failures dominate at scale, accounting for 39.29% of bug reports across C.6 to C.10, so tooling and documentation should be treated as core reliability concerns rather than peripheral ones.","Silent failures, such as lost gradients, invisible checkpoint mismatches, and non-deterministic execution, form the most dangerous bug class and need runtime consistency checks rather than better error messages alone.","User and developer priorities diverge enough that framework roadmaps should be validated against both groups: users ask for workflow transparency while developers focus on low-level runtime robustness.","The five proposed optimizations (simplified setup, high-level LLM abstractions, consistency checks, lightweight observability, and documentation and tooling investment) are concrete starting points for improving framework support."],"supporting_citations":[{"why":"Supplies the multi-framework bug dataset and classification that this study extends toward LLM-specific issues.","marker":"[18]"},{"why":"Documents symptoms, causes, and repairs of bugs inside a deep learning library, giving the defect-focused baseline this taxonomy goes beyond.","marker":"[19]"},{"why":"Establishes silent bugs in deep learning frameworks as a studied phenomenon, which the paper finds especially prominent in LLM workflows.","marker":"[20]"},{"why":"Shows that silent bugs occur in PyTorch programs and motivates the paper's emphasis on silent execution instability.","marker":"[21]"},{"why":"Supplies the methodological precedent of mining developer discussion posts from online communities to understand deep-learning framework usage.","marker":"[23]"},{"why":"Provides the bottom-up bug-taxonomy construction approach that the paper adapts to cluster labeled issue reports into themes and sub-themes.","marker":"[84]"}],"fun_headline_variants":["LLM frameworks fail in 5 predictable ways, study of 38k issues finds","User-developer divide on LLM framework support: 20 themes mapped","Fragile setups, silent crashes top LLM framework challenges","Taxonomy of 75 LLM framework issues reveals key mismatches"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the keyword-filtered issue reports from the selected repositories form a representative sample of LLM-related framework challenges; if the keyword list misses common LLM terms, if silent usability problems are underreported, or if the spot-check precision is optimistic, the taxonomy's completeness and the reported frequencies shift.","fun_headline_variants_meta":{"raw":{"variants":["LLM frameworks fail in 5 predictable ways, study of 38k issues finds","User-developer divide on LLM framework support: 20 themes mapped","Fragile setups, silent crashes top LLM framework challenges","Taxonomy of 75 LLM framework issues reveals key mismatches"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1498,"prompt_tokens":950,"completion_tokens":548,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":469}},"tokens_in":566,"tokens_out":548,"duration_ms":6934,"temperature":1.0,"reasoning_tokens":469,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:36:19.411292+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A replication that applies the published keyword lists to a set of issues independently known to be LLM-related, such as reports maintainers have explicitly tagged as LLM-specific, and finds that a large fraction are missed, or a re-annotation of a random sample by a fresh team that yields substantially different type or theme assignments, would settle whether the taxonomy is representative and stable.","supporting_citations":[],"review_version":1}