{"id":"e42d81fd-4d90-423d-a289-995b6b53cd66","arxiv_id":"2606.05001","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Presents TeleSWEBench, the first commit-driven benchmark with 734 unit-test cases from srsRAN 5G plus TeleJudge LLM evaluator, showing top ASE tools achieve up to 25% functional success on telecom tasks.","lead":"This paper introduces TeleSWEBench, a benchmark built from real developer commits in the srsRAN 5G codebase to test how well LLM agents can edit complex telecom software. It highlights a gap in existing coding benchmarks for stateful, mathematically rigorous wireless systems and reports that current tools reach at most 25% shippable changes.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Benchmark validity: whether srsRAN-mined commits + unit tests + TeleJudge actually measure ability to produce shippable telecom changes remains the unverified premise.","rationale":"The reader's weakest assumption is precisely the load-bearing premise required for the strongest claim. The abstract-only review already isolated it; nothing in the supplied abstract text supplies independent evidence that would falsify or strengthen that assumption. Therefore the appropriate stance remains UNVERDICTED pending a concrete validation of the benchmark itself.","tokens_in":1824,"tokens_out":449,"duration_ms":21631,"concrete_test":"Sample 30 tasks (10 per tier). For each, obtain the original commit diff and pre/post unit-test status from the srsRAN repo; re-run the exact unit tests on the pre-commit code to confirm they fail, then confirm they pass only after the commit. Separately, have two domain engineers (blind to the paper) rate whether solving the task description would produce a change that could be merged without further integration work. If >30% of tasks fail either check, the 25% shippable figure cannot be interpreted as evidence of real-world capability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result (max 25% shippable changes, failures in localization and functional correctness) is only meaningful if the 734 distilled tasks constitute a reliable proxy. The construction mines real commits from srsRAN 5G, tiers them Easy/Medium/Difficult, attaches executable unit tests, and adds a hierarchical LLM judge (TeleJudge) that scores at file level via context and semantic similarity. For the claim to hold, two conditions must be true: (1) the selected commits and their tests capture the stateful, mathematically rigorous wireless-stack requirements that distinguish telecom SE from general coding, and (2) TeleJudge verdicts correlate with actual deployability rather than surface similarity. Neither condition is independently validated in the reported evaluation; the paper provides no cross-check against integration tests, hardware-in-the-loop runs, or expert telecom engineer ratings. If either condition fails, the observed performance gap could be an artifact of benchmark construction rather than a genuine limitation of the ASE tools.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces TeleSWEBench, the first commit-driven benchmark for LLM-powered ASE in telecommunications. It mines real commits from the srsRAN 5G repository, distills them into 734 structured tasks across Easy/Medium/Difficult tiers with executable unit tests, and proposes TeleJudge (a hierarchical LLM-as-a-judge using file-level context and semantic similarity) as a complement to rigid unit-test evaluation. Two-stage evaluation of AIDER, OpenHands, and ClaudeCode (powered by models including Qwen3, GPT OSS, Gemma 4, Kimi, Qwencoder 2.5) finds that agents lack localization accuracy and functional correctness, with the strongest tools achieving at most 25% shippable changes.","tokens_in":2021,"tokens_out":671,"duration_ms":18135,"significance":"If the benchmark construction and TeleJudge verdicts are shown to be reliable proxies for deployable telecom changes, the work would fill a documented gap: general SWE benchmarks do not capture the stateful, mathematically rigorous wireless-stack requirements of telecom. The commit-mining approach and dual (unit-test + semantic) evaluation protocol are concrete strengths that could be extended to other specialized domains. The reported performance ceiling supplies a falsifiable baseline for future agent development in O-RAN/AI-RAN contexts.","major_comments":[{"comment":"§3 (Benchmark Construction): The paper provides no cross-validation (expert telecom-engineer ratings, integration-test runs, or hardware-in-the-loop checks) that the 734 distilled tasks and their unit tests capture the distinguishing stateful and protocol-level requirements of srsRAN-style wireless stacks. This directly undermines the central claim that the observed 25% shippable-change rate reflects genuine limitations rather than benchmark-construction artifacts.","section":"§3"},{"comment":"§4 (TeleJudge and Two-Stage Evaluation): No correlation analysis is reported between TeleJudge file-level semantic scores and either unit-test outcomes or independent expert judgments of deployability. Without such evidence, the hierarchical judge cannot be treated as a validated complement to unit tests for the functional-correctness dimension of the headline result.","section":"§4"},{"comment":"§5 (Results): The 25% shippable-change figure is presented without statistical significance testing, confidence intervals, or breakdown by difficulty tier and model; given that the benchmark validity itself remains unverified, this figure cannot yet be interpreted as a robust performance ceiling.","section":"§5"}],"minor_comments":[{"comment":"The abstract and §2 would benefit from an explicit statement of the exact commit-selection heuristics and inter-rater process used to tier tasks as Easy/Medium/Difficult.","section":"Abstract, §2"},{"comment":"Figure captions and Table 1 should clarify whether the reported percentages are macro- or micro-averaged across the 734 tasks.","section":"Tables/Figures"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the journal's scope well as a domain-specific benchmark contribution, but the citation list should be checked for completeness on prior SWE-bench variants and telecom-specific SE papers."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback highlighting the need for stronger validation of the benchmark tasks, TeleJudge, and statistical presentation of results. We address each major comment below with planned revisions where feasible.","responses":[{"response":"The tasks are distilled directly from real commits authored by srsRAN developers, which inherently encode the stateful, protocol-level, and mathematically rigorous requirements of the wireless stack as implemented in production code. The accompanying unit tests are executable and drawn from the repository's own test infrastructure. We will revise §3 to more explicitly justify the commit-mining approach as a domain-relevant proxy and add an explicit limitations subsection acknowledging the absence of expert ratings or hardware-in-the-loop validation, along with plans for such extensions in future work.","revision_made":"partial","referee_comment":"[§3] §3 (Benchmark Construction): The paper provides no cross-validation (expert telecom-engineer ratings, integration-test runs, or hardware-in-the-loop checks) that the 734 distilled tasks and their unit tests capture the distinguishing stateful and protocol-level requirements of srsRAN-style wireless stacks. This directly undermines the central claim that the observed 25% shippable-change rate reflects genuine limitations rather than benchmark-construction artifacts."},{"response":"We will add a new subsection in §4 reporting correlation (Pearson and Spearman) between TeleJudge file-level semantic scores and unit-test pass/fail outcomes across all evaluated agents and models. This will quantify the degree of alignment. We note that independent expert judgments of deployability were not collected in the current study due to resource constraints and will discuss this as a limitation.","revision_made":"yes","referee_comment":"[§4] §4 (TeleJudge and Two-Stage Evaluation): No correlation analysis is reported between TeleJudge file-level semantic scores and either unit-test outcomes or independent expert judgments of deployability. Without such evidence, the hierarchical judge cannot be treated as a validated complement to unit tests for the functional-correctness dimension of the headline result."},{"response":"We agree that additional statistical rigor is required. In the revised §5 we will include: (i) 95% confidence intervals computed via bootstrapping, (ii) statistical significance tests (e.g., McNemar’s test for paired agent comparisons), and (iii) full breakdowns of success rates by difficulty tier (Easy/Medium/Difficult) and by underlying LLM. These additions will allow readers to interpret the 25% ceiling with appropriate caution.","revision_made":"yes","referee_comment":"[§5] §5 (Results): The 25% shippable-change figure is presented without statistical significance testing, confidence intervals, or breakdown by difficulty tier and model; given that the benchmark validity itself remains unverified, this figure cannot yet be interpreted as a robust performance ceiling."}],"tokens_in":1620,"tokens_out":657,"duration_ms":20893,"standing_objections":["Full cross-validation involving multiple independent telecom experts performing ratings or hardware-in-the-loop integration tests, which would require specialized domain expertise, access to proprietary testbeds, and resources beyond the scope of this initial benchmark release."]},"desk_editor":{"model":"grok-4.3","letter":"TeleSWEBench mines real commits from the srsRAN 5G repository, distills them into 734 tiered tasks, attaches executable unit tests, and adds a hierarchical LLM judge that scores at file level using context and semantic similarity. That construction is the main new piece.\n\nThe paper does a solid job identifying the gap: general SWE benchmarks ignore the stateful logic and strict requirements in wireless stacks. Using actual developer commits instead of synthetic problems, plus running both test-based and judge-based evaluation on tools like AIDER and OpenHands, gives a concrete way to surface localization and correctness failures.\n\nThe soft spot sits in the unverified premise. Nothing in the work checks whether the selected commits and TeleJudge verdicts line up with what would actually ship in a telecom deployment. There is no cross-check against integration tests, hardware runs, or ratings from domain engineers, so the 25% figure could reflect benchmark artifacts rather than tool limits. Commit selection criteria and tiering process also stay thin.\n\nThis is for researchers working on domain-adapted agents or benchmarks in specialized software areas. Someone looking for patterns on how to turn real commits into test suites for complex systems can extract usable ideas even if the numbers need more grounding.\n\nIt deserves a serious referee because the benchmark idea is concrete and the performance observations are worth testing further. I would send it to review and ask for validation steps on the judge and task representativeness.","headline":"TeleSWEBench creates a commit-driven benchmark from srsRAN 5G changes with unit tests and TeleJudge, but leaves open whether those tasks actually track real shippable telecom work.","tokens_in":2531,"tokens_out":376,"would_cite":false,"duration_ms":18179,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"LLM-powered agents achieve at most 25 percent shippable changes when modifying real 5G wireless code.","keywords":["telecom software engineering","LLM agents","code generation benchmark","srsRAN 5G","automated software engineering","commit-driven evaluation","O-RAN","functional correctness"],"falsifier":"A new agent framework that produces more than 40 percent shippable changes across the full set of TeleSWEBench cases under the same two-stage evaluation would falsify the reported performance ceiling.","tokens_in":2718,"feed_emoji":"📡","tokens_out":736,"duration_ms":19185,"temperature":0.7,"pith_summary":"Telecom networks have become large, stateful software systems under frameworks such as O-RAN, yet general coding benchmarks do not test the strict requirements of wireless stacks. The paper therefore builds TeleSWEBench by mining actual developer commits from the srsRAN 5G repository and turning them into 734 executable test cases divided into three difficulty levels. Each case includes unit tests plus a hierarchical LLM judge called TeleJudge that scores file-level changes for semantic fit. When current agent frameworks powered by strong reasoning models are run on the benchmark, they show weak ability both to locate the correct code and to produce changes that pass tests, reaching a maximum of 25 percent shippable outcomes.","feed_headline":"LLM agents fix at most 25% of real telecom code changes","feed_subtitle":"Benchmark mined from srsRAN 5G commits shows current tools struggle with both locating edits and making them pass tests.","key_machinery":"TeleSWEBench, the commit-driven benchmark of 734 questions drawn from srsRAN 5G commits together with executable unit tests and the TeleJudge hierarchical LLM-as-a-judge scoring system.","core_discovery":"TeleSWEBench distills real commits from the srsRAN 5G repository into structured tasks with unit tests. Evaluation of AIDER, OpenHands, and ClaudeCode powered by models such as Qwen3, GPT OSS, Gemma 4, Kimi, and Qwencoder 2.5 shows that these tools lack both localization accuracy and functional correctness, so that even the strongest configurations produce at most 25 percent shippable changes.","pith_inferences":["The same commit-mining approach could expose similar gaps in other specialized domains that rely on stateful or protocol-heavy code.","Performance on TeleSWEBench may improve if agents are given explicit models of wireless protocol constraints rather than generic code context.","The gap between localization success and functional success points to a need for better intermediate verification steps inside agent loops.","Over time the benchmark itself could serve as a training signal for agents that must operate inside live network management systems."],"forward_implications":["Agents must improve at identifying the precise files and functions that need modification inside large wireless codebases.","Evaluation in telecom will require both strict unit tests and semantic judgment because many valid changes are not captured by tests alone.","Domain-specific data or fine-tuning may be needed before agents can handle the mathematically constrained logic of 5G stacks.","Operators adopting automated tools will still require substantial human review for changes to production wireless software."],"fun_headline_variants":["LLM agents fix 25% of srsRAN 5G commits","TeleSWEBench rates AI agents at 25% on real changes","Top LLM tools achieve 25% on telecom code benchmark","srsRAN benchmark reveals 25% success for coding agents"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Commits mined from the srsRAN 5G repository plus their unit tests and TeleJudge scores form a representative measure of an agent's ability to produce shippable changes in real telecom engineering.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents fix 25% of srsRAN 5G commits","TeleSWEBench rates AI agents at 25% on real changes","Top LLM tools achieve 25% on telecom code benchmark","srsRAN benchmark reveals 25% success for coding agents"]},"model":"grok-4.3","cost_usd":0.00736,"raw_usage":{"total_tokens":3440,"prompt_tokens":777,"num_sources_used":0,"completion_tokens":73,"cost_in_usd_ticks":73599500,"prompt_tokens_details":{"text_tokens":777,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2590,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":777,"tokens_out":73,"duration_ms":19711,"temperature":1.0,"reasoning_tokens":2590,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T05:08:58.723165+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A new agent framework that produces more than 40 percent shippable changes across the full set of TeleSWEBench cases under the same two-stage evaluation would falsify the reported performance ceiling.","supporting_citations":[],"review_version":1}