{"id":"c50caaca-ba34-42f0-b14c-23aa6bbb63a2","arxiv_id":"2507.11014","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SIMCODE provides 400 verified ns-3 simulation coding tasks with tests and shows current LLMs compile and run fewer than a third of generated programs.","lead":"SIMCODE is a new benchmark of 400 programming tasks that ask large language models to write ns-3 network simulation code from natural language descriptions. The paper reports that GPT-4.1 performs best at around 30 percent execution accuracy, leaving ample room for improvement.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SIMCODE's unit tests are not a validated correctness metric: the paper itself says functionally correct code can fail them, while the headline 'execution accuracy' only measures compile-and-run success.","rationale":"The reader's weakest assumption was that the reference solutions and unit tests are correct and define the intended notion of success, mainly because they are internally validated and unpublished. My stress-test confirms that this assumption is load-bearing and adds a more precise internal reason to doubt it: the paper itself reports that functionally correct code often fails the tests because it deviates from the reference implementation. That makes Pass@1 a measure of reference-solution mimicry rather than functional correctness. Independently, the paper's headline execution-accuracy metric is explicitly defined as compile-and-execute success, so the most emphasized numbers do not establish correct simulation behavior. The model-ranking conclusion is also fragile: under the stricter Pass@1 metric, Qwen-3 beats GPT-4.1 for both the Instruction and CoT prompts in Table III, contradicting the claim that GPT-4.1 consistently outperforms. These issues are internal to the paper and do not depend on artifact availability, so they deserve to be flagged even if the dataset is released. Nevertheless, the benchmark concept is valuable, the task collection process is described in reasonable detail, and the flaws are fixable by redesigning or validating tests and reporting correctness metrics. Therefore I do not move the verdict away from CONDITIONAL; the paper should be accepted only with these validations and artifact release.","tokens_in":10274,"tokens_out":8020,"duration_ms":96382,"concrete_test":"Release the dataset and test harness, then draw a stratified random sample of 50 generated programs that compile and run but fail the provided unit tests. Have two independent ns-3 experts judge functional correctness against the natural-language prompt without seeing the reference solution. If a substantial fraction (e.g., >20%) are judged functionally correct, the unit tests are overfit to the reference implementations; re-scoring the full benchmark with functionally-equivalence-aware tests would settle whether the reported Pass@1 values and model rankings change materially.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's two success metrics measure different things. Section IV-D defines execution accuracy as the fraction of generated samples that 'successfully compile and execute,' not that pass any behavioral test. Pass@1, defined as passing all unit tests, is the only reported correctness measure. Yet Section V states that test failures 'often' occur because 'implementation logic deviates from the reference, even when functionally correct.' This admission means Pass@1 conflates functional equivalence with reference-solution mimicry: valid alternative implementations are penalized, so the benchmark's test cases do not define the intended notion of task success. The headline results (GPT-4.1 up to 30.6% execution accuracy, 29.3% with CoT) therefore largely report compilability and runnability, not simulation correctness. The claimed model ranking is also metric-dependent: Table III shows Qwen-3's Pass@1 under CoT is 0.193 versus GPT-4.1's 0.153, and under Instruction 0.163 versus 0.088, so 'GPT-4.1 consistently outperforms' holds for execution accuracy but not for the stricter correctness metric. This is an internal-validity problem that merely releasing the dataset would not resolve; the test cases themselves need verification against functionally equivalent solutions. The dataset construction is described in detail and the benchmark idea is useful, but the current evaluation evidence does not support the central claim of measuring correct and executable ns-3 code generation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"SIMCODE is presented as the first benchmark for evaluating LLMs' ability to generate ns-3 network-simulation C++ code from natural-language prompts. The dataset comprises 400 tasks across introductory, intermediate, and advanced levels, each with a prompt, a reference C++ solution, and unit tests. The paper evaluates Gemini-2.0-flash, GPT-4.1, Qwen-3, and a fine-tuned GPT-4.1 under six prompt templates, reporting CodeBLEU, CodeBERTScore, execution accuracy (compile-and-run success), and Pass@1 (passing all unit tests). The main results are that GPT-4.1 reaches up to 29.3% execution accuracy with CoT and 30.6% after fine-tuning, while error analysis attributes 48% of failures to missing/outdated headers and 21% to API mismatches. The dataset and evaluation code are not available during the review process.","tokens_in":10513,"tokens_out":7405,"duration_ms":82390,"significance":"The benchmark idea fills a real gap: existing LLM/ns-3 work focuses on interactive automation rather than reproducible evaluation, and a static benchmark with difficulty levels and per-topic breakdowns would be a useful community resource if it is made public and its test oracle is sound. The paper is not circular: models are evaluated on held-out tasks, and there are no fitted constants that reduce to the measured outcomes. The prompt taxonomy, the fine-tuning comparison, and the error-category table are useful contributions. However, the current evaluation does not establish functional correctness: execution accuracy measures only compilability/runnability, and Pass@1 conflates functional equivalence with reference-solution mimicry. The reported model ranking is metric-dependent, and the fine-tuning comparison is not matched on prompt type. Because the central artifact is absent, independent verification of the ground truth is impossible. These issues are fixable but currently prevent the paper from supporting its strongest claims.","major_comments":[{"comment":"The manuscript states that the dataset and evaluation code were not shared during the review process and will be released only upon acceptance. For a benchmark paper, the dataset is the primary contribution; without access to the 400 prompts, reference solutions, and unit tests, no reviewer or reader can verify the correctness of the ground truth, reproduce Table III or Table IV, or confirm the claimed difficulty distribution and 'verified' status. Please provide the full benchmark and evaluation harness as supplementary material or a permanent public repository in the revised version, and include model outputs for each configuration.","section":"Section VI.B / data availability statement"},{"comment":"Execution accuracy is defined in Section IV-D as the fraction of generated samples that compile and execute, not as a correctness measure. Section V then concedes that unit-test failures 'often' occur because implementation logic deviates from the reference even when functionally correct. This admission invalidates Pass@1 as a correctness oracle: valid alternative implementations are penalized as failures. The problem is load-bearing because the abstract and conclusion claim evaluation of 'correct and executable' code, and because the ranking reverses under Pass@1: in Table III, Qwen-3 outperforms GPT-4.1 on Pass@1 under both Instruction (0.163 vs 0.088) and CoT (0.193 vs 0.153). Please validate the unit tests against a set of functionally equivalent implementations or define a metric that rewards behavioral equivalence, and report execution accuracy and Pass@1 separately without claiming a consistent winner.","section":"Section IV-D and Section V"},{"comment":"The fine-tuning comparison is not apples-to-apples. The fine-tuned model is evaluated only with the Instruction prompt, yet the text says execution accuracy increases 'from 29.3% (best prompt for GPT-4.1) to 30.6%.' The proper baseline is GPT-4.1 with the Instruction prompt, which is 26.5% in Table III; under a matched comparison the improvement is +4.1 percentage points, not +1.3. In addition, Pass@1 is reported as '–' for the fine-tuned model, so there is no evidence that fine-tuning improves correctness rather than mere compile/run success. Please report matched-prompt comparisons, Pass@1, and standard deviations or confidence intervals across the five runs.","section":"Section V-D and Table III"},{"comment":"The quality-assurance description says each data point was independently reviewed by at least two authors and that test cases were validated against reference solutions and erroneous variants, but this is an internal process with no quantitative outcome (e.g., inter-annotator agreement, number of corrections, or post-hoc audits). Since the benchmark's value depends on the correctness of its ground truth, please document the review protocol in more detail and, if possible, include an external audit of a random sample of tasks (e.g., 20–30 tasks) with results reported in the paper.","section":"Section III-A.3"}],"minor_comments":[{"comment":"The average solution lengths are inconsistent: Section III-A.2 reports roughly 40, 80, and over 100 lines for the three difficulty levels, while Table I reports ~60, ~110, and 150+. Please reconcile these numbers and state the unit of measurement.","section":"Section III-A.2 and Table I"},{"comment":"Table II describes the Instruction prompt as a 'bare prompt' supplying only the minimal directive, but Figure 1 shows a detailed multi-sentence task description plus system-level instructions. Please clarify the distinction between the benchmark task prompt and the instruction template.","section":"Section IV-B and Figure 1"},{"comment":"CodeBLEU and CodeBERTScore are presented as principal evaluation metrics, but Section V-B reports that they are nearly constant across models and 'give an illusion of uniform quality.' Please either justify their inclusion as main metrics or relegate them to secondary diagnostics.","section":"Section IV-D vs Section V-B"},{"comment":"Reference [14] is listed as 'V. Mavroudis, \"LangChain v0.3\"' without a paper title; please update it to the full preprint title and provide the correct bibliographic details.","section":"References"},{"comment":"The figure contains spacing artifacts ('Refer ence Solution', 'Unit T ests') and the reference solution enables log component 'NCEx', which is not a standard ns-3 component; please correct the figure and remove the debug leftover.","section":"Figure 1"},{"comment":"Section V-C refers to difficulty-level subtotal rows for advanced, intermediate, and introductory tasks, but Table IV as printed shows only topic rows and an overall row. Please add the subtotal rows or adjust the text to refer to the values that actually appear in the table.","section":"Section V-C and Table IV"}],"recommendation":"major_revision","confidential_remarks":"The paper is a timely fit for the LCN special track on LLMs and networking. The strongest claim—being the first evaluation benchmark for ns-3 code generation—is plausible but is currently undercut by the absence of artifacts and by the mismatch between 'execution accuracy' and functional correctness. Please ask the authors to release the dataset during revision, to re-analyze the evaluation with a correctness oracle that tolerates functionally equivalent solutions, and to make the fine-tuning comparison matched on prompt type. If the dataset is withheld for anonymity reasons, the paper should not be considered a complete benchmark contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nSIMCODE is a legitimate first step: a 400-task benchmark for turning natural-language descriptions into ns-3 simulation code, with prompts, reference solutions, and unit tests. That fills a real gap—GenOnet and the LangChain pipelines are interactive tools, not evaluation suites. The construction process is described in enough detail to be replicated, and the error analysis on missing headers and API mismatches is useful. The evaluation across three models and six prompting styles is standard and honestly reported.\n\nThe soft spots are real but fixable. Most importantly, the dataset and evaluation code are not released. The paper says this is due to double-anonymized review, which doesn't match the public arXiv listing; either way, every empirical claim rests on ground truth we can't check. The second issue is metric validity. Execution accuracy only measures compile-and-run success. Pass@1 requires passing all unit tests, but the paper itself admits that functionally correct code often fails because it deviates from the reference implementation. That makes Pass@1 a mimicry score as much as a correctness score, and it muddies the model ranking: on execution accuracy GPT-4.1 leads, but on Pass@1 Qwen-3 beats it under several prompts. That's an internal-validity problem that artifact release alone won't fix; the test cases need to be checked against alternative valid implementations.\n\nMinor notes: the fine-tuning results are averages over five runs with no variance, and only one prompt type is used for the fine-tuned model. The per-topic breakdown in Table IV is consistent, which is good.\n\nThe core idea is sound and the paper is written carefully. It deserves a serious referee, but acceptance should be conditional on releasing the artifacts and either revising the test suite or redefining what Pass@1 measures.","headline":"Useful first benchmark for ns-3 code generation, but no public artifacts and the test-based metric may penalize valid alternatives.","tokens_in":11064,"tokens_out":2255,"would_cite":false,"duration_ms":24398,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper presents SIMCODE, the first benchmark for LLM-generated ns-3 simulation code, and uses it to show the strongest model reaches only 30.6% execution accuracy.","keywords":["LLM code generation","ns-3 simulation","network simulation benchmark","execution accuracy","natural language to code","domain-specific code generation","prompt engineering","unit test evaluation"],"falsifier":"Publish the SIMCODE dataset and independently re-run one model on a random subsample under the same prompts on a pinned ns-3 version; then check whether every reference solution compiles and passes its own unit tests. If a material fraction of gold solutions fails their own tests, or if two experts disagree on whether a generated program that passes the tests actually satisfies the prompt, the benchmark's validity as a measure of task success collapses.","tokens_in":10058,"feed_emoji":"📡","tokens_out":7466,"duration_ms":75277,"temperature":0.7,"pith_summary":"The paper introduces SIMCODE, a benchmark of 400 ns-3 network-simulation tasks, each pairing a natural-language prompt with a verified C++ solution and unit tests. Using it, the paper measures how well large language models can turn written descriptions into compilable, runnable simulation scripts on the first try. Across three models and six prompting strategies, the best results reach only about 30% execution accuracy, with most failures coming from missing headers and wrong API usage. The benchmark's purpose is to give researchers a fixed yardstick for a domain where earlier work demonstrated automation only as interactive demos.","feed_headline":"LLMs pass under 31% of ns-3 simulation coding tasks","feed_subtitle":"GPT-4.1 leads with about 30.6% execution accuracy; missing headers and API mismatches explain most failures.","key_machinery":"The load-bearing object is the SIMCODE dataset itself: 400 tasks organized into introductory, intermediate, and advanced levels, each as a triplet of a natural-language prompt, a verified C++ reference solution, and 3-10 ns-3 unit-test cases implemented through the simulator's tracing and assertion mechanisms. Evaluation runs the generated code in an ns-3 environment and scores execution accuracy and Pass@1 against those unit tests, while CodeBLEU and CodeBERTScore capture token- and syntax-level similarity. The six prompt templates (instruction, chain-of-thought, few-shot, ReAct, expert, self-consistency) and the fine-tuned GPT-4.1 variant serve as the experimental apparatus for probing what moves performance.","core_discovery":"The central claim is that SIMCODE supplies the first standardized, static benchmark for evaluating LLMs on ns-3 simulation code generation, and that the resulting measurements show current models are far from reliable. GPT-4.1, the strongest tested model, achieves 29.3% execution accuracy with chain-of-thought prompting, rising to 30.6% after task-specific fine-tuning, while Qwen-3 and Gemini-2.0 trail; Pass@1 scores are substantially lower, indicating that most generated code compiles or runs only after failed attempts. Similarity metrics such as CodeBLEU and CodeBERTScore stay high and nearly flat across models, which the paper reads as evidence that surface-level similarity masks real functional failures. An error breakdown attributes roughly half of all failures to missing or outdated headers and about a fifth to API mismatches, pointing toward retrieval-based and compile-feedback remedies.","pith_inferences":["If the public release matches the described data, the benchmark could serve as a drop-in evaluation suite for any future ns-3 code-generation system, allowing head-to-head comparison that the field currently lacks.","Because the two biggest error classes are header selection and API naming, a plausible testable extension is to give the model a small, current header-and-API reference at inference time; the gap between reported accuracy and this retrieval-augmented variant would quantify how much failure is purely a knowledge-timing problem.","The near-saturation of similarity metrics suggests that report readers should treat CodeBLEU and CodeBERTScore rows as sanity checks rather than quality signals; an equally useful next metric would be whether generated simulations actually configure the network the prompt asks for, not just whether they run."],"forward_implications":["Model choice matters more than prompt design: switching from Gemini-2.0 to GPT-4.1 roughly doubles execution accuracy, while prompt engineering moves any single model by only a few points.","Similarity scores overstate quality: because CodeBERTScore stays near 0.93 for all models while execution accuracy varies widely, future benchmark users should rely on execution-based metrics as the primary signal.","The dominant failure modes—missing headers (48%) and API mismatches (21%)—give concrete targets for retrieval-augmented generation and compile-feedback loops, which the paper recommends as mitigations.","Task-specific fine-tuning on a modest training split yields consistent gains (from 29.3% to 30.6% execution accuracy), suggesting domain adaptation is a viable path even with a small dataset.","Several topics, such as advanced Wi-Fi and MANET tasks, are solved by no model; these unsolved clusters define where LLM capability ends for network simulation."],"supporting_citations":[{"why":"Supplies the general-purpose code-generation benchmark and pass@1 metric that this paper's evaluation format builds on.","marker":"[1]"},{"why":"Provides the prompt-code-test benchmark structure and few-shot prompting precedent that SIMCODE adapts.","marker":"[2]"},{"why":"The network simulator whose public APIs and runtime behavior all generated code must target.","marker":"[5]"},{"why":"Prior LLM-driven ns-3 automation work that demonstrates feasibility but lacks systematic evaluation, motivating the benchmark.","marker":"[6]"},{"why":"Recent LLM-integration work that again shows interactive automation without rigorous benchmarking, reinforcing the gap SIMCODE fills.","marker":"[7]"},{"why":"The coding-challenge benchmark whose difficulty grouping and prompt statistics SIMCODE follows in dataset construction.","marker":"[15]"},{"why":"Chain-of-thought prompting, the prompt technique that yields the best average execution accuracy in the reported experiments.","marker":"[18]"},{"why":"Defines CodeBLEU, the code-similarity metric used alongside execution accuracy for evaluation.","marker":"[21]"}],"fun_headline_variants":["First ns-3 coding benchmark: LLMs run under 31% of tasks","GPT-4.1 leads ns-3 code test at 30.6%, still low","Missing headers and API mismatches sink most LLM ns-3 code","SIMCODE reveals LLMs fail most ns-3 simulation code challenges","LLMs score high on code similarity but low on execution in ns-3"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire evaluation rests on the assumption that the reference solutions and unit tests in SIMCODE are correct and unambiguously define success; if a reference solution is wrong, its test set is buggy, or the ground truth depends on subtle simulator-version behavior, every reported accuracy number shifts accordingly.","fun_headline_variants_meta":{"raw":{"variants":["First ns-3 coding benchmark: LLMs run under 31% of tasks","GPT-4.1 leads ns-3 code test at 30.6%, still low","Missing headers and API mismatches sink most LLM ns-3 code","SIMCODE reveals LLMs fail most ns-3 simulation code challenges","LLMs score high on code similarity but low on execution in ns-3"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000291,"raw_usage":{"total_tokens":1684,"prompt_tokens":912,"completion_tokens":772,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":669}},"tokens_in":528,"tokens_out":772,"duration_ms":8124,"temperature":1.0,"reasoning_tokens":669,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:18:21.862031+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Publish the SIMCODE dataset and independently re-run one model on a random subsample under the same prompts on a pinned ns-3 version; then check whether every reference solution compiles and passes its own unit tests. If a material fraction of gold solutions fails their own tests, or if two experts disagree on whether a generated program that passes the tests actually satisfies the prompt, the benchmark's validity as a measure of task success collapses.","supporting_citations":[{"cited_title":"The ns-3 network simulator,","cited_arxiv_id":null,"evidence_quote":"The network simulator whose public APIs and runtime behavior all generated code must target."},{"cited_title":"Genonet: Generative open xg network simulation with multi-agent llm and ns- 3,","cited_arxiv_id":null,"evidence_quote":"Prior LLM-driven ns-3 automation work that demonstrates feasibility but lacks systematic evaluation, motivating the benchmark."},{"cited_title":"Toward Generative 6G Simulation: An Experimental Multi-Agent LLM and ns-3 Integration","cited_arxiv_id":"2503.13402","evidence_quote":"Recent LLM-integration work that again shows interactive automation without rigorous benchmarking, reinforcing the gap SIMCODE fills."},{"cited_title":"Measuring coding challenge competence with APPS,","cited_arxiv_id":null,"evidence_quote":"The coding-challenge benchmark whose difficulty grouping and prompt statistics SIMCODE follows in dataset construction."},{"cited_title":"Chain-of- thought prompting elicits reasoning in large language models,","cited_arxiv_id":null,"evidence_quote":"Chain-of-thought prompting, the prompt technique that yields the best average execution accuracy in the reported experiments."}],"review_version":1}