{"id":"f2e874a2-f8f3-43c8-907f-06774586f371","arxiv_id":"2502.15754","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Text2Net converts plain-text network descriptions into EVE-NG simulations using a prompted LLM and regex parsing, cutting setup time in three test scenarios.","lead":"Text2Net is a tool that turns plain-English descriptions of a computer network into a working simulation in the EVE-NG emulator. It uses ChatGPT to translate text into configuration commands, which could save students and engineers time compared to clicking and typing in traditional network simulators.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The timing advantage is only measured on the happy path; no end-to-end accuracy or reliability metric is reported, so a single parsing failure could erase the claimed 45–58% savings.","rationale":"Good-faith read: the paper is a proof-of-concept for an LLM-based natural-language-to-network-simulation pipeline, and the static-routing scope is acknowledged. The architecture is coherent, and the three-scenario comparison is a reasonable first demonstration. However, the load-bearing premise of the claimed time savings is correctness: a faster pipeline that frequently produces wrong configurations is not faster in practice. The manuscript never measures this. Section III-B demonstrates three input phrasings for the same topology and one missing-information interaction, but no accuracy statistics are given. Algorithm 1's regex extraction is deterministic once SCS is produced, but SCS generation by GPT-4T is stochastic, and no prompt-output conformance rate is reported. The manual timing is explicitly best-case, making the comparison favorable to Text2Net; the Text2Net timing is not shown to include failure handling. This does not invalidate the prototype, but it makes the central quantitative claim conditional on reliability evidence. The reader's weakest assumption identifies the same missing accuracy measurement, so the conditional verdict should remain until that evidence is supplied.","tokens_in":7849,"tokens_out":3103,"duration_ms":27120,"concrete_test":"Run 30–50 independent end-to-end trials per scenario on a held-out set of paraphrased topology descriptions (including near-miss inputs, omitted details, and alternate IP phrasing). For each trial, record whether the generated JSON exactly matches the ground-truth topology and whether EVE-NG applies it without manual intervention, and measure wall-clock time including any LLM retry or user-correction loop. If the end-to-end success rate is below ~95% or the average time including failures exceeds the manual baseline, the central timing claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV's central comparison (110/250/310 s vs. 200/510/730 s) assumes that Text2Net's LLM-then-regex pipeline converts the user's plain text into a correct, deployable EVE-NG topology on the first attempt. The paper reports no end-to-end success rate, no number of trials, and no handling statistics for malformed SCS or parser failures. Algorithm 1 (Section III-C) is a hand-written extraction routine whose correctness depends on the instructed LLM emitting exactly the expected SCS format; the only evidence offered is three hand-picked scenarios, one of which deliberately triggers the 'missing static route' prompt. The manual baseline in Section IV explicitly excludes thinking and troubleshooting time, while the Text2Net timings appear to be single successful runs. If even one user utterance yields non-conforming SCS, the pipeline either regenerates (adding latency) or produces a wrong JSON silently. In real use the deployment time must include detecting and fixing such failures. Therefore the measured 45–58% time savings do not yet establish the abstract claim of reduced 'time and effort' for deployment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Text2Net, a system that takes plain-English descriptions of network topologies and, through an instructed LLM (ChatGPT-4T) and a regex/SpaCy-based extractor, produces JSON configuration files that are used to provision live network topologies in the EVE-NG emulator. The system is demonstrated on three static-routing scenarios of increasing complexity. The authors evaluate Text2Net by comparing task completion time and number of steps with manual EVE-NG configuration, and by collecting qualitative feedback from 15 participants. They report that Text2Net reduces deployment time from 200/510/730 seconds to 110/250/310 seconds across the three scenarios and claim that the system significantly reduces the time and effort required to deploy network scenarios.","tokens_in":8033,"tokens_out":3184,"duration_ms":30373,"significance":"If the quantitative and reliability claims were fully supported, Text2Net would be a useful contribution to network-education tooling by lowering the barrier to using realistic emulators such as EVE-NG. The paper's strengths are that it describes a concrete, end-to-end prototype with a clear system flow, it explicitly states its scope limitation to static routing, and its manual baseline is conservative in the sense that thinking and troubleshooting time are excluded. However, the central efficiency claim rests on single measurements per condition with no variance or statistical testing, and the pipeline's end-to-end correctness is never measured. The significance is therefore conditional on additional evaluation evidence.","major_comments":[{"comment":"The central quantitative claim is based on one timing measurement per scenario per method: Text2Net at 110/250/310 seconds versus manual EVE-NG at 200/510/730 seconds. No repeated trials, standard deviations, confidence intervals, or statistical tests are reported, so the phrase \"significantly reduces\" in the abstract and Section IV is not supported by the data. Please report multiple runs per scenario, describe how start and end times are defined for both conditions (including whether Text2Net times include LLM API latency, parsing, and EVE-NG provisioning), and provide a paired statistical comparison or per-run distributions.","section":"Section IV, Figs. 6-8"},{"comment":"The evaluation contains no end-to-end accuracy or reliability metric. The pipeline's correctness depends on the instructed LLM emitting the expected SCS format and on Algorithm 1 parsing it correctly, but the only supporting evidence is three hand-written scenarios, one of which is deliberately incomplete. If a user utterance produces non-conforming SCS, the system may either regenerate (adding latency) or silently generate a wrong JSON, and such outcomes are not reflected in the reported timings. Please add an accuracy evaluation on a held-out set of paraphrased topology descriptions, reporting at least the first-attempt success rate, exact-match JSON validity, and failure-recovery behavior.","section":"Sections III.B-III.D and Section IV"},{"comment":"Algorithm 1 is not fully specified and appears to contain errors: line 16 reads \"if ace<- ExtractInterfaceDetails(l)\" without a defined variable or comparison, lines 17-18 then refer to \"if ace\", and line 6's condition \"if k has no comma\" is unexplained. Because this algorithm is the core extraction routine that converts SCS into the JSON used for provisioning, please provide corrected, unambiguous pseudocode or the actual parser logic so that the construction of key-value pairs can be reproduced.","section":"Algorithm 1, Section III.C"},{"comment":"The abstract and Section IV claim that the results \"showcase its scalability across various network complexities,\" but the evaluation covers only three static-routing scenarios with up to three routers. This is too narrow to support a general scalability claim. Please either temper the statement to reflect the demonstrated scope or add experiments with larger topologies and more diverse configuration tasks.","section":"Section IV and Abstract"}],"minor_comments":[{"comment":"The text says the model \"was trained\" to interpret and generate SCS, but no fine-tuning or training procedure is described; it appears to be prompt engineering or in-context instruction. Please clarify whether any model weights were updated or whether the behavior is obtained solely through prompting.","section":"Section III.A"},{"comment":"The sentence \"as well as a static router on 'R-3'\" should read \"a static route on 'R-3'.\"","section":"Section III.B"},{"comment":"The definition of \"steps\" is not uniform: manual steps include launching the simulator, logging in, and creating a lab, while Text2Net's two steps are not enumerated. Please clarify what counts as a step for each method and whether Text2Net's step count includes typing the prompt and waiting for deployment.","section":"Section IV"},{"comment":"The qualitative survey is summarized only by an average score of 4.66 out of 5; no questionnaire items, response scales, or per-item results are provided. Please include the survey instrument and a breakdown of responses.","section":"Section IV, Fig. 9"},{"comment":"The paper contains inconsistent spelling of the regular-expression component (\"RegEX\" and \"regex\") and refers to the model as both \"ChatGPT-4T\" and \"GPT-4T\"; please standardize the terminology.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable system description with a clear prototype, but the evaluation is currently too thin for the claims made. The main issues—missing trial repetitions/statistics and missing end-to-end accuracy—are addressable through additional experiments and should be within the scope of a revision. The related-work discussion of NetGPT/NetLM is somewhat tangential and could be streamlined to focus on configuration-generation systems such as CONFPILOT and ABC."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a coherent little proof-of-concept for using an LLM to provision EVE-NG simulations from plain English, but the time-savings claim is built on three single-run measurements and no accuracy data. Treat the 45–58% numbers as illustrative, not established.\n\nWhat's new: the specific pipeline — prompted LLM → Structured Command Strings → regex → JSON → EVE-NG API — is a real integration that I haven't seen in the cited work, which mostly targets configuration generation for real devices. The SCS intermediate format is a sensible design choice because it constrains the LLM's output into something the deterministic parser can handle. I also give credit for the robustness check where two different phrasings of the same topology yield the same SCS, and for the conservative manual baseline that excludes thinking and troubleshooting time. The authors are open about the static-routing scope.\n\nSoft spots: the evaluation is the weak link. Timings are single runs per scenario — no repetition, no variance, no statistical test. So the word \"significantly\" does a lot of work. More important, there's no end-to-end accuracy measurement. We never learn how often the LLM produces well-formed SCS or how often the regex parser fails. If a user's phrasing produces non-conforming SCS, the pipeline either regenerates (adding latency) or fails silently. The three scenarios are hand-picked, and one deliberately incomplete, which isn't a substitute for a robustness study. The \"first initiative\" claim in Section II is overstated relative to CONFPILOT, ABC, and NetCR; the real novelty is the simulation provisioning angle, not LLM-to-config translation. There are no public artifacts, which makes reproduction harder. These are fixable issues, but they sit exactly on the paper's central claim.\n\nBottom line: the paper is a reasonable prototype description for a networking-education venue. The architecture makes sense and the limitations are acknowledged. A serious referee should ask for repeated trials, an accuracy study with varied phrasings, and a clearer success metric before the time-savings claim can be trusted.\n\nRecommendation: send it to peer review, but expect heavy revision.","headline":"Useful LLM-to-EVE-NG proof-of-concept with a clean architecture, but the time-savings claim rests on single-run timings and no accuracy data; worth refereeing with major revision.","tokens_in":8566,"tokens_out":2229,"would_cite":false,"duration_ms":19252,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Text2Net is a system that converts plain-English network descriptions into working emulated topologies, and its three-test comparison reports a 45 to 58 percent reduction in configuration time over manual setup.","keywords":["text-to-network simulation","large language models","natural language processing","network configuration automation","static routing","network emulation","educational technology"],"falsifier":"Give Text2Net 50 plain-English topology descriptions that vary in wording, detail level, and phrasing, including ambiguous or partially invalid IP addresses, and count how many produce a complete deployable topology without human correction; if the success rate is far below perfect, the timing comparison overstates real-world savings.","tokens_in":7653,"feed_emoji":"🌐","tokens_out":5122,"duration_ms":48472,"temperature":0.7,"pith_summary":"The paper introduces Text2Net, a pipeline that takes a plain-English description of a network topology, uses an instructed large language model to turn it into structured command strings, parses those into JSON, and feeds the JSON to a network emulator to provision the topology automatically. The central claim is that this makes deploying network simulations much faster than manually configuring the emulator: in three static-routing scenarios, deployment took 110, 250, and 310 seconds versus 200, 510, and 730 seconds manually, a 45 to 58 percent saving. The authors also report a qualitative survey rating ease of use and educational value at 4.66 out of 5. The point of the paper is that students and professionals can focus on network design rather than vendor-specific syntax.","feed_headline":"Text2Net cuts network setup time by up to 58 percent","feed_subtitle":"An LLM-driven pipeline turns plain-English topology descriptions into live emulations, beating manual configuration in three tests.","key_machinery":"The central mechanism is the Structured Command String (SCS), a short command segment containing one or a few key-value pairs that an instructed large language model produces from a plain-text scenario description. A regex-based extractor then converts these SCS segments into a JSON blueprint for the emulator, and this two-stage design is what lets different wordings of the same topology converge to the same configuration.","core_discovery":"Text2Net demonstrates that a natural-language description of devices, interfaces, IP addresses, and static routes can be carried through an LLM-and-regex pipeline into a working emulated topology. The load-bearing result is the timing comparison: across all three test scenarios, Text2Net needed fewer steps and less time than the manual best case, with the gap widening as complexity grows from one to three routers. The system also detects missing information, such as absent static-route details, and asks the user for it before generating command strings.","pith_inferences":["Because the manual timings deliberately exclude thinking and troubleshooting time, real-world manual deployments are likely to take even longer, so the actual time savings in practice may exceed 58 percent.","The unmeasured end-to-end success rate of the LLM-to-SCS-to-JSON conversion is the main robustness risk; a small pilot with diverse user phrasings could turn the timing comparison into a reliability claim.","The pipeline's clean split between extraction and emulator provisioning means a future version could add a JSON validation step to catch malformed command strings before calling the emulator.","If the paper's planned retrieval-augmented generation replaces the regex parser, the same architecture should extend beyond static routing to Layer 2 protocols and NAT/VPN scenarios."],"forward_implications":["For static-routing configurations, plain-text input alone is sufficient to produce a deployable emulated topology with no manual CLI work.","Manual configuration time grows roughly linearly as nodes are added, while Text2Net's time grows more slowly, so the advantage widens with topology complexity.","Users can describe the same network in different narrative styles and still get the same structured output, which supports use in classrooms with varied student phrasing.","Incomplete descriptions do not fail blindly: the system detects missing static-route details and asks the user for them before proceeding."],"supporting_citations":[{"why":"Supplies the EVE-NG emulator platform that Text2Net automates and that serves as the manual-configuration baseline for the timing comparison.","marker":"[4]"},{"why":"Supports the choice of EVE-NG as a realistic emulator suitable for comparing Text2Net against traditional tools.","marker":"[5]"},{"why":"Documents that traditional simulation tools require repetitive, vendor-specific command syntax, which motivates the time savings Text2Net claims.","marker":"[6]"},{"why":"Motivates using LLMs in network management, positioning Text2Net within the broader move toward AI-native network operations.","marker":"[8]"},{"why":"Provides the closest prior approach that translates natural-language intents into network configuration commands, which Text2Net extends into the simulation domain.","marker":"[12]"},{"why":"Shows the existing gap in manual network configuration assistance, which Text2Net addresses by automating simulation setup rather than recommending manual steps.","marker":"[13]"}],"fun_headline_variants":["Text2Net: Plain text to live network sims","LLM pipeline turns text into network topologies","Text2Net cuts setup time by up to 58%","Ask for your network in plain English","Text2Net: From words to working network sims"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole speed advantage assumes that the language model will always produce command strings the regex parser can turn into valid configuration JSON, and this conversion is never measured for accuracy anywhere in the paper.","fun_headline_variants_meta":{"raw":{"variants":["Text2Net: Plain text to live network sims","LLM pipeline turns text into network topologies","Text2Net cuts setup time by up to 58%","Ask for your network in plain English","Text2Net: From words to working network sims"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000584,"raw_usage":{"total_tokens":2675,"prompt_tokens":802,"completion_tokens":1873,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":418,"completion_tokens_details":{"reasoning_tokens":1797}},"tokens_in":418,"tokens_out":1873,"duration_ms":12847,"temperature":1.0,"reasoning_tokens":1797,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T13:43:18.455214+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give Text2Net 50 plain-English topology descriptions that vary in wording, detail level, and phrasing, including ambiguous or partially invalid IP addresses, and count how many produce a complete deployable topology without human correction; if the success rate is far below perfect, the timing comparison overstates real-world savings.","supporting_citations":[{"cited_title":"Emulated Virtual Environment - Next Generation,","cited_arxiv_id":null,"evidence_quote":"Supplies the EVE-NG emulator platform that Text2Net automates and that serves as the manual-configuration baseline for the timing comparison."},{"cited_title":"Comparison of packet tracer and eve-ng tools for efficient network design,","cited_arxiv_id":null,"evidence_quote":"Supports the choice of EVE-NG as a realistic emulator suitable for comparing Text2Net against traditional tools."},{"cited_title":"Teaching of computer networks with using simulation and emulation environments,","cited_arxiv_id":null,"evidence_quote":"Documents that traditional simulation tools require repetitive, vendor-specific command syntax, which motivates the time savings Text2Net claims."},{"cited_title":"Netgpt: An ai-native network architecture for provisioning beyond per- sonalized generative services,","cited_arxiv_id":null,"evidence_quote":"Motivates using LLMs in network management, positioning Text2Net within the broader move toward AI-native network operations."},{"cited_title":"Confpilot: A pilot for faster configuration by learning from device manuals,","cited_arxiv_id":null,"evidence_quote":"Provides the closest prior approach that translates natural-language intents into network configuration commands, which Text2Net extends into the simulation domain."},{"cited_title":"Netcr: Knowledge graph based recommendation framework for manual network configuration,","cited_arxiv_id":null,"evidence_quote":"Shows the existing gap in manual network configuration assistance, which Text2Net addresses by automating simulation setup rather than recommending manual steps."}],"review_version":1}