RepLLM: Toward Automatically Reproducing Network Research Results

Haiying Huang; Jiwu Shu; Letian Zhu; Linghe Kong; Lizhao You; Lu Tang; Qiang Su; Qiao Xiang; Qingyu Song; Rui Liu

REVIEW 5 major objections 7 minor 52 references

RepLLM: Toward Automatically Reproducing Network Research Results

T0 review · 5 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims an LLM pipeline can turn networking papers into code that reproduces most benchmarks in hours.

desk verdict The abstract advertises a multi-agent framework the body never describes; the body's RePro evaluation is interesting but too contaminated to support the paper's claims. read the letter →

arxiv 2509.21074 v4 pith:IXK227EW submitted 2025-09-25 cs.NI

Yining Jiang , Yunxin Xu , Wenyun Xu , Yufan Zhu , Rui Liu , Tangtang He , Haiying Huang , Letian Zhu

show 9 more authors

Qingyu Song Qiang Su Lizhao You Lu Tang Wanjian Feng Yuchao Zhang Linghe Kong Qiao Xiang Jiwu Shu

This is my paper · ORCID

classification cs.NI

keywords largelanguagemodelscodegenerationresearchreproductioncomputernetworkingpromptengineeringchain-of-thoughtreasoningmulti-agentframeworknetworksystems

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that a structured, LLM-driven pipeline can take a published computer-networking paper and turn it into runnable code that reproduces the paper's own measured results. The abstract frames this as RepLLM, a four-agent end-to-end framework with shared memory that recovers 95% of original benchmarks in about two hours; the body, however, describes a semi-automated framework named RePro with a three-stage prompt-engineering pipeline and reports an average reproduction time of 38.6 hours across six systems. Read as a whole, the manuscript's central thesis is that decomposing a paper into functional modules, generating code framework-first with structured chain-of-thought reasoning, then debugging module by module with semantic chain-of-thought, cuts the human effort of reproduction from weeks of manual work to about thirty human-written prompts per system. That matters because only about 40% of top networking papers release their code, so a cheaper paper-to-code path would let researchers verify and build on results that would otherwise sit behind missing implementations.

What carries the argument

The load-bearing machinery is the prompt-engineering pipeline. Structured Chain-of-Thought (SCoT) generates a pseudo-code skeleton constrained to the three fundamental control structures—sequence, branch, and loop—before any code is written, forcing the model to decide module inputs, outputs, and logical flow first. Semantic Chain-of-Thought (SeCoT) then guides function-level implementation by making the model state data flow and control flow explicitly before emitting code. The third piece is the paper-content mapping: every generated function is annotated with a verbatim excerpt of the source paper as its requirement, so the LLM's later reasoning is anchored to the original text rather than to a paraphrase. These pieces are orchestrated in a four-stage workflow—system description extraction, framework-level generation, function-level generation, and code optimization—and evaluated against five LLMs with different context windows and output limits.

What would settle it

Run the Section 7 protocol on the same six systems with participants who have never seen the papers and no access to the public repositories while recording end-to-end wall-clock time. If the mean comes out near 38.6 hours rather than the abstract's roughly two hours, or errors exceed 5%, the headline claim is contradicted by the paper's own evaluation setup.

Watch

Extended reading notes

Core claim

On the body's own terms, the paper's discovery is that a semi-automated prompt-engineering workflow can faithfully re-implement heterogeneous network systems from their paper text. The workflow extracts the system's sub-field, problem, inputs, outputs, and architecture; divides the reproduction into independent functional modules; generates a framework-level scaffold using Structured Chain-of-Thought and few-shot in-context learning; maps verbatim paper content onto each function; fills in functions using Semantic Chain-of-Thought that tracks data and control flow; and then repairs errors by feeding compiler and test feedback back to the LLM, with humans supplying targeted prompts only for stubborn semantic bugs. Across six open-source systems—a traffic-engineering solver, a graph-pattern miner, a DNS verifier, a control-plane verifier, a datacenter transport protocol, and a congestion-control algorithm—the reproduced implementations come within 0–3.1% of original performance on average, with normalized latencies near 1.0 in most cases. The abstract separately claims that a four-agent end-to-end variant, RepLLM, reproduces 95% of original benchmarks in roughly two hours with up to 10% lower token consumption; the body does not evaluate that variant, and its own reported average is 38.6 hours.

Load-bearing premise

The evaluation assumes that reproducing six systems that already have public open-source code, with participants who are already familiar with the papers and may guide repairs by hand, measures the framework's ability to generate code from paper text rather than the models' memory of existing code plus human repair.

Editorial extensions

If this is right

If the body's results hold, an LLM prompt pipeline can reproduce six heterogeneous network systems with 0–3.1% average error, making papers without public code viable reproduction targets.
Reproduction drops to an average of 38.6 hours and 32.2 human-written prompts—against the 294.5-prompt baseline for unstructured prompting—making error repair, not paper reading or code writing, the main bottleneck.
Semantic errors are the expensive class, taking about 100 minutes and 2.68 prompts per fix versus 37 minutes and 1.57 for syntactic errors, so future gains depend on automated semantic validation.
If the abstract's RepLLM variant delivers its claimed two-hour, 95% reproduction, the multi-agent coordination and shared-memory design would mark an order-of-magnitude improvement over the body's own 38.6-hour pipeline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A decisive test the paper leaves implicit: run the pipeline on papers whose code is not public and with participants unfamiliar with the work; without that, reported reproduction could be retrieval plus human repair.
The modular paper-to-code recipe likely transfers to other system-heavy fields with design-centric papers, such as operating systems, distributed databases, or programmable hardware.
Because semantic repair dominates cost, one natural extension is to close the loop on benchmark numbers: parse the paper's result tables and feed mismatches between reproduced and published values back into the audit stage automatically.
The large gap between the abstract's two-hour claim and the body's 38.6-hour average suggests the authors credit the multi-agent architecture, not prompt structure alone, for most of the speedup; a controlled ablation would show which.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this arXiv record has two different papers stapled together. The abstract advertises RepLLM, an end-to-end multi-agent framework with four specialized agents, Shared Memory, and a two-hour, 95%-benchmark, 10%-token-saving result. The full text describes RePro, a semi-automated prompt-engineering pipeline whose own evaluation reports 38.6 hours average, 32.2 human prompts, and never measures tokens. Figure 8's caption says 'RepLLM Architecture' but the diagram and surrounding text are RePro's stages. The headline numbers in the abstract cannot be checked against the body.

Now the fair part. The body does real work. RePro's four-step pipeline—system description extraction, framework-level code generation with SCoT, function-level generation with SeCoT, and iterative debugging—is a legitimate extension of the authors' HotNets'23 study and of published SCoT/SeCoT prompting. The evaluation covers six named network systems across traffic engineering, graph mining, transport protocols, and verification. The prompt counts (32.2 human-written, about 95 total) are substantially lower than the 294.5 in the prior work, and the error analysis distinguishing syntactic vs semantic repair is a useful datapoint for anyone building on this. That is a genuine empirical contribution, albeit a modest one.

The soft spots are serious. First, the abstract/body mismatch is not a typo—it is a load-bearing failure. If the arXiv record is the version of record, the paper claims results it does not present. Second, the evaluation has no baselines: no comparison against plain LLM prompting, no comparison against LASER or the HotNets'23 approach. Third, the systems are all open-source and well-known (DCTCP, HPCC, etc.), so LLM memorization and participant familiarity (explicitly admitted in Section 7.1) make it ambiguous whether the pipeline generates code from the paper or retrieves memorized code plus human repair. Fourth, Table 3 omits DCTCP and HPCC rows despite textual claims, and there are no error bars or statistical tests anywhere. No artifacts are released. These are not minor polish issues.

Who is this for? A reader interested in prompt engineering for systems code might skim the body for the template designs. But I would not cite it in its current form, and I would not bring it to reading group. The underlying idea has merit, but the submission is not honest as written. My recommendation: desk reject, with an invitation to resubmit a corrected manuscript whose abstract matches the body, adds baselines, releases artifacts, and addresses the memorization concern.

Referee Report

5 major / 7 minor

Summary. The manuscript as submitted presents RepLLM in its abstract, described as an end-to-end multi-agent framework with four specialized agents, a Shared Memory mechanism, and sandbox-isolated static-dynamic debugging, and it claims reproduction of 95% of original benchmarks within about two hours and up to 10% token reduction over state-of-the-art baselines. The full text, however, defines and evaluates a different system, RePro, a semi-automated prompt-engineering framework with a three-stage pipeline (system description extraction, framework-level and function-level code generation, and code optimization). The body's evaluation covers six open-source networking systems reproduced with five LLMs and reports an average reproduction time of 38.6 hours, an average of 32.2 human-written prompts per reproduction, and detailed syntactic/semantic error-repair analyses. No section of the body defines RepLLM's agents, Shared Memory, or sandbox debugging, and no measurement of token consumption or of a 95% benchmark-reproduction rate appears anywhere in the evaluation.

Significance. If the body's claims for RePro were rigorously established, this would be a useful empirical study of LLM-assisted reproduction of networking systems: the prompt templates in Figures 2–7 are concrete and reusable, and the breakdown of syntactic versus semantic error repair times and human prompt involvement (Results 4–7) is informative for practitioners building similar tools. However, the manuscript's headline contribution, as stated in the abstract, is entirely unsupported by the body: the framework name, architecture, and quantitative claims are different. The evaluation also has confounds that preclude attributing the observed behavior to the proposed framework rather than to LLM memorization of widely available open-source code plus substantial human repair. The paper does provide detailed prompt designs and a transparent error analysis, which I credit as useful material, but the central claims as written are not verifiable from the submitted text.

major comments (5)

[Abstract; §6.1; §7.3] The abstract's central claim describes RepLLM and reports "reproduce 95% of the original benchmarks within approximately two hours" and "reducing token consumption by up to 10% compared with state-of-the-art baselines," but the full text defines and evaluates RePro, a different framework. §7.3, Result 2 reports an average reproduction time of 38.6 hours, no section reports token consumption, and RepLLM's four specialized agents, Shared Memory mechanism, and sandbox-isolated static-dynamic debugging methodology are never defined in the body; the only occurrence of the name "RepLLM" in the body is the caption of Figure 8, which actually depicts RePro's pipeline. The abstract's headline numbers therefore cannot be checked or reproduced from the manuscript.
[§7.2; Table 3] Table 3 omits DCTCP and HPCC despite §7.2 claiming that "For NS-3 simulations, DCTCP and HPCC results align closely with the originals" and despite Figure 9 showing qualitative comparisons for both systems. The "Avg. Error (%)" metric in Table 3 is never defined, no number of runs or error bars is provided, and NCFlow's normalized latency of 13.031 is inconsistent with the text's statement that "most systems have normalized latency near 1.0." This undermines the accuracy claim embodied in Result 1 and makes the fidelity comparison for two of the six evaluated systems non-quantitative.
[§7.1; §5.3] The evaluation design does not separate the framework's contribution from LLM memorization and human repair. Participants were asked to reproduce papers with which they were "already familiar" (§7.1), all six selected systems have publicly available open-source implementations (Table 2), and semantic error correction explicitly uses a Human-in-the-Loop procedure in which humans identify the error location and design targeted prompts (§5.3). Given that the evaluated LLMs were likely trained on these widely available codebases, the reported reproductions may reflect retrieval of memorized code combined with iterative human guidance rather than generation from the paper text. A concrete remedying test would be to evaluate on papers without public code, with participants unfamiliar with the target paper, or with an additional condition that ablates the human repair component.
[§7.1–§7.5; Abstract] The evaluation contains no comparison to any state-of-the-art baseline framework, despite the abstract's claim of token reduction "compared with state-of-the-art baselines," and token consumption is never measured. The paper compares reproduced systems only against the original open-source implementations; it does not compare RePro with prior LLM-based reproduction approaches discussed in §3.3, such as the 294.5-prompt pipeline of [35] or LASER [33]. Without such a baseline, the reported 38.6 hours and 32.2 manual prompts cannot substantiate the claimed efficiency advantage, and the abstract's quantitative claim is unverifiable.
[§7.4; Results 4–7] Results 4–7 report precise quantitative values—for example, 100 versus 37 minutes per semantic/syntactic repair, 2.68 versus 1.57 prompts per repair, and 62.1% versus 26.3% human prompt proportions—but none of these numbers appear in any table and they cannot be derived from the figures, which use glyph-only labels and do not include axis legends. The regression in Figure 15 is described as having "error bands," yet no regression coefficients, confidence intervals, R² values, or sample sizes are reported. These omissions leave the paper's quantitative error-analysis conclusions unsupported.

minor comments (7)

[Title; Abstract] The full-text title is "RePro: Leveraging Large Language Models for Semi-Automated Reproduction of Networking Research Results," while the abstract describes RepLLM; the manuscript's own front matter is internally inconsistent about which system is being presented.
[§6.1; Figure 8] Figure 8's caption reads "RepLLM Architecture," but the diagram and the surrounding text describe RePro's multi-stage pipeline; the caption should be reconciled with the content.
[§7.2; Table 3] GRooT's "Absolute Latency" values are negative (-0.029 and -0.028), which is impossible for a latency; clarify whether this is a speedup, a different metric, or a labeling error.
[§7.3; Figure 10; Figure 11; Figure 12; Figure 14] Several evaluation figures (Figures 10–14) use encoded glyph labels in place of readable textual axis labels and legends, making the plots very difficult to interpret; plain-text labels and legends are needed.
[§1; §3.3; Appendix B] The contribution statement "First Semi-Automated Framework for Network Research Reproduction" appears to conflict with §3.3 and Appendix B, which describe LASER [33] as a semi-automated LLM-assisted reproduction framework, albeit for LEO satellite networks; the claim should be qualified to acknowledge this prior work.
[§7.1] The sentence "For DCTCP and HPCC, we use Cursor [2] to facilitate error correction" is inconsistent with the same section's statement that participants were "explicitly prohibited from manual coding" and with the advertised automation; the role of Cursor should be clarified.
[§8; §7.3] The conclusion claims that RePro "substantially reduces reproduction time," but the evaluation does not include a manual-reproduction baseline; a direct comparison with manual effort would be needed to support this claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found; the abstract/RepLLM body/RePro mismatch is a support failure, not a self-referential reduction.

full rationale

The paper's derivation chain—paper decomposition, SCoT/SeCoT-guided framework and function code generation, and error-feedback optimization—is not shown to reduce to its inputs by construction. The body reports measured reproduction outcomes (Result 2: 38.6 hours average; Result 3: 32.2 human-written prompts) rather than fitting parameters and relabeling them as predictions. The abstract's RepLLM claims (95% of benchmarks in about two hours, 10% token reduction, four agents with Shared Memory) are not reported in the body, which describes RePro and reports 38.6 hours and no token-consumption comparison; this is a serious internal-support and consistency problem but not a circularity. The Section 7.1 use of six open-source systems and participants 'already familiar' with the papers, plus the Section 5.3 human-in-the-loop semantic repair, threatens external validity and may let memorization or human repair drive results, but the reported quantities are measured outputs, not inputs renamed as predictions. The only overlapping-author citation ([35], HotNets '23) supplies motivation and a prompt-count baseline (294.5 versus 32.2) but is not load-bearing for the framework's derivation. Under the quoted-reduction standard, no circular step is exhibited.

Assumptions & free parameters 1 free parameters · 6 assumptions · 3 invented entities

The abstract's architecture and metrics are asserted without body support. The body's evaluation assumes LLM competence, open-source ground truth, unbiased participant familiarity, and that human-in-the-loop repair is not the primary driver. The structured program theorem is a standard background assumption.

free parameters (1)

Few-shot example count = 3
The prompt templates in Figures 4-7 each provide three requirement/SCoT/code examples; this number is chosen by hand and no ablation tests its effect on reproduction quality.

assumptions (6)

domain assumption LLMs can generate executable network systems code from paper descriptions when guided by structured prompts.
Invoked throughout Sections 3.2, 5, and 7; no proof or controlled test establishes that the paper text alone is sufficient input.
domain assumption Open-source implementations of the six selected systems are the correct ground truth for reproduction fidelity.
Section 7.1 selects systems 'since all selected systems provide open-source code'; the evaluation assumes these prototypes define the correct behavior.
domain assumption Participant familiarity with the papers does not bias the reproduction outcome.
Section 7.1 tasks each participant with reproducing a paper with which they were 'already familiar', but the analysis does not control for prior knowledge.
domain assumption Except for the human-in-the-loop semantic fixes in Section 5.3, the framework is the driver of the reproduction.
The evaluation counts 32.2 human-written prompts and 50.4% human prompts for semantic errors, so human effort is substantial; the contribution of the automated pipeline alone is not isolated.
standard math The structured program theorem (Bohm and Jacopini) justifies restricting generated frameworks to sequential, conditional, and iterative control structures.
Section 5.1 cites [11]; this is a standard theorem and is not a point of failure, but it is a background assumption for SCoT.
ad hoc to paper The abstract's RepLLM architecture with Shared Memory and sandbox-isolated debugging exists as described.
The abstract asserts these components, but the body does not specify or evaluate them; if we take the abstract's claim at face value, we must assume an undescribed system.

invented entities (3)

Shared Memory mechanism
purpose: Coordinate the four specialized agents to maintain global context consistency
Mentioned only in the abstract; no interface, implementation, or evaluation appears in the body.
Four specialized agents (Content Parsing, Architecture Design, Code Generation, Audit and Repair)
purpose: Collaboratively decompose and execute the reproduction workflow
Described only in the abstract; the body instead presents a three-module RePro pipeline and never names these agents.
Sandbox-isolated static-dynamic debugging methodology
purpose: Resolve semantic discrepancies and runtime errors during reproduction
Only the abstract mentions this debugging methodology; the body's optimization section describes test-driven and human-in-the-loop repair without a sandbox.

how reviews work

0 comments

Cite this review

Pith. "Pith review of RepLLM: Toward Automatically Reproducing Network Research Results." pith.science (2026). https://pith.science/paper/IXK227EW

@misc{pith2026250921074,
  author       = {Pith},
  title        = {Pith review of: RepLLM: Toward Automatically Reproducing Network Research Results},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IXK227EW}},
  note         = {Machine review of arXiv:2509.21074}
}

read the original abstract

Result reproduction of computer networking research is challenging as the scarcity of open-source implementations and the complexity of heterogeneous system architectures. Even though Large Language Models have demonstrated potential in code generation, existing code generation frameworks often fail to address the long-context constraints and intricate logical dependencies, which are vital in reproducing network systems from academic papers. Thus, we introduce RepLLM, an end-to-end multi-agent framework designed to automate code reproduction from paper content. RepLLM features a collaborative architecture comprising four specialized agents -- Content Parsing, Architecture Design, Code Generation, and Audit&Repair, which are coordinated through Shared Memory mechanism to ensure global context consistency. With the enhancement of Structured Chain-of-Thought LLM reasoning and a sandbox-isolated static-dynamic debugging methodology, our framework effectively resolves semantic discrepancies and runtime errors, thereby improving reliable reproductions. Extensive evaluations on representative papers in top conferences demonstrate that RepLLM outperforms state-of-the-art system-level LLM frameworks in generating compile-ready and logically correct systems. Our results show that, with the aid of RepLLM, we can reproduce 95% of the original benchmarks within approximately two hours while reducing token consumption by up to 10% compared with state-of-the-art baselines.

Figures

Figures reproduced from arXiv: 2509.21074 by the authors.

**Figure 3.** illustrates our prompt template module information extraction. The prompt template explicitly forces the LLM to divide a system into relatively independent functional modules (steps), each provides both a brief and detailed description with required inputs and produced outputs. Moreover, the placeholder “UNKNOWN” is used to avoid LLM hallucinations caused by inaccurate information. 𝐿𝐿𝑀 𝑦ො: { "Step1": { "brief_des… view at source ↗

**Figure 2.** Prompt engineering for extracting basic information. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Figures from the paper (11 more)

**Figure 5.** Figure 5: Prompt engineering for generating a framework-level code. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: Prompt engineering for mapping paper content. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: Prompt engineering for generating function code. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

**Figure 8.** Figure 8: RepLLM Architecture. Code Generation, which incrementally produces modulelevel implementations. Finally, the generated modules are integrated into a complete, reproducible network system. 6.2 System Description Extraction Preprocessing module contextualizes the LLM fo…

**Figure 9.** Figure 9: Overall comparison of DCTCP and HPCC performance. [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]

**Figure 11.** Figure 11: Number of prompts of different systems prompts and DCTCP the fewest. GPT-o1 used fewer prompts than GPT-4o on GRooT, GPT-4o outperformed DeepSeek on Tiramisu, and Claude outperformed GPT-4o on Arya. Overall, GPT-o1 and Claude most effectively reduced human interventi…

**Figure 10.** Figure 10: Reproduction time of different systems Moreover, DCTCP required the least time due to its simpler design, whereas GRooT took the longest with complex formulas. GPT-o1 reproduced GRooT slightly faster than GPT-4o by spending less time on exception handling, while Clau…

**Figure 12.** Figure 12: Comparison of error counts and repair time across systems. [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]

**Figure 13.** Figure 13: Distribution of error counts and repair prompts across systems. [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]

**Figure 15.** Figure 15: Regression relationship between exception repair time and number [PITH_FULL_IMAGE:figures/full_fig_p012_15.png]

**Figure 16.** Figure 16: Comparison between CoT, SCoT and SeCoT B Related Work Reproducing scientific papers by LLMs has already seen notable progress within the field of AI. Xiang et al. [36] introduce SciReplicate-Bench, a benchmark designed to evaluate LLMs’ ability to reproduce NLP algor…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 38 canonical work pages

[35]

Xiang, Y

Q. Xiang, Y . Lin, M. Fang, B. Huang, S. Huang, R. Wen, F. Le, L. Kong, and J. Shu. Toward reproducing net- work research results using large language models. In Proceedings of the 22nd ACM Workshop on Hot Topics in Networks, HotNets ’23, page 56–62, New York, NY , USA, 2023. Association for Computing Machinery

work page 2023
[33]

Y . Wang, Y . Hou, Z. Lai, H. Li, Q. Wu, J. Liu, Y . Li, X. Xie, and Z. Han. How llm saved me from struggling with experiment reproduction: Leo networking as a case study. InProceedings of the 2025 3rd Workshop on LEO Networking and Communication, LEO-NET ’25, page 1–7, New York, NY , USA, 2025. Association for Computing Machinery

work page 2025
[1]

ChatGPT.https://chat.openai.com/

work page
[2]

Cursor - The AI Code Editor.https://cursor.com/

work page
[3]

https://github.com/features/ copilot/

GitHub Copilot. https://github.com/features/ copilot/

work page
[4]

https://github

Open-source Prototype of Flash. https://github. com/snlab/flash

work page
[5]

PyTorch: An Imperative Style, High-Performance Deep Learning Library.https://pytorch.org/

work page
[6]

Abadi, P

M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V . Vasudevan, P. Warden, M. Wicke, Y . Yu, and X. Zheng. Tensorflow: a sys- tem for large-scale machine learning. InProceedings of the 12th USENIX Conference on Operating S...

work page 2016

Show all 52 references

[7]

Abhashkumar, A

A. Abhashkumar, A. Gember-Jacobson, and A. Akella. Tiramisu: Fast multilayer network verification. In17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20), pages 201–219, Santa Clara, CA, Feb. 2020. USENIX Association

2020
[8]

Abuzaid, S

F. Abuzaid, S. Kandula, B. Arzani, I. Menache, M. Za- haria, and P. Bailis. Contracting wide-area network topologies to solve flow problems quickly. In18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21), pages 175–200. USENIX Association, Apr. 2021

2021
[9]

Alizadeh, A

M. Alizadeh, A. Greenberg, D. A. Maltz, J. Padhye, P. Patel, B. Prabhakar, S. Sengupta, and M. Sridharan. Data center tcp (dctcp). volume 40, page 63–74, New York, NY , USA, Aug. 2010. Association for Computing Machinery

2010
[10]

Beurer-Kellner, M

L. Beurer-Kellner, M. Fischer, and M. Vechev. Prompt- ing is programming: A query language for large lan- guage models.Proc. ACM Program. Lang., 7(PLDI), June 2023

2023
[11]

Böhm and G

C. Böhm and G. Jacopini. Flow diagrams, turing ma- chines and languages with only two formation rules. Commun. ACM, 9(5):366–371, May 1966

1966
[12]

H. Chen, Y . Miao, L. Chen, H. Sun, H. Xu, L. Liu, G. Zhang, and W. Wang. Software-defined network assimilation: bridging the last mile towards centralized network configuration management with nassim. In Proceedings of the ACM SIGCOMM 2022 Conference, SIGCOMM ’22, page 281–29...

2022
[13]

W. Chen, X. Ma, X. Wang, and W. W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.arXiv preprint arXiv:2211.12588, 2022

2022 arXiv
[14]

Z. Chen, K. Zhou, B. Zhang, Z. Gong, W. X. Zhao, and J.-R. Wen. Chatcot: Tool-augmented chain-of-thought reasoning on chat-based large language models.arXiv preprint arXiv:2305.14323, 2023

2023 arXiv
[15]

L. Cui, Y . Wu, J. Liu, S. Yang, and Y . Zhang. Template- based named entity recognition using bart.arXiv preprint arXiv:2106.01760, 2021

2021 arXiv
[16]

Devlin, M.-W

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio, editors,Proceedings of the 2019 Con- ference of the North American Chapter of the Associa- tion for Com...

2019
[17]

Q. Dong, L. Li, D. Dai, C. Zheng, J. Ma, R. Li, H. Xia, J. Xu, Z. Wu, T. Liu, et al. A survey on in-context learning.arXiv preprint arXiv:2301.00234, 2022

2022 arXiv
[18]

J. Gu, Z. Han, S. Chen, A. Beirami, B. He, G. Zhang, R. Liao, Y . Qin, V . Tresp, and P. Torr. A systematic survey of prompt engineering on vision-language foun- dation models.arXiv preprint arXiv:2307.12980, 2023

2023 arXiv
[19]

Handigol, B

N. Handigol, B. Heller, V . Jeyakumar, B. Lantz, and N. McKeown. Reproducible network experiments using container-based emulation. InProceedings of the 8th In- ternational Conference on Emerging Networking Exper- iments and Technologies, CoNEXT ’12, page 253–264, New York, NY ...

2012
[20]

Jiang, K

J. Jiang, K. Zhou, Z. Dong, K. Ye, W. X. Zhao, and J.-R. Wen. Structgpt: A general framework for large language model to reason over structured data.arXiv preprint arXiv:2305.09645, 2023

2023 arXiv
[21]

S. K. R. Kakarla, R. Beckett, B. Arzani, T. Millstein, and G. Varghese. Groot: Proactive verification of dns config- urations. InProceedings of the Annual Conference of the ACM Special Interest Group on Data Communication on the Applications, Technologies, Architectures, and P...

2020
[22]

Lester, R

B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameter-efficient prompt tuning.arXiv preprint arXiv:2104.08691, 2021

2021 arXiv
[23]

J. Li, G. Li, Y . Li, and Z. Jin. Structured chain-of-thought prompting for code generation.ACM Trans. Softw. Eng. Methodol., 34(2), Jan. 2025

2025
[24]

Y . Li, R. Miao, H. H. Liu, Y . Zhuang, F. Feng, L. Tang, Z. Cao, M. Zhang, F. Kelly, M. Alizadeh, and M. Yu. Hpcc: high precision congestion control. InProceedings of the ACM Special Interest Group on Data Communica- tion, SIGCOMM ’19, page 44–58, New York, NY , USA,
[25]

Z. Lin, Y . Shen, Q. Cai, H. Sun, J. Zhou, and M. Xiao. Autop2c: An llm-based agent framework for code repos- itory generation from multimodal content in academic papers.arXiv preprint arXiv:2504.20115, 2025

2025 arXiv
[26]

Y . Luo, L. Shi, Y . Li, A. Zhuang, Y . Gong, L. Liu, and C. Lin. From intention to implementation: automating biomedical research via llms.Science China Informa- tion Sciences, 68(7):1–18, 2025

2025
[27]

Y . Ma, Y . Yu, S. Li, Y . Jiang, Y . Guo, Y . Zhang, Y . Xie, and X. Liao. Bridging code semantic and llms: Semantic chain-of-thought prompting for code generation.arXiv preprint arXiv:2310.10698, 2023

2023 arXiv
[28]

Petroni, T

F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y . Wu, A. H. Miller, and S. Riedel. Language models as knowl- edge bases?arXiv preprint arXiv:1909.01066, 2019

1909 arXiv
[29]

Rahmani, M

K. Rahmani, M. Raza, S. Gulwani, V . Le, D. Morris, A. Radhakrishna, G. Soares, and A. Tiwari. Multi-modal program inference: a marriage of pre-trained language models and component-based synthesis.Proc. ACM Program. Lang., 5(OOPSLA), Oct. 2021

2021
[30]

Schach.Software Engineering

S. Schach.Software Engineering. Aksen Associates series in electrical and computer engineering. Aksen Associates, 1990

1990
[31]

M. Seo, J. Baek, S. Lee, and S. J. Hwang. Paper2code: Automating code generation from scientific papers in machine learning.arXiv preprint arXiv:2504.17192, 2025

2025
[32]

F. Shi, X. Chen, K. Misra, N. Scales, D. Dohan, E. H. Chi, N. Schärli, and D. Zhou. Large language mod- els can be easily distracted by irrelevant context. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors,Proceedings of the 40th Internatio...

2023
[34]

J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou. Chain-of- thought prompting elicits reasoning in large language models. 2022

2022
[36]

Xiang, H

Y . Xiang, H. Yan, S. Ouyang, L. Gui, and Y . He. Scireplicate-bench: Benchmarking llms in agent-driven algorithmic reproduction from research papers.arXiv preprint arXiv:2504.00255, 2025

2025 arXiv
[37]

Yan and N

L. Yan and N. McKeown. Learning networking by repro- ducing research results.SIGCOMM Comput. Commun. Rev., 47(2):19–26, May 2017

2017
[38]

Yang and S

H. Yang and S. S. Lam. Real-time verification of net- work properties using atomic predicates.IEEE/ACM Transactions on Networking, 24(2):887–900, 2016

2016
[39]

J. Yen, T. Lévai, Q. Ye, X. Ren, R. Govindan, and B. Raghavan. Semi-automated protocol disambiguation and code generation. SIGCOMM ’21, page 272–286, New York, NY , USA, 2021. Association for Computing Machinery

2021
[40]

Zhang, J

J. Zhang, J. P. Cambronero, S. Gulwani, V . Le, R. Piskac, G. Soares, and G. Verbruggen. Pydex: Repairing bugs in introductory python assignments using llms.Proc. ACM Program. Lang., 8(OOPSLA1), Apr. 2024

2024
[41]

Zhang, D

J. Zhang, D. Li, J. C. Kolesar, H. Shi, and R. Piskac. Automated feedback generation for competition-level code. ASE ’22, New York, NY , USA, 2023. Association for Computing Machinery

2023
[42]

Zhang, X

P. Zhang, X. Liu, H. Yang, N. Kang, Z. Gu, and H. Li. APKeep: Realtime verification for real networks. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20), pages 241–255, Santa Clara, CA, Feb. 2020. USENIX Association

2020
[43]

J. X. Zhao, Y . Xie, K. Kawaguchi, J. He, and M. Q. Xie. Automatic model selection with large language models for reasoning.arXiv preprint arXiv:2305.14333, 2023. 14

2023 arXiv
[44]

Z. Zhu, K. Wu, and Z. Liu. Arya: Arbitrary graph pattern mining with decomposition-based sampling. In20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 1013–1030, Boston, MA, Apr. 2023. USENIX Association. 15 A Comparison of Different CoT Unl...

2023
[47]

Create a List to Store Odd Numbers
[48]

Use a `for-each` loop to traverse each integer in the input array
[49]

Check if the number is odd, If true, add the number to the `oddNumbers` list
[50]

No odd numbers found

Check if the `oddNumbers` list is empty. If empty, print: `"No odd numbers found."`. Otherwise, print: `"Odd numbers: " + oddNumbers`, which leverages the `toString()` method of the list for formatting. Input: numbers: an array of integers Output: odd_numbers: a list of intege...
[51]

Iterate over each element in the array numbers: Check if the number is odd (num % 2 != 0)

Control Flow: Start → Call findOddNumbers(int[] numbers) Initialize an empty list oddNumbers to store the odd numbers. Iterate over each element in the array numbers: Check if the number is odd (num % 2 != 0). If true, add the number to oddNumbers. After iterating through all ...
[52]

Process: Each element of the array is checked for its odd/even property

Data Flow: Input: An array of integers numbers. Process: Each element of the array is checked for its odd/even property. Odd numbers are added to the oddNumbers list. Output: A list of integers containing only the odd numbers from the input array
[53]

The modular design ensures clarity, scalability, and testability, making it suitable for realworld applications

Finally： This implementation effectively filters odd numbers from an input array. The modular design ensures clarity, scalability, and testability, making it suitable for realworld applications. SeCoT Figure 16: Comparison between CoT, SCoT and SeCoT B Related Work Reproducing...
[2019]

Association for Computing Machinery

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [35]

Xiang, Y

Q. Xiang, Y . Lin, M. Fang, B. Huang, S. Huang, R. Wen, F. Le, L. Kong, and J. Shu. Toward reproducing net- work research results using large language models. In Proceedings of the 22nd ACM Workshop on Hot Topics in Networks, HotNets ’23, page 56–62, New York, NY , USA, 2023. Association for Computing Machinery

work page 2023

[2] [33]

Y . Wang, Y . Hou, Z. Lai, H. Li, Q. Wu, J. Liu, Y . Li, X. Xie, and Z. Han. How llm saved me from struggling with experiment reproduction: Leo networking as a case study. InProceedings of the 2025 3rd Workshop on LEO Networking and Communication, LEO-NET ’25, page 1–7, New York, NY , USA, 2025. Association for Computing Machinery

work page 2025

[3] [1]

ChatGPT.https://chat.openai.com/

work page

[4] [2]

Cursor - The AI Code Editor.https://cursor.com/

work page

[5] [3]

https://github.com/features/ copilot/

GitHub Copilot. https://github.com/features/ copilot/

work page

[6] [4]

https://github

Open-source Prototype of Flash. https://github. com/snlab/flash

work page

[7] [5]

PyTorch: An Imperative Style, High-Performance Deep Learning Library.https://pytorch.org/

work page

[8] [6]

Abadi, P

M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V . Vasudevan, P. Warden, M. Wicke, Y . Yu, and X. Zheng. Tensorflow: a sys- tem for large-scale machine learning. InProceedings of the 12th USENIX Conference on Operating S...

work page 2016