REVIEW 3 major objections 4 minor 33 references
LLM-Based Config Synthesis requires Disambiguation
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read LLM-generated route-map and ACL updates are often ambiguous because overlapping rules make insertion order underdetermined; Clarify resolves this by asking users targeted differential behavioral questions.
desk verdict Real problem and useful overlap data, but the formal conditions in Section 4 don't actually guarantee a single insertion point, and the binary-search disambiguator is never exercised in the evaluation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The Disambiguator is the load-bearing component. Given a verified new stanza and an existing ordered configuration, it determines possible insertion positions by comparing behaviors of candidate configurations, generating differential behavioral examples—concrete input routes or packets that are treated differently depending on where the stanza is placed—and asking the user which behavior matches their intent. Because the user's intended update is assumed to be expressible as a single inserted rule satisfying the three conditions, the problem reduces to locating one insertion point in an ordered list of interacting rules, and binary search over the candidate rules yields the answer with a logarithmic number of user queries. Verification of the stanza in isolation is done with existing ACL and route-map analysis methods before insertion, so disambiguation only has to resolve ordering, not the stanza's own correctness.
What would settle it
Run Clarify on a corpus of real production route-map and ACL change requests. If a substantial fraction of requests cannot be expressed as a single inserted rule satisfying the three conditions, or if operators presented with the generated differential examples frequently pick an option that does not match their stated intent, the central claim that disambiguation-by-binary-search resolves the problem would be refuted.
Extended reading notes
Core claim
The central claim is that incremental synthesis of routing policies and access-control lists with LLMs fails unless the system disambiguates where a newly generated stanza belongs, because route-maps and ACLs are ordered rule lists whose rules often overlap in header space. The paper formalizes a single-rule update as one new rule $S^*$ inserted into an existing list $S$, with the intended semantics $M'$ satisfying three conditions: every input is either handled as before or by the new rule; inputs handled by the new rule must match it; and there is a single insertion point consistent with the intended behavior. Given these conditions, the Disambiguator collects the rules that potentially interact with $S^*$, shows the user a differential example between two candidate insertion positions, and uses binary search to locate the correct position with a logarithmic number of questions. The paper's measurements of 237 ACLs (69 with overlaps, one with over 100 overlapping rule pairs) and 800 route-maps (140 with overlaps) support the premise that ambiguity is a practical problem, and the synthetic evaluation on a three-router topology shows the workflow can produce verified configurations end to end.
Load-bearing premise
The load-bearing assumption is that every intended update can be represented as a single new rule that either leaves each route's handling unchanged or hands it to the new rule, with one insertion point that realizes the intended behavior, and that users can reliably choose the correct option from the differential examples.
Editorial extensions
If this is right
- If disambiguation is accepted as necessary, LLM-based config synthesis tools should include an interactive clarification phase rather than emitting a single config from a prompt.
- The binary-search disambiguation algorithm means the number of user questions grows only logarithmically with the number of interacting rules, so even configs with hundreds of overlaps remain practical to clarify.
- The same disambiguation step is needed for manual configuration updates, not just LLM-generated stanzas, because the insertion-order ambiguity exists independent of the generator.
- The approach generalizes to other program updates where the new code is correct in isolation but its placement in an existing program determines global behavior.
Reading between the lines
- A natural extension would be to test Clarify against real operator change requests from production networks, since the paper's evaluation is limited to a small synthetic topology; that test would reveal whether real intents satisfy the single-rule conditions.
- The disambiguator's interface could be made harder to misread by presenting multiple differential examples at once, because a single example may not isolate the dimension of intent the user cares about; this is a usability question the paper leaves open.
- The binary-search approach could be adapted to other ordered structures beyond route-maps and ACLs, such as firewall rule sets or compiler optimization passes, wherever overlapping match conditions make insertion order consequential.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that LLM-based incremental synthesis of network configuration stanzas (route-maps and ACLs) faces a distinct problem beyond hallucination: even when an LLM correctly generates a stanza in isolation, the stanza's placement in an existing ordered list of rules is ambiguous because rules frequently overlap in header space. The paper offers overlap statistics from a cloud provider and a campus network, formalizes the disambiguation problem in Section 4, and proposes Clarify, a prototype that verifies LLM-generated stanzas and asks users differential behavioral questions to determine insertion placement. The evaluation is a small synthetic three-router topology, and the authors explicitly describe the work as a toy demonstration in Section 7.
Significance. If the central claim holds, the paper identifies a real and underexplored bottleneck in LLM-based configuration synthesis and provides a reasonable starting point for addressing it. The strengths are the clear problem statement, the use of real-network overlap measurements, the crisp (though currently flawed) formal model, and the candid acknowledgment of limitations. However, the formalization in Section 4 has a correctness gap, and the evaluation in Section 5 does not exercise the proposed binary-search algorithm; as written, the evidence is not yet sufficient to support the claim that Clarify resolves insertion ambiguity in general.
major comments (3)
- The non-strict inequality in condition 3 is insufficient to guarantee that a single insertion point can realize the intended semantics M'. Consider S=[A,B] and let S* match exactly the inputs that A matches. Define M' so that one such input is handled by A (kept) and another is handled by S* (new). Then conditions 1 and 2 hold, and condition 3 holds because M(r)=M(r')=A for the two inputs, so M(r)<=M(r') is true. Yet no insertion point realizes M': inserting S* before A makes both inputs use S*, while inserting after A keeps both on A. The condition should use strict inequality (M(r)<M(r')) or an additional constraint preventing the same original rule from appearing on both sides; with strict inequality the condition does characterize realizable single-insertion semantics. This is load-bearing because the binary-search correctness argument and the claim that Clarify resolves ambiguity both rest on this characterization.
- The evaluation does not validate the Section 4 algorithm. Section 2.2 states that the prototype only supports inserting a stanza at the top or bottom of the initial route-map, so the 17 reported disambiguation choices (5, 6, and 6 for routers M, R1, R2) exercise only the two-location special case, not the general binary search over arbitrary insertion positions described in Section 4. In addition, the evaluation has no baseline comparison (e.g., LLM-only insertion without disambiguation, random insertion, or human expert placement), and Section 7 concedes the experiment is a toy demonstration. The central claim that Clarify resolves insertion ambiguity therefore goes beyond what the current evidence supports.
- The overlap measurements are presented without error analysis, confidence intervals, or a precise statement of what an 'overlap count' counts (number of overlapping pairs? number of rules involved?). For route-maps, the overlap definition deliberately ignores actions and is an upper bound on behavioral ambiguity, so the reported 140/800 route-maps with overlaps do not directly establish that insertion ambiguities would occur in practice. For campus ACLs, the fraction with conflicting overlaps drops from 37.7% to 18.6% when proper-subset matches are excluded, and the cloud ACL numbers do not provide the corresponding breakdown. The connection from overlap frequency to actual disambiguation need is asserted rather than measured, for instance by simulating realistic incremental updates over the measured configurations.
minor comments (4)
- There are several typographical errors, including 'LLLLM' instead of 'LLM', 'to to produce' instead of 'to produce', and 'preferen ce' in the example route-map.
- The Related Work text contains 'os then filled' instead of 'is then filled', and reference [19] appears to have incorrect author formatting ('Sean Welleck Pranjal Aggarwal, Bryan Parno').
- The notation in condition 3 should make explicit that M(r) and M(r') are compared by their order in the original list S, and that the consequent uses M(r'), the original rule handling r', not the new rule S*; the surrounding prose can be read either way.
- The caption could state the totals (33 LLM calls and 17 disambiguation choices) to make the scale of the evaluation immediately clear.
Circularity Check
No circularity: the disambiguation algorithm treats user answers as ground truth and never derives a prediction from its own outputs.
full rationale
The paper's load-bearing claims are (i) that route-maps/ACLs frequently overlap, (ii) that overlap makes insertion-point choice ambiguous, and (iii) that Clarify's user-in-the-loop differential questioning resolves that ambiguity. Claim (i) is supported by direct measurement (Section 3), not by a fitted model or by a theorem imported from the authors' prior work. Claim (ii) follows from the structural definition of leftmost-match semantics in Section 4, where an inserted S* can change behavior only on inputs it matches; the existence of multiple insertion points with different behaviors is an observable property, not a derived prediction. Claim (iii) is implemented by asking the user to select the desired behavior and then verifying the resulting configuration with Batfish against a JSON specification that the user manually approves (Section 2.1); user choice is independent ground truth. The self-citations ([18], [26]) supply a synthesis-with-verification loop and a Lightyear-style decomposition for the synthetic workload; neither is the sole justification for the disambiguation result. The paper's own limitations—top/bottom-only insertion in Section 2.2, the toy demonstration in Section 7, and the Section 4 condition that a single insertion point exists—are explicit and are correctness/scope risks, not circular reductions. In particular, the non-strict inequality in the third formal condition may fail to characterize single-insertion intents, but that is a formal adequacy gap, not an input-output equivalence.
Assumptions & free parameters
assumptions (4)
- domain assumption Route-maps and ACLs are ordered lists of rules with leftmost-match (first-match) semantics and an implicit final deny.
- domain assumption The user's intended behavior M' satisfies the three conditions in Section 4: unchanged routes keep their old rule, new-rule routes match S*, and an ordering condition ensures a single insertion point exists.
- domain assumption Batfish's searchRoutePolicies, searchFilters, and compareRoutePolicies correctly and completely identify behavioral differences and can generate representative example inputs.
- domain assumption Users can correctly and consistently choose between the differential examples to express their true intent.
Cite this review
Pith. "Pith review of LLM-Based Config Synthesis requires Disambiguation." pith.science (2026). https://pith.science/paper/3JLDHYB6
@misc{pith2026250712443,
author = {Pith},
title = {Pith review of: LLM-Based Config Synthesis requires Disambiguation},
year = {2026},
howpublished = {\url{https://pith.science/paper/3JLDHYB6}},
note = {Machine review of arXiv:2507.12443}
}
read the original abstract
Beyond hallucinations, another problem in program synthesis using LLMs is ambiguity in user intent. We illustrate the ambiguity problem in a networking context for LLM-based incremental configuration synthesis of route-maps and ACLs. These structures frequently overlap in header space, making the relative priority of actions impossible for the LLM to infer without user interaction. Measurements in a large cloud identify complex ACLs with 100's of overlaps, showing ambiguity is a real problem. We propose a prototype system, Clarify, which uses an LLM augmented with a new module called a Disambiguator that helps elicit user intent. On a small synthetic workload, Clarify incrementally synthesizes routing policies after disambiguation and then verifies them. Our treatment of ambiguities is useful more generally when the intent of updates can be correctly synthesized by LLMs, but their integration is ambiguous and can lead to different global behaviors.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Anubhavnidhi Abhashkumar, Aaron Gember-Jacobson, and Aditya Akella. 2020. AED: incrementally synthesizing policy-compliant and manageable configurations. In Proceedings of the 16th International Conference on Emerging Networking EXperiments and Technologies (Barcelona, Spain) (CoNEXT ’20). Association for Computing Machin- ery, New York, NY, USA, 482–495....
-
[2]
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. arXiv:2310.11511 [cs.CL] https: //arxiv.org/abs/2310.11511
arXiv 2023
-
[3]
Ryan Beckett, Ratul Mahajan, Todd Millstein, Jitendra Padhye, and David Walker. 2016. Don’t Mind the Gap: Bridging Network-wide Objectives and Device-level Configurations. In Proceedings of the 2016 ACM SIGCOMM Conference (Florianopolis, Brazil) (SIGCOMM ’16). Association for Computing Machinery, New York, NY, USA, 328–341. https://doi.org/10.1145/2934872.2934909
-
[4]
Ryan Beckett, Ratul Mahajan, Todd Millstein, Jitendra Padhye, and David Walker. 2017. Network configuration synthesis with abstract topologies. SIGPLAN Not. 52, 6 (June 2017), 437–451. https://doi.org/ 10.1145/3140587.3062367
arXiv 2017
-
[5]
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. 2025. From Local to Global: A Graph RAG Approach to Query-Focused Summarization. arXiv:2404.16130 [cs.CL] https://arxiv.org/abs/2404.16130
arXiv 2025
-
[6]
Ahmed El-Hassany, Petar Tsankov, Laurent Vanbever, and Martin T. Vechev. 2017. Network-Wide Configuration Synthesis. In Computer Aided Verification - 29th International Conference, CA V 2017, Heidelberg, Germany, July 24-28, 2017, Proceedings, Part II (Lecture Notes in Com- puter Science, Vol. 10427), Rupak Majumdar and Viktor Kuncak (Eds.). Springer, 261...
- [7]
-
[8]
Yunfan Gao, Yun Xiong, Meng Wang, and Haofen Wang. 2024. Modular RAG: Transforming RAG Systems into LEGO-like Reconfigurable Frameworks. arXiv:2407.21059 [cs.CL] https://arxiv.org/abs/2407. 21059
arXiv 2024
Show all 33 references
-
[10]
Zhenbei Guo, Fuliang Li, Jiaxing Shen, Tangzheng Xie, Shan Jiang, and Xingwei Wang. 2024. ConfigReco: Network Configuration Recom- mendation With Graph Neural Networks. IEEE Network 38, 1 (2024), 7–14. https://doi.org/10.1109/MNET.2023.3336239
2024
- [11]
-
[12]
Beni Ifland, Elad Duani, Rubin Krief, Miro Ohana, Aviram Zilber- man, Andres Murillo, Ofir Manor, Ortal Lavi, Hikichi Kenji, Asaf Shabtai, Yuval Elovici, and Rami Puzis. 2024. GeNet: A Multi- modal LLM-Based Co-Pilot for Network Topology and Configuration. arXiv:2407.08249 [cs...
2024 arXiv
-
[13]
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela
-
[14]
Fuliang Li, Haozhi Lang, Jiajie Zhang, Jiaxing Shen, and Xingwei Wang. 2024. PreConfig: A Pretrained Model for Automating Network Configuration. arXiv:2403.09369 [cs.NI] https://arxiv.org/abs/2403. 09369
2024 arXiv
-
[15]
Smith, and Yejin Choi
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2023. We’re Afraid Language Models Aren’t Modeling Ambigu- ity. In Proceedings of the 2023 Conference on Empirical Methods in Nat- ural Language...
2023 doi
-
[16]
Jianmin Liu, Li Chen, Dan Li, and Yukai Miao. 2025. CEGS: Configu- ration Example Generalizing Synthesizer. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25) . USENIX Association, Philadelphia, PA, 1327–1347. https://www.usenix.org/ conference/n...
2025
-
[17]
Declan McCullagh. [n. d.]. How Pakistan knocked YouTube offline (and how to make sure it never happens again)
-
[18]
Rajdeep Mondal, Alan Tang, Ryan Beckett, Todd Millstein, and George Varghese. 2023. What do LLMs need to Synthesize Correct Router Con- figurations? (HotNets ’23). Association for Computing Machinery, New York, NY, USA, 189–195. https://doi.org/10.1145/3626111.3628194
2023
-
[19]
Sean Welleck Pranjal Aggarwal, Bryan Parno. 2024. AlphaVerus: Boot- strapping Formally Verified Code Generation through Self-Improving Translation and Treefinement. arXiv:2405.19616 [cs.AI] https: //arxiv.org/abs/2412.06176
2024 arXiv
-
[20]
Sivaramakrishnan Ramanathan, Ying Zhang, Mohab Gawish, Yogesh Mundada, Zhaodong Wang, Sangki Yun, Eric Lippert, Walid Taha, Minlan Yu, and Jelena Mirkovic. 2023. Practical Intent-driven Routing Configuration Synthesis. In 20th USENIX Symposium on Networked Systems Design and I...
2023
-
[21]
Tibor Schneider, Rüdiger Birkner, and Laurent Vanbever. 2021. Snow- cap: synthesizing network-wide configuration updates. In Proceedings 7 Rajdeep Mondal, Nikolaj Bjorner, Todd Millstein, Alan Tang, and George Varghese of the 2021 ACM SIGCOMM 2021 Conference (Virtual Event, US...
2021
-
[22]
Anastasiia Sedova, Robert Litschko, Diego Frassinelli, Benjamin Roth, and Barbara Plank. 2024. To Know or Not To Know? Analyz- ing Self-Consistency of Large Language Models under Ambiguity. arXiv:2407.17125 [cs.CL] https://arxiv.org/abs/2407.17125
2024 arXiv
-
[23]
Aditi Singh, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. 2025. Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG. arXiv:2501.09136 [cs.AI] https://arxiv.org/abs/2501.09136
2025 arXiv
-
[24]
Chuyue Sun, Ying Sheng, Oded Padon, and Clark Barrett. 2024. Clover: Closed-Loop Verifiable Code Generation. In Proceedings of the First International Symposium on AI Verification (SAIV ’24) . Springer-Verlag, 134–155. https://doi.org/10.1007/978-3-031-65112-0_7 Montreal, Canada
2024 doi
-
[25]
Wong, and Hongyi Zeng
Yu-Wei Eric Sung, Xiaozheng Tie, Starsky H.Y. Wong, and Hongyi Zeng. 2016. Robotron: Top-down Network Management at Facebook Scale. In Proceedings of the 2016 ACM SIGCOMM Conference (Floria- nopolis, Brazil) (SIGCOMM ’16). Association for Computing Machin- ery, New York, NY, U...
2016 doi
-
[26]
Alan Tang, Ryan Beckett, Steven Benaloh, Karthick Jayaraman, Tejas Patil, Todd Millstein, and George Varghese. 2023. Lightyear: Using Modularity to Scale BGP Control Plane Verification. In Proceedings of the ACM SIGCOMM 2023 Conference (ACM SIGCOMM ’23) . ACM. https://doi.org/...
2023
-
[27]
Changjie Wang, Mariano Scazzariello, Alireza Farshin, Simone Ferlin, Dejan Kostic, and Marco Chiesa. 2024. NetConfEval: Can LLMs Facili- tate Network Configuration? Proc. ACM Netw.2, CoNEXT2 (2024), 7:1– 7:25. http://dblp.uni-trier.de/db/journals/pacmnet/pacmnet2.html# WangSFFKC24
2024
-
[28]
Changjie Wang, Mariano Scazzariello, Alireza Farshin, Dejan Kostic, and Marco Chiesa. 2023. Making Network Configuration Human Friendly. arXiv:2309.06342 [cs.NI] https://arxiv.org/abs/2309.06342
2023 arXiv
-
[29]
Ke Wang, Jiahui Zhu, Minjie Ren, Zeming Liu, Shiwei Li, Zongye Zhang, Chenkai Zhang, Xiaoyu Wu, Qiqi Zhan, Qingjie Liu, and Yunhong Wang. 2024. A Survey on Data Synthesis and Augmen- tation for Large Language Models. arXiv:2410.12896 [cs.CL] https: //arxiv.org/abs/2410.12896
2024 arXiv
-
[30]
Zilong Wang, Zifeng Wang, Long Le, Steven Zheng, Swaroop Mishra, Vincent Perot, Yuwei Zhang, Anush Mattapalli, Ankur Taly, Jingbo Shang, Chen-Yu Lee, and Tomas Pfister. 2025. Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting. In The Thirteenth Internat...
2025
-
[31]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain- of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903 [cs.CL] https://arxiv.org/abs/2201.11903
2023 arXiv
-
[32]
2025.Graph- Synth: Synthesis of Network Configuration Templates Using Large Lan- guage Models
Xiaofeng Zhang, Xianming Gao, Peilin Tao, and Tao Feng. 2025.Graph- Synth: Synthesis of Network Configuration Templates Using Large Lan- guage Models. Association for Computing Machinery, New York, NY, USA, 108–114. https://doi.org/10.1145/3728725.3728742 8
2025
-
[2020]
CoRR abs/2005.11401 (2020)
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. CoRR abs/2005.11401 (2020). arXiv:2005.11401 https://arxiv. org/abs/2005.11401
2020 arXiv
-
[2024]
In 2024 IEEE 10th International Conference on Network Softwarization (NetSoft)
An Intent-based Networks Framework based on Large Lan- guage Models. In 2024 IEEE 10th International Conference on Network Softwarization (NetSoft). 7–12. https://doi.org/10.1109/NetSoft60951. 2024.10588879
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.