REVIEW 4 major objections 6 minor 30 references
ATLAS is a command-line tool that extracts aligned syntax, control-flow, and data-flow graphs from C/C++ source without a build, and the paper reports that its control-flow paths improve an LLM's unit-test line coverage by about 35 percenta
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 16:37 UTC pith:4KNBNPWG
load-bearing objection A genuinely useful no-build multi-view graph tool for C/C++, but the core correctness claim rests on self-inspection and the coverage experiment is under-specified. the 4 major comments →
ATLAS: Multi-View Code Representation Tool for C and C++ Source Programs
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central discovery is that aligned, source-level multi-view graphs can be produced directly from source text, without a compiler or build database, and that their statement-level control flow paths are quantitatively useful to an LLM for test generation. The authors report correct CFG generation for 96.80% of C files and 91.67% of C++ files, and correct DFG generation for 91.38% (C) and 90.56% (C++), with correctness established by the authors' own manual verification. When a large language model is given one execution path per unique control-flow path extracted from ATLAS's CFG, it achieves an average of 34.71% higher line coverage and 32.66% higher branch coverage than the same model wi
What carries the argument
The machine at the center is the aligned multi-view graph, where every AST, CFG, and DFG node carries one shared identity across all views and distinct edge classes distinguish syntax, control, and data flow. The CFG is statement-level and inter-procedural: calls become edges from the caller statement to the callee's first statement, indirect calls branch to all possible callees, and recursion appears as a self-loop. The DFG is built from reaching-definition analysis, tracking definitions that reach a use without being overwritten, and is extended across function boundaries through parameter aliasing and class-member state. The build-free operation rests on a preprocessing pass that consolid
Load-bearing premise
The tool's correctness rates come only from the authors checking their own output, with no independent ground truth or verification rubric, so every downstream coverage number inherits that unverified premise.
What would settle it
Take a set of C/C++ files whose correct CFGs and DFGs are established by an independent static analyzer or compiler-derived ground truth, run ATLAS on them, and compute exact node/edge match rates; if the match rates fall materially below the reported 90–97%, the central correctness claim is falsified.
If this is right
- C/C++ code that currently cannot be analyzed because it fails to compile or lacks headers becomes analyzable; the authors report handling such partial projects directly.
- An LLM given execution paths extracted from ATLAS's CFG generates one test per unique path, which the paper reports raises line coverage by an average of 34.71 percentage points and branch coverage by 32.66 percentage points compared to the same LLM without the CFG context.
- The aligned multi-view graph means a single JSON output serves AST-only, CFG-only, and DFG-only consumers, and any view can be recovered by filtering edges in the shared namespace.
- Graph optimization flags (variable collapsing, node blacklisting) let users shrink graphs to fit LLM context windows, trading detail against token budget.
- The authors report polynomial scaling with median runtime under 7 seconds and memory under ~135 MB on the largest benchmark programs (up to ~300 LOC), suggesting per-file extraction is cheap enough for dataset generation.
Where Pith is reading between the lines
- The 34.71-point coverage gain is measured only against the LLM with no CFG context; a baseline with random execution paths (or function signatures only) would be needed to confirm that the gain comes from control-flow structure rather than any extra tokens in the prompt.
- Should the self-verified correctness rates survive independent ground-truth comparison, ATLAS would fill a genuine tooling gap: a single no-build extractor for aligned AST/CFG/DFG, replacing pipelines that stitch together separate parsers and analyzers. An independent re-run on a corpus with externally known-correct graphs would settle this.
- The failure modes the authors list — goto, pointer arithmetic, multi-threading, operator overloading — are precisely the constructs that C/C++ machine-learning models handle worst, so even a successful ATLAS leaves a measurable accuracy gap on those patterns; a follow-up adding lightweight points-to or thread analysis is the natural next step.
- The one-test-per-unique-path prompting recipe is transferable: any source of disambiguated execution paths (a symbolic executor, a tracer, or another CFG generator) could be dropped into the same LLM pipeline, which suggests the improvement may generalize beyond this specific tool.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. ATLAS is a Python command-line tool that claims to generate aligned abstract syntax trees (ASTs), statement-level inter-procedural control flow graphs (CFGs), and reaching-definition data flow graphs (DFGs) for C and C++ source files without requiring a build or compilation. The paper reports success rates on the TheAlgorithms corpus (406 C files, 360 C++ files): CFG 96.80% and 91.67%, DFG 91.38% and 90.56%, with failures attributed to goto, multi-threading, pointer arithmetic, operator overloading, and static variables. In a downstream experiment, using ATLAS-generated CFG paths as context for DeepSeek V3.2 during unit-test generation improved line coverage by 34.71 percentage points and branch coverage by 32.66 percentage points compared to the same LLM without CFG context, and was on par with or better than KLEE on several programs. The paper also reports polynomial scalability on 20 benchmark programs. The tool is presented as open-source with a Docker image and video demonstrations.
Significance. If the quantitative claims are upheld, ATLAS would fill a real gap: a no-build, source-level, multi-view graph generator for C/C++ that aligns AST, CFG, and DFG views, enabling ML4SE pipelines to consume rich structure without a resolved build environment. The downstream result, that statement-level CFG context materially improves LLM-based test coverage, is a meaningful and practically relevant finding. The paper's strengths include the candid failure analysis (Table 2), the use of established datasets, and the release of the tool as open source with Docker support, which makes independent verification possible. However, the central evidence—the graph correctness rates and the coverage improvement—currently relies on self-assessment and a small, non-representative benchmark, so the reported numbers should be treated as preliminary until external validation is added.
major comments (4)
- [Section 4.1] The graph success rates (CFG 96.80% C / 91.67% C++; DFG 91.38% C / 90.56% C++) are established only by the authors' manual verification of their own tool's output on 766 files. No verification rubric is defined, no second annotator/inter-annotator agreement is reported, and no external ground truth (e.g., Clang's source-level CFG, a gold-standard dataset, or a second static analyzer) is used. Since the downstream coverage claim in Table 1 inherits this premise, the paper must provide a precise correctness criterion, a comparison against an independent oracle on a sample, and per-file success/failure data. Without these, the headline numbers are not reproducible.
- [Section 4.1 / Section 3.1] The no-build capability is a core differentiator, described as 'accepts partial input such as a project with missing headers' and 'including files that do not compile.' Yet the paper never states how many of the 406 C and 360 C++ files are non-compilable, what 'non-compilable' means, or how ATLAS behaves when an #include or #define cannot be resolved (skip, stub, error?). This leaves the key differentiator empirically unverified. Please report the count of non-compilable files and a breakdown of the failure modes, or temper the claim.
- [Section 4.2 / Table 1] The 34.71 pp line / 32.66 pp branch coverage improvement is based on only 10 C programs, a single LLM (DeepSeek V3.2), and no statistical significance testing. The protocol does not report the number of generated tests per condition, token budget, or actual repair iterations consumed, so the comparison may be confounded: generating one test per CFG path could yield more tests for DS w/ CFG than the baseline. The 'test efficiency' metric (75% vs 67%) is also undefined—how is 'unique execution path' measured? Please report test counts, token usage, and use standard per-test coverage metrics.
- [Section 4.3 / Figure 3] The claim of 'polynomial scalability' is supported only by a 'best-fit polynomial curve' for 20 programs up to 309 LOC, with no fit degree, coefficients, R², or residual analysis. Given a median execution time of 6.89s on ~119-LOC programs, the Abstract's 'large-scale C/C++ projects' and the Conclusion's 'polynomial scalability' overstate the evidence. Report the fitted model and evaluate on a broader size range, or revise the wording to match the data.
minor comments (6)
- [General] The paper's title in the arXiv metadata ('Multi-View Code Representation Tool for C and C++ Source Programs') differs from the title inside the full text ('Automated Tree-based Language Analysis System for C and C++ Source Programs'). The two video links (abstract: youtu.be/50DvEbenp14; full text: youtu.be/QGuJZhj9CTA) also differ. Please make these consistent.
- [Section 4.1, 4.2] There are formatting/spacing errors: 'EvaluatingATLAS', 'ATLASbypasses' should be separated. Also 'TheAlgorithms' is one word, but references [27][28] are fine.
- [Table 1] The 'Total/Average' row mixes sums (Function/Line/Branch) with averages (coverage percentages). Label the row clearly, e.g., 'Total' for counts and 'Average' for percentages, or provide two rows.
- [Section 3.2.4] The claim of 'fifteen combination variants' is not enumerated. Either list the fifteen combinations or provide a reference to a figure/table that does so.
- [Section 4.2] The 'test efficiency' ratio is vaguely described. Define it operationally (e.g., using gcov traces or a coverage tool) and report the raw numbers behind the 75% and 67% figures.
- [Abstract / Availability] The full text shows 'Tool Github repository: ATLAS' without a clickable URL. Provide the actual repository link and the Docker image name so the tool is directly accessible.
Circularity Check
No circular derivation: the headline accuracy and coverage numbers are empirical measurements, not quantities derived by construction from their own inputs.
full rationale
ATLAS's central claims are empirical evaluations rather than derived predictions, so the circularity patterns do not apply. The graph-generation success rates (CFG 96.80%/91.67%, DFG 91.38%/90.56%) are counts of outputs that the authors manually inspected, not the result of fitting a parameter and then predicting it. The absence of an external ground truth, rubric, or inter-annotator agreement is a serious validity/reproducibility limitation, and the skeptic is right to flag it, but a self-assessed empirical measurement is not a circular derivation: the success rate is not an equation-level reduction of output to input. The coverage experiment is a controlled comparison (DS w/ CFG vs DS w/o CFG) with an external coverage metric; using ATLAS-generated CFGs as the intervention is the point of the experiment, and the outcome is not defined by the tool's own output. KLEE is an independent baseline from the literature. No load-bearing self-citations appear: the dataset references (TheAlgorithms) and the KLEE/DeepSeek citations are external, and no uniqueness theorem or prior work by the same authors is invoked to force the conclusion. The scalability measurements are also direct empirical observations. The paper does make unusually strong claims based on self-verification, but that is an evidence-quality problem, not a circularity problem.
Axiom & Free-Parameter Ledger
free parameters (3)
- Polynomial trend-line fit (degree and coefficients) =
unspecified (best-fit curves in Fig. 3)
- LLM test-repair iteration cap =
15
- Test discard rule =
discard tests failing after 15 repair iterations
axioms (3)
- domain assumption Tree-sitter concrete syntax trees carry enough information for statement-level, 'type-aware' data-flow analysis without a C/C++ type resolver or build.
- domain assumption Adding ATLAS CFG paths as LLM context is the causal driver of the coverage gains.
- domain assumption Line/branch coverage values in Table 1 were produced by an accurate, independent coverage tool.
read the original abstract
Multi-view code graphs that align abstract syntax trees, control flow graphs, and data flow graphs are now central to machine-learning models for software engineering. For C and C++, no single tool produces these aligned views without a complete build. We present ATLAS, a command-line tool that takes one or more C or C++ source files and emits an AST, a source-level inter-procedural CFG, a reaching-definition DFG, or any combination. The output is available as JSON, DOT, or PNG. ATLAS runs directly on source code and accepts partial input such as a project with missing headers, so it needs no compilation or build database. All views share one node namespace, so a downstream consumer can recover any single view by filtering edges. Command-line flags select views, collapse variables, and blacklist node categories to resize the emitted graph. On the TheAlgorithms corpora, ATLAS produces a correct CFG for 96.80% of C files and 91.67% of C++ files, including files that do not compile. It already serves as the CFG front-end of an LLM-based unit-test generation framework for C. ATLAS is open source and ships as a Docker image with a screencast walkthrough. Demo link: https://youtu.be/50DvEbenp14.
Figures
Reference graph
Works this paper leans on
-
[1]
[n. d.]. Joern - The Bug Hunter’s Workbench. https://joern.io/. Accessed: 2023-11-17
2023
-
[2]
Abhinav Anand, Shweta Verma, Krishna Narasimhan, and Mira Mezini. 2024. A Critical Study of What Code-LLMs (Do Not) Learn. arXiv:2406.11930 [cs.SE] https://arxiv.org/abs/2406.11930
Pith/arXiv arXiv 2024
-
[3]
Brooks. 1987. No Silver Bullet Essence and Accidents of Software Engineering. Computer20, 4 (1987), 10–19. doi:10.1109/MC.1987.1663532
arXiv 1987
-
[4]
Max Brunsfeld and Contributors. 2023. Tree-sitter. https://tree-sitter.github.io/ tree-sitter/. Accessed: 2023-11-17
2023
-
[5]
Cristian Cadar, Daniel Dunbar, and Dawson Engler. 2008. KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs. InProceedings of the 8th USENIX Conference on Operating Systems Design and Implementation(San Diego, California)(OSDI’08). USENIX Association, USA, 209–224
2008
-
[6]
Sridhar Chimalakonda, Debeshee Das, Alex Mathai, Srikanth Tamilselvam, and Atul Kumar. 2023. The Landscape of Source Code Representation Learning in AI-Driven Software Engineering Tasks. In2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). 342–343. doi:10.1109/ICSE-Companion58688.2023.00098
arXiv 2023
-
[7]
Debeshee Das, Noble Saji Mathews, Alex Mathai, Srikanth Tamilselvam, Kranthi Sedamaki, Sridhar Chimalakonda, and Atul Kumar. 2023. COMEX: a tool for generating customized source code representations. In2023 38th IEEE/ACM Inter- national Conference on Automated Software Engineering (ASE). IEEE, 2054–2057
2023
-
[8]
DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenhao Xu, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Erhang Li, Fangqi Zhou, Fangyun Lin, Fucong Dai, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Ha...
Pith/arXiv arXiv 2025
-
[9]
Dubniczky, Krisztofer Zoltán Horvát, Tamás Bisztray, Mo- hamed Amine Ferrag, Lucas C
Richard A. Dubniczky, Krisztofer Zoltán Horvát, Tamás Bisztray, Mo- hamed Amine Ferrag, Lucas C. Cordeiro, and Norbert Tihanyi. 2025. CASTLE: Benchmarking Dataset for Static Code Analyzers and LLMs towards CWE Detec- tion. arXiv:2503.09433 [cs.CR] https://arxiv.org/abs/2503.09433
Pith/arXiv arXiv 2025
-
[10]
Vinicius H. S. Durelli, Rafael S. Durelli, Simone S. Borges, Andre T. Endo, Marcelo M. Eler, Diego R. C. Dias, and Marcelo P. Guimarães. 2019. Machine Learning Applied to Software Testing: A Systematic Mapping Study.IEEE Trans- actions on Reliability68, 3 (2019), 1189–1212. doi:10.1109/TR.2019.2892517
arXiv 2019
-
[11]
Afonso Fontes and Gregory Gay. 2023. The integration of machine learning into automated test generation: A systematic mapping study.Software Testing, Verification and Reliability33, 4 (2023), e1845
2023
-
[12]
Sijia Gu, Noor Nashid, and Ali Mesbah. 2025. LLM Test Generation via Iterative Hybrid Program Analysis.arXiv preprint arXiv:2503.13580(2025)
Pith/arXiv arXiv 2025
-
[13]
Sonia Haiduc, Jairo Aponte, and Andrian Marcus. 2010. Supporting program comprehension with source code summarization. InProceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 2. 223–226
2010
-
[14]
Ali Reza Ibrahimzada, Kaiyao Ke, Mrigank Pawagi, Muhammad Salman Abid, Rangeet Pan, Saurabh Sinha, and Reyhaneh Jabbarvand. 2025. AlphaTrans: A Neuro-Symbolic Compositional Approach for Repository-Level Code Translation and Validation.Proceedings of the ACM on Software Engineering2, FSE (June 2025), 2454–2476. doi:10.1145/3729379
doi:10.1145/3729379 2025
-
[15]
Siyuan Jiang, Ameer Armaly, and Collin McMillan. 2017. Automatically generat- ing commit messages from diffs using neural machine translation. In2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 135–146
2017
-
[16]
Chris Lattner, Mehdi Amini, Uday Bondhugula, Albert Cohen, Andy Davis, Jacques Pienaar, River Riddle, Tatiana Shpeisman, Nicolas Vasilache, and Olek- sandr Zinenko. 2020. MLIR: A compiler infrastructure for the end of Moore’s law. arXiv preprint arXiv:2002.11054(2020)
Pith/arXiv arXiv 2020
-
[17]
Yihe Li, Ruijie Meng, and Gregory J Duck. 2025. Large language model pow- ered symbolic execution.Proceedings of the ACM on Programming Languages9, OOPSLA2 (2025), 3148–3176
2025
-
[18]
Changshu Liu and Reyhaneh Jabbarvand. 2025. A Tool for In-depth Analysis of Code Execution Reasoning of Large Language Models. arXiv:2501.18482 [cs.SE] https://arxiv.org/abs/2501.18482
Pith/arXiv arXiv 2025
-
[19]
Alex Mathai, Kranthi Sedamaki, Debeshee Das, Noble Saji Mathews, Srikanth Tamilselvam, Sridhar Chimalakonda, and Atul Kumar. 2024. CodeSAM: Source Code Representation Learning by Infusing Self-Attention with Multi-Code-View Graphs. arXiv:2411.14611 [cs.SE] https://arxiv.org/abs/2411.14611
Pith/arXiv arXiv 2024
-
[20]
Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. 2016. Convolutional neural net- works over tree structures for programming language processing. InProceedings of the AAAI conference on artificial intelligence, Vol. 30
2016
-
[21]
Annibale Panichella, Bogdan Dit, Rocco Oliveto, Massimilano Di Penta, Denys Poshynanyk, and Andrea De Lucia. 2013. How to effectively use topic models for software engineering tasks? an approach based on genetic algorithms. In2013 35th International conference on software engineering (ICSE). IEEE, 522–531
2013
-
[22]
Vishnu S. Pendyala and Neha B. Thakur. 2025. Performance and interpretability analysis of code generation large language models.Neurocomputing656 (2025), 131461. doi:10.1016/j.neucom.2025.131461
arXiv 2025
-
[23]
Hazem Peter Samoaa, Firas Bayram, Pasquale Salza, and Philipp Leitner. 2022. A systematic mapping study of source code representation for deep learning in software engineering.IET Software16, 4 (June 2022), 351–385. doi:10.1049/sfw2. 12064
-
[24]
Philipp Dominik Schubert, Ben Hermann, and Eric Bodden. 2019. Phasar: An inter-procedural static analysis framework for c/c++. InInternational Conference on Tools and Algorithms for the Construction and Analysis of Systems. Springer, 393–410
2019
-
[25]
Philipp Dominik Schubert, Ben Hermann, and Eric Bodden. 2021. Lossless, Per- sisted Summarization of Static Callgraph, Points-To and Data-Flow Analysis. In 35th European Conference on Object-Oriented Programming (ECOOP 2021) (Leibniz International Proceedings in Informatics (LIPIcs), Vol. 194), Anders Møller and Manu Sridharan (Eds.). Schloss Dagstuhl – L...
-
[26]
Bjarne Stroustrup. 2020. Thriving in a Crowded and Changing World: C++ 2006–2020.Proceedings of the ACM on Programming Languages4, HOPL (2020), 1–168. doi:10.1145/3386320
-
[27]
TheAlgorithms. 2025. TheAlgorithms / C: Collection of various algorithms in C for educational purposes. https://github.com/TheAlgorithms/C. GitHub repository, accessed 17 November 2025
2025
-
[28]
TheAlgorithms. 2025. TheAlgorithms / C-Plus-Plus: Collection of various algo- rithms in C++ for educational purposes. https://github.com/TheAlgorithms/C- Plus-Plus. GitHub repository, accessed 17 November 2025
2025
-
[29]
Michele Tufano, Jevgenija Pantiuchina, Cody Watson, Gabriele Bavota, and Denys Poshyvanyk. 2019. On learning meaningful code changes via neural machine translation. In2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 25–36
2019
-
[30]
Jiayimei Wang, Tao Ni, Wei-Bin Lee, and Qingchuan Zhao. 2025. A Con- temporary Survey of Large Language Model Assisted Program Analysis. arXiv:2502.18474 [cs.SE] https://arxiv.org/abs/2502.18474 , , Jaid et al. A ATLAS: Tool Demonstration Walkthrough This appendix comparesATLAS’s CFG representation with LLVM IR and provides step-by-step demonstrations sho...
Pith/arXiv arXiv 2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.