REVIEW 3 major objections 3 minor 33 references
Comprendia: AI-Augmented Code Comprehension
T0 review · 3 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Comprendia uses the visible dependency graph as the relevance signal for LLM code-explanation prompts, making callee selection deterministic, reproducible across LLM families, and auditable against graph nodes.
desk verdict A well-specified, honest tool paper with a genuinely novel deterministic prompt-builder, but the inheritance-collapse phase as written can silently drop distinct sibling implementations, and the evaluation cannot detect it. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is GACP (Graph-Aware Callee Pruning), a pure static function that turns an AST-extracted callee list into a ranked, budgeted prompt fragment using the dependency graph. Phase 1 admits callees whose declaring class is within two hops of the enclosing class (self-calls at distance 0 always admitted), rendering distance-1 callees as full bodies and distance-2 callees as signatures. Phase 2 replaces any group of two or more surviving candidates whose classes share an immediate inheritance ancestor with that ancestor's body plus one concrete delegate, eliminating near-duplicate siblings. Phase 3 scores survivors by edge-type weight divided by squared graph distance, giving inheritance and interface edges the highest weight, and emits candidates greedily under the token budget. Each emitted callee carries an inclusion reason that maps to a visible graph node, which is what makes the prompt auditable.
What would settle it
Run a behavior-based user study on the same six code fixtures comparing selection-only prompts against GACP-assembled prompts on explanation accuracy or task completion; if GACP does not improve outcomes, the practical value of the algorithm is unestablished.
Extended reading notes
Core claim
The central claim is that the graph the developer sees should be the same data structure the LLM prompt builder reasons over. Concretely, GACP is a pure, static, deterministic function: from the enclosing class, AST-extracted project-internal callees, a read-only graph index, and a token budget, it returns a ranked list of callee records, and each record's inclusion reason maps one-to-one to a visible node. The paper demonstrates this on six method-body selections from an open-source Java project. Across three LLM families, prompt-token counts are identical for every fixture and mode, confirming that the algorithm's output is gateway-independent. On the three fixtures with clone groups, the inheritance-collapse phase replaces $k$ sibling bodies with the shared parent contract plus one concrete delegate, cutting or limiting token growth. The paper is explicit that it makes no human-comprehension claim; what it establishes is the behavior of the prompt-construction algorithm.
Load-bearing premise
The load-bearing premise is that including project-internal callees selected by graph distance and edge-type weighting improves, or at least does not hurt, LLM code explanations compared with showing only the selected code.
Editorial extensions
If this is right
- The same selection and budget produce identical prompt tokens on three different LLM families for every fixture, so explanations are backend-independent.
- When Phase 2 fires, the prompt shape changes from $k$ duplicate sibling bodies to one shared parent contract plus one delegate, which the paper argues better reflects the code's structure.
- A hard two-hop cutoff can admit structurally proximate helpers that a fixed five-callee cap would drop, while deeper helpers are excluded because they add tokens without enough comprehension signal.
- Every callee citation in the explanation can be followed back to a visible graph node, giving the developer a concrete way to verify what the LLM was shown.
- Because GACP is deterministic and free of LLM calls, its output can be replayed, tested, and compared across tools, a property selection-only and embedding-retrieval approaches do not offer.
Reading between the lines
- The paper's own coarse accuracy rubric saturates at 2/2 for all modes, so the decisive next test is a human or behavior-based study that asks whether GACP-assembled prompts actually improve explanation quality, not just whether they change the prompt.
- If GACP's principle generalizes, method-level call graphs could give finer context than class-level two-hop reachability; the paper lists this as future work but does not test it.
- Because GACP is deterministic and parameterized, a head-to-head comparison with embedding-retrieval context selection on explanation quality is directly runnable, and the paper explicitly says it did not run one.
- A parameter sweep of the hop cutoff and edge-type weights is an obvious stress test: the claimed structural advantage could shift on codebases with deep call chains or different clone patterns.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents Comprendia, an Eclipse plugin that combines a multi-edge-type dependency graph with LLM-powered code explanation on a shared interactive substrate. Its main algorithmic contribution is Graph-Aware Callee Pruning (GACP), a deterministic, static function that selects project-internal callees for an explanation prompt using graph distance, inheritance-based collapse, and edge-type weighting, with every inclusion mapped to a visible graph node. The paper also contributes a clone-detection overlay and a CVE risk overlay, and it demonstrates the system on six method-body fixtures from the QUICKBITE benchmark, reporting prompt-token counts across three LLM families and comparing four prompt modes. The authors explicitly state in Section VI(d) that the paper measures the algorithm's effect on the prompt, not whether developers comprehend code faster.
Significance. If GACP is taken on its own terms, it offers a novel and attractive alternative to embedding-similarity retrieval and perplexity-based compression: selection decisions are purely structural, deterministic, and traceable to a developer-visible graph. The paper is unusually precise about the algorithm specification and ships a public artifact, and the reproducibility claim is verified by construction—the same prompt string is sent to every backend. However, the practical value of GACP for code comprehension is not established by the evaluation, which is limited to token counts and callee-set composition on a single benchmark system. The central risk is that the inheritance-collapse rule, as formally specified, is not actually a near-duplicate collapse and can delete behaviorally distinct sibling implementations from the prompt, undermining the very grounding the paper claims to provide.
major comments (3)
- [Section IV-B, Phase 2] The formal rule for Phase 2 replaces any group of at least two surviving candidates whose declaring classes share an immediate inheritance ancestor A with A rendered FULL_BODY plus a single concrete delegate, without any clone or near-duplicate check. This contradicts the contribution statement in Section I-B, which describes the phase as collapsing "near-duplicate siblings." As specified, two sibling classes that share an immediate ancestor but implement a method differently, both called by the enclosing method, will be collapsed to the ancestor contract plus one delegate; the dropped sibling is then explained by name only, which is exactly the failure mode attributed to selection-only prompts in Section I. The six QUICKBITE fixtures all exercise seeded clone groups, so Tables III and IV cannot reveal this over-collapse, and the saturated rubric in Section VI(b) would not detect it either. Please add a similarity or clone check to Phase 2 (or rename the phase and re-scope the claim), and demonstrate behavior on non-clone sibling hierarchies.
- [Section VI(b) and VI(d)] The evaluation cannot support the paper's comprehension-oriented claims. The keyword-match accuracy rubric saturates at 2/2 across all modes and LLM families, and Section VI(d) explicitly disclaims any human-comprehension measurement. The remaining quantitative evidence consists of prompt-token counts and callee-set composition, which show that GACP produces different prompts from selection-only baselines but do not show that these prompts support comprehension better, or even not worse. The abstract's statement that the tool "supports comprehension" and the Introduction's framing of GACP as closing the selection-only gap accordingly outrun the evidence. Since this is the load-bearing motivation for the algorithm, either add a behavioral or human evaluation, or explicitly reframe the contribution as a deterministic prompt-construction mechanism whose downstream effects on explanation quality are untested.
- [Section V-A] The claim that prompt-token counts are "identical across all three LLM families" is an implementation check rather than an empirical finding, because GACP assembles the prompt before any LLM call and the AiGateway sends the same string to all backends. Presenting identical token counts as evidence about LLM families is therefore misleading: the counts are fixed by construction. The section would be more informative if it reported the actual LLM responses (for example, whether the three families produced similar explanations from the same prompt), or if the cross-LLM framing were removed and the reproducibility claim were stated as a deterministic property of the algorithm.
minor comments (3)
- [Table I] The row for CodeCity/Zest appears to mark all of graph-grounded, IDE-native, and auditable with checks, but Section II-C describes these tools as visualization-only without AI explanation; please clarify the criteria for the "AI-explanation" and "Auditable" columns so the comparison is consistent.
- [Section VI(f)] The paper acknowledges that no comparison to embedding-retrieval systems was performed; adding even a small illustrative comparison (e.g., GACP versus an embedding-based selector on one fixture) would make the positioning against Cody/RepoFuse more concrete, though it is not required for the algorithmic claim.
- [Throughout] Several instances of "COMPRENDIAis" and "COMPRENDIA" lack a separating space before the following verb; please correct the typesetting in the camera-ready version.
Circularity Check
No significant circularity: GACP's determinism, traceability, and reproducibility are properties of its explicit specification, not derived from fitted data or self-citations.
full rationale
GACP is specified in Section IV as a pure, static function with no randomness, no LLM calls, and no fitted parameters; K_max=2 and the edge-type weights are stated design choices, not estimated from the six QUICKBITE fixtures (Section IV-A, IV-C, VI-f). The central claims of reproducibility across LLM families and traceability to visible graph nodes are direct consequences of this specification: the prompt is assembled before the AI gateway is invoked, and each inclusion reason is emitted alongside the graph node that justifies it. Section V's identical token counts are accordingly a sanity check of the implementation rather than an empirical prediction. Self-citations ([5], [6], [17], [21]) support background motivations (selection-only prompts are insufficient; Maven vulnerabilities; an existing clone engine) and are externally published, empirically grounded, or non-essential; none is used to prove GACP's formal behavior. The acknowledged limitations—saturating 2/2 rubric (VI-b), no human-comprehension claim (VI-d), single benchmark (VI-a), no parameter sweep (VI-f)—are evaluation gaps, not circular reductions. The discrepancy between the advertised near-duplicate guard and Phase 2's ancestry-only collapse rule (IV-B) is a correctness risk about over-collapsing non-clone siblings, but it is not circular because no output is fed back as an input and no fitted value is renamed as a prediction. Overall, the paper is self-contained with respect to its algorithmic claims.
Assumptions & free parameters
free parameters (4)
- K_max =
2
- edge_type_weights =
inheritance/interface=1.0, field-type=0.6, method-parameter/method-return=0.4
- snippet_mode_threshold =
distance <=1 FULL_BODY, distance=2 SIGNATURE_ONLY
- token_budget
assumptions (5)
- domain assumption JDT AST parsing and project-internal symbol filtering correctly identify callees and edges.
- ad hoc to paper Two-hop reachability in the class-level dependency graph captures the callees relevant to explaining a method.
- ad hoc to paper Edge-type weights reflect semantic relevance for comprehension.
- ad hoc to paper Including project-internal callees in LLM prompts improves explanation quality or reduces hallucination.
- domain assumption CloMan clone detection results are correct for the overlay and Phase 2 collapse.
Cite this review
Pith. "Pith review of Comprendia: AI-Augmented Code Comprehension." pith.science (2026). https://pith.science/paper/GM6OY5K7
@misc{pith2026260810290,
author = {Pith},
title = {Pith review of: Comprendia: AI-Augmented Code Comprehension},
year = {2026},
howpublished = {\url{https://pith.science/paper/GM6OY5K7}},
note = {Machine review of arXiv:2608.10290}
}
read the original abstract
Comprendia is an Eclipse plugin that integrates structural dependency visualization with LLM-powered code explanation on a shared interactive graph for Java program comprehension. The tool rests on four pillars: (1) a multi-edge-type dependency graph with live search and multiple layouts; (2) LLM explanations grounded in Graph-Aware Callee Pruning (GACP), an auditable strategy that selects relevant callees using the same graph the developer navigates; (3) a clone-detection overlay that highlights duplication and suggests extract-to-parent refactoring opportunities; and (4) a CVE risk overlay powered by OSV.dev. GACP uses graph distance, inheritance collapse, and edge-type weighting to produce prompts that are reproducible across LLM families and traceable to visible graph nodes. We demonstrate Comprendia on a Java project containing known clones and vulnerabilities, showing how the unified graph substrate supports comprehension while keeping the developer in control. Screencast: https://youtu.be/1wlh_RYehzA
Figures
Reference graph
Works this paper leans on
-
[1]
Analyz- ing chatgpt assistance in programming,
C. Nachuma, M. F. Rabbi, A. I. Champa, and M. F. Zibran, “Analyz- ing chatgpt assistance in programming,” inSoftware Engineering and Management: Theory and Applications: Volume 17. Springer, 2025, pp. 203–215
work page 2025
-
[2]
H. Li, H. Zhang, and A. E. Hassan, “The rise of ai teammates in software engineering (se) 3.0: How autonomous coding agents are reshaping software engineering,” 2025. [Online]. Available: https://arxiv.org/abs/2507.15003
arXiv 2025
-
[3]
GitHub, “GitHub Copilot Chat,” https://github.com/features/copilot, 2026, accessed: May 2026
work page 2026
-
[4]
Cursor: The AI-First Code Editor,
Anysphere, “Cursor: The AI-First Code Editor,” https://cursor.com, 2026, accessed: May 2026
work page 2026
-
[5]
ChatGPT in Action: Analyzing Its Use in Software Development,
A. I. Champa, M. F. Rabbi, C. Nachuma, and M. F. Zibran, “ChatGPT in Action: Analyzing Its Use in Software Development,” inProceedings of the 21st International Conference on Mining Software Repositories (MSR), Lisbon, Portugal, 2024, pp. 182–192
work page 2024
-
[6]
C. Nachuma and M. Zibran, “When AI Teammates Meet Code Review: Collaboration Signals Shaping the Integration of Agent-Authored Pull Requests,” inProceedings of the 23rd International Conference on Mining Software Repositories (MSR), Brazil, 2026, pp. 1–5
work page 2026
-
[7]
Understanding understanding source code with functional magnetic resonance imaging,
J. Siegmund, C. Kästner, S. Apel, C. Parnin, A. Bethmann, T. Leich, G. Saake, and A. Brechmann, “Understanding understanding source code with functional magnetic resonance imaging,” in36th international conference on software engineering, 2014, pp. 378–389
work page 2014
-
[8]
Hard-to-answer questions about code,
T. LaToza and B. Myers, “Hard-to-answer questions about code,” in Evaluation and usability of programming languages and tools, 2010, pp. 1–6
work page 2010
Show all 33 references
-
[9]
A Comprehensive Model for Code Readability,
S. Scalabrino, M. Linares-Vásquez, R. Oliveto, and D. Poshyvanyk, “A Comprehensive Model for Code Readability,”Journal of Software: Evolution and Process, vol. 30, no. 6, p. e1958, 2018
2018
-
[10]
Cody: AI Coding Assistant That Knows Your Codebase,
Sourcegraph, “Cody: AI Coding Assistant That Knows Your Codebase,” https://sourcegraph.com/cody, 2026, accessed: May 2026
2026
-
[11]
RepoFuse: Repository-Level Code Completion with Fused Dual Context,
M. Liang, X. Xie, G. Zhang, X. Zheng, P. Di, W. Jiang, H. Chen, C. Wang, and G. Fan, “RepoFuse: Repository-Level Code Completion with Fused Dual Context,”CoRR, vol. abs/2402.14323, 2024, arXiv preprint
2024 arXiv
-
[12]
Using Pre-Trained Models to Boost Code Review Automation,
R. Tufano, S. Masiero, A. Mastropaolo, L. Pascarella, D. Poshyvanyk, and G. Bavota, “Using Pre-Trained Models to Boost Code Review Automation,” inProceedings of the 44th International Conference on Software Engineering (ICSE), 2022, pp. 2291–2302
2022
-
[13]
LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models,
H. Jiang, Q. Wu, C.-Y . Lin, Y . Yang, and L. Qiu, “LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023, pp. 13 358–13 376
2023
-
[14]
RECOMP: Improving Retrieval-Augmented LMs with Context Compression and Selective Augmentation,
F. Xu, W. Shi, and E. Choi, “RECOMP: Improving Retrieval-Augmented LMs with Context Compression and Selective Augmentation,” in12th International Conference on Learning Representations, 2024
2024
-
[15]
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression,
H. Jiang, Q. Wu, X. Luo, D. Li, C.-Y . Lin, Y . Yang, and L. Qiu, “LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression,” in62nd Annual Meeting of the Association for Computational Linguistics, 2024, pp. 1658–1677
2024
-
[16]
LLMLOOP: Improving LLM-Generated Code and Tests through Au- tomated Iterative Feedback Loops,
R. Ravi, D. Bradshaw, S. Ruberto, G. Jahangirova, and V . Terragni, “LLMLOOP: Improving LLM-Generated Code and Tests through Au- tomated Iterative Feedback Loops,” inIEEE International Conference on Software Maintenance and Evolution, Tool Demo Track, 2025
2025
-
[17]
Decoding Dependency Risks: A Quantitative Study of Vulnerabilities in the Maven Ecosystem,
C. Nachuma, M. M. Hossan, A. K. Turzo, and M. F. Zibran, “Decoding Dependency Risks: A Quantitative Study of Vulnerabilities in the Maven Ecosystem,” inProceedings of the 22nd International Conference on Mining Software Repositories (MSR), 2025, pp. 270–280
2025
-
[18]
IDE-based real-time focused search for near-miss clones,
M. F. Zibran and C. K. Roy, “IDE-based real-time focused search for near-miss clones,” inProceedings of the 27th ACM Symposium On Applied Computing (SAC). ACM, 2012, pp. 1235–1242
2012
-
[19]
The vision of software clone man- agement: Past, present, and future,
C. Roy, M. Zibran, and R. Koschke, “The vision of software clone man- agement: Past, present, and future,” inproceedings of the IEEE CSMR- 18/WCRE-21 Software Evolution Week (SEW’14), Vision Keynote. IEEE, 2014, pp. 18–33
2014
-
[20]
Towards implementation of an integrated clone man- agement infrastructure,
M. F. Zibran, “Towards implementation of an integrated clone man- agement infrastructure,” inProceedings of the 10th IEEE International Workshop on Software Clones (IWSC). IEEE, 2016, pp. 60–61
2016
-
[21]
Analysis and Visualization for Clone Refactoring,
M. F. Zibran, “Analysis and Visualization for Clone Refactoring,” in Proceedings of the 9th IEEE International Workshop on Software Clones (IWSC), Montréal, Canada, 2015, pp. 47–53
2015
-
[22]
CodeCity: 3D Visualization of Large-Scale Software,
R. Wettel and M. Lanza, “CodeCity: 3D Visualization of Large-Scale Software,” inCompanion Proceedings of the 30th International Confer- ence on Software Engineering (ICSE Companion), 2008, pp. 921–922
2008
-
[23]
Do Dynamic Object Process Graphs Support Program Understanding? A Controlled Experiment,
J. Quante, “Do Dynamic Object Process Graphs Support Program Understanding? A Controlled Experiment,” in16th IEEE International Conference on Program Comprehension, 2008, pp. 73–82
2008
-
[24]
Toward the Automatic Classifi- cation of Self-Affirmed Refactoring,
A. AlOmar, W. Mkaouer, and A. Ouni, “Toward the Automatic Classifi- cation of Self-Affirmed Refactoring,”J. of Systems and Software, 2022
2022
-
[25]
Poisoning Programs by Un-Repairing Code: Security Concerns of AI-Generated Code,
C. Improta, “Poisoning Programs by Un-Repairing Code: Security Concerns of AI-Generated Code,” inProceedings of the International Symposium on Software Reliability Engineering Workshops, 2023
2023
-
[26]
Comprendia: AI-Augmented Code Com- prehension (ICSME 2026 Artifact),
C. Nachuma and M. F. Zibran, “Comprendia: AI-Augmented Code Com- prehension (ICSME 2026 Artifact),” https://doi.org/10.5281/zenodo. 20436759, 2026
2026 doi
-
[27]
Code Comprehension Gaps in AI-Assisted Development: A Conceptual Framework for Knowledge Erosion Patterns,
M. S. Patel, R. Dhawan, L. Bramhanwade, and D. J. Mishra, “Code Comprehension Gaps in AI-Assisted Development: A Conceptual Framework for Knowledge Erosion Patterns,”IEEE Access, vol. 14, pp. 66 740–66 757, 2026
2026
-
[28]
Project Usus: Eclipse Plug-Ins for OO Design Metrics and Dependency Visualization,
Project Usus Contributors, “Project Usus: Eclipse Plug-Ins for OO Design Metrics and Dependency Visualization,” https://github.com/usus/ usus-plugins/wiki, 2017, accessed: May 2026
2017
-
[29]
OSV: Open Source Vulnerabilities Database,
Google Open Source Security Team, “OSV: Open Source Vulnerabilities Database,” https://osv.dev/, 2026, accessed: May 2026
2026
-
[30]
Cytoscape.js: a graph theory library for visualisation and analysis,
M. Franz, C. T. Lopes, G. Huck, Y . Dong, O. Sumer, and G. D. Bader, “Cytoscape.js: a graph theory library for visualisation and analysis,” Bioinformatics, vol. 32, no. 2, pp. 309–311, 2016
2016
-
[31]
Eclipse Java Development Tools (JDT),
Eclipse Foundation, “Eclipse Java Development Tools (JDT),” https:// www.eclipse.org/jdt/, 2026, accessed: May 2026
2026
-
[32]
Eclipse JGit: Java Implementation of Git,
Eclipse Foundation, “Eclipse JGit: Java Implementation of Git,” https: //www.eclipse.org/jgit/, 2026, accessed: May 2026
2026
-
[33]
CodeWatcher: IDE Telemetry Data Extraction Tool for Understanding Coding Interactions with LLMs,
M. Basha, A. Ribeiro, J. Javahar, G. Rodriguez-Perez, and C. de Souza, “CodeWatcher: IDE Telemetry Data Extraction Tool for Understanding Coding Interactions with LLMs,” inIEEE International Conference on Software Maintenance and Evolution, Tool Demo Track, 2025
2025
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.