{"id":"5907d098-accb-4a46-a93d-74c48140696d","arxiv_id":"2608.10290","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Comprendia's GACP selects callees for LLM prompts using a visible dependency graph, making context inclusion auditable and reproducible across LLM families.","lead":"This paper presents Comprendia, an Eclipse plugin that combines dependency graph visualization with AI-generated code explanations. Its GACP algorithm chooses which helper methods to show the AI, using only the graph the developer can see.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Phase 2 over-collapses non-clone siblings: any pair sharing an immediate ancestor is replaced by the ancestor plus one delegate, so behaviorally distinct callee bodies can vanish from the prompt; the saturated rubric in Section VI(b) would not detect this.","rationale":"The reader's weakest assumption correctly identifies the missing value evidence: no human-comprehension study, coarse rubric, and no baseline against embedding retrieval. This stress-test pass sharpens that concern into a specific algorithmic failure mode. Section IV-B's collapse rule is broader than its motivating clone scenario: without a similarity predicate, it can delete semantically distinct sibling implementations that are direct callees of the selection. That directly threatens the central practical claim that GACP produces prompts that support comprehension, because the prompt may omit the very code needed to explain a call. The deterministic, auditable, and reproducible properties of GACP are credible and machine-checkable; those claims are not the target here. The concrete micro-fixture would distinguish between a benign design choice and a genuine over-collapse. If the test shows the dropped sibling's behavior is lost, the paper's conclusion needs either a similarity guard before Phase 2 or a method-level call graph before collapsing siblings. The overall disposition remains conditional: the tool's algorithmic claims are plausible, but the relevance guarantee and the comprehension value need verification with a discriminating test.","tokens_in":7905,"tokens_out":7456,"duration_ms":83416,"concrete_test":"Add a micro-fixture to the public replay harness: abstract class Price { abstract int value(); }, Cheap extends Price returning 1, and Expensive extends Price returning 100; select a method that calls both cheap.value() and expensive.value(). Run GACP with K_max=2 and inspect the emitted callee list. If Phase 2 emits Price plus only one concrete delegate, the other sibling's behavior is absent from the prompt. Then compare SELECTION-ONLY and GACP-BODY explanations for this fixture to see whether either correctly states both behaviors. This settles whether Phase 2 preserves relevant callee information outside seeded clone groups.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section IV-B defines Phase 2 as: any group of ≥2 surviving candidates whose declaring classes share an immediate inheritance ancestor A is replaced with A rendered FULL_BODY plus one concrete delegate. The advertised \"near-duplicate\" guard from Contribution 2 is not part of the formal rule: no clone or similarity check precedes the collapse. If the selected method calls methods in two sibling classes that share an ancestor but implement the same method differently, GACP discards all but one concrete implementation from the prompt. The resulting prompt describes the ancestor contract (possibly abstract) plus one delegate; the call to the dropped sibling is explained only by name, which is exactly the failure mode attributed to selection-only prompts in Section I. The six QUICKBITE fixtures all exercise seeded clone groups, so Tables III and IV cannot reveal this over-collapse; the keyword-match rubric saturates at 2/2 (Section VI-b), and Section VI-d explicitly disclaims human-comprehension measurement. The load-bearing premise that GACP's context selection is at least not worse than selection-only is therefore not established and is plausibly false on non-clone sibling hierarchies. This is a correctness risk in the algorithm as specified, not merely a missing evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Comprendia, an Eclipse plugin that combines a multi-edge-type dependency graph with LLM-powered code explanation on a shared interactive substrate. Its main algorithmic contribution is Graph-Aware Callee Pruning (GACP), a deterministic, static function that selects project-internal callees for an explanation prompt using graph distance, inheritance-based collapse, and edge-type weighting, with every inclusion mapped to a visible graph node. The paper also contributes a clone-detection overlay and a CVE risk overlay, and it demonstrates the system on six method-body fixtures from the QUICKBITE benchmark, reporting prompt-token counts across three LLM families and comparing four prompt modes. The authors explicitly state in Section VI(d) that the paper measures the algorithm's effect on the prompt, not whether developers comprehend code faster.","tokens_in":8156,"tokens_out":4277,"duration_ms":44558,"significance":"If GACP is taken on its own terms, it offers a novel and attractive alternative to embedding-similarity retrieval and perplexity-based compression: selection decisions are purely structural, deterministic, and traceable to a developer-visible graph. The paper is unusually precise about the algorithm specification and ships a public artifact, and the reproducibility claim is verified by construction—the same prompt string is sent to every backend. However, the practical value of GACP for code comprehension is not established by the evaluation, which is limited to token counts and callee-set composition on a single benchmark system. The central risk is that the inheritance-collapse rule, as formally specified, is not actually a near-duplicate collapse and can delete behaviorally distinct sibling implementations from the prompt, undermining the very grounding the paper claims to provide.","major_comments":[{"comment":"The formal rule for Phase 2 replaces any group of at least two surviving candidates whose declaring classes share an immediate inheritance ancestor A with A rendered FULL_BODY plus a single concrete delegate, without any clone or near-duplicate check. This contradicts the contribution statement in Section I-B, which describes the phase as collapsing \"near-duplicate siblings.\" As specified, two sibling classes that share an immediate ancestor but implement a method differently, both called by the enclosing method, will be collapsed to the ancestor contract plus one delegate; the dropped sibling is then explained by name only, which is exactly the failure mode attributed to selection-only prompts in Section I. The six QUICKBITE fixtures all exercise seeded clone groups, so Tables III and IV cannot reveal this over-collapse, and the saturated rubric in Section VI(b) would not detect it either. Please add a similarity or clone check to Phase 2 (or rename the phase and re-scope the claim), and demonstrate behavior on non-clone sibling hierarchies.","section":"Section IV-B, Phase 2"},{"comment":"The evaluation cannot support the paper's comprehension-oriented claims. The keyword-match accuracy rubric saturates at 2/2 across all modes and LLM families, and Section VI(d) explicitly disclaims any human-comprehension measurement. The remaining quantitative evidence consists of prompt-token counts and callee-set composition, which show that GACP produces different prompts from selection-only baselines but do not show that these prompts support comprehension better, or even not worse. The abstract's statement that the tool \"supports comprehension\" and the Introduction's framing of GACP as closing the selection-only gap accordingly outrun the evidence. Since this is the load-bearing motivation for the algorithm, either add a behavioral or human evaluation, or explicitly reframe the contribution as a deterministic prompt-construction mechanism whose downstream effects on explanation quality are untested.","section":"Section VI(b) and VI(d)"},{"comment":"The claim that prompt-token counts are \"identical across all three LLM families\" is an implementation check rather than an empirical finding, because GACP assembles the prompt before any LLM call and the AiGateway sends the same string to all backends. Presenting identical token counts as evidence about LLM families is therefore misleading: the counts are fixed by construction. The section would be more informative if it reported the actual LLM responses (for example, whether the three families produced similar explanations from the same prompt), or if the cross-LLM framing were removed and the reproducibility claim were stated as a deterministic property of the algorithm.","section":"Section V-A"}],"minor_comments":[{"comment":"The row for CodeCity/Zest appears to mark all of graph-grounded, IDE-native, and auditable with checks, but Section II-C describes these tools as visualization-only without AI explanation; please clarify the criteria for the \"AI-explanation\" and \"Auditable\" columns so the comparison is consistent.","section":"Table I"},{"comment":"The paper acknowledges that no comparison to embedding-retrieval systems was performed; adding even a small illustrative comparison (e.g., GACP versus an embedding-based selector on one fixture) would make the positioning against Cody/RepoFuse more concrete, though it is not required for the algorithmic claim.","section":"Section VI(f)"},{"comment":"Several instances of \"COMPRENDIAis\" and \"COMPRENDIA\" lack a separating space before the following verb; please correct the typesetting in the camera-ready version.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is a tool paper with a clean algorithmic specification, but the evaluation is largely a sanity check of the implementation. The Phase 2 over-collapse is a genuine correctness risk that I believe can be fixed by adding a similarity check or by re-scoping the claim; the lack of any comprehension measurement is a more serious gap for a paper whose title promises 'Code Comprehension.' An editor may wish to consider whether the scope of the journal requires an empirical user study or at least a behavioral proxy before acceptance, beyond the algorithmic contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one with the stress-test note in hand, because the note lands. The headline is that GACP—a deterministic, graph-traced callee-selector for LLM prompts—is a genuinely new and cleanly specified contribution, but its Phase 2 inheritance-collapse rule has a real correctness gap that the evaluation cannot expose.\n\nWhat's actually new: GACP is the first prompt-construction strategy I know that uses the IDE's own dependency graph as a hard structural filter, with a two-hop admission rule, edge-type-weighted budget allocation, and per-callee inclusion reasons that map one-to-one onto visible graph nodes. The determinism claim is backed by construction and by identical token counts across three LLM families, and the plugin, fixtures, and replay harness are public. That is reproducible, formal work, and it earns credit.\n\nThe soft spots are concentrated in Phase 2 and the evaluation. Section IV-B replaces any group of at least two candidates sharing an immediate ancestor with the ancestor plus one delegate. No clone check precedes this. So two sibling classes that share a parent but implement a method differently get collapsed to one delegate; the other concrete implementation vanishes from the prompt, and the explanation falls back to name-based guessing for that callee—exactly the failure mode the paper attributes to selection-only prompts. The paper's own Contribution 2 says 'near-duplicate siblings,' but the formal rule doesn't enforce near-duplication. The six fixtures all exercise seeded clone groups, so Tables III-IV can't reveal this; the keyword-match rubric saturates at 2/2, and Section VI-d explicitly disclaims human comprehension. The load-bearing premise that GACP's context is at least not worse than selection-only is therefore unestablished and, on non-clone sibling hierarchies, plausibly false.\n\nThe evaluation is also thin: one benchmark, no embedding-retrieval baseline, no parameter sweep, and a token estimator the paper itself says is plus-or-minus 10-15%. The abstract overreaches when it says the tool 'supports comprehension' after the limitations section disclaims any human-comprehension measurement.\n\nStill, the core idea is sound and the paper is honest about most of its own limits. It belongs in front of a referee who will ask for a corrected Phase 2 (either add a clone/similarity check or narrow the claim to clone-touching hierarchies), an expanded evaluation on non-clone hierarchies, and a softened abstract. I'd send it out.","headline":"A well-specified, honest tool paper with a genuinely novel deterministic prompt-builder, but the inheritance-collapse phase as written can silently drop distinct sibling implementations, and the evaluation cannot detect it.","tokens_in":8645,"tokens_out":2471,"would_cite":true,"duration_ms":24442,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Comprendia uses the visible dependency graph as the relevance signal for LLM code-explanation prompts, making callee selection deterministic, reproducible across LLM families, and auditable against graph nodes.","keywords":["program comprehension","large language models","prompt construction","dependency graphs","code clones","software refactoring","IDE plugin","Eclipse"],"falsifier":"Run a behavior-based user study on the same six code fixtures comparing selection-only prompts against GACP-assembled prompts on explanation accuracy or task completion; if GACP does not improve outcomes, the practical value of the algorithm is unestablished.","tokens_in":1777,"feed_emoji":"🤖","tokens_out":4929,"duration_ms":128655,"temperature":0.7,"pith_summary":"Comprendia is an Eclipse plugin that tries to fix a blind spot in LLM-assisted code comprehension: when a developer asks an AI to explain a method, the prompt usually contains only the selected code, leaving the model to guess at internal helper functions. The paper's unifying claim is that the project's dependency graph, already visible and navigable in the IDE, should be the relevance signal for prompt construction. Its Graph-Aware Callee Pruning (GACP) admits callees within two graph hops, collapses near-duplicate siblings sharing an inheritance ancestor into that ancestor plus one delegate, and allocates a token budget by edge-type weighting. The result is a prompt in which every callee inclusion carries a reason that maps one-to-one to a visible graph node, and the paper reports identical prompt-token counts for all six fixtures across three LLM families. If that is right, developers can verify why the AI was shown a given helper, and changing LLM backends does not silently change the reasoning context.","feed_headline":"Dependency graph, not embeddings, chooses LLM context in IDE tool","feed_subtitle":"Six code fixtures gave identical prompt tokens on three LLM families, with every callee traceable to a visible graph node.","key_machinery":"The central object is GACP (Graph-Aware Callee Pruning), a pure static function that turns an AST-extracted callee list into a ranked, budgeted prompt fragment using the dependency graph. Phase 1 admits callees whose declaring class is within two hops of the enclosing class (self-calls at distance 0 always admitted), rendering distance-1 callees as full bodies and distance-2 callees as signatures. Phase 2 replaces any group of two or more surviving candidates whose classes share an immediate inheritance ancestor with that ancestor's body plus one concrete delegate, eliminating near-duplicate siblings. Phase 3 scores survivors by edge-type weight divided by squared graph distance, giving inheritance and interface edges the highest weight, and emits candidates greedily under the token budget. Each emitted callee carries an inclusion reason that maps to a visible graph node, which is what makes the prompt auditable.","core_discovery":"The central claim is that the graph the developer sees should be the same data structure the LLM prompt builder reasons over. Concretely, GACP is a pure, static, deterministic function: from the enclosing class, AST-extracted project-internal callees, a read-only graph index, and a token budget, it returns a ranked list of callee records, and each record's inclusion reason maps one-to-one to a visible node. The paper demonstrates this on six method-body selections from an open-source Java project. Across three LLM families, prompt-token counts are identical for every fixture and mode, confirming that the algorithm's output is gateway-independent. On the three fixtures with clone groups, the inheritance-collapse phase replaces $k$ sibling bodies with the shared parent contract plus one concrete delegate, cutting or limiting token growth. The paper is explicit that it makes no human-comprehension claim; what it establishes is the behavior of the prompt-construction algorithm.","pith_inferences":["The paper's own coarse accuracy rubric saturates at 2/2 for all modes, so the decisive next test is a human or behavior-based study that asks whether GACP-assembled prompts actually improve explanation quality, not just whether they change the prompt.","If GACP's principle generalizes, method-level call graphs could give finer context than class-level two-hop reachability; the paper lists this as future work but does not test it.","Because GACP is deterministic and parameterized, a head-to-head comparison with embedding-retrieval context selection on explanation quality is directly runnable, and the paper explicitly says it did not run one.","A parameter sweep of the hop cutoff and edge-type weights is an obvious stress test: the claimed structural advantage could shift on codebases with deep call chains or different clone patterns."],"forward_implications":["The same selection and budget produce identical prompt tokens on three different LLM families for every fixture, so explanations are backend-independent.","When Phase 2 fires, the prompt shape changes from $k$ duplicate sibling bodies to one shared parent contract plus one delegate, which the paper argues better reflects the code's structure.","A hard two-hop cutoff can admit structurally proximate helpers that a fixed five-callee cap would drop, while deeper helpers are excluded because they add tokens without enough comprehension signal.","Every callee citation in the explanation can be followed back to a visible graph node, giving the developer a concrete way to verify what the LLM was shown.","Because GACP is deterministic and free of LLM calls, its output can be replayed, tested, and compared across tools, a property selection-only and embedding-retrieval approaches do not offer."],"supporting_citations":[{"why":"Identifies the selection-only interaction paradigm whose insufficiency motivates adding project-internal callees.","marker":"[3]"},{"why":"Supplies empirical grounding that selection-only prompts can lead to shallow or hallucinated explanations.","marker":"[5]"},{"why":"Represents embedding-similarity retrieval, the alternative selection strategy GACP contrasts with.","marker":"[10]"},{"why":"Represents perplexity-based prompt compression, another non-structural context-selection method.","marker":"[13]"},{"why":"Provides the clone-detection engine whose overlay creates the clone groups that trigger the collapse phase.","marker":"[21]"},{"why":"Makes the plugin, benchmark fixtures, and replay harness available for reproducibility.","marker":"[26]"},{"why":"Provides the interactive graph rendering layer so the developer's visible graph is also the graph GACP queries.","marker":"[30]"},{"why":"Provides the AST parsing that extracts callees and constructs the graph edges.","marker":"[31]"}],"fun_headline_variants":["Graph, not embeddings, selects LLM context in IDE","Developer's graph picks LLM prompt callees, reproducibly","Same visible graph drives LLM context in Java IDE","Callee graph yields identical LLM prompts across families","Traceable LLM context from the developer's own graph"],"cache_read_input_tokens":10752,"weakest_assumption_plain":"The load-bearing premise is that including project-internal callees selected by graph distance and edge-type weighting improves, or at least does not hurt, LLM code explanations compared with showing only the selected code.","fun_headline_variants_meta":{"raw":{"variants":["Graph, not embeddings, selects LLM context in IDE","Developer's graph picks LLM prompt callees, reproducibly","Same visible graph drives LLM context in Java IDE","Callee graph yields identical LLM prompts across families","Traceable LLM context from the developer's own graph"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1368,"prompt_tokens":911,"completion_tokens":457,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":376}},"tokens_in":527,"tokens_out":457,"duration_ms":4314,"temperature":1.0,"reasoning_tokens":376,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:09:59.264084+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a behavior-based user study on the same six code fixtures comparing selection-only prompts against GACP-assembled prompts on explanation accuracy or task completion; if GACP does not improve outcomes, the practical value of the algorithm is unestablished.","supporting_citations":[{"cited_title":"GitHub Copilot Chat,","cited_arxiv_id":null,"evidence_quote":"Identifies the selection-only interaction paradigm whose insufficiency motivates adding project-internal callees."},{"cited_title":"ChatGPT in Action: Analyzing Its Use in Software Development,","cited_arxiv_id":null,"evidence_quote":"Supplies empirical grounding that selection-only prompts can lead to shallow or hallucinated explanations."},{"cited_title":"Cody: AI Coding Assistant That Knows Your Codebase,","cited_arxiv_id":null,"evidence_quote":"Represents embedding-similarity retrieval, the alternative selection strategy GACP contrasts with."},{"cited_title":"LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models,","cited_arxiv_id":null,"evidence_quote":"Represents perplexity-based prompt compression, another non-structural context-selection method."},{"cited_title":"Analysis and Visualization for Clone Refactoring,","cited_arxiv_id":null,"evidence_quote":"Provides the clone-detection engine whose overlay creates the clone groups that trigger the collapse phase."},{"cited_title":"Cytoscape.js: a graph theory library for visualisation and analysis,","cited_arxiv_id":null,"evidence_quote":"Provides the interactive graph rendering layer so the developer's visible graph is also the graph GACP queries."},{"cited_title":"Eclipse Java Development Tools (JDT),","cited_arxiv_id":null,"evidence_quote":"Provides the AST parsing that extracts callees and constructs the graph edges."}],"review_version":1}