{"id":"a03a42b4-f30d-41a6-aa71-12f3caf2dbd9","arxiv_id":"2504.17426","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Using LLM-generated summaries as input to BERTopic can extract coherent topics from Python source code even when comments and function names are removed.","lead":"This paper tests whether summarizing source code with a large language model, then applying topic modeling to those summaries, can reveal meaningful topics even when code comments and function names are removed. The authors report that the LLM-based topics align reasonably well with topics derived from original docstrings and outperform topics from function names alone.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Obfuscation leaves identifiers inside function bodies intact; the 'solely code structure' claim is confounded by residual natural-language cues.","rationale":"The paper combines LLM-generated summaries with BERTopic to infer code topics, and its headline contribution is that this works without natural-language cues. The preprocessing, however, removes only function declaration names and comments; all other identifiers remain. Since summaries are generated from code containing those identifiers, the experiment cannot distinguish structure-based understanding from identifier-based reading. This is a treatment-confounding problem, not a disagreement with consensus. The reader's concern about docstring quality is secondary: even a noisy reference could support a relative comparison, but residual identifiers directly undermine the independent variable. I would keep the paper's conditional verdict and add the full-identifier obfuscation ablation as the decisive check. If the check fails, the claim should be narrowed to 'without comments or function names'; if it passes, the structure-only claim is supported. The paper does provide a public artifact and a coherent internal-quality analysis, which is why I do not recommend rejection without testing.","tokens_in":8386,"tokens_out":7925,"duration_ms":80847,"concrete_test":"Run an AST-based full-identifier obfuscation ablation on the same 10,000 CodeSearchNet functions: replace every local variable, parameter, attribute, and class name with placeholders while preserving control flow, literals, and external API calls, instead of only regex-replacing declaration names. Rebuild Msumm and recompute the Table 2 rows (d_TOP_w and d_cap) and topic coherence. If the summary-vs-docstring alignment drops toward the function-names baseline or coherence falls well below the reported ~0.60, the published results depend on residual identifiers and the 'solely code structure' claim fails. If the metrics remain comparable, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that topics are modeled 'by relying solely on code structure rather than leveraging natural-language elements' (Section 5) is not actually isolated by the experiment. Section 3 says only 'function names in their declarations' were replaced with a placeholder and comments removed; no AST-level renaming of local variables, parameters, attributes, or class names is described. Section 2 states that function names are obfuscated 'so as to remove the intrinsic code naturalness,' and Section 5 claims identifiers were obfuscated, but the described preprocessing does not do that. CodeSearchNet Python functions are identifier-rich, so Gemma2 summaries can exploit surviving names such as 'image', 'response', 'request', or 'dataframe'. Because these summaries are the sole input to Msumm and are also fed into Mdoc for the first row of Table 2, the measured alignment with docstring topics may be driven by residual identifier semantics rather than by code structure. The function-names baseline does not control for this: it removes body identifiers entirely and therefore tests a much weaker signal. The strongest claim is thus unsupported by the current design.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a topic-modeling pipeline for Python source code that replaces function names with a placeholder, removes comments, asks an LLM (Gemma2 2B-it) to generate natural-language summaries of the obfuscated code, and then applies BERTopic to those summaries. The authors compare the resulting topics, via four distance metrics, against topics inferred from docstrings and from tokenized function names, reporting that summary-based topics align more closely with docstring-based reference topics than do function-name-based topics, and that the summary-based topics have higher internal coherence. The stated central claim is that this approach models topics from source code by relying solely on code structure rather than natural-language elements. The paper includes public data and code on Zenodo.","tokens_in":8548,"tokens_out":3138,"duration_ms":29865,"significance":"If the central claim held, the paper would offer a practical method for topic modeling on poorly documented or identifier-poor code, and it would extend the line of work on LLM-based code understanding. The main strengths are the simple, reproducible pipeline, the public release of data and code, and the use of multiple complementary evaluation metrics. However, the strongest conceptual claim (that the method relies solely on code structure) is not yet supported because the obfuscation step leaves most identifiers inside function bodies intact, and the evaluation lacks statistical significance testing. The paper is best viewed as a promising preliminary study whose headline conclusion requires substantially stronger evidence.","major_comments":[{"comment":"The preprocessing described in Section 3 only replaces 'function names in their declarations' and removes comments; it does not rename local variables, parameters, attributes, class names, or other identifiers appearing inside function bodies. CodeSearchNet Python functions are identifier-rich, so the LLM summaries can be driven by surviving names such as 'response', 'request', or 'dataframe'. Consequently, the claim in Section 5 that the method models topics 'by relying solely on code structure rather than leveraging natural-language elements' is not supported by the experiment as described. The authors should either obfuscate all identifiers (or otherwise control for body-level naturalness) and rerun the experiments, or substantially qualify the claim.","section":"Sections 3 and 5"},{"comment":"Table 2 reports average distances without any variance estimates, confidence intervals, or significance tests. For instance, the d_TOP values 3.46 vs 3.11 and the d_cap values 3.71 vs 2.18 are presented as evidence that summaries outperform function names, but with only 500 evaluation instances it is unclear whether these differences are statistically reliable. A paired test (e.g., Wilcoxon signed-rank or bootstrap) over the per-document distances is straightforward and should be added before the word 'significantly' is used.","section":"Section 4, Table 2"},{"comment":"The number of topics, nr_topics = 40, was selected after 'running the model multiple times with varying values and evaluating the results' on what appears to be the same corpus used for the reported evaluation. This post-hoc selection on the evaluation data can inflate coherence scores and makes the reported absolute values hard to interpret. The authors should report sensitivity to nr_topics (e.g., a range of values) and clarify whether the held-out 500-instance evaluation set was used at all during parameter selection.","section":"Section 3"},{"comment":"The coherence comparison between summary-based topics (average C_v ≈ 0.60) and docstring-based topics (0.38) is not a fair comparison, because C_v is sensitive to corpus size, vocabulary, and document length, and the two corpora differ on all of these dimensions. The docstring corpus and the summary corpus are different document sets with different vocabularies, so the difference in coherence may reflect corpus properties rather than the quality of the underlying topic representations. A matched comparison, or at least an explicit discussion of this confound, is needed.","section":"Section 4"},{"comment":"Treating docstrings as the 'golden standard' for code topics is questionable, since docstrings can be noisy, outdated, or only loosely related to the actual code behavior. Additionally, the only metric that can be computed between Msumm and Mdoc is d_cap, which counts shared top words across different topic vocabularies; this is a weak basis for the claim that the two topic models are 'close.' The authors should discuss the limitations of this reference standard and of the cross-model metric, and ideally validate against an external human-judged topic assignment.","section":"Section 2, Table 2"}],"minor_comments":[{"comment":"The definitions of the four metrics are muddled: the text defines d_TOP three times and never gives the explicit formula for d_cap. The third definition should be labeled d_cap, and the formula 'number of common words between the 5 most relevant words' should be written consistently with the notation introduced earlier.","section":"Section 2"},{"comment":"There are typos in the top-word lists: 'respnse' should be 'response' and 'rended' should likely be 'rendered'. Also, some tokens such as 'colormathcolorconversions' and 'colormathcolorobjects' appear as single vocabulary items, which reduces interpretability; consider whether post-processing should split or filter such tokens.","section":"Table 1"},{"comment":"The description of the prompt template is unclear: the 'Base query' and 'Prompt template' lines suggest concatenation but the exact ordering and the role of the '##### Description:' marker are not fully specified. Providing the exact prompt string used would improve reproducibility.","section":"Section 3"},{"comment":"The paper does not report the decoding parameters for Gemma2 2B-it (e.g., temperature, top-p, max new tokens). Since the summaries are the sole input to the topic model, these parameters can affect the results and should be stated.","section":"Section 3"},{"comment":"There is an internal inconsistency in describing the obfuscation: Section 1 and Section 5 say 'functions’ identifiers' are obfuscated, while Section 3 says only 'function names in their declarations' are replaced. This inconsistency should be resolved, especially in light of the major comment about residual identifiers.","section":"Section 1 and Section 5"},{"comment":"Figures 2 and 3 have minimal captions and no axis labels. Figure 3's color-based topic assignments are hard to interpret without a legend or topic-number labels; consider adding these.","section":"Figures 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a plausible initial study with a public artifact, which I view positively. However, the central claim appears to be confounded: the obfuscation step is too weak to rule out reliance on residual identifiers, and the evaluation lacks error bars. If the authors can rerun the experiments with full identifier obfuscation (or a convincing control) and add significance testing, the paper could become acceptable. As it stands, the contribution is more of an exploratory result than a supported claim of structure-only topic modeling."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this one. The combination is genuinely new as a pipeline: LLM-generated summaries of code fed into BERTopic for topic modeling, evaluated against docstring-derived topics and function-name-only topics. The paper is clearly written, well-scoped, and ships a public artifact. The qualitative topics in Table 1 look plausible and the idea of using summaries as a surrogate for missing documentation is worth testing.\n\nBut the central claim as worded does not hold. The authors say they \"obfuscated the functions' identifiers\" and conclude the approach works \"by relying solely on code structure rather than leveraging natural-language elements.\" The experimental section, though, says only that function names in their declarations were replaced with a placeholder and comments removed. No AST-level renaming of parameters, local variables, attributes, or class names is described. CodeSearchNet functions are full of meaningful body identifiers like `image`, `response`, `dataframe`, and `request`. Gemma2 sees those, and so do the summaries that feed both Msumm and the first row of Table 2. The function-names baseline removes all body identifiers by construction, so it tests a much weaker signal. The comparison therefore does not isolate code structure from residual identifier semantics.\n\nOther soft spots are secondary but real. Docstrings are treated as a golden standard without discussion of how noisy or out-of-date they can be. There are no error bars or significance tests on any of the Table 2 numbers. The number of topics was chosen after inspecting results, which is fine for exploration but not for confirmation. And the coherence comparison of 0.60 for summary topics vs 0.38 for docstring topics compares two different text genres, not two equivalent representations; a cross-genre coherence gap tells you little about model quality.\n\nNone of this kills the paper. The pipeline is sensible, the artifact is a plus, and the negative result that function names alone do worse is a useful data point. What needs to change is the framing: the claim should be softened from \"solely code structure\" to \"with function names and comments removed,\" or the authors should actually obfuscate all identifiers and re-run. A serious referee could push the paper to that point. I'd send it to review rather than desk reject, but with the expectation of heavy revision and a narrower claim.","headline":"The summary+BERtopic pipeline is a reasonable new combination, but the 'solely code structure' claim is undercut because the preprocessing only replaced function names, leaving body identifiers fully intact.","tokens_in":9071,"tokens_out":2458,"would_cite":false,"duration_ms":22368,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-generated summaries of stripped-down code can serve as the input for topic modeling, producing meaningful topics without comments or meaningful function names.","keywords":["source code analysis","topic modeling","LLM summarization","code comprehension","docstring alignment","code structure semantics","software maintenance","function name obfuscation"],"falsifier":"Re-run the comparison using human expert labels instead of docstrings as the reference on a new set of Python functions; if summary-based topics do not beat function-name topics at predicting expert labels, the central claim fails.","tokens_in":8173,"feed_emoji":"🧩","tokens_out":6940,"duration_ms":62445,"temperature":0.7,"pith_summary":"The paper proposes that topic modeling for source code can be driven by LLM-generated summaries of code rather than by natural-language clues inside the code. It strips comments, replaces function names with placeholders, asks a language model to describe each function's purpose, and runs topic modeling on those descriptions. The comparison against topics derived from docstrings shows that summary-based topics are close to docstring-based reference topics, and clearly closer than topics derived from function names alone. If this holds, code repositories without documentation can still be organized and searched by topic using only code structure.","feed_headline":"LLM summaries beat function names at finding code topics","feed_subtitle":"Stripped of comments and names, code still yields coherent topics — closer to docstring models than name-only baselines.","key_machinery":"The load-bearing mechanism is a three-stage pipeline. First, preprocessing strips comments and replaces each function name with a placeholder, so the language model sees only code structure. Second, an instruction-tuned language model generates a plain-text purpose description for each function. Third, a transformer-embedding topic modeling method clusters these descriptions into topics, and each function gets a probability distribution over topics. The validation machinery is comparative: an identical topic model is built from the original docstrings as the golden reference, and four distance metrics, mean squared error over topic distributions, overlap of top-10 topics, cosine similarity of top-10 topic vectors, and shared top-5 words, measure how close summary-based and name-based inferences come to that reference.","core_discovery":"The central claim is that the structure of source code alone, with comments removed and identifiers obfuscated, carries enough semantics for an LLM to write descriptions from which a topic model can recover coherent, interpretable topics. The paper reports an average topic coherence near 0.60 for summary-based topics, versus 0.38 for topics built from original docstrings, and reports that summary-based topic assignments are more similar to docstring-based reference topics than are assignments made from function names on every metric where the comparison is defined. The conclusion is that LLM summaries compensate for missing documentation and outperform the natural semantics encoded in function identifiers.","pith_inferences":["If the summaries are what carry the semantics, the pipeline should transfer to other languages and other code granularities such as classes or modules with only prompt changes; a direct test would be to rerun it on Java or C++ snippets.","The docstring golden standard anchors the evaluation to one human-written documentation style; a stricter test would compare against task performance, such as code search retrieval or bug localization, where topic quality shows up behaviorally.","The coherence gap between summaries and docstrings may partly reflect that LLM summaries are more uniform in vocabulary than docstrings; controlling for summary length or vocabulary would test whether the gain is semantic or stylistic."],"forward_implications":["Topic models can be built for code with no comments, no docstrings, and no meaningful identifiers, making undocumented legacy or third-party code accessible to topic-based organization.","Summary-based topic assignment can serve as a proxy when documentation exists but is incomplete, with performance close to using the docstrings themselves.","Function names alone are a weaker signal than whole-code summaries, which implies identifier naturalness should not be the primary target for code topic extraction.","High topic coherence from summaries suggests LLM-generated descriptions can be a reusable substrate for software maintenance tasks such as automatic tagging, documentation, and repository reorganization."],"supporting_citations":[{"why":"Supplies the 10,000 Python functions and their docstrings; without it there is no experimental corpus or reference target.","marker":"[10]"},{"why":"Provides the transformer-embedding topic modeling method used to turn summaries, docstrings, and names into topics.","marker":"[9]"},{"why":"The language model that generates the summaries from obfuscated code, serving as the central source of semantic descriptions.","marker":"[6]"},{"why":"Defines the C_v coherence measure used to judge topic interpretability in the results.","marker":"[17]"},{"why":"Establishes the prior result that removing natural content lowers code summarization performance, the baseline this work aims to improve on.","marker":"[8]"},{"why":"Motivates the combination of transformer embeddings with topic modeling for text, which the method relies on.","marker":"[16]"}],"fun_headline_variants":["LLM summaries beat names for code topics","Code topic modeling improves with LLM summaries","LLM summaries outperform function names in topic modeling","Summaries from LLMs yield better code topics than names"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes docstrings are the correct ground truth for what a code topic should be; if many docstrings are stale, generic, or mismatched to the code, the reference targets that everything is compared against are unreliable.","fun_headline_variants_meta":{"raw":{"variants":["LLM summaries beat names for code topics","Code topic modeling improves with LLM summaries","LLM summaries outperform function names in topic modeling","Summaries from LLMs yield better code topics than names"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000452,"raw_usage":{"total_tokens":2219,"prompt_tokens":834,"completion_tokens":1385,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":1326}},"tokens_in":450,"tokens_out":1385,"duration_ms":9638,"temperature":1.0,"reasoning_tokens":1326,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:39:57.241037+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the comparison using human expert labels instead of docstrings as the reference on a new set of Python functions; if summary-based topics do not beat function-name topics at predicting expert labels, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the C_v coherence measure used to judge topic interpretability in the results."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the prior result that removing natural content lowers code summarization performance, the baseline this work aims to improve on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the combination of transformer embeddings with topic modeling for text, which the method relies on."}],"review_version":1}