{"id":"df22ac01-ece2-40f8-aa85-23ae7db1837c","arxiv_id":"2511.00215","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A zero-shot LLM prompting scheme (local categorization + external filtering) detects code-documentation incorrectness with low flag rates and about 0.6 precision across Python, TypeScript, C++, and Java.","lead":"DocPrism uses off-the-shelf LLMs to find places where a function's documentation promises or states something the code does not actually do, and it filters out less-important 'under-promise' gaps. Its LCEF prompting method cuts the rate at which code gets falsely flagged from about 98% to about 14% in the authors' tests, with precision around 0.6 across four languages.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ablation supporting LCEF is computed on the 52-function construction set used to develop the method; no holdout ablation is reported, so the headline 98%→14% flag-rate improvement may be inflated by tuning.","rationale":"The reader's verdict is CONDITIONAL, and the missing holdout ablation is a concrete internal-validity threat that the reader noted in the rationale but did not make the primary weakest assumption. The reader's weakest assumption focuses on the author-made ground-truth labels; that is also important, especially because the kappa was computed on only 50 functions in one language, and extension precision has no recall. However, the more directly load-bearing issue for the central claim is the in-sample ablation: Table 6 is the only evidence for the headline 98%→14% flag-rate and F1 0.22→0.77 improvements, and it comes from the construction set used during method development. The 72-benchmark holdout is mentioned but never used for the ablation or even reported separately in Table 1. This is a fixable methodological gap, so not a reason to reject outright, but it must be addressed before the central LCEF claim can be accepted. The missing SOTA comparison from the abstract is also serious, but the core contribution is LCEF, so the in-sample evaluation is the more fundamental issue. I agree with the reader's CONDITIONAL verdict: the paper is plausible and the qualitative analysis is useful, but the numerical headline claims need stronger out-of-sample support.","tokens_in":18708,"tokens_out":3496,"duration_ms":32503,"concrete_test":"Use the artifact's 72 held-out Python benchmarks (excluding the 52 construction-set functions) and run all seven prompt variants plus DocPrism with LLaMA 3.1-70B at temperature 0, exactly as in §4.3. Compute flag rate, precision, recall, accuracy, F1, and under-promise rate, and compare to Table 6. A successful replication would show DocPrism's flag rate near 14% and F1 near 0.77 on the holdout; if flag rate exceeds ~30% or F1 falls below ~0.4, the headline LCEF claim is not supported on unseen data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—LCEF makes zero-shot LLM inconsistency detection usable by cutting flag rate from 98% to 14% and raising F1 from 0.22 to 0.77—is supported almost entirely by Table 6, which reports the ablation on the construction dataset (§4.7.2, Table 6). §4.2.1 states this 52-benchmark subset was examined while creating DocPrism and LCEF; it is therefore a tuning set, not a test set. The paper never reports the same ablation on the 72 held-out Python benchmarks or on the full 124-benchmark labeled set. Because the prompt wording, JSON schema, and external-filter design could have been selected to minimize flag rate on these 52 cases, the observed improvement may be an artifact of selection. The extension datasets (§4.6) only report precision on flagged functions, with no recall or accuracy, so they cannot validate F1 or flag-rate gains. The abstract's missing SOTA numbers further weaken the comparative claim. If the holdout ablation does not reproduce the improvement, the core contribution of LCEF is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents DocPrism, a zero-shot LLM-based tool for detecting incorrectness inconsistencies between function-level documentation and code. To avoid the high false-positive rates of plain LLM prompting, the authors introduce Local Categorization, External Filtering (LCEF): the LLM is prompted to answer per-category yes/no check-in questions encoded in a JSON schema, and under-promise findings are removed by a deterministic external filter after generation. The tool is evaluated on a manually labeled Python dataset (124 functions, 14 inconsistent) and on extension datasets in TypeScript, C++, and Java (1,491 functions), with manual inspection of flagged items. The paper reports low flag rates (11–18%), precision around 0.6, and a headline ablation result in which LCEF reduces the flag rate from 98% to 14% and increases F1 from 0.22 to 0.77 on a 52-benchmark construction dataset.","tokens_in":18959,"tokens_out":7370,"duration_ms":65113,"significance":"If the reported results hold, the paper makes a useful practical contribution: it demonstrates that a simple, fine-tuning-free prompt structure can make LLM-based inconsistency detection tractable for developers by keeping the flag rate low while preserving moderate precision, and it goes beyond binary classification by producing highlighted snippets and explanations. The LCEF idea—decomposing a categorization task into local check-in questions and delegating filtering to deterministic code—is simple, plausible, and potentially transferable to other LLM-based SE tasks. The paper is also transparent about its design trade-offs, releases an artifact, performs manual error analysis, and includes a limited cross-LLM check with GPT-4.1. The main weakness is that the central comparative claim for LCEF is supported only by an in-sample ablation, and some abstract-level claims are not backed by experiments in the body.","major_comments":[{"comment":"The headline ablation result (98%→14% flag rate, F1 0.22→0.77) is computed on the 52-benchmark construction dataset that was explicitly used while developing DocPrism and LCEF. Section 4.2.1 states that this subset was examined at tool-creation time. The paper does not report the same ablation on the 72 held-out Python benchmarks or on the full 124-benchmark labeled set; Table 1 reports only DocPrism's final performance on the full set, not the V1–V7 variants. Because the JSON schema, check-in wording, and external-filter design could have been selected on these 52 cases, the observed improvement may be substantially inflated by selection. This is load-bearing for the abstract's central claim. The authors should either run the ablation on the held-out or full labeled Python set, or explicitly restrict the claim to the construction dataset and temper the general statement.","section":"§4.2.1, §4.7.2, Table 6"},{"comment":"The provided abstract states: 'DocPrism achieves precision comparable to the state-of-the-art on an established synthetic dataset, but substantially outperforms it on our real-world Java dataset in precision (DocPrism: 0.47-0.67 vs. SOTA: 0.05-0.14).' It also claims 'a conservative lower bound across four programming languages... 11% of code-documentation pairs.' Neither of these claims is supported by any experiment, table, or derivation in the manuscript body. Section 5 only mentions prior precision numbers (icomment 0.61, tcomment 0.48, clean-dataset precision 0.54–0.61) but does not compare them to DocPrism on the same data. If these comparisons exist, they must be added to the paper; if not, the abstract must be corrected. An unsupported state-of-the-art comparison is a serious accuracy issue in the submission's summary.","section":"Abstract (arXiv metadata)"},{"comment":"The fully labeled Python evaluation rests on only 14 inconsistent functions (9 TP + 5 FN). The reported precision, recall, and F1 of 0.64 are therefore estimates based on a small positive denominator, and no confidence intervals are provided. For example, a 95% Wilson interval for 9/14 is roughly 0.39–0.84, which spans values that would change the qualitative conclusion. The same issue affects the construction-set ablation in Table 6, where the positive count is likely even smaller. The paper should report confidence intervals or at least explicitly acknowledge the width of these estimates. This is load-bearing for the absolute metric claims, though it does not invalidate the design itself.","section":"§4.5.1, Table 1"},{"comment":"For the extension datasets, recall, accuracy, and F1 cannot be computed because only flagged functions were manually labeled. The paper states this clearly, but the multi-language generalization claim in the abstract ('maintains a low flag rate... precision') is consequently limited to precision and flag rate. This is acceptable as a scoping statement, but the paper should be explicit that the LCEF benefit—particularly the recall/F1 improvement—is not validated across languages. The current wording in the introduction ('across this diverse dataset it achieves... a flag rate of 15%, and a precision of 0.62') reads as a broad performance claim without this caveat. Please add the caveat to the abstract and introduction.","section":"§4.2.2, §4.6, Table 4"}],"minor_comments":[{"comment":"The arXiv metadata abstract and the paper's own abstract disagree on several numbers: metadata says F1 0.22→0.77 and flag rate 17%, precision 0.63; the paper text says accuracy 14%→94% and flag rate 15%, precision 0.62. Both are based on the same tables, so one version should be aligned. Also, RQ3 heading in Section 4 misspells 'External' as 'Exernal'.","section":"Abstract vs. full text"},{"comment":"The JSON key for the direct-mismatch check-in is printed as 'Does_the_code_correctly_implement_what_is_mentioned in_the_documentation' with a space before 'in_the_documentation'. This appears to be a formatting artifact but should be corrected for clarity, since the schema is central to the method.","section":"§3.1.2"},{"comment":"The GPT-4.1 robustness check reports only flag rates (87%→20%) and no precision, recall, or number of benchmarks. This is a useful sanity check but is too coarse to support the claim that 'LCEF remains effective' beyond flag-rate reduction. Consider reporting the same metrics as Table 6 if feasible.","section":"§6 (Generalization to Other LLMs)"},{"comment":"The definition of inconsistency-level precision is never explicitly written, though it is used in Tables 1 and 4. Please add the formula or a verbal definition for consistency with the function-level metrics.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The missing SOTA comparison in the body despite its presence in the abstract is the most serious issue; I would ask the authors to either add the experiment or remove the claim. The in-sample ablation is the second key concern; adding a held-out ablation on the 72 Python benchmarks is feasible and should be required. The small number of positives is a broader concern but can be addressed with confidence intervals and cautious wording."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the core idea is smaller than the packaging. DocPrism's LCEF—check-in yes/no questions in a JSON schema paired with follow-up keys, then dropping under-promises externally—is a concrete, portable technique that is genuinely different from plain instructed categorization. The paper gives it a strong motivation (LLMs over-flag under-promises) and shows it works across four languages, with a useful error taxonomy. That is a real contribution to the SE/LLM tooling space.\n\nThe paper is also transparent about its main weakness, which is more than many do: it states explicitly that 52 of the 124 Python benchmarks were used while building DocPrism. But that means Table 6, the ablation that produces the headline numbers (98%→14% flag rate, F1 0.22→0.77), is in-sample. The construction set is a tuning set, not a test set. The stress-test note is right about that, and the paper doesn't report the same ablation on the 72 held-out Python pairs. It does show flag rates on three other languages in Table 7, which supports the qualitative direction, but those datasets have no recall/labels, so they can't confirm F1 or precision gains.\n\nOther soft spots are smaller but real. The fully-labeled Python set is 124 functions with 14 inconsistencies; the headline F1 rests on that. Extension precision is computed only on flagged functions, which is fine for precision but says nothing about missed inconsistencies. The inter-coder reliability is one kappa on 50 functions in one language. And the abstract and body disagree on flag rate/precision (17%/0.63 vs 15%/0.62), which suggests sloppy final-number-checking. More importantly, the abstract's claim about SOTA comparison does not appear in the body—no such evaluation is reported. That is a real mismatch.\n\nI don't think any of this is fatal. The LCEF idea is plausible and the mechanism—making the LLM answer local yes/no questions rather than recall category definitions—is well argued. The qualitative analysis of true and false positives is careful and useful. But the central quantitative claim needs a proper holdout ablation and a reported SOTA comparison before I'd trust the magnitude of the effect.\n\nWho should read this: people building LLM-based tools for code/docs tasks, and anyone working on prompt structures for classification with LLMs. It deserves a serious referee, but with requests for: (1) the ablation on the 72 held-out Python benchmarks, (2) the missing SOTA numbers, (3) consistency fixes between abstract/body, and (4) ideally, code release with commit hash.","headline":"LCEF is a real and useful prompt-engineering idea, and the paper reads honestly, but the headline ablation is in-sample, so the 98%-to-14% flag-rate drop is not yet established.","tokens_in":19464,"tokens_out":2162,"would_cite":true,"duration_ms":22398,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DocPrism claims decomposing doc-code inconsistency detection into local yes/no questions plus an external under-promise filter cuts LLM flag rate from 98% to 14%.","keywords":["code-documentation inconsistency","LLM zero-shot detection","over-promise","direct mismatch","under-promise filtering","local categorization","external filtering","multi-language software analysis"],"falsifier":"Have a fresh set of annotators who are blind to the authors' labels independently mark which of the 1,615 code-documentation pairs contain an over-promise or direct mismatch, then run DocPrism and compute precision against those labels; the central claim fails if precision falls below roughly 0.5 or if a majority of the fresh annotators judge a substantial share of the filtered-away under-promises as actionable. A simpler observational check: if a replication on a new language (e.g., Go or Rust) under independent labeling yields a flag rate above 25% or precision below 0.4, the claimed general","tokens_in":18528,"feed_emoji":"🐛","tokens_out":8288,"duration_ms":73180,"temperature":0.7,"pith_summary":"DocPrism sets out to show that large language models can be made useful for spotting incorrectness inconsistencies between function-level documentation and code—specifically over-promises and direct mismatches—without fine-tuning. The paper argues that plain prompting makes LLMs flag nearly every function because they report benign under-promises, cases where the code does more than the documentation says. The proposed remedy, Local Categorization and External Filtering (LCEF), splits detection into per-category yes/no questions encoded in a JSON schema and then discards under-promise answers during post-processing. Across 1,615 real functions in Python, TypeScript, C++, and Java, DocPrism holds a 14-18% flag rate with roughly 0.6 precision; an ablation shows flag rate dropping from 98% to 14% and F1 rising from 0.22 to 0.77. If correct, developers can run practical post-hoc doc-code mismatch detection on existing codebases in many languages with a standard LLM and no training data.","feed_headline":"Local yes/no checks cut LLM doc-code false flags from 98% to 14%","feed_subtitle":"Local yes/no checks plus a filter make zero-shot LLM detection of doc-code bugs practical in four languages.","key_machinery":"Local Categorization, External Filtering (LCEF). The system prompt is a JSON schema whose entries are check-in keys (CK)—short yes/no questions, one per inconsistency category (over-promise, direct mismatch, under-promise)—paired with self-contained, verbose follow-up keys (FK) that are filled only when the check-in answer calls for it. This forces the LLM to act as a completion engine on each local field rather than performing open-ended reasoning, and the external filtering stage simply removes the under-promise JSON entries before producing the report. The mechanism does two jobs: it makes category assignment reliable enough that the filter can trust the labels, and it is the primary leve","core_discovery":"The paper's central claim is that the standard way of prompting LLMs for code-documentation consistency is structurally wrong: asking an LLM to 'find inconsistencies' exploits its tendency to agree and its difficulty ignoring benign abstraction gaps, so it flags 90% or more of functions. The fix is to make the model answer three narrow yes/no questions—one per inconsistency kind—embedded as JSON keys with self-contained follow-ups, and to drop the under-promise answers after generation. With this Local Categorization + External Filtering recipe, the paper reports flag rate falling from 98% to 14% and F1 rising from 0.22 to 0.77 on its 52-function construction set, and, on 1,615 real function","pith_inferences":["The check-in/follow-up schema is a generic antidote to LLM over-reporting in verification-style tasks; analogous 'is this specific property violated?' schemas could reduce false positives in security triage, test generation, or compliance review, where the same sycophantic bias exists.","The paper assumes under-promises are never worth showing; a tool that surfaces them as optional low-severity warnings, rather than dropping them outright, might capture the cases where an omitted detail is a real trap (the Tesseract example in the paper hints the LLM thought so) without losing the flag-rate benefit.","Because the ground-truth labels were produced and adjudicated by the authors themselves, with only one 50-function kappa check, an independent re-labeling study is the natural next step; if other developers disagree about which under-promises are harmless, the 0.52-0.70 precision numbers may not transfer to new settings.","The 11% lower bound likely understates the real rate: recall is not measured on the extension datasets, and the construction dataset's labeled recall of 0.64 suggests actual inconsistencies may be more frequent than the flagged ones."],"forward_implications":["A standard LLM, prompted with LCEF and no fine-tuning, can flag over-promises and direct mismatches in Python, TypeScript, C++, and Java at a flag rate of 14-18% and precision around 0.6, tractable for manual developer review.","Because DocPrism needs no diff or edit history, it can be run post hoc on existing repositories to detect latent inconsistencies that just-in-time tools miss.","Across four popular real-world projects, at least 11% of function-documentation pairs contain an incorrectness inconsistency (over-promise or direct mismatch); this is a conservative lower bound since unflagged pairs were not exhaustively checked.","The ablation isolates the gain: pairing check-in questions with verbose follow-up keys reduces the under-promise rate from 50% to 0% relative to the closest instructing-and-filtering alternative, and the effect appears to transfer to a different LLM family in a supplementary run reported in the paper."],"fun_headline_variants":["Local yes/no checks trim LLM doc-code false flags from 98% to 14%","Ask LLM three yes/no questions, not open-ended, to catch doc-code bugs","Zero-shot LLM doc-code check: flag rate drops 98% to 14%","Three yes/no prompts make LLM doc-code detection practical","DocPrism: local LLM queries cut false inconsistency flags by 84%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire evaluation rests on the authors' own hand-written labels of what counts as a reportable inconsistency—with only a single Cohen's kappa of 0.78 computed on 50 functions in one language—so if other developers draw the under-promise boundary differently, both the precision numbers and the design decision to suppress under-promises lose their foundation.","fun_headline_variants_meta":{"raw":{"variants":["Local yes/no checks trim LLM doc-code false flags from 98% to 14%","Ask LLM three yes/no questions, not open-ended, to catch doc-code bugs","Zero-shot LLM doc-code check: flag rate drops 98% to 14%","Three yes/no prompts make LLM doc-code detection practical","DocPrism: local LLM queries cut false inconsistency flags by 84%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000829,"raw_usage":{"total_tokens":3525,"prompt_tokens":875,"completion_tokens":2650,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":2553}},"tokens_in":619,"tokens_out":2650,"duration_ms":17164,"temperature":1.0,"reasoning_tokens":2553,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T06:52:22.612733+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a fresh set of annotators who are blind to the authors' labels independently mark which of the 1,615 code-documentation pairs contain an over-promise or direct mismatch, then run DocPrism and compute precision against those labels; the central claim fails if precision falls below roughly 0.5 or if a majority of the fresh annotators judge a substantial share of the filtered-away under-promises as actionable. A simpler observational check: if a replication on a new language (e.g., Go or Rust) under independent labeling yields a flag rate above 25% or precision below 0.4, the claimed general","supporting_citations":[],"review_version":1}