{"id":"ed90b61c-be8b-4094-8fda-72688eff3a85","arxiv_id":"2504.18702","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Codetations keeps rich, interactive annotations outside source files, tracks them through code edits, and lets LLMs generate new annotation types from a single prompt.","lead":"Codetations is a VSCode extension that stores notes and interactive widgets separately from code, attaching them to text spans that move as the code changes. It matters because it gives both developers and AI coding assistants a way to keep useful context attached to the exact code it describes without cluttering source files.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM repair results are confounded: the Appendix D context includes the full corrected code, so success with context may be copying rather than improved reasoning.","rationale":"The reader's weakest_assumption correctly identifies the small, hand-picked LLM evaluation as load-bearing, but the more severe problem is that Appendix D's context includes the complete corrected code. This is not merely a matter of sample size or generalizability; it undermines the causal interpretation of the with-context success. Section 7 describes constructing context with 'a hint at the true explanation mixed into a set of extraneous information,' but D.1 and D.2 show full before/after code in the context block. Unless those blocks were not actually passed to the model — which is nowhere stated — the model could have succeeded by copying. The user study and system design remain plausible qualitative contributions, and no mathematical or logical errors appear elsewhere. The appropriate disposition is therefore the same CONDITIONAL verdict the reader reached, with the acceptance conditions strengthened: either reframe Section 8.1.3 as illustrating a potential benefit under an optimistic reading, or run the concrete test above and report the outcome. I do not see grounds for outright rejection, since the paper's design-space contributions do not depend solely on the LLM repair result, but the abstract's claim that relevant notes 'improve LLM performance during code repair' should not stand without a valid demonstration.","tokens_in":26089,"tokens_out":4451,"duration_ms":47629,"concrete_test":"Re-run the two Appendix D repair tasks using the exact context specification, but delete all 'code after', 'debugged version', and corrected-mode snippets from the prompt, keeping only the original code, documentation snippets, debug outputs, and distractor notes. If GPT-4o still solves both cases, the conclusion survives; if performance collapses, the reported gains are attributable to answer leakage. Run at least 10 trials per condition with fixed temperature and report pass rates. As an additional check, add a third repair problem whose correct fix is never shown in any context block and verify whether context still improves repair.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim, stated in Section 8.1.3, is that GPT-4o solved repair tasks 'reliably only when the relevant slice of annotation context was present.' But the context blocks in Appendix D contain the exact fix. In D.1, the supplied context includes a 'Code before and after debugging' block with the complete corrected implementation that adds the authentication header. In D.2, it includes a 'Code comparison' block with the full HSU-mode initialization that resolves the NFC bug. As written, the 'relevant slice' is not a hint, a debug output, or a Codetations-style note; it is the patch itself. The described with-context behavior ('adds boilerplate to provide the authentication', 'switch the module to HSU mode') is fully consistent with the model copying or lightly adapting the provided correct code. This makes the comparison uninterpretable as evidence that external notes improve LLM repair. The evaluation also rests on only two examples with no repeated trials, no ablation separating documentation from code-after, no variation in distractor sets, and no exact prompt/context transcript. Section 7 candidly says the authors 'evaluated a few examples,' but Section 8.1.3 calls these 'controlled experiments' and says GPT-4o solved tasks 'reliably' only with context. That overstatement matters because the abstract and conclusion use this example to motivate the whole system. Unless the code-after blocks were withheld from the LLM prompt — which the appendix does not state — the reported result is an answer-leakage artifact, not a demonstration that curated local context helps LLMs reason about code.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Codetations, a VSCode extension that lets users attach persistent, interactive annotations to spans of code without modifying the source file. The system uses a hybrid edit-tracking/LLM-based anchoring method to keep annotations attached as the document evolves, and exposes an API that lets annotation types read/write document text, call the editor's LLM API, and host interactive UI widgets. The authors report a qualitative need-finding and prototype evaluation with nine experienced programmers, describe several annotation types they generated in minutes with LLM prompts, and present two worked examples in which GPT-4o is said to repair code bugs only when supplied with context that mimics Codetations-style annotations. The paper concludes that external annotations can improve LLM program repair and motivate building persistent annotation layers into IDEs.","tokens_in":26284,"tokens_out":4405,"duration_ms":40707,"significance":"The system design and qualitative user study are the main strengths of the paper. The idea of document-external, semantically anchored, interactive annotations is well-motivated by prior work, and the demonstration that new annotation types can be generated by prompting an LLM with API examples is a credible and useful contribution. The anchoring approach builds on the authors' prior Misback et al. work and is appropriately situated relative to Catseye, Sodalite, and other annotation systems. The user study (N=9) is modest but honestly reported, and the qualitative findings about documentation fragmentation, maintenance burden, and users' desire for non-intrusive rich annotations are plausible and useful for future tool builders. However, the paper's central empirical claim about LLM repair is not supported by the evidence as presented: the with-context condition in both worked examples includes the full corrected code, so the reported behavior is consistent with copying the patch rather than reasoning from annotation context, and the evaluation rests on only two examples with no repeated trials or statistics.","major_comments":[{"comment":"The 'with context' condition in both worked examples embeds the complete corrected code. In D.1, the context bullet 'Code before and after debugging' contains the full 'Debugged version' of list_repos with the authentication header; in D.2, the 'Code comparison' block contains the complete HSU-mode initialization sequence. The reported with-context responses ('adds boilerplate to provide the authentication', 'switch the module to HSU mode') are therefore consistent with copying or lightly adapting the supplied patch rather than with reasoning from annotation context. The comparison is uninterpretable as evidence that Codetations-style external notes improve LLM repair. The authors must either withhold the code-after blocks from the prompt and re-run the comparison, or add an ablation in which the context contains only the relevant hint bullet (e.g., the authentication requirement or the HSU-mode note) without the patch.","section":"Section 7.1 and Appendix D"},{"comment":"The paper's central RQ3 claim is stated in Section 8.1.3 as 'Our controlled experiments demonstrate that GPT-4o solved repair tasks reliably only when the relevant slice of annotation context was present.' Section 7, however, describes the evaluation as 'we also evaluated a few examples.' With only two examples, no repeated trials, no variation in distractor sets, and no reported prompt/context transcripts, the term 'controlled experiments' and the adverb 'reliably' overstate the evidence. Either report a systematic evaluation (e.g., multiple trials per condition, multiple models, ablation of context components) or explicitly present the two examples as illustrative anecdotes and temper the abstract and conclusion accordingly.","section":"Sections 7 and 8.1.3"},{"comment":"The method for guaranteeing that the bug information is 'latent in the model' is to generate the problems with GPT-4o without tool calling. This assumes that a model that can generate a problem statement can also solve the repair task from a zero-context prompt, which is not verified. The repair evaluation then uses GPT-4o again, so the zero-context failure may reflect prompt underspecification or the model's inability to retrieve the latent knowledge on demand, rather than the absence of contextual annotations. Add a sanity check: run each problem with the explicit bug explanation (or with only the relevant hint bullet) to confirm the model can solve it when given the key information, and also test at least one other model family to rule out model-specific effects.","section":"Section 7"}],"minor_comments":[{"comment":"The phrase 'interactively-collected data' in the abstract is not operationalized; the appendix shows hand-assembled context bullets, not data collected through the Codetations UI. Clarify what the annotations actually contained and how they were produced.","section":"Abstract and Section 7"},{"comment":"The exact prompts and the full context strings shown to the LLM are not reported; include a transcript or the exact template so the conditions can be reproduced.","section":"Appendix D"},{"comment":"The limitations paragraph discusses the user study but omits the limitations of the LLM evaluation (n=2, code-after confound, single model family); add a sentence acknowledging the anecdotal nature of the repair examples.","section":"Section 8.3"},{"comment":"The 'Show Debugged Example' annotation is said to execute via a 'webview interpreter'; specify whether this is a JavaScript interpreter embedded in the webview and whether it can execute code of languages other than JavaScript.","section":"Section 6.1 and Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest in Section 7 about the LLM evaluation being a few examples, but the abstract, Section 8.1.3, and conclusion elevate it to a demonstrated result. The code-after confound is fixable with an ablation or by withholding the corrected code from the prompt. The user study itself is a reasonable qualitative contribution, and the system description is clear enough for replication. The main revision needed is to bring the reported LLM-repair claim in line with the evidence, either by strengthening the evaluation or by clearly labeling the two examples as anecdotes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the Codetations system and the user study are worth your time, but the headline claim that annotation context makes GPT-4o reliably repair code is confounded — the appendix's \"context\" includes the full corrected code, so the model could be copying rather than reasoning.\n\nWhat's actually new: document-external annotations that stay out of the file, a hybrid edit-tracking/LLM anchoring method (built on their earlier Magic Markup work), and an API that lets annotations be stateful, interactive widgets. The demonstration that a single LLM prompt can generate a new annotation type — the Show Debugged Example — is a real, concrete contribution. The user study, though N=9, is honestly reported and the participants' reactions ring true: people want context attached to code without cluttering the source.\n\nThe soft spots are in the evaluation, not the architecture. Section 7 says they \"evaluated a few examples.\" Section 8.1.3 calls those same examples \"controlled experiments\" and says GPT-4o solved repair tasks \"reliably\" only with context. It is only two examples, with no repeated trials, no ablation separating documentation from the code-after blocks, and no exact prompt/context transcript. More seriously, the Appendix D context blocks include both the original and the fixed code — in D.1 \"Code before and after debugging\" and in D.2 \"Code comparison.\" The \"relevant slice\" isn't a hint or a debug output; it's the patch itself. The described behaviors — adding authentication, switching to HSU mode — are exactly what you'd get from lightly adapting the provided correct code. So the comparison is uninterpretable as evidence that external notes improve LLM reasoning. The paper's own limitations section is candid about the convenience sample and the cost of creating annotations, but the abstract and conclusion lean on the LLM example anyway.\n\nWho it's for: HCI and SE tool-builders who care about persistent context, annotations, and IDE integration. They will find the design space mapping and the qualitative results useful. A serious referee should engage with it — the system deserves attention — but the LLM-repair claim needs to be reframed as illustrative or dropped, and the artifact (code or data) should be released if the claim stays.\n\nI'd send it to peer review, conditional on the authors fixing or cutting the Section 8.1.3 claim.","headline":"A solid systems paper whose central LLM-repair claim leaks the answer in its own appendix.","tokens_in":26905,"tokens_out":2658,"would_cite":true,"duration_ms":26372,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Codetations argues that document-external, editor-hosted annotations keep code context alive and that relevant annotation context materially improves LLM program repair.","keywords":["document annotation","live programming","visual programming","end-user programming","language models","code generation","program repair","semantic anchoring"],"falsifier":"Run the paper's two repair prompts (the GitHub API authentication bug and the PN532 NFC bug) on a current sample of LLMs with no context and with randomized distractor sets; if a substantial fraction of models solve them without context, or if relevant hints do not beat distractors alone, the claim that relevant annotation context is necessary for reliable repair collapses.","tokens_in":25795,"feed_emoji":"📌","tokens_out":7077,"duration_ms":64323,"temperature":0.7,"pith_summary":"Codetations is a VSCode extension that attaches rich, interactive annotations to selected spans of code without modifying the source file. The paper's central claim is that these document-external notes solve two problems at once: developers get a clutter-free place to keep design rationale, examples, and debug data, and LLMs get the external context they need to repair unfamiliar code. Its controlled experiments, reported in Section 8.1.3, showed that GPT-4o solved repair tasks reliably only when the relevant slice of annotation context was present, and produced confident wrong fixes without it. A nine-developer user study found consistent interest in the non-intrusive, self-updating annotation model. If the claim holds, annotation layers that belong to the editor rather than the file become a cheap way to improve both human and machine understanding of code.","feed_headline":"Relevant notes make GPT-4o fix bugs it otherwise guesses wrong","feed_subtitle":"A new editor layer attaches live notes to code; without that context, the model confidently misdiagnosed the same repairs.","key_machinery":"The central machinery is the split between anchor records and document content. Anchors live in a separate annotation data file, so nothing is inserted into the source; online edits move anchors through low-latency editor heuristics, while offline changes trigger LLM-based semantic reattachment, using the method the paper cites as [25]. On top of that, annotation types act as living UIs: they receive the document text, their own data, and an API to call the editor's language model, so an annotation can regenerate its contents or run code when the anchored span changes. External storage, hybrid anchoring, and LLM-equipped annotation types together are what allow notes to survive code evolution and to act on their surroundings.","core_discovery":"Codetations treats context as a first-class, editor-hosted layer: an annotation record stores an ID, start and end positions in a document version, a way to obtain that version, an annotation type, and a data field, all kept in a JSON file outside the source. While the user types, anchor positions move with edit heuristics; when a file changes offline, the system asks permission and re-attaches anchors semantically using an LLM. Annotation types are React components with an API for reading and writing document text, reading and writing their own data, and calling the editor's language model, which is what lets notes regenerate their content and re-execute as code changes. The paper reports that a single prompt with API examples was enough to generate new annotation types, including the Show Debugged Example annotation that writes, runs, and displays test code for the anchored region. The empirical anchor is a small controlled comparison: for two GPT-4o-generated debugging problems, one a GitHub API authentication bug and one an NFC/UART hardware bug, the model misdiagnosed without context and diagnosed correctly when relevant annotation-style context was included alongside distractors; the paper concludes that curated local context still materially boosts state-of-the-art models and can guard against confident error.","pith_inferences":["If the two-case result generalizes, the bottleneck shifts from model capability to context curation: benchmarks for code repair should report which external hints were supplied, and annotation systems could standardize that reporting.","A testable extension would automate note generation from git history, runtime traces, or review comments, then measure whether the LLM-repair benefit persists for automatically produced notes rather than hand-picked hints.","Because unrelated context actively misled the model, one could build a relevance-scoring layer that ranks candidate annotations for each repair task; the paper leaves that design open.","The same anchoring machinery could attach notes to non-code documents such as specs and configuration files, and the LLM-repair result suggests a direct test for whether such notes help LLM agents in other structured-text domains."],"forward_implications":["Relevant, curated context is not optional for current LLM program repair: in the paper's two tasks, GPT-4o repaired the bugs only when the right hint was present, even though the knowledge was latent in the model.","Persistent annotation layers can be built without polluting source files, because anchor records live outside the document; this lets comments, diagrams, and debugging widgets scale without bloating code.","LLM-generated annotation types are practical: a single prompt plus API examples produced a working Show Debugged Example annotation, suggesting the ecosystem can grow quickly.","Irrelevant context is dangerous: when the model was given only unrelated documentation, it confidently produced incorrect diagnoses, so systems need context filtering and provenance tracking, not just retrieval.","Editor-hosted annotations lower the barrier to keeping context: study participants understood and valued the system once it lived in their IDE, and specifically valued robust re-anchoring and live widgets."],"supporting_citations":[{"why":"Supplies the LLM-based semantic reattachment method used when offline edits detach an annotation.","marker":"[25]"},{"why":"Documents the hallucination and missing-context problem that motivates giving LLMs explicit external context.","marker":"[22]"},{"why":"Provides the traditional text-comparison source-location tracking method that the hybrid anchoring approach builds on and compares against.","marker":"[28]"},{"why":"Prior system for attaching rich comments to source code, illustrating the manual reattachment limitation that semantic anchoring addresses.","marker":"[18]"},{"why":"The editor's language model API that Codetations exposes to annotation types, enabling LLM-generated debug code and unit tests.","marker":"[23]"},{"why":"The Odyssey floating-point workbench, used as a concrete annotation type that demonstrates wrapping an external tool through an LLM translation layer.","marker":"[24]"}],"fun_headline_variants":["Curated code notes cure confident LLM misdiagnosis","Persistent notes stop LLM overconfidence in code repair","GPT-4o stops guessing wrong when code notes stay live","Live notes for code: the missing context for LLM debugging"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the two debugging problems generated by GPT-4o have repairs latent in the model, so that failing without context is a fair measure of context's value, and that the hand-picked hints embedded in the distractor materials represent real annotation content.","fun_headline_variants_meta":{"raw":{"variants":["Curated code notes cure confident LLM misdiagnosis","Persistent notes stop LLM overconfidence in code repair","GPT-4o stops guessing wrong when code notes stay live","Live notes for code: the missing context for LLM debugging"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001027,"raw_usage":{"total_tokens":4330,"prompt_tokens":947,"completion_tokens":3383,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":3314}},"tokens_in":563,"tokens_out":3383,"duration_ms":22722,"temperature":1.0,"reasoning_tokens":3314,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:11:10.665638+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's two repair prompts (the GitHub API authentication bug and the PN532 NFC bug) on a current sample of LLMs with no context and with randomized distractor sets; if a substantial fraction of models solve them without context, or if relevant hints do not beat distractors alone, the claim that relevant annotation context is necessary for reliable repair collapses.","supporting_citations":[{"cited_title":"Magic Markup: Maintaining Document-External Markup with an LLM","cited_arxiv_id":"2403.03481","evidence_quote":"Supplies the LLM-based semantic reattachment method used when offline edits detach an annotation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the hallucination and missing-context problem that motivates giving LLMs explicit external context."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the traditional text-comparison source-location tracking method that the hybrid anchoring approach builds on and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior system for attaching rich comments to source code, illustrating the manual reattachment limitation that semantic anchoring addresses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The editor's language model API that Codetations exposes to annotation types, enabling LLM-generated debug code and unit tests."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Odyssey floating-point workbench, used as a concrete annotation type that demonstrates wrapping an external tool through an LLM translation layer."}],"review_version":1}