{"id":"5f94625d-454e-4c66-88cf-759fc5233155","arxiv_id":"2412.08041","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Automated refactoring of deprecated Java APIs succeeds 71 to 82 percent of the time when Javadoc contains code hints, and at most 14 percent without them.","lead":"The paper finds that deprecated Java methods whose Javadoc comments contain a usable 'replaced by' hint are successfully refactored by automated tools 71 to 82 percent of the time, while methods without such hints succeed at best 14 percent. If that gap is causal, adding replacement hints to API documentation is a cheap way to make legacy code migration automatable.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 71% versus 14% headline compares different task sets; without a same-task hint-stripped ablation, the causal claim that Javadoc hints boost refactoring is not established.","rationale":"The reader's weakest assumption is that the with-hint and without-hint benchmark groups are not comparable in intrinsic difficulty, and that no ablation strips hints from the same tasks. This is exactly the load-bearing concern. The paper's causal recommendation is actionable ('Adding more code hints to Javadoc can hence boost the refactoring'), so establishing that hints, rather than task difficulty, drive the result is essential. The reader's CONDITIONAL verdict is appropriate: the observed effect is large and consistent across two very different engines, which makes the qualitative direction plausible, but the design is confounded. My read does not change the verdict; it reinforces the need for the proposed ablation and statistical reporting. I would therefore keep the verdict as CONDITIONAL rather than ACCEPT or REJECT. The concrete test above is feasible with the existing artifact and would settle whether the central claim survives on the same tasks. I agree with the reader that this is the weakest assumption, and no other concern in the manuscript appears more load-bearing: the verifier incompleteness is documented and counted as unsound cases, but the headline effect size dwarfs the unsound counts, and the central causal attribution rests on the uncontrolled comparison.","tokens_in":25795,"tokens_out":6952,"duration_ms":72500,"concrete_test":"Run a same-task ablation on a random sample of 30-50 benchmarks from the with-hint group (131 tasks). For RefNeural-Claude3, create two prompt variants per task: the original Javadoc including the hint, and the same Javadoc with the replacement suggestion (the @code text after 'replaced by') redacted; keep all other context identical. For RefSym, run the same tasks with CodeHints-library seeding and again with Types-library only. Compare success rates within tasks, not across groups. If the hint-stripped variants succeed at a rate similar to the hinted variants, the headline 71% versus 14% gap is an artifact of task difficulty. If success on the same tasks collapses toward the no-hint level, the causal role of hints is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is causal: adding code hints to Javadoc should boost automated refactoring. The evidence, however, is an observational comparison between two disjoint benchmark groups. Section 5.2 splits the dataset into benchmarks where the authors' parser could extract a code hint and benchmarks where it could not. These groups differ not only in hint presence but almost certainly in intrinsic difficulty: methods with a documented replacement tend to have simple, well-specified substitutions, while methods without an extractable hint may be awkward, underdocumented, or require knowledge that the engines lack. The paper provides no ablation that strips the hint from the same task and reruns the engines, so the 5-to-13-fold success gap could be entirely a task-selection effect. Two details make the confound concrete. First, for the neural engine, the 'without code hints' condition still feeds the full Javadoc comment into the prompt (Figure 3), so the LLM is only deprived of parser-extracted hints, not of replacement suggestions written in prose; the paper even notes that some Javadocs contain useful hints not tagged with @code. Second, for the symbolic engine, the no-hint condition uses the deliberately restricted Types-library (§3.1), which omits classes such as Calendar that appear only in hints, making the no-hint symbolic baseline an artifact of library construction rather than a measure of hint value. The abstract's 'even the worst engine... at best 14%' comparison also mixes engines (worst with hints versus best virtual without), further obscuring the effect. Section 6 lists threats to validity but never addresses this selection confound, which is the primary threat to the prescriptive conclusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents two automated refactoring engines for replacing deprecated Java API methods: a symbolic engine based on type-directed and component-based synthesis, and a neural engine based on Claude LLMs. Both use a CEGIS loop with a fuzzing-based equivalence checker. The evaluation covers 236 deprecated methods from the Oracle JDK 15 deprecated API list, split into benchmarks where Javadoc code hints were extractable and benchmarks without such hints. The headline result is that with code hints, success rates range from 71% to 82%, while without code hints they range from 6% to 14%. The paper concludes that adding code hints to Javadoc would substantially boost automated refactoring of deprecated APIs.","tokens_in":25968,"tokens_out":5668,"duration_ms":49168,"significance":"If the causal claim holds, the result is practically important: it would provide quantitative evidence that a concrete, low-cost documentation practice (adding structured replacement hints to Javadoc) materially improves automated migration of deprecated APIs. The paper also contributes a careful artifact, including two engine families, a fuzzing-based verifier, and a public benchmark derived from the JDK 15 deprecated list. Strengths include the transparency about verifier incompleteness (Sections 4.2, 4.4, 5.2, 6), the manual adversarial review of unsound refactorings, and the availability of the implementation and experiments. However, the central causal interpretation is currently threatened by a task-selection confound: the with-hint and without-hint conditions are evaluated on disjoint benchmark sets, so the reported 71% vs 14% gap may reflect intrinsic task difficulty rather than the benefit of hints.","major_comments":[{"comment":"The headline comparison contrasts two disjoint benchmark sets: tasks where the authors' parser could extract a code hint from the Javadoc versus tasks where it could not. Methods whose deprecation notes document a clean replacement are likely to be intrinsically simpler to refactor than methods without such documentation. The paper provides no same-task ablation that strips the hint from the same benchmark and reruns the engines, so the 71% versus 14% gap could be entirely a task-difficulty effect. This is load-bearing for the abstract's causal claim that 'adding more code hints to Javadoc can hence boost the refactoring.' Please add a condition in which the same tasks are run with the hint removed, or otherwise control for intrinsic difficulty (e.g., through matched pairs of methods with comparable signatures).","section":"§5.2, Table 1"},{"comment":"For the neural engine, the 'without code hints' condition still includes the full Javadoc comment in the prompt, so the LLM receives any replacement suggestions written in prose even when the parser failed to extract a structured @code hint. The paper itself acknowledges in §3 (Phase 1) that 'there are scenarios where, while the Javadoc does contain a useful code hint, it is not tagged accordingly with the @code tag.' Thus the neural comparison actually measures the presence of parser-extractable structured hints, not the presence of hints in general. To support the paper's broader conclusion, the no-hint condition should also suppress the @deprecated text from the prompt, or the analysis should be restricted to tasks where the only difference is the structured @code hint.","section":"§2.1 and Figure 3"},{"comment":"For the symbolic engine, the no-hint baseline uses the Types-library, which by construction excludes classes that appear only in code hints. The paper states in §3.1 that the Calendar class 'is only mentioned by the code hints and would not be included in the seeding of the Types-library.' Consequently, the symbolic no-hint success rate of 6% is in part an artifact of an intentionally restricted component library, not a measure of what type-directed synthesis could achieve from the type signature alone. The claim that code hints are 'enabling' for the symbolic engine requires a baseline where the same task is, in principle, solvable from types alone (e.g., a broader type-based library, or a control where the hint is replaced by an equivalent non-hint signal such as the textual deprecation reason).","section":"§3.1 and §5.2"}],"minor_comments":[{"comment":"The term 'code hint' is defined conceptually in the introduction as 'sometimes, what can be used in its place,' but the experimental condition is 'code hints could be extracted from the Javadoc.' Please clarify the distinction between the conceptual definition and the operational parser-based criterion, and state this explicitly when reporting the split.","section":"§1 and §5.2"},{"comment":"The table header '⌀ runtime' is nonstandard; rename to 'avg. runtime' and add a note that the neural-engine counts are averages over three runs, as described in §5.1.","section":"Table 1"},{"comment":"The sentence 'we bound the search by at most 500 inputs and 5 minutes per verification phase, and at most 2 minutes per synthesis phase' should clarify whether these bounds apply identically to the neural engine's verification calls and to the symbolic engine's fallback to the Types-library.","section":"§5.1"},{"comment":"The curated Mockito-avoidance constructor list is a manual, task-specific artifact; please describe its size and selection criteria, since it could influence the symbolic engine's success on particular benchmarks.","section":"§4.1"},{"comment":"The 'Threats to validity' section does not discuss the selection confound between hint and no-hint benchmarks; this is a more serious threat than the ones currently listed and should be addressed explicitly.","section":"§6"},{"comment":"The phrasing 'even the worst engine correctly refactors 71% of the tasks with code hints, which drops to at best 14% on tasks without' could mislead readers into thinking the same engine achieves 14%; rephrase to make clear that 14% is the best virtual engine without hints.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely acceptable after a same-task ablation study, which the available artifact makes feasible. The central claim is causal, and the current disjoint-set comparison does not control for task difficulty; this is a fixable experimental gap rather than an irreparable flaw. Also, the arXiv header indicates publication at FSE Companion 2025; if the journal requires a substantive extension over the companion paper, the novelty and extended content should be clarified in the submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the paper builds two real refactoring engines, releases their code, and reports a 71–82% vs ≤14% success gap that is almost certainly too large to be pure noise. But the headline comparison rests on a confound—the with-hint and without-hint tasks are different benchmark sets, and there is no same-task ablation where the hint is stripped from the same method. The causal claim that Javadoc hints boost refactoring is plausible but not proven.\n\nWhat’s new: a component library seeded from parsed Javadoc @code hints, and a systematic measurement on the JDK 15 deprecated API set. That’s a real contribution. The symbolic engine (CEGIS + JQF) and the neural engine (Claude 2.1/3 with iterative counterexample prompting) are both implemented and tested; the paper is transparent about verifier incompleteness (Sections 4.2, 5.2) and counts unsound refactorings it manually found. That honesty is worth credit.\n\nThe soft spots are real, and the stress-test note is on target. The 71% vs 14% comparison is not apples-to-apples. For the symbolic engine, the no-hint condition uses a Types-library that the authors built by restricting seeding to the method’s signature; the paper itself admits that Calendar-class generators like getInstance would not be included without the hint. So the symbolic no-hint baseline is partly an artifact of library construction. For the neural engine, the no-hint prompt still includes the full Javadoc (Figure 3), so the LLM only loses the @code-tagged hint, not all replacement information—and the paper notes useful hints appear in prose. Also, the “Best Virtual Engine” comparison mixes engines (worst-with vs best-without), which inflates the apparent effect. No confidence intervals or significance tests are reported, and the with/without split is defined by the authors’ parser.\n\nOne minor point: the abstract’s “even the worst engine… drops to at best 14%” obscures that it is comparing different engines’ best conditions, not the same engine with and without hints.\n\nWho’s this for? SE researchers working on automated refactoring, API migration, and documentation-for-code-generation. The artifact is a useful baseline, and the research question matters. A serious referee should engage; the paper is worth reviewing but needs a same-task ablation or a treated-control design, plus basic statistical reporting, before the prescriptive conclusion is justified. My recommendation: send it out, but make clear the confound is the central issue to fix.","headline":"Real engines, transparent reporting, and a big observed gap—but the causal claim about Javadoc hints rests on a confounded comparison that needs a same-task ablation before it can be believed.","tokens_in":26680,"tokens_out":1802,"would_cite":true,"duration_ms":17945,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Javadoc replacement hints are the enabling factor for automated refactoring of deprecated Java APIs.","keywords":["program refactoring","deprecated APIs","code hints","program synthesis","large language models","CEGIS","Javadoc","Java"],"falsifier":"Rerun the engines on the same call sites after mechanically deleting the @code replacement text from the Javadoc of the hinted benchmarks, keeping the @deprecated tag otherwise intact. If the success rate stays near 71%, hints are not the cause; if it falls toward the 14% no-hint level, the paper's causal claim is confirmed. The symmetric experiment would add a synthetic hint to the no-hint tasks.","tokens_in":25474,"feed_emoji":"💡","tokens_out":7109,"duration_ms":66564,"temperature":0.7,"pith_summary":"This paper asks whether automated tools can replace calls to deprecated Java APIs, and how much the replacement suggestion written in Javadoc contributes to that task. It builds two automatic refactoring engines, one symbolic and one neural, and applies them to 236 deprecated methods from the JDK 15 API. The central finding is that code hints are enabling: when the Javadoc names a replacement, even the weaker engine refactors 71% of the tasks successfully, while without a hint the best engine succeeds on at most 14%. If the comparison is fair, the practical consequence is that adding a one-line replacement hint to API documentation is a direct way to make legacy-code migration automatable.","feed_headline":"Javadoc hints lift refactoring success from 14% to 71%","feed_subtitle":"When docs say what to use instead, symbolic and LLM engines solve most tasks; without the hint, almost all fail.","key_machinery":"The carrying mechanism is a counterexample-guided inductive synthesis (CEGIS) loop shared by both engines. A synthesis phase proposes a candidate refactoring, and a verification phase fuzzes for an input on which the original and candidate behave differently; if it finds one, that input is fed back to refine the candidate. For the symbolic engine, the synthesis space is controlled by a seeded component library: the three-phase seeding algorithm starts from constants and instructions in the Javadoc hint, adds realisable generators for required target types, then adds transformers that consume the original inputs. For the neural engine, synthesis is an iterated LLM query whose prompt carries the Javadoc hint, the code snippet, formatting constraints, and past counterexamples. Equivalence is checked against a predicate that compares exceptions, live variables, loaded classes, static fields, and alias equivalence classes, so the reported refactorings are observational equivalences over stack and heap state, not mere textual rewrites.","core_discovery":"On the paper's own terms, the discovery is that a Javadoc code hint—a @code block inside a @deprecated tag naming the replacement API—is what makes deprecated-API refactoring tractable for both families of code generators. With hints, the worst of the three engine configurations refactors 71% of benchmarks and the best virtual engine reaches 82%; without hints, the best configuration is at 14%. The engines themselves are a symbolic synthesizer that weaves instructions from a library seeded by types and hints, and a neural synthesizer that prompts an LLM with the Javadoc, the method definition, and any counterexamples. Both depend on the hint, though in different ways: the symbolic engine needs it to prune the component library, while the neural engine uses it as additional context. The paper concludes from this that adding more code hints to Javadoc can boost automation of refactoring.","pith_inferences":["Editorial inference: a direct ablation—stripping only the @code replacement text from the hinted benchmarks and rerunning the same engines on the same call sites—would test whether the gap is caused by the hint or by a correlation between having a documented replacement and having an easier migration.","Editorial inference: the result suggests that API documentation quality, not model or solver capability, may be the current bottleneck for automated migration, so investment in writing replacement hints could have higher leverage than further engine tuning.","Editorial inference: the seeded-library mechanism should transfer to other languages and frameworks whose deprecation documentation follows a similar 'use X instead' convention, making the hint-extraction step the portability question rather than the synthesis itself."],"forward_implications":["On tasks whose Javadoc contains a code hint, all engine configurations refactor at least 71% of the 236-benchmark suite, and the union of engine outputs reaches 82%.","On tasks without code hints, no engine exceeds 14%, so the hint, not the engine choice, dominates whether automation succeeds.","The symbolic engine slightly outperforms the neural engines on hinted tasks while using less compute, indicating type- and hint-guided search can beat large language models in this specialised setting.","A practical policy follows: for deprecated methods that have a replacement, adding an explicit @code hint to the Javadoc should make client-code migration substantially more automatable."],"supporting_citations":[{"why":"Supplies the JDK 15 Deprecated API list from which the 236 benchmark tasks are drawn.","marker":"[41]"},{"why":"Introduces the CEGIS refinement loop that both the symbolic and neural engines follow.","marker":"[48]"},{"why":"Provides the coverage-guided fuzzing platform used as the safety checker in the synthesis and verification phases.","marker":"[43]"},{"why":"Establishes the component-based synthesis technique whose library composition the symbolic engine adapts.","marker":"[12]"},{"why":"Supplies type-directed synthesis background for using types to guide construction of the component library.","marker":"[42]"},{"why":"Provides the proprietary LLM family that the neural engine queries with its prompt template.","marker":"[1]"}],"fun_headline_variants":["Javadoc hints: from 14% to 71% refactoring success","Without Javadoc hints, best engine only hits 14% refactoring","Deprecated API refactoring: hints in Javadoc make it work","Code hints in Javadoc turn auto-refactoring from 14% to 71%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The causal claim assumes that the with-hint and without-hint benchmark groups are equally hard on average; if methods with documented replacements happen to be simpler to migrate, the 71%-versus-14% gap could reflect task difficulty rather than the hint's contribution.","fun_headline_variants_meta":{"raw":{"variants":["Javadoc hints: from 14% to 71% refactoring success","Without Javadoc hints, best engine only hits 14% refactoring","Deprecated API refactoring: hints in Javadoc make it work","Code hints in Javadoc turn auto-refactoring from 14% to 71%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000959,"raw_usage":{"total_tokens":4059,"prompt_tokens":893,"completion_tokens":3166,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":3076}},"tokens_in":509,"tokens_out":3166,"duration_ms":23001,"temperature":1.0,"reasoning_tokens":3076,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:18:09.135473+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the engines on the same call sites after mechanically deleting the @code replacement text from the Javadoc of the hinted benchmarks, keeping the @deprecated tag otherwise intact. If the success rate stays near 71%, hints are not the cause; if it falls toward the 14% no-hint level, the paper's causal claim is confirmed. The symmetric experiment would add a synthetic hint to the no-hint tasks.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the JDK 15 Deprecated API list from which the 236 benchmark tasks are drawn."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the CEGIS refinement loop that both the symbolic and neural engines follow."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the proprietary LLM family that the neural engine queries with its prompt template."}],"review_version":1}