{"id":"ee4a3125-2006-4dab-926f-8e0df8c618c8","arxiv_id":"2606.08135","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"TICoder improves repository-level code generation by 11.52% over prior methods through test-driven planning and implementation-aware code reuse on standard benchmarks.","lead":"TICoder is a framework for repository-level code generation with LLMs that adds test-driven iterative planning and a dual-view similarity strategy for reusing existing functions. A smart generalist might read it to see how AI coding tools are being refined for handling large, interdependent codebases.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"11.52% improvement claim lacks evidence that gains are driven by test-driven planning or dual-view reuse rather than other factors","rationale":"The reader's weakest assumption matches the load-bearing gap for the experimental claim. Because the original review was abstract-only, the absence of ablation evidence means the verdict should move from UNVERDICTED to CONDITIONAL pending verification that the novel components are responsible for the reported gains.","tokens_in":1724,"tokens_out":368,"duration_ms":18518,"concrete_test":"On the same benchmarks and LLMs, re-run TICoder with two ablations: (1) replace test-driven iterative planning with non-iterative standard planning while keeping all other components fixed; (2) replace dual-view similarity + dual-stage selection with standard embedding-based retrieval. If either ablation drops Pass@1 or equivalent metric to within 3 points of the strongest SOTA baseline, the 11.52% average improvement cannot be attributed to the proposed mechanisms.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that TICoder outperforms SOTA by 11.52% on average. This rests on the experimental results being attributable to the two novel mechanisms (test-driven iterative planning using tests as behavioral specs, and implementation-aware reuse via dual-view similarity + dual-stage selection). The abstract provides no indication of ablation studies, component-wise breakdowns, or controls that isolate these mechanisms from baseline RAG, prompt variations, or model differences. If the observed gains persist when either mechanism is removed or replaced with standard alternatives, the headline attribution would not hold. The reader's weakest assumption directly identifies the unverified reliability of tests for planning and the reuse strategy for surfacing patterns; without empirical isolation, the performance delta cannot be confidently linked to those components.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript proposes TICoder, a repository-level code generation framework for LLMs. It addresses limitations in prior RAG and planning methods by introducing test-driven iterative planning that uses test cases as behavioral specifications to refine steps, and an implementation-aware reuse strategy that retrieves callee functions via dual-view similarity (functional and implementation aspects) followed by dual-stage selection (structure-based clustering and perplexity-based filtering). Experiments on standard benchmarks with multiple LLMs are reported to yield an average 11.52% improvement over SOTA baselines.","tokens_in":1884,"tokens_out":509,"duration_ms":18925,"significance":"If the performance gains are robustly demonstrated and causally linked to the two proposed mechanisms via appropriate controls, the work could meaningfully extend RAG-based repository code generation by incorporating behavioral test guidance during planning and implementation-level signals during reuse. These ideas build directly on existing retrieval and planning literature in software engineering and could inform future systems that treat tests as first-class planning artifacts.","major_comments":[{"comment":"Abstract and Experimental Results section: The central claim of an average 11.52% improvement over SOTA is presented without any mention of ablation studies, component-wise breakdowns, statistical significance tests, variance across runs, or controls that isolate the contribution of test-driven iterative planning versus the dual-view/dual-stage reuse strategy. This absence directly undermines attribution of the reported gains to the novel mechanisms rather than prompt variations, model differences, or baseline RAG enhancements.","section":"Abstract and Experimental Results section"},{"comment":"Methodology (planning and reuse subsections): The assumption that test cases reliably serve as behavioral specifications for iterative plan refinement, and that dual-view similarity plus dual-stage selection will surface genuinely reusable implementation patterns, is stated without empirical isolation or counter-example analysis. If these assumptions fail on the evaluated benchmarks, the headline performance delta cannot be confidently linked to the proposed components.","section":"Methodology (planning and reuse subsections)"}],"minor_comments":[{"comment":"The abstract would be strengthened by naming the specific benchmarks, metrics (e.g., pass@k), and SOTA baselines used to compute the 11.52% figure.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The experimental validation appears under-specified relative to the strength of the central empirical claim; this may affect fit for a journal that expects reproducible, ablated results in LLM-for-SE papers."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on strengthening the empirical attribution of our results. We address each major comment below and outline the revisions we will make.","responses":[{"response":"We agree that the abstract and experimental results section do not include ablation studies, component-wise breakdowns, statistical significance tests, variance across runs, or explicit controls isolating the two proposed mechanisms. The reported 11.52% figure reflects end-to-end comparisons against baselines. In the revised manuscript we will add a dedicated ablation subsection, report run-to-run variance, include statistical significance tests, and provide controls that separate the contribution of test-driven iterative planning from the dual-view/dual-stage reuse strategy.","revision_made":"yes","referee_comment":"[Abstract and Experimental Results section] Abstract and Experimental Results section: The central claim of an average 11.52% improvement over SOTA is presented without any mention of ablation studies, component-wise breakdowns, statistical significance tests, variance across runs, or controls that isolate the contribution of test-driven iterative planning versus the dual-view/dual-stage reuse strategy. This absence directly undermines attribution of the reported gains to the novel mechanisms rather than prompt variations, model differences, or baseline RAG enhancements."},{"response":"We acknowledge that the methodology subsections present the design rationale without dedicated empirical isolation of the assumptions or counter-example analysis. The current results show overall gains but do not directly demonstrate where the assumptions hold or break. We will add targeted analysis in the revised version, including counter-examples on the evaluated benchmarks, to better link the assumptions to the observed performance improvements.","revision_made":"yes","referee_comment":"[Methodology (planning and reuse subsections)] Methodology (planning and reuse subsections): The assumption that test cases reliably serve as behavioral specifications for iterative plan refinement, and that dual-view similarity plus dual-stage selection will surface genuinely reusable implementation patterns, is stated without empirical isolation or counter-example analysis. If these assumptions fail on the evaluated benchmarks, the headline performance delta cannot be confidently linked to the proposed components."}],"tokens_in":1413,"tokens_out":441,"duration_ms":16476,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"TICoder targets two gaps in repository-level code generation: plans that ignore behavioral specs and reuse that stays at surface level. It adds test-driven iterative planning that treats test cases as specs to refine steps, plus an implementation-aware reuse path that combines functional and implementation similarity, then applies structure clustering and perplexity filtering to pick patterns.\n\nThose two mechanisms are the actual new elements. Earlier RAG and planning work left those aspects open, so the framing is direct and the fixes line up with the stated problems.\n\nThe paper reports an average 11.52% lift over SOTA across benchmarks and several LLMs. If the full experiments include component ablations and fair baselines, that would be a usable data point for tool builders.\n\nThe clear soft spot is the absence of any experimental details in the abstract—no setup description, no statistical tests, no breakdown showing the planning or reuse steps are what produce the delta. The stress-test note is accurate here: without isolation, the attribution stays unproven. The assumptions that tests will reliably guide plans and that dual-view similarity will surface useful patterns are plausible but rest on the results, which are not shown.\n\nThis is for people working on AI code tools at repository scale. A reader hunting for concrete ways to fold tests into planning or to improve retrieval could extract ideas even if the numbers need checking.\n\nIt should go to peer review so the experiments can be examined directly. The core thinking is coherent and engages the literature on the gaps without obvious internal contradictions.","headline":"TICoder adds test-driven iterative planning and dual-view reuse to repo-level generation but the 11.52% claim has no visible ablations or setup details to tie gains to those pieces.","tokens_in":2364,"tokens_out":392,"would_cite":false,"duration_ms":18567,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"TICoder improves repository-level code generation by 11.52% on average by adding test-driven iterative planning and implementation-aware reuse.","keywords":["repository-level code generation","test-driven planning","implementation-aware reuse","large language models","code generation benchmarks","retrieval-augmented generation"],"falsifier":"A controlled run on the same benchmarks with the test-driven planning loop removed or replaced by non-test planning, checking whether the 11.52% average gain disappears.","tokens_in":2627,"feed_emoji":"💻","tokens_out":638,"duration_ms":14996,"temperature":0.7,"pith_summary":"The paper presents TICoder to fix two gaps in LLM-based code generation across whole repositories: plans that ignore expected behaviors and reuse that misses how functions are actually implemented. It adds an iterative loop where test cases serve as behavioral specs to refine the sequence of implementation steps. It also retrieves candidate functions by matching both their purpose and their internal logic, then narrows them with clustering on code structure and filtering by perplexity. Experiments across standard benchmarks and multiple LLMs show the combined changes produce the reported average gain over prior methods.","feed_headline":"TICoder lifts code generation accuracy by 11.52% on average","feed_subtitle":"Test-driven iterative planning and dual-view function reuse better match generated code to repository behaviors.","key_machinery":"Test-driven iterative planning mechanism combined with dual-view similarity retrieval and dual-stage selection for implementation-aware code reuse.","core_discovery":"TICoder introduces a test-driven iterative planning mechanism that leverages test cases as behavioral specifications to refine implementation steps, together with an implementation-aware code reuse strategy that retrieves potential callee functions using dual-view similarity capturing both functional and implementation aspects and then identifies relevant usage patterns through a dual-stage selection strategy combining structure-based clustering and perplexity-based filtering.","pith_inferences":["The same test-driven refinement loop could be applied to other LLM tasks that require step-by-step plans, such as API composition or test-case generation itself.","If high-quality test cases are unavailable, the planning component would lose its main signal and the reported gains would likely shrink.","Dual-view similarity retrieval might extend to other software-engineering retrieval problems where both intent and structural patterns matter."],"forward_implications":["Generated plans align more closely with the behaviors specified by the provided test cases.","Functions retrieved from the repository are more likely to be integrated correctly because both purpose and implementation details are considered.","Performance on repository-level code generation benchmarks rises by an average of 11.52% across the tested LLMs compared with prior retrieval-plus-planning methods.","Complex inter-function dependencies become easier to satisfy because reuse decisions are guided by actual usage patterns inside the repository."],"fun_headline_variants":["TICoder employs test-driven iterative planning","Dual-view similarity aids function reuse in TICoder","Perplexity filtering refines TICoder usage patterns","TICoder achieves 11.52% average improvement in code gen"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The approach assumes that test cases serve as reliable behavioral specifications that can iteratively refine plans and that the dual-view similarity plus dual-stage selection will surface genuinely reusable implementation patterns.","fun_headline_variants_meta":{"raw":{"variants":["TICoder employs test-driven iterative planning","Dual-view similarity aids function reuse in TICoder","Perplexity filtering refines TICoder usage patterns","TICoder achieves 11.52% average improvement in code gen"]},"model":"grok-4.3","cost_usd":0.007947,"raw_usage":{"total_tokens":3618,"prompt_tokens":663,"num_sources_used":0,"completion_tokens":62,"cost_in_usd_ticks":79474500,"prompt_tokens_details":{"text_tokens":663,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2893,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":663,"tokens_out":62,"duration_ms":16561,"temperature":1.0,"reasoning_tokens":2893,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T19:32:05.951561+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A controlled run on the same benchmarks with the test-driven planning loop removed or replaced by non-test planning, checking whether the 11.52% average gain disappears.","supporting_citations":[],"review_version":1}