{"id":"3b97092a-96ca-40d2-aeb1-3bbb7be1787c","arxiv_id":"2501.05062","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A T5-based study of 8 context types for Java code completion finds that combining coding contexts yields +11% relative improvement, and a confidence-based ensemble of context models yields +22%.","lead":"This paper tests whether giving a deep-learning code completion model extra context, such as project structure, open issues, or the developer's history, improves its accuracy. Across 18 trained T5 models, the best combined approach improved correct predictions by a relative 22% over the baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The +22.4% headline relies on comparing raw T5 log-likelihoods across models without length normalization or calibration; shorter, incorrect completions can win, so the improvement may largely be an artifact of the selection rule.","rationale":"The paper is a careful empirical study: it builds eight datasets from 1,072 compiled Java repositories, uses git-blame-based masking to create realistic completion instances, controls for duplicate methods, applies Holm-corrected McNemar tests, and releases a replication package. The individual-context results (Table II) are plausible and largely internally consistent; I do not object to the +11% relative gain from coding contexts as a finding, though the 'most similar method' context carries a near-duplicate leakage risk that should be separately disclosed. The single most load-bearing concern is the confidence-based ensemble, because it is the source of the abstract's '+22%' and of the reader's strongest claim. The issue is not that the models are wrong; it is that the selection rule's input, raw T5 sequence log-likelihood, is not a valid cross-model confidence measure as used. Sequence log-likelihood is length-dependent and per-model scales are not calibrated; the paper gives no evidence that the highest log-likelihood among 18 heterogeneous models is the most likely to be correct. Figure 3 shows a useful within-model correlation but does not establish cross-model comparability. Absent a length-normalized or calibrated comparison, or at least a random-selection control, the +22.4% cannot be confidently attributed to a reliable confidence-based ensemble. This is not an attack on the authors' integrity; it is a request for a specific robustness check that the paper should have included. I therefore recommend maintaining the CONDITIONAL verdict: the study is conditionally acceptable, with the confidence-based claim needing verification before it is cited as the headline result.","tokens_in":18793,"tokens_out":5606,"duration_ms":55523,"concrete_test":"On the test set, recompute the confidence-based ensemble using a length-normalized score: divide each generated prediction's log-likelihood by its number of tokens before taking the maximum across models. If the resulting accuracy is substantially below 37.43% (e.g., below the best single-context model's 33.88%), the raw-score comparison is load-bearing. As a control, run a random-selection baseline that chooses among the 18 models with the same marginal selection frequencies as the confidence selector; if random selection approaches the confidence selector's accuracy, the confidence signal itself is not driving the gain. This single experiment directly tests whether the +22.4% result survives a minimal correction for cross-model comparability.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is the confidence-based model in Section V (Table IV): selecting the prediction with the highest T5 confidence among 18 models improves correct predictions from 30.58% to 37.43% (OR=6.56). The selection rule assumes that the raw sequence log-likelihood produced by each fine-tuned T5 is a comparable, calibrated confidence score across models. This assumption is not supported. First, the score is the joint log-likelihood of the generated token sequence, and the paper never length-normalizes it; completions are one or two statements of variable token length, so raw log-likelihood systematically favors shorter outputs. A correct two-statement completion with log-likelihood -10 will lose to an incorrect one-token completion with log-likelihood -2, even though the former is the goal. Second, the 18 models are fine-tuned on input representations of very different lengths (Table I lengths range from mean 243 to 1,079 tokens) and formats, which can shift the entire score scale; a model that is generally 'more confident' can dominate the selector regardless of correctness. Figure 3 only shows within-model confidence-vs-accuracy trends, not cross-model comparability. The paper also does not compare the confidence selector against trivial baselines such as always using the best single contextual model (33.88%) or random model selection, so it is unclear how much of the +22% comes from the confidence signal versus from the ensemble having access to more models. Until the scores are made comparable (e.g., length-normalized and calibrated on the evaluation set), the headline improvement is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical study of how different types of contextual information affect the performance of T5-based code completion at the Java method level. It defines three families of contexts (coding, process, and developer), builds 18 fine-tuned models on a dataset of 85,266 instances derived from 1,072 Java projects, and compares their correct-prediction rates against a baseline that sees only the incomplete method. The baseline achieves 30.58% correct predictions; the best within-family combination of coding contexts reaches 33.88% (a relative +11% improvement). A confidence-based model that selects the highest-confidence prediction among all 18 models reaches 37.43% (a relative +22.4% improvement, OR=6.56). The paper concludes that additional contextual information can substantially improve code completion and that combining models by confidence is promising.","tokens_in":19116,"tokens_out":6743,"duration_ms":66384,"significance":"If the results hold, this is a valuable large-scale empirical contribution that quantifies the benefit of different context families for code completion and demonstrates a practical way to combine context-augmented models. The study uses a substantial dataset, adopts a realistic masking strategy based on git-blame history, and applies standard significance testing (McNemar with Holm correction). The finding that coding contexts provide the largest gains is plausible and internally consistent. However, the headline confidence-based result rests on an unvalidated assumption about cross-model comparability of raw T5 log-likelihoods, and the 'most similar method' context may be inflated by same-project similarity. Both concerns are addressable with additional analysis.","major_comments":[{"comment":"The confidence-based model's +22.4% improvement relies on treating T5's raw sequence log-likelihood as a calibrated confidence score that is comparable across 18 models. This assumption is not justified. Completions are one or two statements of varying token length, so raw log-likelihood intrinsically favors shorter outputs; a correct longer completion can lose to an incorrect short one. In addition, the models are fine-tuned on inputs of very different lengths (Table I means range from 243 to 1,079 tokens), which can shift the score scales across models. The paper should provide a length-normalized or otherwise calibrated confidence score and show that the result is robust to that choice.","section":"Section V, Table IV"},{"comment":"The 'most similar method' context is retrieved from the training set, but the train/test split is at method level, not project level. Because the dataset is built from 1,072 repositories and methods from the same project can appear in both training and test, the retrieved similar method may come from the same project as the incomplete method, giving the model access to project-specific style and near-duplicate code. This could inflate the impact of the MSM context, which is part of the best-performing within-family combination (MSM+MC+CS, 33.88%). The authors should report the proportion of same-project similar-method retrievals and, ideally, re-evaluate the MSM context using a project-level split or a filter that excludes same-project methods.","section":"Section III-A, Section IV-A"},{"comment":"The confidence-based selector is not compared against trivial baselines that would disentangle the effect of the confidence signal from the effect of having an ensemble. Since the best single contextual model achieves 33.88% and the combinational models already reach 33.54-33.75%, the paper should report the performance of always selecting the best single model, random model selection, and an oracle selector. Without these comparisons, it is unclear whether the jump to 37.43% comes from the confidence ranking or simply from having access to more models, and the OR=6.56 should be interpreted with caution.","section":"Section V, Figure 3"}],"minor_comments":[{"comment":"The row label 'IT + IBy' appears to be a typo for 'IT + IB', and 'Best Code+ Best Developer' is missing a space after the plus sign.","section":"Table II"},{"comment":"Reference [9], the replication package, uses placeholder author 'A. authors' and a generic URL; if the package is available, the final version should provide the actual authors and URL. In addition, References [40] and [41] duplicate the same T5 paper and should be merged.","section":"References"},{"comment":"The x-axis of the confidence plots is labeled in a confusing way ('0.5 Confidence 10.1'); it should be a clear 0.0 to 1.0 scale with evenly spaced bucket labels.","section":"Figure 3"},{"comment":"The hyperparameter configuration is tuned only on the baseline dataset, and the paper argues this may only improve the contextual models. This is a reasonable assumption but should be stated as an assumption rather than a fact, since a configuration that is optimal for the baseline is not guaranteed to be optimal for longer input representations.","section":"Section IV-A"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution, but the confidence-based ensemble result is currently over-sold relative to the validation provided. The same-project leakage concern for the most-similar-method context is important and should be resolved before publication. Please also verify that the replication package reference is real and accessible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper is a solid, systematic empirical study of how contextual information affects DL code completion. The genuinely new piece is the comparison of eight context types across coding, process, and developer families, on a large dataset (85k instances from 1,072 compiled Java repos), with git-blame-based masking that mimics real edits. The coding-context result is credible: combining most similar method, method calls, and class signatures gives +11% relative over baseline, with McNemar tests and Holm correction. The data collection and replication package are real assets.\n\nThe weak spot is the confidence-based ensemble that drives the headline +22%. The authors select the prediction with the highest raw T5 log-likelihood across 18 fine-tuned models. They never validate that these scores are comparable across models, which differ in input length (mean 243 to 1,079 tokens) and format. Figure 3 only shows within-model confidence-vs-accuracy, not cross-model calibration. They also omit the natural baselines: always use the best single contextual model (33.88% vs 37.43% for the ensemble) and random model selection. So the +22% relative improvement over baseline is not established as a robust property of the confidence signal; some of it likely comes from having access to more models, and some may be an artifact of scale differences. The stress-test note is right about the missing baselines, though I'd soften 'largely an artifact'—the ensemble does beat the best single model by 3.5 points absolute, so there's probably real signal, but the magnitude needs better support. Length-normalizing the scores and adding a random/best-single baseline would settle it.\n\nSecond soft spot: the 'most similar method' context may leak via near-duplicate methods from the same project. The split is by method, not by project, so a similar method from the same repository can appear in the training set and contain the masked statements. The paper removes exact duplicates but not near-duplicates. This could inflate the MSM results. Worth checking how often the retrieved method comes from the same project.\n\nThe paper is otherwise careful: hyperparameter tuning on the baseline is acknowledged, external validity limits (Java, T5, statement-level) are stated, and the process context uses a trained TSDAE with MRR 0.34 vs 0.14 random, so it's not obviously broken.\n\nBottom line: this deserves a serious referee. I'd send it out, but the confidence-based analysis needs additional baselines and calibration before the headline claim can be accepted.","headline":"Solid empirical study of code-completion contexts; the coding-context result holds up, but the headline +22% confidence-based ensemble needs baselines and calibration before I'd trust it.","tokens_in":19647,"tokens_out":4944,"would_cite":true,"duration_ms":51225,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Giving a code completion model project context improves accuracy, and choosing predictions by model confidence yields a relative +22.4% over the baseline.","keywords":["code completion","deep learning","contextual information","T5","empirical study","Java","confidence-based selection","software engineering"],"falsifier":"Re-run the confidence-based selector after separately calibrating each of the 18 models on a held-out set; if the +22.4% gain shrinks or vanishes once overconfident models are rescaled, the cross-model comparability assumption fails. Also compare the selector with a random baseline and a majority-vote ensemble of the same models, checking whether the gain comes from confidence calibration rather than from one model dominating the selection.","tokens_in":18605,"feed_emoji":"⌨️","tokens_out":9038,"duration_ms":82148,"temperature":0.7,"pith_summary":"This paper asks whether a code completion model predicts missing Java statements better when it is given more than just the unfinished method, and its answer is yes. The authors define three families of context—coding (methods that call or are called by the target, the rest of the class, a retrieved similar method), process (the title or body of a related open issue), and developer (statements and method calls recently used by the person typing)—and fine-tune 18 T5 models on 85,266 real completion instances that differ only in context. Single contexts improve correct predictions by up to a relative +7%, the three coding contexts together give +11%, and a confidence-based rule that outputs, for each completion, the prediction of whichever model is most confident reaches 37.43% correct predictions versus 30.58% for the baseline, a relative +22.4%. The upshot is that feeding models project and developer signals can substantially improve code assistants, and that pooling several context models by confidence is a more effective use of those signals than training one model on one input format.","feed_headline":"Adding context lifts code completion accuracy 22.4%","feed_subtitle":"Eight context types tested; the biggest gain comes from picking the most confident model.","key_machinery":"The mechanism that carries the argument is a pool of T5 models with different input representations plus a selection rule. T5, a transformer-based text-to-text model, is fine-tuned on eight datasets that share the same target completions but differ in what context is appended to the incomplete method; combinations of contexts produce 18 models in total. Each generated prediction carries a log-likelihood, and the paper converts that value into a confidence score on a 0-to-1 scale. The confidence-based model is then the central object: it compares these scores across all context models for a given completion and outputs the prediction with the highest confidence. This design does the explanatory work because it turns context engineering from choosing one best input into building a set of complementary specialists and delegating each instance to the most confident one.","core_discovery":"The paper's central discovery is that the context appended to an incomplete method is a first-order performance factor, and that different contexts are complementary. Using the same 85,266 statement-completion instances, the baseline T5 model (a transformer text-to-text model) that sees only the incomplete method produces 30.58% correct predictions, while a model augmented with the most similar method reaches 32.68% and one with method calls reaches 32.46%. Combining the three coding contexts (most similar method, method calls, class signatures) yields 33.88% correct predictions, a relative +11% over the baseline. The largest result comes from the confidence-based model, defined as follows: for each test instance, compute the T5 confidence score of every trained context model and emit the prediction with the highest score; this reaches 37.43% correct predictions, a relative +22.4% over the baseline. Complementarity analysis shows the context models and the baseline are not redundant: 20.90% of correct predictions come only from the confidence selector, while only 3.19% come only from the baseline. The intended takeaway is that contextual information should be supplied to multiple specialist models, and their confidence scores can be used to choose the best answer per instance.","pith_inferences":["Editorial inference: if T5 confidence is approximately calibrated, the same highest-confidence selection rule can be applied to pools of models from different architectures or languages, without needing a fusion layer or retraining.","Editorial inference: the weaker performance of process and developer contexts may be a representation problem; dense embeddings of issue text or developer activity could transfer more of that signal than raw token strings do.","Editorial inference: the 85,266-instance intersection requires projects to have callable code, open issues, and developer history, so the measured gains are conditional on rich-project metadata; coding-context gains are the most likely to generalize to more modest repositories.","Editorial inference: a serving-time implementation could first consult the coding-context models, since they contribute the most correct predictions, and only fall back to other contexts when confidence is low."],"forward_implications":["A single context already helps: most similar method gives a relative +7% over the baseline and method calls give +6%.","The three coding contexts together reach 33.88% correct predictions, a relative +11% over the baseline.","Choosing the highest-confidence prediction across all 18 models gives 37.43% correct predictions, a relative +22.4% over the baseline, with only 3.19% of correct predictions coming solely from the baseline.","Issue body and most similar statements are the only single contexts not statistically distinguishable from the baseline, and cross-family combinations do not beat the all-coding-context combination."],"supporting_citations":[{"why":"Defines the baseline task—mask one or two statements in a Java method and generate them—plus the T5-based model all 18 variants extend.","marker":"[13]"},{"why":"Supplies the T5 architecture and its pre-training/fine-tuning procedure; every experimental model is a fine-tuned T5.","marker":"[41]"},{"why":"Provides the project sampling tool used to select the compilable Java projects from which all context datasets are built.","marker":"[16]"},{"why":"Introduces the TSDAE embedding model trained to rank open issues, the mechanism behind the issue-title and issue-body contexts.","marker":"[48]"},{"why":"Defines CrystalBLEU, the code similarity metric used to retrieve the most similar method for the coding context.","marker":"[18]"},{"why":"Prior study showing that adding context to code representations helps deep-learning tasks; motivates the coding-context family and the comparison point.","marker":"[45]"}],"fun_headline_variants":["Context-rich code completion gains 22% accuracy","Picking best context model lifts code completion 22%","Specialist models + confidence = +22% code completion","Eight contexts, one best: code completion up 22%","Code completion: context variety yields 22% improvement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that T5's confidence score for a generated completion is a real probability and is comparable across models fine-tuned on different contexts, so the prediction with the highest score is the one most likely to be correct.","fun_headline_variants_meta":{"raw":{"variants":["Context-rich code completion gains 22% accuracy","Picking best context model lifts code completion 22%","Specialist models + confidence = +22% code completion","Eight contexts, one best: code completion up 22%","Code completion: context variety yields 22% improvement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1387,"prompt_tokens":1113,"completion_tokens":274,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":729,"completion_tokens_details":{"reasoning_tokens":195}},"tokens_in":729,"tokens_out":274,"duration_ms":3990,"temperature":1.0,"reasoning_tokens":195,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:19:22.501553+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the confidence-based selector after separately calibrating each of the 18 models on a held-out set; if the +22.4% gain shrinks or vanishes once overconfident models are rescaled, the cross-model comparability assumption fails. Also compare the selector with a random baseline and a majority-vote ensemble of the same models, checking whether the gain comes from confidence calibration rather than from one model dominating the selection.","supporting_citations":[{"cited_title":"An Empirical Study on the Usage of Transformer Models for Code Completion","cited_arxiv_id":"2108.01585","evidence_quote":"Defines the baseline task—mask one or two statements in a Java method and generate them—plus the T5-based model all 18 variants extend."},{"cited_title":"Exploring the limits of transfer learning with a unified text-to-text transformer,","cited_arxiv_id":null,"evidence_quote":"Supplies the T5 architecture and its pre-training/fine-tuning procedure; every experimental model is a fine-tuned T5."},{"cited_title":"Sampling Projects in GitHub for MSR Studies","cited_arxiv_id":"2103.04682","evidence_quote":"Provides the project sampling tool used to select the compilable Java projects from which all context datasets are built."},{"cited_title":"Crystalbleu: precisely and efficiently measuring the similarity of code,","cited_arxiv_id":null,"evidence_quote":"Defines CrystalBLEU, the code similarity metric used to retrieve the most similar method for the coding context."},{"cited_title":"Adding context to source code representations for deep learning,","cited_arxiv_id":null,"evidence_quote":"Prior study showing that adding context to code representations helps deep-learning tasks; motivates the coding-context family and the comparison point."}],"review_version":1}