{"id":"2bd882bb-4361-462e-a151-9ab055310640","arxiv_id":"2505.13073","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"The authors introduce LCP and ROUGE-LCP as user-perception-aligned code completion metrics and an SPSR-Graph corpus construction method, with experiments on ZTE's internal copilot logs.","lead":"This paper proposes two code completion evaluation metrics, LCP and ROUGE-LCP, and claims they align better with how programmers actually accept code suggestions. It also presents a graph-based method for building training corpora that encode cross-file code dependencies, tested on ZTE's internal code assistant data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The LCP–adoption correlation is largely mechanical because the logged reference is the user-confirmed final code; an independent ground-truth test is needed before metric superiority is accepted.","rationale":"The reader's weakest assumption identifies exactly the load-bearing weakness: the logged reference is defined by the user's final confirmed content, which is directly shaped by whether the user accepted the suggestion. This makes the empirical validation of the headline metric claim circular. The paper's central claim is that LCP and ROUGE-LCP are superior to EM, ROUGE-L, and LCS because they correlate more strongly with adoption rate. But because the reference includes the suggestion for adopted cases, LCP is high for adopted cases and low for rejected cases by construction, so the correlation is partly mechanical. The theoretical derivation in Sec 3.2 models LCP against a fixed reference and does not address this data-generation dependency. The SPSR-Graph experiments are a separate contribution, but they do not rescue the metric claim, and they are also weakened by the absence of strong baselines. The proposed independent-reference test is the minimal check that would determine whether the correlation survives when the reference is not contaminated by the acceptance event. Since the reader already rejected the paper on this basis, my assessment does not change the verdict.","tokens_in":13302,"tokens_out":2473,"duration_ms":26410,"concrete_test":"Re-run the correlation analysis on a held-out dataset with an independent reference: for each completion event, log the suggestion, the final accepted code, and also obtain a ground-truth reference from a source that does not depend on the suggestion, e.g., the next commit of the file in version control or a pre-recorded 'correct completion' judged by a human who never saw the model output. Compute LCP(S, R_independent) and the adoption label, and recompute Table 2. If the correlation with adoption rate persists (e.g., r remains above 0.6 with p < 0.05) on this independent reference, the circularity concern is resolved; if the correlation drops toward zero or reverses, the metric's claimed user-perception alignment is an artifact of the logging protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 defines the logged reference answer as 'the content confirmed by the user pressing the Enter key.' In an on-the-fly completion session, the sequence is: the model displays a suggestion; if the user presses Tab, the suggestion is inserted into the editor; the user may edit further and then presses Enter; the final buffer content becomes the reference. Therefore the reference is not an independent ground truth: for adopted suggestions, the reference contains the suggestion prefix by construction, so LCP(S,R) is inflated exactly for the cases labeled as adopted. For rejected suggestions, R is user-typed code bearing no particular relation to S, so LCP is low. The observed Pearson correlations in Tables 1-2 (r > 0.6-0.9) therefore partly follow from the logging definition rather than from an intrinsic property of LCP as a perception-aligned metric. The theoretical distribution in Sec 3.2 does not resolve this: it assumes R is the target code and S is the model output, but it does not model the fact that R was generated after S was presented. The comparison with EM/LCS/ROUGE-L in Table 2 is subject to the same artifact because all metrics are computed against the same contaminated reference; the ranking among metrics may survive, but the claim that LCP is user-perception-aligned is not established independently of the data collection protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper makes two contributions for on-the-fly code completion with large language models. First, it proposes LCP and ROUGE-LCP as evaluation metrics that are claimed to align with user perception, supported by a probabilistic derivation in Sec. 3.2 and by Pearson correlations with user adoption rates computed from ZTE-Code-Copilot logs in Sec. 4.2. Second, it proposes SPSR-Graph, a structure-preserving, semantically reordered code graph for building repository-level training corpora via AST-based semantic unit extraction and graph traversal, evaluated in Sec. 4.4 by fine-tuning Qwen2.5-7B-Coder on C/C++ corpora and comparing pipeline variants.","tokens_in":13624,"tokens_out":3925,"duration_ms":37366,"significance":"If the metric-alignment claim were valid, the paper would provide a practical, easy-to-compute alternative to EM and ROUGE-L for evaluating code completion in interactive settings, and the SPSR-Graph pipeline would be a plausible engineering contribution for repository-level pretraining. The paper deserves credit for collecting a real user-behavior dataset and for attempting to connect metric design to a probabilistic model of prefix correctness. However, the central validation is compromised by the data collection protocol: the reference answer is defined as the user-confirmed final content, which is directly influenced by whether the user accepted the suggestion. As a consequence, the observed correlations between LCP and adoption rate are partly mechanical and do not establish that LCP reflects user perception independently of the logging definition. The theoretical derivation is essentially the definition of first-error probability and does not connect to adoption behavior. The corpus experiments also lack external baselines and significance testing. The paper's core claim is therefore not supported as written.","major_comments":[{"comment":"The reference answer used for computing LCP is defined as 'the content confirmed by the user pressing the Enter key' (§4.1). In an on-the-fly completion session, if the user presses Tab, the suggestion text is inserted into the editor buffer, so the final Enter-confirmed content inherits the suggestion's prefix. Consequently, LCP(S,R) is inflated exactly for the cases labeled as adopted and suppressed for rejected suggestions. The correlations in Tables 1 and 2 (r > 0.6–0.9) therefore follow in part from the logging definition rather than from an intrinsic property of LCP as a perception-aligned metric. The comparison with EM, LCS, and ROUGE-L in Table 2 is computed against the same contaminated reference, so the claim that LCP is superior for capturing user perception is not established independently of the data collection protocol.","section":"§4.1, §4.2, Tables 1–2"},{"comment":"The derived expression for P(n=k) is the standard definition of the probability that the first error occurs at position k+1: it is the product of conditional probabilities of correct prefixes followed by the first mismatch. The derivation does not introduce any model of user behavior, so it does not 'explain the relationship between these metrics and user adoption behavior' as claimed in §3.2. Moreover, the assumption that the reference R is a fixed target sequence is violated in the logged data, where R is generated after S has been presented, as noted in the previous comment.","section":"§3.2, Eq. (1)"},{"comment":"The piecewise definition of P(Rouge-LCP(S,R)) is internally inconsistent with the definition ROUGE-LCP(S,R)=LCP(S,R)/|R|. In the third case, where LCP(R,S)=|R| and S≠R, the formula writes (LCP(S,R)+|S_ext|)/|R|, which can exceed 1, contradicting the preceding definition that ROUGE-LCP never exceeds 1. Additionally, the notation switches between LCP(S,R) and LCP(R,S) without clarification. Please reconcile the definition and the distributional formula, and clarify which argument order is intended for a symmetric evaluation metric.","section":"§3.2, piecewise formula for ROUGE-LCP"},{"comment":"The experiments for the SPSR-Graph contribution compare only the authors' own pipeline variants (Pipeline, +AST, +KGF, +KGF S). There is no comparison with existing repository-level completion methods such as CoCoMIC, RepoFusion, or GraphCoder, which are cited in §2.2, and no statistical significance tests or variance estimates are reported. The improvements in Table 3 are small (e.g., LCP stays at 5.2 for C across all conditions), and Fig. 5 is presented without error bars or multiple runs, so the claim of 'significant improvement' is not supported.","section":"§4.4, Table 3, §4.5"}],"minor_comments":[{"comment":"The header 'Blue' should be 'BLEU' for consistency with the rest of the paper.","section":"Table 3"},{"comment":"In the conclusion, 'we proposes a data processing method' should be 'we propose a data processing method'.","section":"§5"},{"comment":"The pseudocode formatting is confusing: the line 'S←∅ ;' appears to be part of the first line, and the function definitions are not clearly separated from the main body. Please restructure for readability.","section":"Algorithm 1"},{"comment":"The abbreviations 'R-L' and 'R-LCP' are used in the heatmaps but defined only indirectly in the caption; define them explicitly in each caption.","section":"Figure 4 and Figure 8 captions"},{"comment":"The statement 'According to the Central Limit Theorem, the length of the reference text |R| typically follows an approximately Gaussian distribution in real-world settings' is asserted without justification and is not used later. If it is meant to support the mixed-distribution claim, explain how the CLT applies to code lengths; otherwise remove it.","section":"§3.2"},{"comment":"The phrase 'the traversal depth is fixed at 1' combined with the stated complexity O(n + nd + n·d^D·m) in §3.3.2 is unclear; clarify whether the complexity formula is intended for depth D and how the breadth parameter k is incorporated into the traversal.","section":"§4.5"}],"recommendation":"reject","confidential_remarks":"The metric-validation artifact is load-bearing and would require re-collection of data with an independent ground truth to fix, which is beyond a typical revision. The corpus-construction portion is a plausible engineering direction but is insufficiently validated with external baselines and statistical rigor. The paper also presents proprietary data that cannot be independently checked. I recommend rejection, while noting that a revised version with an independent-reference study and proper baseline comparisons could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know that the central empirical claim—LCP correlates with user adoption—is built on a circular logging definition. The reference answer is the post-Enter content; if the user pressed Tab, that content contains the suggestion's prefix, so LCP is inflated for adopted suggestions by construction. The paper's own definitions (Sec 3.1 and Sec 4.1) make this explicit. That doesn't kill everything, but it means the correlation tables are not evidence that LCP captures user perception better than other metrics.\n\nWhat's actually new: this is a real industrial deployment with over ten thousand logged sessions in a communication-domain code assistant, and the SPSR-Graph corpus method is a reasonable combination of AST-based segmentation and call-graph reordering. The ablation in Table 3 shows consistent EM and BLEU gains from adding function- and struct-level graph context. That part is worth taking seriously, though it compares only against the authors' own pipeline—no GraphCoder, RepoFusion, or other strong repository-level baselines—and reports no variance or significance tests.\n\nThe metric theory in Sec 3.2 is mostly a restatement of the definition of first-error probability, not a derivation. The ROUGE-LCP mixed distribution is a case split, and the claim that the long-tail property matches the theoretical model is not tested against any alternative. Table 2's comparison is subject to the same contaminated reference; the ranking among metrics may survive, but the user-perception alignment claim is not independently established. A proper test would use an independent ground truth—pre-written tests, or human judgment on the completion alone without the Tab/Enter interaction.\n\nThe corpus experiments are interesting but narrow: C/C++ only, and the performance differences in Table 3 are small (a couple of EM points). The breadth analysis in Sec 4.5 is a single plot without error bars, so the non-monotonicity claim is weak.\n\nBottom line: the metric half of the paper does not support its load-bearing claim. The corpus half has some promise but needs stronger baselines and uncertainty quantification. I would send this to peer review because the industrial data and corpus method deserve scrutiny, but the authors need to redo the metric validation with an independent ground truth before the LCP claims can be accepted. If they do that, the result could be useful; as it stands, the paper is not ready for acceptance.","headline":"The metric validation is circular—the reference answer is constructed from user acceptance—so the headline claim about LCP is not established; the corpus method is plausible but under-benchmarked.","tokens_in":14167,"tokens_out":1650,"would_cite":false,"duration_ms":18003,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The longest common prefix between a suggested completion and the final code, not exact or partial overlap, predicts whether developers accept the suggestion.","keywords":["code completion","evaluation metrics","user perception","longest common prefix","ROUGE-LCP","code knowledge graph","cross-file dependency","abstract syntax tree"],"falsifier":"Recompute the daily correlation using a reference that is not derived from the accepted suggestion, such as expert-authored expected completions or the user's final code from sessions where suggestions were not accepted, and check whether LCP still predicts adoption; if the correlation falls to near zero, the claimed perception alignment is an artifact of the logging definition.","tokens_in":13121,"feed_emoji":"⌨️","tokens_out":9640,"duration_ms":91386,"temperature":0.7,"pith_summary":"The paper claims that user acceptance of an on-the-fly code completion is governed by the length of the exact prefix run that matches the final code, not by overall overlap. To capture this, it proposes LCP and its length-normalized variant ROUGE-LCP, and reports that across more than ten thousand logged completion events from a commercial code assistant, LCP correlates with the daily adoption rate more strongly than exact match, ROUGE-L, or LCS. The same paper argues that repository-level completion accuracy improves when the training corpus is rebuilt around AST-complete semantic units and reordered along directed call and reference paths, a construction it calls the SPSR-Graph. If these claims hold, evaluation of code assistants can move toward prefix-focused metrics that mirror how developers actually edit.","feed_headline":"Longer matching prefixes predict user adoption of AI code suggestions","feed_subtitle":"A two-month log of 10,769 completions makes LCP and ROUGE-LCP the perception-aligned metrics, and a code graph boosts C/C++ completion.","key_machinery":"LCP is the number of consecutive characters, starting at the first character, that a suggested completion shares with the reference; ROUGE-LCP divides that count by the reference length $|R|$ so samples of different sizes can be compared. The probability model treats the first mismatch as the end of a success run, giving $P(n=k)$ as the product of conditional match probabilities up to $k$ times one minus the conditional mismatch probability at the first error position, which yields the long-tail shape the logs exhibit. The second mechanism is the SPSR-Graph: a directed graph whose nodes are AST-extracted semantic units, such as functions, structs, and classes, and whose edges encode calls, references, type usages, macro expansions, and file includes. Training samples are built by concatenating nodes along breadth-first paths up to a maximum depth $D$, with file-path comments inserted at cross-file boundaries, so the model sees dependency order explicitly during pretraining.","core_discovery":"The central discovery is that the longest common prefix (LCP) between a suggested completion and the reference text is the best logged predictor of whether a developer presses Tab to adopt the suggestion. The paper models LCP as a long-tail distribution: the run of correct prefix characters survives only while each conditional next-character probability holds, and the first mismatch ends it. Empirically, LCP's Pearson correlation with daily adoption rate was above 0.7 across the full two-month window, beating LCS, ROUGE-L, exact match, and its own normalized variant ROUGE-LCP. The paper also claims that its SPSR-Graph corpus construction, which reorders code into samples that follow function-level and struct-level dependency paths, raises exact-match and BLEU scores on the on-the-fly completion task while keeping prefix quality stable.","pith_inferences":["If the logging artifact is removed, with reference answers taken from independent ground truth rather than from what the user finally typed, and LCP still tracks adoption, the metric becomes a cheap always-on training signal for live code assistants; that test is not in the paper.","The SPSR-Graph construction is a pretraining-time analogue of retrieval-augmented generation; a natural extension would be to retrieve the same graph paths at inference time and compare latency and accuracy against pretraining-only exposure.","The observed LCP long tail suggests most accepted completions are short correct prefixes, so product decisions about when to trigger a suggestion may matter more than maximizing whole-line exact match; this follows from the paper's data but is not stated by the authors.","A controlled experiment that inserts syntactically valid but semantically wrong tokens at different prefix positions could separate LCP's sensitivity to the first error from ROUGE-L's sensitivity to total overlap, which the current logged data cannot disentangle."],"forward_implications":["Fine-tuning a completion model with a loss aligned to LCP should increase the fraction of suggestions whose opening run is correct, which is the behavior users actually reward with Tab.","ROUGE-LCP normalizes by reference length, making it usable for fair comparisons across completions of different lengths, unlike raw LCP.","Replacing token-level Fill-in-the-Middle masking with AST-complete subtree masking preserves structure without hurting prefix quality, since LCP stays stable while exact match rises.","Adding function-level and struct-level code-graph reordering to the pretraining corpus improves C and C++ completion, with the largest exact-match gain from function-level graphs in C.","Graph traversal breadth should be kept near the observed dependency count, about 4, because wider traversal can add irrelevant context and reduce accuracy."],"supporting_citations":[{"why":"Supplies the standard practice of using exact match and ROUGE-L for code completion, the baseline the paper argues misses user perception.","marker":"(Li et al., 2025; Jiang et al., 2025)"},{"why":"Provides a practical evaluation methodology for code-completion models that the user-perception comparison extends.","marker":"(Izadi et al., 2024)"},{"why":"Contributes the AST-based structure-aware pretraining approach that the paper's AST semantic segmentation builds on.","marker":"(Gong et al., 2024)"},{"why":"Introduces code-context-graph construction and retrieval-based context injection, a direct precursor to the SPSR-Graph.","marker":"(Liu et al., 2024b)"},{"why":"Provides evidence that long irrelevant context distracts large language models, motivating cross-file dependency modeling.","marker":"(Shi et al., 2023)"},{"why":"Represents a long-context cross-file completion method that the corpus construction aims to improve upon.","marker":"(Ding et al., 2024)"},{"why":"Supplies the base code model used in all pretraining and fine-tuning experiments.","marker":"(Hui et al., 2024)"},{"why":"Establishes the precedent of using real-world logged user behavior to learn and assess code autocompletion.","marker":"(Aye et al., 2021)"}],"fun_headline_variants":["Prefix length is the real predictor of devs accepting AI code suggestions","Longest common prefix: the metric that matches how devs judge code completions","SPSR graph reorders code to boost completion quality while keeping prefixes strong","Tab or not? Longest prefix match says it all for code completion metrics","New code-graph method sharpens AI completion without sacrificing prefix accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The logged reference answer is defined as whatever the user ended up with after pressing Enter, so accepted suggestions write their own prefix into the reference; the measured link between longer common prefixes and adoption therefore partly follows from how the data was collected, not from an independent property of the metric.","fun_headline_variants_meta":{"raw":{"variants":["Prefix length is the real predictor of devs accepting AI code suggestions","Longest common prefix: the metric that matches how devs judge code completions","SPSR graph reorders code to boost completion quality while keeping prefixes strong","Tab or not? Longest prefix match says it all for code completion metrics","New code-graph method sharpens AI completion without sacrificing prefix accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000306,"raw_usage":{"total_tokens":1713,"prompt_tokens":867,"completion_tokens":846,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":748}},"tokens_in":483,"tokens_out":846,"duration_ms":8188,"temperature":1.0,"reasoning_tokens":748,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:20:19.635704+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the daily correlation using a reference that is not derived from the accepted suggestion, such as expert-authored expected completions or the user's final code from sessions where suggestions were not accepted, and check whether LCP still predicts adoption; if the correlation falls to near zero, the claimed perception alignment is an artifact of the logging definition.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the AST-based structure-aware pretraining approach that the paper's AST semantic segmentation builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents a long-context cross-file completion method that the corpus construction aims to improve upon."}],"review_version":1}