{"id":"59274a16-2b72-41dc-8a8b-25d5fc4febbd","arxiv_id":"2411.17538","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Soft-ZCA whitening, an eigenvalue-regularized ZCA transform, is proposed and shown to improve semantic code search MRR on CodeSearchNet and a low-resource R dataset, although test-set whitening and test-set-tuned epsilon weaken the evidence.","lead":"This paper applies a softened version of ZCA whitening, a standard embedding decorrelation transform, to code embeddings and reports improved semantic code search across several code language models. The significance is that a cheap post-processing step could boost retrieval without retraining, but the paper's evaluation uses test set statistics, so the claimed gains may not transfer to real systems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported MRR gains rely on whitening matrices computed on the test set and epsilon tuned on test-set MRR; the central claim lacks a held-out evaluation.","rationale":"Reader's weakest_assumption is exactly the load-bearing concern I find: the whitening statistics and epsilon are derived from the test set, so the evaluation is transductive and the reported MRR gains may not generalize. This is confirmed by explicit statements in the manuscript (Section 3, Section 4, Figure 1). The paper does not provide any held-out evidence—the appendix's separate-vs-combined whitening comparison also computes matrices on the test set. The method could still be valid; a clean held-out evaluation would settle it. Because my concern matches the reader's, I recommend no change to the verdict (REJECT). The protocol concern is not a matter of consensus but of correctness of the empirical evidence, so the appropriate outcome is rejection of the current submission, with the path to resubmission being a proper train/validation/test evaluation.","tokens_in":6083,"tokens_out":4274,"duration_ms":39232,"concrete_test":"Re-run the experiments with a strict held-out protocol. For each CodeSearchNet language, fit the code and comment whitening matrices on the original training split (not the test set), select ε on the validation split using MRR, and then report MRR on the test split. Compare these held-out deltas with Table 3. If the positive improvements shrink to near zero or reverse, the reported gains are artifacts of test-set leakage. For StatCodeSearch R, use nested cross-validation or a separate training corpus to fit whitening and select ε, given its small size.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Soft-ZCA whitening reliably improves code search is not supported by the current evidence, because the evaluation leaks test-set information into the method. Section 3 states: \"The whitening matrices are calculated independently for code and comments, using the full test sets.\" Thus the covariance matrix used to build W = U(Λ+εI)^{-1/2}U^T is estimated from the same query and code instances that are later ranked. Section 4 describes selecting ε by test-set MRR (\"we found that moderate whitening ... results in the best performance\"), which is hyperparameter tuning on the evaluation set. In a deployment setting, a query is unseen and the whitening transform must be fixed before retrieval; fitting it to the test items makes the reported gains transductive artifacts. The risk is visible in the scale of the improvements: base CodeBERT Ruby jumps from MRR 0.006 to +0.230, and Code Llama Ruby +0.476, where tiny amounts of test-set adaptation can produce large relative changes. The StatCodeSearch R set (1,070 pairs) is especially susceptible. Consequently, Tables 2–3 and Figure 1 do not establish generalization to unseen code-comment pairs, so the central claim is currently unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies anisotropy in the embedding spaces of three pre-trained code language models (CodeBERT, CodeT5+, and Code Llama) and proposes a modified ZCA whitening technique, Soft-ZCA, which adds an eigenvalue regularizer epsilon to control the degree of whitening. The authors evaluate semantic code search using MRR on CodeSearchNet and a low-resource R-language dataset, reporting consistent improvements after whitening, with the largest gains for Code Llama (e.g., +0.476 MRR on Ruby). They conclude that Soft-ZCA whitening improves code search performance and can complement contrastive fine-tuning.","tokens_in":6318,"tokens_out":4393,"duration_ms":38530,"significance":"If the reported results were obtained under a sound evaluation protocol, the paper would offer a useful practical contribution: a lightweight, training-free post-processing step that improves retrieval performance, with source code released. The descriptive analysis of isotropy across code LMs and the observation that contrastive fine-tuning has limited effect on isotropy are also of interest. However, the evaluation protocol as described leaks test-set information into the method, which means the quantitative claims are not currently supported; the core evidence needs to be re-generated under a properly held-out setting before the contribution can be assessed.","major_comments":[{"comment":"The whitening matrices are calculated using the full test sets, as stated: \"The whitening matrices are calculated independently for code and comments, using the full test sets.\" This means the covariance statistics used to build W = U(Λ+εI)^{-1/2}U^T are estimated from the same queries and code snippets that are subsequently ranked. In a deployment setting, queries are unseen and the whitening transform must be fixed before retrieval; fitting it to the test items makes the reported MRR gains in Tables 2-3 transductive artifacts. This affects every result row and is especially problematic for the StatCodeSearch R set (1,070 pairs), where the whitening matrix is fit on the entire evaluation set. The current experiments therefore do not establish generalization to unseen code-comment pairs.","section":"Section 3, Procedure"},{"comment":"The eigenvalue regularizer epsilon is selected on the test set. Section 4 states that \"moderate whitening (epsilon in {0.1, 0.01}) results in the best performance\" and Table 3 is labeled as using \"the best epsilon,\" indicating that epsilon was chosen per model and language by maximizing test-set MRR. This is hyperparameter tuning on the evaluation set, which inflates the reported improvements. A valid protocol would fix epsilon on a validation split (or use nested cross-validation) and then evaluate once on a held-out test set with a whitening transform computed only from training or validation data.","section":"Section 4 and Figure 1"},{"comment":"No error bars, confidence intervals, or significance tests are reported. Many of the claimed improvements are small (e.g., CodeT5+ deltas of +0.007, +0.003, and 0.000 in Table 3), and several baseline MRR values are near zero (e.g., CodeBERT Ruby MRR 0.006 in Table 2). Without multiple runs or statistical testing, these differences cannot be distinguished from noise, and the conclusion that the technique \"reliably\" improves code search is not statistically supported.","section":"Tables 2-3"},{"comment":"The paper cites prior whitening work [9, 13] but does not compare Soft-ZCA against existing whitening methods such as PCA whitening, Cholesky whitening, or the whitening approach of Su et al. [9]. The experiments only contrast Soft-ZCA with no whitening and with full ZCA (epsilon=0). As a result, the reader cannot tell whether Soft-ZCA offers an improvement over established post-processing techniques, which weakens the stated contribution of introducing the regularizer.","section":"Section 2 and Section 4"}],"minor_comments":[{"comment":"There are typos: \"seperate\" should be \"separate,\" and \"combines\" in the table caption should be \"combined.\"","section":"Appendix 6.2"},{"comment":"The model name is written inconsistently as \"Code LLama\" in Table 1 and Figure 1, while the rest of the paper uses \"Code Llama.\"","section":"Table 1 and Figure 1"},{"comment":"The language name \"Javascript\" should be capitalized as \"JavaScript\" to match standard usage.","section":"Table 2"},{"comment":"The paper refers to the \"StatCodeSearch test dataset [7],\" but reference [7] is titled \"GenCodeSearchNet.\" The relationship between StatCodeSearch and GenCodeSearchNet should be clarified explicitly, since it is currently confusing to the reader.","section":"References [7]"}],"recommendation":"major_revision","confidential_remarks":"The test-set leakage in the evaluation protocol is the central barrier to publication. If the authors re-run the experiments with a proper train/validation/test split, freeze epsilon on validation data, and add significance testing or confidence intervals, the paper could become publishable. The current version should not be accepted because the reported MRR gains do not support the central claim as stated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline: Soft-ZCA is a small twist on standard ZCA whitening — add an eigenvalue regularizer — and the paper's actual new contribution is applying it to code embeddings and showing that isotropy correlates with retrieval performance across several code LMs. That part is fine and worth knowing. The problem is the evaluation protocol: the whitening matrices are computed on the full test sets, and epsilon is chosen by looking at test-set MRR. That leaks the test set into the method, so the reported MRR gains (up to +0.476 for Code Llama on Ruby) are not evidence of generalization to unseen queries. The authors say this mirrors production, but in production you don't have the queries or codes before you start searching.\n\nWhat's genuinely good: the paper is clearly written, the code is available, and the isotropy analysis (separating code and comment embeddings, measuring with IsoScore) is a useful data point. The finding that fine-tuning barely changes isotropy is consistent with prior NLP work, and the supplementary comparison of separate vs. combined whitening is a nice sanity check.\n\nWhere it's soft: besides the test-set leak, there are no error bars or significance tests, and the R dataset (1,070 pairs) is small enough that a handful of queries can move MRR a lot. The novelty bar is low — regularized ZCA is not new, and the soft version is essentially what people call shrinkage whitening — but domain transfer is a legitimate contribution, not a fake one.\n\nMy take: the central claim as stated is currently unsupported. But the flaw is fixable. Estimate the whitening statistics on the training or validation split, choose epsilon on a validation split, and report the test numbers once. If the gains survive that, the paper is solid. If they shrink to nothing, the honest result is still informative. I'd send this to peer review with a demand for a held-out evaluation rather than desk-reject it; the question is practical and the method is simple enough that the field would benefit from a clean answer.\n\nWho's it for: people working on code retrieval, embedding post-processing, and anyone who wants a quick isotropy check. I wouldn't cite the current numbers, but I'd cite it if the revised version holds up.\n\nRecommendation: send to review, but make the held-out evaluation a condition.","headline":"Soft-ZCA is a minor variant of ZCA whitening applied to code search, but the central MRR claim is compromised by test-set whitening and epsilon tuning on the test set.","tokens_in":6809,"tokens_out":2071,"would_cite":false,"duration_ms":18460,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Soft-ZCA whitening of code and comment embeddings reliably improves semantic code search retrieval across pre-trained code models, and it composes with contrastive fine-tuning.","keywords":["semantic code search","embedding isotropy","ZCA whitening","Soft-ZCA","code language models","mean reciprocal rank","post-processing","retrieval"],"falsifier":"Re-run the Soft-ZCA pipeline with the whitening matrix and $\\epsilon$ estimated only from a training or validation split, then evaluate MRR on a held-out query-code set; if the gains over the non-whitened baseline vanish or turn negative, the test-set-based tuning is responsible for the reported improvements.","tokens_in":5906,"feed_emoji":"🔍","tokens_out":6010,"duration_ms":49995,"temperature":0.7,"pith_summary":"This paper argues that the poorly shaped geometry of code-model embeddings—specifically their low isotropy—hurts semantic code search, and that a post-processing step can fix much of the damage without retraining. It introduces Soft-ZCA, a regularized variant of ZCA whitening that replaces the whitening matrix $\\Sigma^{-1/2}$ with $U(\\Lambda+\\epsilon I)^{-1/2}U^{\\top}$, letting the user control how much variance is flattened. Across CodeBERT, CodeT5+, and Code Llama on six CodeSearchNet languages plus the low-resource R benchmark, the transform raises retrieval MRR on nearly all settings, with the largest gains for pre-trained models (up to +0.476 for Code Llama on Ruby). If the results hold in deployment, code search systems can gain accuracy by whitening embeddings at index time, and the technique composes with contrastive fine-tuning.","feed_headline":"Whitening code embeddings lifts search MRR by up to 0.476","feed_subtitle":"A soft ZCA transform fixes skewed embedding geometry and stacks on top of contrastive fine-tuning.","key_machinery":"The load-bearing object is the Soft-ZCA whitening matrix $W = U(\\Lambda + \\epsilon I)^{-1/2} U^{\\top}$, computed from the covariance of each modality's embeddings (code and comments separately). The parameter $\\epsilon$ is an eigenvalue regularizer: it sets a floor on the inverse-square-root eigenvalues, so the transform can be tuned from no whitening to full ZCA. Isotropy is measured with IsoScore, a bounded $[0,1]$ metric of how uniformly vectors occupy the embedding space; the paper uses it to show that the best retrieval performance sits at intermediate isotropy for pre-trained models and near-perfect isotropy for the fine-tuned one.","core_discovery":"The central discovery is that code language models produce strongly anisotropic embedding spaces—IsoScores often below 0.1 for code and comments—and that standard ZCA whitening, which forces the covariance to the identity, overshoots: it helps only weakly for pre-trained encoders and hurts fine-tuned or already-well-shaped models. Soft-ZCA dampens the whitening by adding a regularizer $\\epsilon$ to the eigenvalues before inversion, giving a continuum from the original space ($\\epsilon \\to \\infty$) to full ZCA ($\\epsilon=0$). With modest regularizers ($\\epsilon \\in \\{0.1,0.01\\}$), pre-trained CodeBERT and Code Llama embeddings become substantially more isotropic (IsoScores roughly 0.2–0.9) and retrieval improves; fine-tuned CodeBERT, which already benefits from contrastive training, needs a much smaller $\\epsilon=0.0001$ to reach near-perfect isotropy and still gains MRR. CodeT5+ gains are marginal, suggesting that the benefit of whitening depends on how anisotropic the starting space is. The paper states this as: Soft-ZCA whitening improves the performance of pre-trained code language models and can complement contrastive fine-tuning.","pith_inferences":["A natural stress test the paper leaves open is whether the gains survive when the whitening matrix and $\\epsilon$ are estimated from a training or validation pool rather than the full test set; if test-set statistics leak into the transform, the reported gains are an upper bound for streaming deployment.","The same regularized-whitening recipe could be applied to other retrieval modalities where anisotropy is known to hurt, such as multilingual sentence or biomedical embeddings, since the mechanism is purely geometric.","The result suggests isotropy is not a monotonic good: forcing embeddings to be perfectly uniform can destroy discriminative signal, so per-model tuning of $\\epsilon$ is likely to be necessary in practice."],"forward_implications":["Post-processing with Soft-ZCA can improve code search MRR for pre-trained code language models without any fine-tuning, with gains as large as +0.476 (Code Llama, Ruby).","The technique composes with contrastive fine-tuning: fine-tuned CodeBERT still gains +0.042 to +0.075 MRR after Soft-ZCA, so whitening can be applied on top of already-tuned systems.","Separate whitening of code and comment embeddings, computed from each modality's own statistics, performs at least as well as a shared transform, matching how production search pipelines index code independently of queries.","The benefit transfers to a low-resource language (R) not seen in pre-training, where Code Llama gains +0.337 MRR after Soft-ZCA.","The optimal amount of whitening is architecture-dependent: pre-trained encoders prefer moderate isotropy (IsoScore roughly 0.2–0.8), while the fine-tuned model prefers near-perfect isotropy."],"supporting_citations":[{"why":"Defines ZCA whitening, the base transform that Soft-ZCA regularizes.","marker":"[10]"},{"why":"Provides the optimal-whitening properties used to justify choosing ZCA over other decorrelation transforms.","marker":"[13]"},{"why":"Shows that whitening sentence embeddings improves semantics and retrieval, motivating the post-processing approach for code.","marker":"[9]"},{"why":"Introduces IsoScore, the metric used throughout to quantify embedding isotropy.","marker":"[14]"},{"why":"Supplies the CodeSearchNet benchmark with six programming languages used for the main evaluation.","marker":"[4]"},{"why":"Supplies the StatCodeSearch test set for the low-resource R language generalization experiment.","marker":"[7]"},{"why":"Provides CodeBERT, one of the evaluated pre-trained code language models.","marker":"[11]"},{"why":"Provides CodeT5+, the contrastively pre-trained model in the evaluation.","marker":"[6]"},{"why":"Provides Code Llama, the largest evaluated model and source of the largest MRR gains.","marker":"[12]"},{"why":"Supplies evidence that fine-tuning does not fully repair embedding geometry, motivating the whitening post-processing.","marker":"[8]"}],"fun_headline_variants":["Soft-ZCA whitening boosts code search by fixing skewed embeddings","Dampened whitening makes code embeddings isotropic, lifts MRR","Gentle ZCA transform improves code search retrieval","Anisotropic code embeddings fixed with Soft-ZCA whitening","Soft-ZCA whitening complements contrastive fine-tuning for code search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains assume that whitening statistics and the regularization strength chosen on the full test set remain representative when the same pipeline meets unseen queries; if that assumption fails, the improvements will shrink or disappear in real deployment.","fun_headline_variants_meta":{"raw":{"variants":["Soft-ZCA whitening boosts code search by fixing skewed embeddings","Dampened whitening makes code embeddings isotropic, lifts MRR","Gentle ZCA transform improves code search retrieval","Anisotropic code embeddings fixed with Soft-ZCA whitening","Soft-ZCA whitening complements contrastive fine-tuning for code search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00093,"raw_usage":{"total_tokens":3948,"prompt_tokens":879,"completion_tokens":3069,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":2984}},"tokens_in":495,"tokens_out":3069,"duration_ms":21423,"temperature":1.0,"reasoning_tokens":2984,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:59:02.899758+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Soft-ZCA pipeline with the whitening matrix and $\\epsilon$ estimated only from a training or validation split, then evaluate MRR on a held-out query-code set; if the gains over the non-whitened baseline vanish or turn negative, the test-set-based tuning is responsible for the reported improvements.","supporting_citations":[{"cited_title":"Edges are the ’independent compo- nents’ of natural scenes","cited_arxiv_id":null,"evidence_quote":"Defines ZCA whitening, the base transform that Soft-ZCA regularizes."},{"cited_title":"Salesforce/codet5p-110m-embedding","cited_arxiv_id":null,"evidence_quote":"Introduces IsoScore, the metric used throughout to quantify embedding isotropy."},{"cited_title":"GenCodeSearchNet: A benchmark test suite for evaluating generalization in programming language understanding","cited_arxiv_id":null,"evidence_quote":"Supplies the StatCodeSearch test set for the low-resource R language generalization experiment."},{"cited_title":"How does fine-tuning affect the geometry of embedding space: A case study on isotropy","cited_arxiv_id":null,"evidence_quote":"Supplies evidence that fine-tuning does not fully repair embedding geometry, motivating the whitening post-processing."}],"review_version":1}