{"id":"e53f687d-2529-46ff-b223-46b023fe589e","arxiv_id":"2412.05159","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Task-specific embedding alignment via soft InfoNCE on CodeBLEU similarities lifts RAG-based Fortran-to-C++ translation quality by 14-15% relative without LLM fine-tuning.","lead":"This paper trains a retrieval model to pick code examples that maximize CodeBLEU, and uses it to improve Fortran to C++ translation inside a retrieval-augmented generation system. The authors report 14-15% relative CodeBLEU gains without fine-tuning the large language model, which matters for cheaply modernizing legacy HPC code.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 4 reports RAG retrieval and evaluation on the same HPC/Numerical Recipes benchmarks without stating whether query snippets are excluded from the retrieval corpus; if not, aligned retrieval can return the ground-truth pair and inflate CodeBLEU.","rationale":"The reader's weakest_assumption concerns the §3.2 training signal: pairwise CodeBLEU between LLaMA-3.1-8B-generated translations, not ground-truth translations, may cause the aligned retriever to reinforce systematic generator errors. That is a legitimate methodological concern. However, the more direct threat to the paper's headline claim is the evaluation protocol. Section 4 states that the HPC Fortran2C++ and Numerical Recipes datasets were used for 'RAG retrieval and evaluation,' and the vector database is described only as storing 'Fortran-C++ pairs,' with no statement that query snippets are excluded. With only 315 and 298 pairs in these benchmarks, including the query itself or its near-duplicates in the retrieval corpus would let the aligned retriever supply the ground-truth C++ translation as the few-shot example, trivially inflating CodeBLEU. This is a concrete, testable issue: a leave-one-out evaluation or a disjoint retrieval corpus would settle it. I do not accuse the authors of intentional leakage; the paper simply omits the detail needed to rule it out. Because the reader's verdict is already CONDITIONAL, I keep that verdict, but the primary condition should be explicit query-exclusion and disjoint retrieval-corpus construction, not only the training-signal proxy. If the leakage check fails, the central empirical claim would need to be rejected.","tokens_in":12476,"tokens_out":7230,"duration_ms":80412,"concrete_test":"Reconstruct or inspect the retrieval corpus construction and rerun all Table 1 evaluations under a strict leave-one-out protocol (query removed from the vector database) and, separately, with retrieval restricted to a held-out corpus disjoint from HPC Fortran2C++ and Numerical Recipes (e.g., Stack-V2 pairs). If the aligned-vs-unaligned CodeBLEU deltas collapse under either condition, retrieval leakage rather than embedding alignment explains the reported gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is the CodeBLEU improvement from aligned vs. unaligned retrieval (0.64→0.73 and 0.52→0.60). Section 4 says HPC Fortran2C++ and Numerical Recipes were 'employed for RAG retrieval and evaluation with LLMs,' and the vector database is described only as 'storing Fortran-C++ pairs' with no explicit train/eval split or query-exclusion rule. If the query itself (or a near-duplicate from the same small benchmark) is present in the retrieval pool, the aligned retriever can return the ground-truth C++ translation as the few-shot example, trivially raising CodeBLEU relative to an unaligned retriever that picks less relevant neighbors. Because the evaluation sets contain only 315 and 298 pairs, even retrieving from the same benchmark without self-exclusion constitutes a form of test-set leakage. The §3.2 training-signal proxy (pairwise CodeBLEU among LLaMA-8B-generated translations rather than ground truth) is a secondary concern; the missing retrieval-corpus protocol directly threatens the headline numbers.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a retrieval-augmented generation (RAG) method for Fortran-to-C++ translation in which the retrieval embedding model is fine-tuned with a contrastive loss (S-InfoNCE) that uses CodeBLEU scores between LLM-generated C++ translations as soft similarity labels. The aligned embeddings are then used to retrieve few-shot example pairs for several LLMs. Experiments on the HPC Fortran2C++ and Numerical Recipes datasets report average CodeBLEU improvements from 0.64 to 0.73 and from 0.52 to 0.60, respectively, without fine-tuning the LLM. The paper also includes a theoretical lemma characterizing stationary points of the proposed loss.","tokens_in":12709,"tokens_out":3525,"duration_ms":34737,"significance":"If the reported gains are reliable, the work is a useful empirical contribution: it shows that retrieval can be adapted to a downstream code-translation metric with a contrastive objective, and it demonstrates a plausible way to exploit CodeBLEU when aligned source-target datasets are scarce. The method is clearly described and the experiments cover multiple LLMs and shot counts. However, the current manuscript has load-bearing weaknesses: the evaluation protocol does not specify whether query snippets are excluded from the retrieval corpus; the theoretical lemma is a restatement of the loss rather than a substantive optimality result; the training signal is based on pairwise similarities among generator outputs rather than ground truth, contrary to parts of the abstract; and the headline numbers are not actually averaged over all models as claimed. These issues must be resolved before the central empirical claim can be accepted.","major_comments":[{"comment":"The evaluation section states that the HPC Fortran2C++ (315 pairs) and Numerical Recipes (298 pairs) datasets were 'employed for RAG retrieval and evaluation with LLMs' and that the vector database stores 'Fortran-C++ pairs', but it never states whether the query snippet (or near-duplicates from the same small benchmark) are excluded from the retrieval corpus. If the query itself is in the retrieval pool, the aligned retriever can return the ground-truth C++ translation as a few-shot example, which would trivially inflate CodeBLEU relative to an unaligned retriever. Given the small evaluation sets, the authors must specify the exact retrieval corpus for evaluation, the train/evaluation split, and the self-exclusion rule, and rerun or justify the results with that protocol.","section":"Section 4 (Retrieval and Evaluation Protocol)"},{"comment":"The proof of Lemma 1 treats the softmax probabilities p_ij(Ψ) as independent free variables in the Lagrangian (Eq. 14) and derives the stationary condition by differentiating with respect to p_ij. However, p_ij(Ψ) are not independent; they are deterministic functions of the embedding parameters Ψ and are constrained by the softmax normalization plus the structure of cosine similarities. The derivation therefore does not establish a stationary point of the loss over Ψ; it only shows that if p_ij could be set arbitrarily, the loss would be minimized by matching the normalized CodeBLEU matrix, which is exactly what the S-InfoNCE loss is designed to encourage. The 'optimal loss' formula in Eq. (21) is consequently a restatement of the loss construction rather than a proven property of the trained embedding model. The lemma should either be removed or replaced with a rigorous statement about the optimization landscape of the actual parameter space.","section":"Section 3.2, Lemma 1 and proof (Eqs. 9-21)"},{"comment":"The training data for alignment is built by computing pairwise CodeBLEU scores between C++ translations generated by LLaMA 3.1-8B (Eq. 5), not between generated translations and ground-truth translations. Yet the abstract says 'We compute pairwise CodeBLEU scores between the generated translations and ground truth examples', and Section 4 repeats a similar claim. The method therefore aligns the retriever to the generator's output distribution, under the assumption that source snippets whose generated translations are similar to each other are the best retrieval examples for producing ground-truth-faithful translations. This assumption is not validated (e.g., by measuring whether the proxy correlates with ground-truth CodeBLEU on a held-out set, or by an ablation using human-verified translations), and the Limitations section does not address it. This is a central methodological concern that should be discussed and, ideally, tested.","section":"Section 3.2, Eqs. (4)-(5) and abstract"},{"comment":"The text says 'averaged over all shot counts and models, the aligned embeddings achieved an average CodeBLEU score of 0.73, whereas unaligned embeddings achieve 0.64' for HPC Fortran2C++. However, Table 1 shows that the 0.64/0.73 values correspond only to the llama3.1 70b model when averaging the 1-, 2-, and 3-shot CodeBLEU deltas onto the zero-shot baseline (0.364 to 0.626/0.710, 0.639/0.735, 0.645/0.741). For llama3.1 8b the averages are approximately 0.58/0.70 (from Figure 2 and Table 1), and for the other models the numbers differ as well. The claim that these are averages over all models is not supported by the reported data and should be corrected with per-model, per-shot statistics.","section":"Section 4 (headline results) and Table 1"}],"minor_comments":[{"comment":"The abstract and Section 4 state that CodeBLEU is computed 'between the generated translations and ground truth examples', but Section 3.2 (Eq. 5) computes it between generated translations. Please make the descriptions consistent.","section":"Abstract / Section 3.2"},{"comment":"The caption says 'One-shot CodeBLEU' and 'across all four tested datasets', but the figure shows 1-, 2-, and 3-shot results for two datasets. Please correct the caption to match the content.","section":"Figure 2 caption"},{"comment":"There is a typo: 'th LLM' should be 'the LLM'.","section":"Conclusion"},{"comment":"The paper does not report the number of training steps/epochs or the stopping criterion for the embedding alignment, nor does it state whether results are averaged over multiple random seeds. Given the small evaluation sets, significance tests (e.g., paired bootstrap or Wilcoxon) over samples would strengthen the claims; at present only mean and standard deviation are reported.","section":"Section 4 (experimental setup)"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the experimental direction is timely, but the manuscript in its current form does not support the central empirical claim because of the unclear retrieval/evaluation protocol and the misattributed headline numbers. The theoretical lemma is also not sound as stated. I would encourage the authors to fix the evaluation protocol, re-verify the numbers, and reframe the theoretical contribution; after that, the paper could be a solid fit for the venue. The large computational cost (256 GH200 GPUs) is a practical reproducibility concern but not a reason to reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is genuinely useful: instead of retrieving with generic embeddings in a RAG code-translation pipeline, fine-tune the retriever with soft contrastive labels derived from CodeBLEU between LLM-generated target translations. That is a sensible, transferable trick, and the paper shows consistent directional improvements across all four models in Table 1. The pipeline is clean, the limitations section is honest about CodeBLEU not capturing functional equivalence, and the appendix example is a nice qualitative trace. For anyone working on RAG for code or HPC modernization, this is worth reading despite the flaws below.\n\nNow the soft spots, in rough order of importance.\n\nFirst, the retrieval-evaluation protocol is underspecified in a way that threatens the headline numbers. Section 4 says the HPC Fortran2C++ and Numerical Recipes datasets were \"employed for RAG retrieval and evaluation,\" and the vector database is described only as storing Fortran-C++ pairs. It never states whether query snippets were excluded from the retrieval corpus. If the ground-truth pair for a query is in the pool, an aligned retriever can retrieve that exact pair and trivially inflate CodeBLEU. The evaluation sets are only 315 and 298 pairs, so this is a real risk. The authors need to state the train/eval split explicitly and, ideally, evaluate by retrieving from a separate corpus (e.g., Stack-V2 only) rather than from the same benchmark.\n\nSecond, the Lemma 1 proof does not work. The proof treats the normalized softmax probabilities as free variables subject only to sum-to-one constraints, but those probabilities are functions of the embedding parameters. The stationary condition in Eq. 9 is the optimum of the loss over arbitrary probability distributions, not over the embedding parameterization. The claim may still hold in some loose sense, but as written it is not a valid proof. This is fixable by rewriting or removing the lemma, but it should not stand.\n\nThird, the headline averages are overstated. The 0.64 to 0.73 and 0.52 to 0.60 numbers match the llama3.1-70b results in Figure 2, not an average over all shot counts and models as the text claims. The Mistral and Mixtral gains are much smaller. The text should report per-model aggregates or clearly label the headline as the 70b case.\n\nFourth, there are no significance tests, and the reported standard deviations overlap heavily (0.64 ± 0.19 vs. 0.73 ± 0.17). Paired tests across the 315/298 examples would be cheap and would make the claim much more credible. Finally, no code or artifacts are released, and the 256-GPU training run is not easy to reproduce.\n\nMy recommendation: send this to peer review, but request major revision focused on the retrieval protocol, the lemma, and the reporting. The idea deserves referee time; the current presentation does not yet support the strength of the claims.","headline":"A useful retrieval-alignment idea with solid directional gains, but the proof is wrong, the headline averages are overstated, and the retrieval protocol is underspecified enough to worry about leakage.","tokens_in":13226,"tokens_out":2492,"would_cite":false,"duration_ms":27640,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Aligning the retriever to CodeBLEU similarity raises Fortran-to-C++ translation quality by 14–15% without fine-tuning the LLM.","keywords":["code translation","Fortran to C++","retrieval-augmented generation","embedding alignment","contrastive learning","CodeBLEU","soft InfoNCE","few-shot prompting"],"falsifier":"Hold out a set of Fortran snippets with ground-truth C++. Measure the correlation between pairwise CodeBLEU of LLaMA-generated translations and pairwise CodeBLEU of the ground-truth translations; if it is weak for snippets with similar runtime behavior, the training signal is not tracking what retrieval should optimize. The end-to-end test is simpler: with the LLM fixed, compare RAG output CodeBLEU using top-k examples from the aligned retriever against top-k examples from generic embeddings or random selection; if the aligned examples do not consistently win, the central claim fails.","tokens_in":12277,"feed_emoji":"🔁","tokens_out":11117,"duration_ms":98238,"temperature":0.7,"pith_summary":"The paper is trying to establish that in retrieval-augmented Fortran-to-C++ translation, the quality of the retrieved few-shot examples is a bottleneck, and that the retriever can be trained to serve the translation metric directly. It proposes S-InfoNCE, a soft contrastive loss that uses pairwise CodeBLEU scores between LLaMA-generated C++ translations as continuous labels for aligning the StarEncoder embedding space. With this aligned retriever, the paper reports average CodeBLEU gains from 0.64 to 0.73 on HPC Fortran2C++ and from 0.52 to 0.60 on Numerical Recipes, across several LLMs and few-shot settings, with no fine-tuning of the language model. If the method transfers, it provides a way to improve translation quality for language pairs that lack aligned ground-truth training data but have a target-side quality metric.","feed_headline":"Aligned retrieval raises Fortran-to-C++ CodeBLEU by 14-15%","feed_subtitle":"Retraining only the retriever improves translation quality across models and datasets, with no LLM fine-tuning.","key_machinery":"The central object is S-InfoNCE, a soft version of the InfoNCE contrastive loss. For source snippets $c_i^s$ embedded by $\\Psi$ with temperature $\\tau$, it compares the scaled cosine similarity $S^s_{\\Psi ij} = \\frac{1}{\\tau}\\frac{\\Psi(c_i^s)\\cdot\\Psi(c_j^s)}{\\|\\Psi(c_i^s)\\|\\|\\Psi(c_j^s)\\|}$ against the target-side CodeBLEU similarity $S^t_{ij}$ between the C++ translations that the generator produced for the two snippets. The loss is $L_{\\mathrm{S-InfoNCE}}(\\Psi) = -\\sum_{i}\\sum_{j} S^t_{ij} \\log \\frac{\\exp(S^s_{\\Psi ij})}{\\sum_k \\exp(S^s_{\\Psi ik})}$, and the paper's lemma shows that at stationarity these two similarity distributions coincide. This alignment is what makes the RAG retriever choose Fortran examples whose C++ counterparts are likely to push the generator's output toward high CodeBLEU.","core_discovery":"The central claim is that retrieval quality, not generator size alone, determines how much a few-shot RAG system benefits from its examples, and that the embedding model can be aligned to the downstream metric instead of to generic code similarity. The authors generate C++ translations of 25,000 Fortran snippets with LLaMA 3.1-8B, compute pairwise CodeBLEU scores among the generated translations, and train the StarEncoder embedding with S-InfoNCE so that the cosine structure of Fortran embeddings tracks the CodeBLEU structure of their translations. At the optimum of the loss, the normalized embedding similarities match the normalized CodeBLEU similarities. Integrated into RAG, this aligned retriever raises average CodeBLEU by 14–15% on the two benchmarks across four LLMs and one-, two-, and three-shot settings, without modifying the generator.","pith_inferences":["If the soft-label proxy is the bottleneck, swapping CodeBLEU for execution-based equivalence (for example, running generated C++ against test oracles) would be the natural next experiment, and it could reveal how much of the gain is due to n-gram and AST similarity versus functional correctness.","Since the soft labels come from a single generator, the aligned retriever may inherit that generator's systematic errors; using translations from several models or seeding a small ground-truth set would test whether this is a ceiling or a correction.","The recipe is generalizable beyond code: any generation task with a target-side metric and a large unlabeled source set could use the same pairwise-scoring plus S-InfoNCE pipeline, though pairwise scoring grows quadratically in the size of the training set.","A practical implication for RAG systems is that the first one or two retrieved examples matter most, so reranking effort could be concentrated on the top two retrieved neighbors rather than spread across many."],"forward_implications":["The reported gains hold across LLaMA 3.1-8B, LLaMA 3.1-70B, Mistral 123B, and Mixtral 8x22B, so the benefit is not specific to one generator.","Because the LLM is left untouched, the entire training cost is in the embedding model, making the upgrade substantially cheaper than fine-tuning a generator.","The aligned retriever narrows the spread of CodeBLEU scores, so translations become more consistently close to ground truth, not just better on average.","Diminishing returns after two shots imply that retrieval alignment matters most when only one or two examples are affordable.","The authors' stated generalization is that the same alignment strategy applies to other language pairs and tasks as long as a target-side quality metric such as CodeBLEU is available."],"supporting_citations":[{"why":"introduces the retrieval-augmented generation framework that the paper modifies for code translation.","marker":"Lewis et al., 2020"},{"why":"provides the prior RAG setup and unaligned-embedding baselines for Fortran-to-C++ translation that the paper compares against.","marker":"Bhattarai et al., 2024"},{"why":"defines CodeBLEU, the target-side metric used both as the training signal and the evaluation measure.","marker":"Ren et al., 2020"},{"why":"supplies The Stack v2 dataset from which the 25,000 Fortran snippets for embedding alignment are sampled.","marker":"Lozhkov et al., 2024"},{"why":"provides StarCoder, the embedding model that the paper fine-tunes with the S-InfoNCE loss and uses as the RAG retriever.","marker":"Li et al., 2023"},{"why":"introduces InfoNCE, the contrastive loss whose binary-class formulation S-InfoNCE extends to continuous CodeBLEU similarities.","marker":"van den Oord et al., 2018"},{"why":"provides the HPC Fortran2C++ benchmark used for RAG retrieval and evaluation.","marker":"Lei et al., 2023"},{"why":"provides the Numerical Recipes Fortran-C++ pairs used as a second evaluation dataset.","marker":"Press et al., 1988"}],"fun_headline_variants":["Retriever alignment boosts Fortran-to-C++ CodeBLEU by 14-15%","Task-specific embeddings lift code translation in RAG by 14%","No LLM fine-tuning needed: aligned retriever improves RAG translation","CodeBLEU-optimized retrieval lifts Fortran-to-C++ scores 14-15%","Aligning embeddings to translation metric beats generic RAG retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that pairwise CodeBLEU between LLaMA-generated C++ translations is a reliable proxy for which Fortran source pairs are the best retrieval examples for producing ground-truth-faithful translations, even though no ground-truth C++ is seen during alignment.","fun_headline_variants_meta":{"raw":{"variants":["Retriever alignment boosts Fortran-to-C++ CodeBLEU by 14-15%","Task-specific embeddings lift code translation in RAG by 14%","No LLM fine-tuning needed: aligned retriever improves RAG translation","CodeBLEU-optimized retrieval lifts Fortran-to-C++ scores 14-15%","Aligning embeddings to translation metric beats generic RAG retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000253,"raw_usage":{"total_tokens":1605,"prompt_tokens":1026,"completion_tokens":579,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":476}},"tokens_in":642,"tokens_out":579,"duration_ms":4915,"temperature":1.0,"reasoning_tokens":476,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:50:38.897622+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out a set of Fortran snippets with ground-truth C++. Measure the correlation between pairwise CodeBLEU of LLaMA-generated translations and pairwise CodeBLEU of the ground-truth translations; if it is weak for snippets with similar runtime behavior, the training signal is not tracking what retrieval should optimize. The end-to-end test is simpler: with the LLM fixed, compare RAG output CodeBLEU using top-k examples from the aligned retriever against top-k examples from generic embeddings or random selection; if the aligned examples do not consistently win, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the Numerical Recipes Fortran-C++ pairs used as a second evaluation dataset."}],"review_version":1}