{"id":"d1cea23b-6a27-493d-b8ac-62cfc50b7060","arxiv_id":"2412.15178","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Fine-tuning DeepSeek-Coder on a new 122k synthetic HPC instruction dataset yields open-source models that reach 34.1 pass@1 on ParEval parallel code generation, besting other open baselines but trailing GPT-4.","lead":"This paper builds a 122k-sample synthetic instruction dataset for parallel programming, fine-tunes DeepSeek-Coder models on it, and reports that the resulting HPC-Coder-V2 models are the best open-source code LLMs for parallel code generation on the ParEval benchmark. The study also maps how base model choice, instruction masking, data quantity, data source, and model size affect parallel code generation performance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'best open-source' ranking rests on ParEval pass@1 gaps of 1–2 points with no confidence intervals; if those gaps are sampling noise, the headline claim is unsupported, so the paper needs a significance or contamination check before the comparative claim is accepted.","rationale":"I read the paper in good faith: it contributes a large synthetic dataset, releases the fine-tuned models, runs a broad ablation study, and is transparent about limitations such as unverified data quality and cursory hyperparameter tuning. Those are real strengths. However, the central claim is a comparative ranking, and the evidence for that ranking is weaker than it appears. The paper reports single-point pass@1 estimates with no uncertainty quantification, and the differences between the top models are small relative to the noise expected from the stated evaluation protocol. This makes the 'best open-source' claim the least secure part of the paper. The reader's weakest_assumption focused on synthetic-data correctness; I agree that is a genuine concern, particularly for RQ3 and for generalization beyond ParEval, but I consider the statistical reliability of the benchmark comparison more load-bearing for the specific headline claim. The two concerns interact: if the benchmark itself is noisy or contaminated, data-quality conclusions drawn from it are also uncertain. My recommendation is unchanged from the reader's CONDITIONAL verdict, because the paper's contributions are useful and the missing pieces are addressable: add bootstrap confidence intervals, run an external benchmark or contamination check, and soften the comparative claim until those are in place. I have no objection to the paper's methodology beyond this, and I explicitly credit the authors for releasing code, data, and models, which makes the proposed check feasible for the authors or a third party.","tokens_in":18347,"tokens_out":4744,"duration_ms":47685,"concrete_test":"Compute bootstrap 95% confidence intervals for the parallel pass@1 of the models in Table I using the per-cell cp values underlying Appendix C: resample the 84 problem-type × execution-model cells with replacement for 10,000 iterations, preserve the cp/N estimate within each resampled cell, and report the 2.5th and 97.5th percentiles for HPC-Coder-V2-16B, HPC-Coder-V2-6.7B, Phind-V2-34B, and Magicoder-6.7B. If the 16B interval overlaps any of the other open-source intervals, the headline 'best open-source code LLM for parallel code generation' is not supported by the reported evaluation and should be softened to 'competitive with'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section VII-B and Table I, HPC-Coder-V2-16B achieves ParEval parallel pass@1 of 34.1, versus 33.3 for HPC-Coder-V2-6.7B, 32.1 for Phind-V2-34B, and 30.4 for Magicoder-6.7B. The paper reports no confidence intervals, significance tests, or per-problem pass@1 distributions. Appendix C states that pass@1 is estimated from 20 samples per prompt, so the value is the average of per-problem cp/20 over 420 problems. Under that protocol, the standard error of the aggregate mean is roughly sqrt(mean(p_i(1-p_i)/20)/420 + var(p_i)/420); with the between-problem variance typical of this benchmark, a 2-point gap is within about 1.5–2 standard errors. The 0.8-point gap between the 16B and 6.7B models is even more clearly within noise. Consequently, the claimed separation between HPC-Coder-V2-16B and the next-best open-source models is plausibly sampling variation, and the headline 'best performing open-source code LLM for parallel code generation to date' is not statistically established. This concern is compounded by the fact that ParEval is authored by the same group (reference [2] shares the first author) and no decontamination check is reported between HPC-INSTRUCT (seeded from The Stack v2) and ParEval. The synthetic-data-quality issue raised by the reader is real but secondary: it affects transfer of the reported gains to real parallel-code generation, whereas the headline ranking is already fragile on statistical grounds.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces HPC-INSTRUCT, a synthetic parallel-code instruction dataset of roughly 122k problem-solution pairs generated by four LLMs (Gemini-Pro, DBRX, Llama-3-70B, Mixtral-8x7B) from seed snippets collected from The Stack v2, and fine-tunes DeepSeek-Coder models (1.3B, 6.7B, 16B) on it to produce the HPC-Coder-V2 family. The authors report ablation studies on instruction masking, base-versus-instruct models, MPI data quantity, synthetic data source, and model size, and evaluate all models on the ParEval benchmark. The central claim is that HPC-Coder-V2-16B is the best performing open-source code LLM for parallel code generation, with a ParEval parallel pass@1 of 34.1, ahead of Phind-V2-34B (32.1) and Magicoder-6.7B (30.4), and close to GPT-4 (37.8) and GPT-3.5 (39.6).","tokens_in":18720,"tokens_out":5165,"duration_ms":43562,"significance":"The paper makes a substantial practical contribution: a large, publicly released synthetic HPC instruction dataset, three released open models, and a systematic ablation study that addresses several under-explored fine-tuning choices for low-resource parallel languages. The use of a machine-checked correctness harness (ParEval's unit tests), the release of reproducibility scripts, and the breadth of the ablation (28 fine-tuned models) are concrete strengths. The observed improvements over strong open-source baselines are plausible and of clear interest to the HPC-LLM community. However, the headline ranking claim and several quantitative conclusions rest on point estimates from a single self-authored benchmark with no confidence intervals, and the synthetic data is used without any verification of solution correctness. These issues currently limit the strength of the evidence for the paper's strongest claims, even though the underlying methodology and artifacts are valuable.","major_comments":[{"comment":"The headline claim that HPC-Coder-V2-16B is the best performing open-source model for parallel code generation is not statistically established. Table I reports parallel pass@1 values of 34.1 (HPC-Coder-V2-16B), 33.3 (HPC-Coder-V2-6.7B), and 32.1 (Phind-V2-34B). Appendix C states that pass@1 is estimated from 20 samples per prompt over 420 problems, i.e., the aggregate mean of per-problem cp/20. Under that protocol the standard error of the aggregate mean is approximately sqrt( (1/420)*mean(p_i(1-p_i)/20) + var(p_i)/420 ), which, given the large between-problem variance visible in the Figure 12 heatmaps, is on the order of 1.5–2 percentage points. Consequently, the 2-point gap over Phind-V2 and the 0.8-point gap over the 6.7B model are plausibly sampling noise. No confidence intervals, significance tests, or per-problem pass@1 distributions are reported. The same concern applies to the ablation conclusions in Section VI, including the instruction-masking differences in Figure 4 (e.g., 31.8 vs. 31.1 for the 6.7B base model) and the data-source differences in Figure 6. The authors should provide uncertainty quantification (confidence intervals or significance tests) and either demonstrate that the reported separations are statistically meaningful or soften the ranking claims accordingly.","section":"§VII-B, Table I, Appendix C"},{"comment":"All central claims—the comparison with other models, the ranking, and every ablation—are evaluated exclusively on ParEval [2], a benchmark co-authored by two of the present authors. No independent parallel-code benchmark is used to validate the ranking, and no decontamination check is reported between HPC-INSTRUCT (whose seed snippets are drawn from The Stack v2, the same corpus used to pretrain several comparison models) and ParEval. The claim that HPC-Coder-V2 is the best open-source model for parallel code generation therefore rests on a single, self-authored, potentially benchmark-specific evaluation. The authors should validate the ranking on at least one external benchmark or provide a contamination analysis (e.g., checking for n-gram overlap between HPC-INSTRUCT and ParEval prompts), or explicitly acknowledge this as a limitation of the headline claim.","section":"§V-A, §VII-B, §III-A"},{"comment":"The conclusion that the quality of synthetic data significantly impacts fine-tuned model performance (RQ3) uses the generating LLM identity as a proxy for data quality, but no automated or human verification of the correctness of the HPC-INSTRUCT problem-solution pairs is reported. The paper itself notes in Section VI-B that this approach 'will not allow us to infer what makes the data better or worse.' If a substantial fraction of the generated solutions contain subtle correctness bugs—which is plausible given that the generating LLMs are asked to write non-trivial parallel code—the fine-tuned models may be learning flawed patterns, and the reported ParEval gains plus the data-quality conclusions would not transfer to real parallel-code generation. The authors should either add a verification or random-audit step for the synthetic data, or reframe the RQ3 result as an effect of the data source model rather than of 'data quality' as a validated property.","section":"§VI-B, RQ3, §III-A"}],"minor_comments":[{"comment":"Typo: 'Tangetially' should be 'Tangentially'.","section":"§IV-B"},{"comment":"The caption says 'MPI fine-tuning date'; 'date' should be 'data'.","section":"Figure 5 caption"},{"comment":"The sentence 'These problems range across 12 different problem types: sort, scan, dense linear algebra, sparse linear algebra, search, reduce, histogram, stencil, graph, geometry, fourier transform and transform help us show the diversity' is grammatically incomplete; 'and transform help us show' should be rephrased, for example as '... and transform. These problem types help us show the diversity ...'.","section":"§V-A"},{"comment":"The binomial-coefficient notation in the pass@k formula is garbled in the text (the superscript/subscript layout does not render correctly). Please fix the typesetting so that the formula reads as 1 - C(N-c_p, k)/C(N, k).","section":"Equation (1)"},{"comment":"The claim 'best performing open-source code LLM for parallel code generation to date' should be qualified as applying to the ParEval benchmark, since no external benchmark is used.","section":"Abstract and §I"},{"comment":"In the throughput-versus-pass@1 scatter plot, the HPC-Coder-V2-16B dot is not labeled with its memory requirement (GB) or throughput; please add the missing label for completeness.","section":"Figure 11"},{"comment":"The related-work discussion mentions HPC-specific models such as MPIrigen [23] and OMPGPT [31] but does not quantitatively compare against them; a brief statement of why they are not directly comparable (e.g., task scope or evaluation protocol) would help situate the contribution.","section":"§VIII-A"}],"recommendation":"major_revision","confidential_remarks":"The overlap between the authors and the ParEval benchmark (reference [2]) is a genuine independence concern for the review process. Because the headline ranking and all ablations depend on ParEval, I would encourage the editor to request an external validation benchmark or a rigorous contamination analysis before accepting the 'best open-source' claim. The dataset and model releases are a strong practical contribution, and the statistical issues are addressable in principle, so major revision rather than rejection seems appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real news here is the dataset and the study, not the headline number. HPC-INSTRUCT (122k synthetic instruction pairs) and the three fine-tuned models are public, and the ablation work asks the right questions: base vs instruct, instruction masking, data quantity, data source, and model size. The finding that base models beat their instruct variants is a useful data point, and the data-source quality effect (Llama-3-70B data helping more than DBRX) is worth taking seriously. The paper is also honest about its limits — it admits it cannot quantify data quality and that hyperparameters were cursorily tuned. That transparency earns real credit.\n\nThe soft spot is exactly where the stress-test lands. The claim that HPC-Coder-V2-16B is the best open-source parallel code model rests on a 0.8-point gap over the 6.7B model and a 2.0-point gap over Phind-V2-34B on ParEval, with no confidence intervals and no significance testing. Appendix C says pass@1 comes from 20 samples per prompt; with 420 problems, a 2-point gap is comfortably within sampling noise. The fact that ParEval is the same group's benchmark makes the ranking even shakier, and the absence of a contamination check between the training data and ParEval is a genuine omission. The synthetic data quality issue is real but secondary — it affects transfer, while the ranking is already fragile on statistical grounds.\n\nNone of this kills the paper. The dataset and the qualitative ablation insights are valuable regardless of whether 16B is truly #1. The authors could fix the main problem with bootstrap confidence intervals, an external benchmark, or at least a per-problem error analysis. The claim should be softened until that is done.\n\nThis paper is for anyone building specialized code LLMs for HPC or low-resource parallel languages. It deserves a serious referee: the resource is useful, the study is honest, and the flaws are addressable. I would ask for the statistics, not reject the work.","headline":"Solid dataset and ablation study, but the 'best open-source' ranking is statistically fragile — worth reviewing, not desk-rejecting.","tokens_in":19263,"tokens_out":1429,"would_cite":true,"duration_ms":14380,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning on synthetic HPC instruction data lets a 16B open-source model generate parallel code at near-GPT-4 accuracy.","keywords":["large language models","code generation","high performance computing","parallel code","synthetic instruction data","fine-tuning","ParEval","MPI"],"falsifier":"Take a few hundred HPC-INSTRUCT samples, compile and run them against ParEval's unit-test drivers, and compare the dataset's own pass rate with the pass@1 of the fine-tuned models; if the dataset's pass rate is low, or if retraining on only verified-correct samples changes ParEval pass@1 by more than a few points, the unverified-data assumption is the load-bearing weak point.","tokens_in":18135,"feed_emoji":"⚙️","tokens_out":12079,"duration_ms":84953,"temperature":0.7,"pith_summary":"The paper's claim is that generating high-quality, HPC-specific instruction data is a tractable way to improve LLMs for parallel code, and that fine-tuning open models on such data can close much of the gap to frontier commercial models. To support the claim, the authors build HPC-INSTRUCT, roughly 122k problem–solution pairs distilled from four LLMs (Gemini-Pro, DBRX, Llama-3-70B, Mixtral-8x7B) using open-source parallel-code snippets as seeds, then fine-tune DeepSeek-Coder base models of 1.3B, 6.7B, and 16B parameters on this data plus two general code-instruction datasets. Their best model, HPC-Coder-V2-16B, scores 34.1 pass@1 on the ParEval parallel-code benchmark, ahead of Phind-V2-34B (32.1), Magicoder-6.7B (30.4), and StarCoder2-15B (23.1), and within a few points of GPT-4 (37.8) and GPT-3.5 (39.6). The ablation studies identify the levers behind the gain: data source (and by proxy quality) matters more than data volume for larger models, fine-tuning base weights beats fine-tuning instruct variants, instruction masking has little effect, and gains from scaling to 16B are small compared with the jump from 1.3B to 6.7B. If these conclusions hold, HPC teams can build competitive parallel-code assistants from open weights rather than relying on closed APIs.","feed_headline":"Open-source 16B model hits near-GPT-4 accuracy on parallel code","feed_subtitle":"Fine-tuned on 122K synthetic HPC examples, HPC-Coder-V2-16B posts a 34.1 ParEval pass@1, beating 34B Phind-V2.","key_machinery":"The machinery is HPC-INSTRUCT, a synthetic instruction dataset built by wrapping seed snippets from open-source HPC code in four prompt templates (programming, translation, optimization, parallelization), having four LLMs generate problem–solution pairs, and keeping the roughly 122k samples that parse. The authors then fine-tune DeepSeek-Coder base models on this dataset together with Magicoder-OSS-Instruct-75K and Evol-Instruct-Code-80k-v1, and evaluate on ParEval, a 420-problem benchmark that compiles and unit-tests generated code across twelve problem types and seven execution models, reporting pass@1, the estimated probability that a single generated sample passes the tests. The controlled variations of this dataset and the fine-tuning configuration carry the argument: data amount, data source, base-model choice, instruction masking, and model size are each varied while measuring ParEval pass@1.","core_discovery":"The central discovery is that a mid-size open-source code LLM can be made the best open-source generator of parallel code by fine-tuning on synthetic HPC instruction data. The authors report that HPC-Coder-V2-16B achieves the highest ParEval parallel pass@1 among open models (34.1), exceeding 34B Phind-V2 (32.1) and Magicoder-6.7B (30.4); the 6.7B variant reaches 33.3 while using 14.6 GB memory and higher throughput than Phind-V2-34B's 67.1 GB. They also report that fine-tuning the original DeepSeek-Coder base models consistently outperforms fine-tuning their instruct variants, that masking instruction tokens during training makes little difference, that increasing MPI data volume helps the 1.3B model up to 6k samples with diminishing returns while leaving the 6.7B model essentially unchanged, and that synthetic data generated by Llama-3-70B yields up to six pass@1 percentage points more than DBRX-generated data. The paper interprets these results as evidence that the quality of the synthetic data used for fine-tuning is the binding constraint, and that with the right data a modest open-source model can reach near-frontier parallel-code generation.","pith_inferences":["Because the paper uses the generating LLM as a proxy for data quality and never verifies the synthetic solutions, the data-source ranking could reflect stylistic imitation rather than semantic correctness; running a correctness-filtered version of HPC-INSTRUCT would separate the two.","The authors note in Section III-C that their hyperparameters were chosen by cursory experiments, so the exact pass@1 gaps across model sizes could shift under a more exhaustive search even if the main data-quality findings stand.","The small gain from 6.7B to 16B suggests that an intermediate size near 3B parameters might capture most of the benefit, which would make HPC-specialized assistants practical on consumer hardware; the paper does not test this.","The MPI data-volume experiment stops at 12k samples, so the conclusion that larger models are indifferent to more data may only hold within this range; scaling to tens of thousands of MPI samples is a natural stress test."],"forward_implications":["A 16B open-weight model can serve as a practical parallel-code assistant, delivering correctness close to GPT-4 at a fraction of the memory and per-token cost.","HPC data collection should focus on source-model quality rather than sheer volume; fine-tuning on data from a stronger generator can yield several points of pass@1 improvement.","Future HPC specialization should start from code base models rather than their instruct-tuned checkpoints, since base weights fine-tune more effectively for this domain.","MPI and MPI+OpenMP remain the weakest execution models even after targeted training, so they are the highest-value targets for future data generation and verification.","The diminishing returns between 6.7B and 16B mean that most of the parallel-code benefit of this recipe is available in a model that can run on a single GPU."],"supporting_citations":[{"why":"Supplies the ParEval benchmark and the baseline finding that LLMs generate parallel code far worse than serial code.","marker":"[2]"},{"why":"Provides the seed-snippet synthetic data generation method and the Magicoder-OSS-Instruct-75K dataset used in fine-tuning.","marker":"[8]"},{"why":"Source of the open-source seed snippets from The Stack v2 and of the StarCoder2 comparison models.","marker":"[3]"},{"why":"Defines the DeepSeek-Coder 1.3B and 6.7B base models that are fine-tuned throughout the study.","marker":"[10]"},{"why":"Defines the DeepSeek-Coder-V2 16B mixture-of-experts base model that yields the best HPC-Coder-V2 result.","marker":"[11]"},{"why":"Supplies the Evol-Instruct-Code-80k-v1 dataset included alongside HPC-INSTRUCT during fine-tuning.","marker":"[14]"},{"why":"Establishes the pass@k evaluation metric and the HumanEval reference used to contrast base model quality.","marker":"[18]"},{"why":"Previous HPC-Coder fine-tuning work on a GPT-2-era PolyCoder base, which the current study explicitly supersedes.","marker":"[27]"}],"fun_headline_variants":["Open-source 16B tops parallel code, beating 34B","16B HPC model beats 34B rival in parallel code","Fine-tuned 16B beats 34B at parallel code","Synthetic data lifts 16B to top open parallel coder","HPC-Coder-V2-16B: best open parallel code"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes without verification that the roughly 122k synthetic problem–solution pairs in HPC-INSTRUCT are correct and 'high quality,' using only the generating LLM as a proxy for quality, so if many solutions contain subtle parallel bugs the fine-tuned models could be learning flawed patterns and the reported ParEval gains would not transfer to real code.","fun_headline_variants_meta":{"raw":{"variants":["Open-source 16B tops parallel code, beating 34B","16B HPC model beats 34B rival in parallel code","Fine-tuned 16B beats 34B at parallel code","Synthetic data lifts 16B to top open parallel coder","HPC-Coder-V2-16B: best open parallel code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00104,"raw_usage":{"total_tokens":4396,"prompt_tokens":986,"completion_tokens":3410,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":3318}},"tokens_in":602,"tokens_out":3410,"duration_ms":20113,"temperature":1.0,"reasoning_tokens":3318,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:33:32.864629+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a few hundred HPC-INSTRUCT samples, compile and run them against ParEval's unit-test drivers, and compare the dataset's own pass rate with the pass@1 of the fine-tuned models; if the dataset's pass rate is low, or if retraining on only verified-correct samples changes ParEval pass@1 by more than a few points, the unverified-data assumption is the load-bearing weak point.","supporting_citations":[{"cited_title":"Can large language models write parallel code?","cited_arxiv_id":null,"evidence_quote":"Supplies the ParEval benchmark and the baseline finding that LLMs generate parallel code far worse than serial code."},{"cited_title":"Evaluating large language models trained on code,","cited_arxiv_id":null,"evidence_quote":"Establishes the pass@k evaluation metric and the HumanEval reference used to contrast base model quality."},{"cited_title":"Modeling parallel programs using large language models,","cited_arxiv_id":null,"evidence_quote":"Previous HPC-Coder fine-tuning work on a GPT-2-era PolyCoder base, which the current study explicitly supersedes."}],"review_version":1}