{"id":"07280036-48a0-47b2-8d59-6abbd8ff2ede","arxiv_id":"2411.15470","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"PolyCoder's pass@1 performance on translated HumanEval benchmarks is inconsistent across MultiPL-E and HumanEvalSynthesize, reversing the relative ranking of Java and Rust.","lead":"This paper evaluates the PolyCoder code model on two translated versions of the HumanEval benchmark across four programming languages. It finds that the two benchmark translations rank languages differently and that the results are hard to reproduce, so translated benchmarks need careful validation before use.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's validation claim is contradicted by the paper's own MultiPL-E results: Java, the top language, is explicitly said not to align with perplexity.","rationale":"The reader's strongest_claim correctly identified the contradiction in the abstract, but their weakest_assumption focused on the HumanEvalSynthesize implementation fidelity. While the implementation concern is valid and acknowledged in Section 6, it is not the most load-bearing issue for the paper's central claim. Even if the HumanEvalSynthesize results were entirely discarded, the MultiPL-E results alone—with Java outperforming Python and the authors admitting this does not align with perplexity—directly refute the abstract's assertion of alignment. The paper's stated conclusion is unsupported by its own evidence, making the internal inconsistency the primary reason the central claim fails. A concrete test of rank correlation between perplexity and pass@1 would settle whether any alignment exists; given the reported data, the correlation is unlikely to be significantly positive. The reader's verdict of REJECT remains appropriate, so I do not change it.","tokens_in":8506,"tokens_out":3075,"duration_ms":25544,"concrete_test":"Extract per-language perplexity values for PolyCoder 2.7B from Xu et al. [23] and compute the Spearman rank correlation between perplexity (lower = better) and pass@1 across the four languages for each benchmark (Tables 2 and 3). Also compute bootstrap confidence intervals for each pass@1 value, given the small number of problems (156–164), to assess whether the language rankings are even distinguishable. If the correlation is not significantly positive, or if Java and Rust rank opposite to perplexity ordering, the central claim fails unequivocally.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that translated benchmark outcomes 'align well with evaluation metrics used during the training phase, such as perplexity'—is directly contradicted by the paper's reported data and its own narrative. In Section 4, the authors state that MultiPL-E's best-performing language for the larger PolyCoder models is Java (5.63% pass@1 for 2.7B), and they explicitly write that this 'does not align with the perplexity scores' from Xu et al. In HumanEvalSynthesize, the second-best language for the 2.7B model is Rust (3.05%), a low-resource language that presumably has high perplexity; this also contradicts the claimed alignment. The paper provides no statistical analysis, and with only four languages, three model sizes, and pass rates in the 1–6% range, the observed differences are plausibly within sampling noise. The internal contradiction alone is sufficient to invalidate the abstract's validation claim, regardless of whether the HumanEvalSynthesize implementation faithfully reproduces OctoPack. Thus the most load-bearing concern is not the faithfulness of the bigcode harness, but the absence of any evidence for the stated alignment and the presence of direct evidence against it.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a preliminary empirical study of PolyCoder (160M, 0.4B, and 2.7B) on two translated versions of HumanEval, MultiPL-E and HumanEvalSynthesize, across Python, JavaScript, Java, and Rust, using pass@1 at temperature 0.2. It presents pass rates in Tables 2 and 3, observes several inconsistencies between benchmarks and with model-size trends, and discusses reproducibility challenges. The abstract claims the results align well with perplexity-based metrics, thereby validating translated benchmarks; this claim is the load-bearing conclusion the paper must support.","tokens_in":8700,"tokens_out":7228,"duration_ms":62231,"significance":"The research question is relevant: translated benchmarks are increasingly used to evaluate multilingual code generation, and independent empirical checks are valuable. The paper's strengths are that it uses publicly available models and benchmark pipelines, reports raw pass@1 scores in readable tables, and explicitly acknowledges limitations in Section 6. If the alignment-with-perplexity claim were supported, the result would be a useful validity check for MultiPL-E and HumanEvalSynthesize. However, the paper provides no statistical analysis, and its own data contradict the claim; the contribution is therefore only a small, partially interpretable data point, not a validation of translated benchmarks.","major_comments":[{"comment":"The central validation claim in the abstract is contradicted by the paper's own results. Table 2 shows that in MultiPL-E, Java is the best-performing language for the 0.4B and 2.7B PolyCoder models (3.83% and 5.63%), and Section 4 explicitly states that this 'does not align with the perplexity scores' from Xu et al. Table 3 shows that in HumanEvalSynthesize, Rust (a low-resource language) is the second-best language for the 2.7B model at 3.05%, which also contradicts the expected perplexity ordering. No correlation coefficient, confidence interval, or significance test is provided. The abstract's statement that outcomes 'align well with evaluation metrics used during the training phase, such as perplexity' is therefore unsupported and, by the paper's own account, false; the paper's main conclusion must be reframed or the claim must be withdrawn.","section":"Abstract and Section 4 (Tables 2-3)"},{"comment":"The HumanEvalSynthesize measurements were obtained with the bigcode-evaluation-harness rather than the OctoPack authors' original setup. Section 6 concedes that this implementation 'may differ from the original setup described in the OctoPack study.' If the harness is not equivalent, Table 3 does not measure HumanEvalSynthesize, and the cross-benchmark comparisons in Section 4 (e.g., performance being 'almost reversed' between benchmarks) are not valid. The authors should either demonstrate equivalence to the original OctoPack evaluation or explicitly treat Table 3 as results on the BigCode re-implementation, not on HumanEvalSynthesize itself.","section":"Section 3.2 and Section 6"},{"comment":"The language-level conclusions are drawn from very small absolute differences without uncertainty quantification. For instance, in Table 2 the 2.7B model scores 5.63% on Java, 5.56% on Python, and 5.16% on JavaScript, while in Table 3 the 2.7B model scores 6.10% on Python but 3.05%, 2.44%, and 1.22% on Rust, JavaScript, and Java. Given roughly 156-164 problems and a single pass@1 estimate per cell, these differences are plausibly within sampling noise. Confidence intervals, multiple seeds, or a statistical test are needed to support any ranking claim and to support the 'inconsistency' findings that motivate the paper.","section":"Section 4, Tables 2 and 3"}],"minor_comments":[{"comment":"The sentence comparing Rust performance across benchmarks appears to swap the benchmark names: as written, it says Rust's 3.05% score is 'higher than in the MultiPL-E ... compared to HumanEvalSynthesize,' but Table 3 shows 3.05% in HumanEvalSynthesize and Table 2 shows 2.30% in MultiPL-E.","section":"Section 4"},{"comment":"There is a typographical error in 'HumanEval-XL [19]., which translates' with an extra period after the bracket.","section":"Section 2.1"},{"comment":"The text refers to reference [10] as 'OctoCoder,' but the cited paper is 'OctoPack'; this may confuse readers.","section":"Section 2.3"},{"comment":"The capitalization of the benchmark name is inconsistent: 'HumanEvalSyntheSize' in several places versus 'HumanEvalSynthesize' in the benchmark's own name; please standardize.","section":"Section 3.2"},{"comment":"The claimed consistency of the Python pass@1 scores with Xu et al.'s reported values (2.13%, 2.96%, 5.59%) should be quantified; the observed values differ by up to 0.6 percentage points, so the paper should state whether this is within expected sampling error rather than asserting consistency as validation.","section":"Section 4"}],"recommendation":"reject","confidential_remarks":"The paper is a short workshop paper with a useful negative observation, but the abstract overclaims and the main comparison rests on an unverified implementation of HumanEvalSynthesize. I would encourage the authors to rework it as a replication/negative result with a corrected abstract; as submitted, the central claim fails. The rejection should be on the merits of the validation claim, not on novelty or scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this one. It's a six-page workshop paper, but it reports genuinely new numbers: pass@1 for PolyCoder (160M, 0.4B, 2.7B) on MultiPL-E and HumanEvalSynthesize for Python, JavaScript, Java, and Rust. Those measurements aren't in the cited literature, and the cross-benchmark inconsistency (Java best on MultiPL-E, Rust second-best on HumanEvalSynthesize for the 2.7B model) is real and worth documenting.\n\nThe problem is the abstract. It claims the benchmark outcomes \"align well with evaluation metrics used during the training phase, such as perplexity, thereby validating their effectiveness.\" Section 4 then explicitly says the MultiPL-E Java result \"does not align with the perplexity scores\" from Xu et al. That is a direct internal contradiction, and the stress-test note is right that it is the load-bearing flaw. Without that claim, the paper's actual finding is the opposite: translated benchmarks do not reliably reflect perplexity-based expectations, and different translation methodologies produce different rankings. Sections 5 and 6 are more careful and contain that skeptical message, but the abstract and conclusion don't match.\n\nOther soft spots: no statistical tests, only four languages, pass rates in the 1–6% range, so observed differences are plausibly sampling noise. The HumanEvalSynthesize implementation uses the BigCode harness, which the authors admit in Section 6 may differ from the original OctoPack setup. That's a real caveat, but I agree with the stress-test note: even if the harness were faithful, the Java-perplexity contradiction invalidates the abstract's validation claim. The authors are transparent about this limitation and about hardware constraints, which I credit.\n\nWho is this for? People building or using translated benchmarks. The cautionary message—validate before adopting—is valuable even if the paper's positive claim fails. It deserves a serious referee, not a desk reject, because the measurements are new and the framing is fixable. As it stands, I would not accept it with the current abstract; I'd ask for major revision, specifically to align the claims with the paper's own observations. If revised, it would be a modest but honest contribution.\n\nBottom line: useful data, flawed framing. Engage with it, but expect the referee to catch the contradiction.","headline":"Useful new pass@1 measurements, but the abstract's validation claim is contradicted by the paper's own Java finding—send to review, expect major revision.","tokens_in":9215,"tokens_out":2514,"would_cite":true,"duration_ms":23281,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Poly-Coder's language ranking flips between two translated HumanEval benchmarks","keywords":["code language models","code generation","translated benchmarks","HumanEval","MultiPL-E","pass@1","perplexity","multilingual evaluation"],"falsifier":"Run the same 164 HumanEval problems and the same Poly-Coder checkpoints through the original OctoPack evaluation scripts and through the general harness the authors used, keeping temperature at 0.2. If Java stays above Rust in both, the paper's reversal is an artifact or a translation difference; if the two runs disagree, the HumanEvalSynthesize numbers in Table 3 are a property of the harness, not the benchmark.","tokens_in":8282,"feed_emoji":"💻","tokens_out":6290,"duration_ms":52629,"temperature":0.7,"pith_summary":"This preliminary study asks whether translated code-generation benchmarks can reliably estimate how well a multilingual code model performs in languages beyond Python. It runs the open Poly-Coder model at three sizes against two translated versions of the HumanEval benchmark—an automated large-coverage translation and a smaller manual one—in Python, JavaScript, Java, and Rust, measuring pass@1. The authors' central claim is that the benchmark results align with training-time perplexity scores, which would validate translated benchmarks as proxies for CLM capability. Yet their own tables show the alignment is partial: Java is the best language in the automated benchmark despite poor perplexity rankings, and the two benchmarks reverse the order of Java and Rust. The paper's stated finding is therefore cautious: translated benchmarks can be useful, but methodological differences and replication problems must be checked before adopting them.","feed_headline":"Poly-Coder's language ranking flips between two translated benchmarks","feed_subtitle":"A first study of HumanEval translations finds perplexity mostly tracks pass@1, but Java and the harness raise doubts.","key_machinery":"The machinery is Poly-Coder, an open multilingual code model trained on twelve languages with known per-language perplexity, evaluated through the pass@1 metric at temperature 0.2. The two translated benchmarks are the load-bearing comparators: MultiPL-E, which automatically transpiles HumanEval problems into 18 languages with a few incomplete translations, and HumanEvalSynthesize, a manual, complete translation of all 164 problems into five languages shipped with OctoPack. The evaluation harness used for HumanEvalSynthesize does the mediating work, and the paper's central doubt is whether that harness faithfully reproduces the original benchmark.","core_discovery":"On the paper's own terms, the discovery is that translated HumanEval benchmarks give a usable but unstable picture of multilingual code-model performance. Using Poly-Coder's 160M, 0.4B, and 2.7B variants, pass@1 generally rises with model size in both benchmarks, and the overall pattern is close enough to the perplexity rankings reported when Poly-Coder was released that the authors take this as provisional validation of the translated-benchmark approach. The qualification is visible in the data itself: in MultiPL-E, Java outperforms Python at the two larger sizes, which 'does not align with the perplexity scores'; in HumanEvalSynthesize, Rust beats Java and JavaScript for the 2.7B model, nearly reversing the order. The authors attribute such reversals to translation methodology and to implementing HumanEvalSynthesize through a general evaluation framework that may differ from the original OctoPack setup, so they stop short of endorsing any single translated benchmark as authoritative.","pith_inferences":["A per-language correlation test between perplexity and pass@1 would settle the paper's validation claim more directly than the overall pattern, since the Java and Rust reversals show that an aggregate agreement can hide per-language failures.","The same two benchmark families could be compared on a stronger model, such as an instruction-tuned code model, to see whether the ranking reversals are a Poly-Coder quirk or a property of the benchmarks themselves.","If the harness limitation is confirmed, then published translated benchmarks should start reporting which evaluation script they used, with version hashes, because the paper shows the script can change the result.","A natural extension is to translate the same benchmark with both methodologies for the same five languages and compute agreement on a per-problem basis, which would isolate translation quality from evaluation-harness effects."],"forward_implications":["If translated benchmarks do track perplexity, they can serve as a proxy for multilingual capability when perplexity is unavailable or untrusted.","If benchmark choice changes which language appears strongest, then single-benchmark multilingual comparisons are not reliable evidence; a result should be confirmed on a second translated benchmark.","The consistently low pass@1 across all languages, even for the largest Poly-Coder model, warns that small absolute differences between languages can be magnified into misleading conclusions.","The paper's validation recipe—checking a new translated benchmark against an already-known model's performance—becomes a required step before a translated benchmark is adopted.","If the harness mismatch is real, the HumanEvalSynthesize column should be read as a reimplementation result, not as a result for OctoPack's published benchmark."],"supporting_citations":[{"why":"Supplies the MultiPL-E translated benchmark and its pass@1 evaluation protocol, including the temperature recommendation.","marker":"[8]"},{"why":"Introduces OctoPack and HumanEvalSynthesize, the manually translated benchmark used as the second comparator.","marker":"[10]"},{"why":"Provides the Poly-Coder model, its sizes, its per-language perplexity scores, and the original HumanEval pass@1 numbers used as a validity check.","marker":"[23]"},{"why":"Defines the original HumanEval benchmark and pass@k metric that both translations start from.","marker":"[9]"},{"why":"Implements the evaluation harness used for HumanEvalSynthesize; the paper flags that it may differ from the original OctoPack setup.","marker":"[3]"}],"fun_headline_variants":["Translated benchmarks flip Poly-Coder's language order","Multilingual code benchmarks disagree on Poly-Coder","Study: Translated benchmarks only partly validate Poly-Coder","Pass@1 vs perplexity: Translated benchmarks raise doubts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The HumanEvalSynthesize half of the study is only meaningful if the evaluation harness the authors used is a faithful reproduction of the original OctoPack setup; Section 6 says the implementation 'may differ from the original setup,' and if that is true the table does not measure HumanEvalSynthesize.","fun_headline_variants_meta":{"raw":{"variants":["Translated benchmarks flip Poly-Coder's language order","Multilingual code benchmarks disagree on Poly-Coder","Study: Translated benchmarks only partly validate Poly-Coder","Pass@1 vs perplexity: Translated benchmarks raise doubts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1351,"prompt_tokens":990,"completion_tokens":361,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":310}},"tokens_in":606,"tokens_out":361,"duration_ms":4126,"temperature":1.0,"reasoning_tokens":310,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:15:35.304887+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 164 HumanEval problems and the same Poly-Coder checkpoints through the original OctoPack evaluation scripts and through the general harness the authors used, keeping temperature at 0.2. If Java stays above Rust in both, the paper's reversal is an artifact or a translation difference; if the two runs disagree, the HumanEvalSynthesize numbers in Table 3 are a property of the harness, not the benchmark.","supporting_citations":[],"review_version":1}