{"id":"d876b00e-976b-4848-81c8-4ee4c8391432","arxiv_id":"2505.05063","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CodeMixBench adds code-mixed prompt variants to BigCodeBench and shows that code-mixed inputs consistently reduce Pass@1 performance of code LLMs.","lead":"This paper introduces CodeMixBench, a benchmark that rewrites BigCodeBench coding tasks into code-mixed prompts in Hinglish, Spanish-English, and Chinese Pinyin-English. Testing 17 open-source code models, it finds that code-mixed prompts lower Pass@1 accuracy, with the largest drops for small models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The degradation claim hinges on unvalidated semantic fidelity: GAME scores are LLM-based and only Hinglish results are reported, while Table 3 itself contains CMD=0.6 non-degradations, so 'consistently degrade' is not yet established.","rationale":"The reader's weakest-assumption analysis correctly identifies semantic fidelity as the load-bearing requirement. Without human-validated preservation of task meaning, the Pass@1 drops could be caused by translation errors rather than by code-mixing itself. My reading adds two supporting observations that reinforce a conditional verdict rather than overturning it. First, the paper's own results section evaluates only the Hinglish complete split, so the abstract's three-language claim is broader than the reported evidence. Second, Table 3 contains three models that do not degrade at CMD=0.6, which contradicts the literal wording 'consistently degrade' unless that wording is understood as an average trend. These issues are addressable: the benchmark is constructed from BigCodeBench, the evaluation harness is standard, and the CMD framework is a reasonable design. The lack of released artifacts and the absence of error bars add uncertainty but do not by themselves invalidate the contribution. I also note that Algorithm 1 in Appendix 8.3 appears to contain a while-loop that would replace the same word multiple times rather than selecting distinct top-k words; if the implementation follows the pseudocode literally, the CMD manipulation itself would be unreliable, which would be an even more fundamental problem. However, since the actual implementation is not provided, I treat this as a secondary concern. The most direct path to settling the central claim is human evaluation of semantic fidelity, as proposed in the concrete test. If the test shows that GAME is a reliable proxy and only a small fraction of prompts have altered semantics, the conditional verdict can be upgraded; if not, the headline claim should be weakened or the benchmark rebuilt with human-checked prompts.","tokens_in":11726,"tokens_out":5671,"duration_ms":59211,"concrete_test":"Have at least two bilingual annotators independently rate semantic equivalence for 100 randomly sampled prompts per language and CMD level (600 total) against the original English prompt, focusing on task-critical elements such as preconditions, return values, edge cases, and argument order. Compare these human ratings with the GAME scores for the same prompts. Then recompute Pass@1 after excluding prompts that any annotator flags as semantically changed. If the flagged fraction is nontrivial or if GAME fails to correlate with human ratings, the observed degradation cannot be cleanly attributed to code-mixing, and the abstract claim would need to be qualified or re-measured.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that code-mixed prompts degrade Pass@1 relative to English-only prompts, with larger drops at higher CMD. This requires that the code-mixed prompts preserve the original task's semantics; otherwise, observed drops are translation artifacts, not robustness effects. Section 3.3 claims semantic fidelity via GAME, which back-translates each prompt with Gemini-2.0-Flash-Lite and compares all-MiniLM-L6-v2 sentence embeddings. This is weak evidence for task-critical preservation: embedding cosine similarity can miss a single mistranslated condition, a changed return contract, or a swapped argument, all of which would cause test failures in BigCodeBench. The mean GAME of 90 is reported without distribution, per-language/per-CMD breakdown, error bars, or human validation, and it is exactly at CMD=0.9 that translation difficulty and semantic drift risk are highest. Additionally, Section 5 states that all code-mixed evaluations were run only on the Hindi-English complete split, although the abstract generalizes to three language pairs. Finally, Table 3 already contradicts 'consistently degrade' at CMD=0.6: OpenCoder-8B-Instruct improves (51.3 vs 50.9), DeepSeek-R1-Distill-Llama-8B improves (15.9 vs 15.3), and Hermes-2-Theta is unchanged (36.4 vs 36.4). The benchmark may still be useful, but the headline robustness finding is not yet supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CodeMixBench, a code-generation benchmark derived from BigCodeBench by translating and code-mixing natural-language instructions and docstrings into Hinglish (Hindi-English), Spanish-English, and Chinese Pinyin-English at two controlled code-mixing degrees (CMD 0.6 and 0.9). The authors describe an LLM-based augmentation pipeline using Gemini-2.0-Flash-Lite, a GAME-based semantic fidelity check, and an evaluation of 17 open-source code LLMs (1.5B to 15B parameters) using greedy decoding and the Pass@1 metric against BigCodeBench unit tests. The central claim is that code-mixed prompts consistently degrade Pass@1 relative to English-only prompts, with larger drops at higher CMD levels, especially for smaller models.","tokens_in":12060,"tokens_out":7480,"duration_ms":64939,"significance":"If the benchmark's semantic fidelity were convincingly established, CodeMixBench would fill a genuine gap: existing code-generation benchmarks evaluate English-only prompts, while multilingual developers frequently code-mix. The resource itself, with 1,140 tasks across three language pairs and two CMD levels, is potentially useful, and the promised public release under CC-BY 4.0 is a positive feature. The execution-based Pass@1 evaluation using the BigCodeBench harness is also a strength. However, the evidence reported in the manuscript does not yet support the headline claim: the evaluation is limited to the Hinglish complete split, the CMD axis appears internally inconsistent, the semantic fidelity check is weak, and the results show several non-degradations and no statistical significance testing. The paper is transparent about some limitations in Section 6.4, but those limitations directly constrain the generality of the stated conclusions.","major_comments":[{"comment":"The claim that code-mixed prompts 'consistently degrade' Pass@1 is contradicted by the reported numbers in Table 3. At CMD 0.6, OpenCoder-8B-Instruct improves (51.3 vs. 50.9), DeepSeek-R1-Distill-Llama-8B improves (15.9 vs. 15.3), Hermes-2-Theta is unchanged (36.4 vs. 36.4), and Phi-4-multimodal-instruct is essentially flat (46.4 vs. 46.5). The CMD 0.9 vs. CMD 0.6 comparison is also non-monotonic for several models (e.g., StarCoder2-7b: 6.8 vs. 8.9; Phi-4: 46.7 vs. 47.1; DeepSeek-R1-Distill-Qwen-14B: 38.8 vs. 40.3). The term 'consistent degradation' is therefore not supported by the table. I recommend reporting aggregate statistics with paired significance tests (e.g., bootstrap over the 1,140 tasks) and softening the claim accordingly, or providing corrected data if the table contains errors.","section":"§5, Table 3; Abstract; §6.1"},{"comment":"The definition of CMD is internally inconsistent with the reported results and with the prompt examples. Section 3.2 states that CMD = 0.6 retains 60% of switchable words in English and CMD = 0.9 retains more, and Algorithm 1 replaces matrix-language words with English words with probability proportional to CMD. Under this definition, higher CMD should yield prompts closer to English and should presumably produce less degradation, not more. Yet the paper describes CMD = 0.9 as 'heavy mixing' and reports larger performance drops there. Moreover, Table 5's CMD = 0.9 Hinglish example is almost entirely Hindi, with no English tokens except 'Args' and 'Returns', which is opposite to the stated definition. Table 6, by contrast, shows CMD 0.9 with more English tokens than CMD 0.6. This makes the independent variable ambiguous. Please clarify the direction of CMD, correct the algorithm or the examples, and then reinterpret the results accordingly.","section":"§3.2, Algorithm 1, Table 5"},{"comment":"The semantic fidelity validation is insufficient to support the causal claim that code-mixing, rather than translation error or introduced ambiguity, causes the observed Pass@1 drops. The GAME score uses Gemini-based back-translation and cosine similarity of all-MiniLM-L6-v2 embeddings, with only a mean of 90 reported; there is no per-language, per-CMD distribution, no standard deviation, and no human validation. For BigCodeBench tasks, a single mistranslated condition, changed return contract, or swapped argument can cause test failures, and embedding similarity is not a reliable detector for such discrete errors. The authors' own Section 6.5 acknowledges that constructing human-authored datasets would 'help eliminate translation artifacts', which underscores the risk. Stronger validation (e.g., execution-based checks on a sample, human annotation, or per-task manual review) is needed before the benchmark can be said to measure robustness to code-mixing rather than robustness to translation noise.","section":"§3.3, §6.5"},{"comment":"The evaluation does not support generalization to three language pairs as claimed in the Abstract and Section 1. Section 5 states that code-mixed prompts were evaluated 'exclusively in the Hindi-English code-mix subset', and Table 3 contains only Hinglish results. No Pass@1 results are reported for Spanish-English or Chinese Pinyin-English, which are nevertheless claimed in the Abstract and Section 3.4 to be part of the benchmark. Section 6.4 also concedes that 'several models and code-mixing scenarios remain unexplored'. Please either provide experimental results for the other two language pairs or explicitly restrict the conclusions to the Hinglish subset. As written, the headline claim about 'three language pairs' is unsupported by any measurement.","section":"§5 (first sentence), §3.4, §6.4"},{"comment":"No error bars, confidence intervals, or repeated sampling are reported, yet Pass@1 is computed from a single greedy decode per task. Differences of 1–2 points (e.g., OpenCoder-8B-Instruct 51.3 vs. 50.9 at CMD 0.6) are within the range of what could be expected from random variation across 1,140 tasks. Without a paired significance test (e.g., bootstrap over tasks or multiple sampling runs), the claim that code-mixed prompts 'consistently' degrade performance is not statistically established. Please add such analysis or explicitly present the differences as descriptive rather than as evidence of a consistent effect.","section":"§5.1, §4.2, Table 3"}],"minor_comments":[{"comment":"The replacement-score equation is typeset incorrectly: 'si = f (engi) f (hii)' is missing the division symbol and should read 'si = f(engi) / f(hii)', and the summation for f(hii) is garbled as '3X j=1'. Please fix the mathematical formatting.","section":"§3.2.1"},{"comment":"Figures 1 and 2 appear to contain raw CSV fragments and label text such as 'Original CSV' embedded in the plot areas, making them difficult to interpret. Please regenerate these figures with proper axis labels, legends, and a clean caption.","section":"Figures 1 and 2"},{"comment":"The description of DeepSeek-R1-Distill-Llama-8B is factually inaccurate: it is claimed to have been trained from scratch on 2 trillion tokens with 87% code and 13% code-mixed natural language, but that description applies to DeepSeek-Coder, not to the R1-distilled Llama model, which is obtained by distillation from DeepSeek-R1 onto a Llama-3.1-8B architecture. Please correct the model description.","section":"§5.2"},{"comment":"The statement that 'instruction-tuned variants consistently outperforming their base counterparts' is not supported by Table 3, which does not include paired base/instruct model comparisons in the same table. Please either add such comparisons or remove this unsupported generalization.","section":"§6.1"},{"comment":"The GAME score is reported only as a mean across all prompts; please report the distribution (e.g., standard deviation, quantiles) and a per-language/per-CMD breakdown, since a single mean can hide large variance that is directly relevant to the benchmark's validity.","section":"§3.3"},{"comment":"The Hinglish example labeled CMD = 0.9 is almost entirely Hindi, which contradicts the definition in Section 3.2 and the CMD = 0.9 example in Table 6. Please reconcile these examples so that the CMD level is unambiguously illustrated.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is potentially useful, but the current manuscript overreaches relative to the evidence. The most significant issues are the inconsistency of the CMD definition with the reported trend, the restriction of evaluations to the Hinglish subset, and the weak semantic-fidelity check. These are fixable with additional experiments and analysis, so I recommend major revision rather than rejection. I would also suggest the editors ask the authors to verify all factual claims about model training data, as the DeepSeek-R1-Distill description is demonstrably incorrect."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CodeMixBench is a genuinely useful idea: take a solid code generation benchmark, BigCodeBench, and add controlled code-mixing to the natural-language side of the prompts. That is new, and worth pursuing. But the headline finding, as stated, is not supported by the paper's own numbers. Table 3 shows several models at CMD=0.6 doing as well as or better than on English (OpenCoder-8B 51.3 vs 50.9, DeepSeek-R1-Distill-Llama-8B 15.9 vs 15.3, Hermes unchanged), and a few at CMD=0.9 recovering relative to 0.6 (StarCoder2-7B: 6.8 to 8.9; Phi-4: 46.7 to 47.1). So the 'consistently degrade with higher CMD' statement in the abstract is an overclaim.\n\nWhat the paper does well: the CMD mechanism is adapted sensibly from Gupta et al., the model range is broad (17 models, 1.5B to 15B), and the authors correctly identify semantic fidelity as the key threat to validity. The pipeline of translation, PoS-tagging, frequency-based replacement, and romanization is described in enough detail to reproduce.\n\nThe soft spots. First, the actual evaluation is only on the Hinglish subset; Spanish-English and Pinyin-English are generated but never scored. The abstract and conclusion claim 'three language pairs' — that is misleading. Second, the GAME validation is weak for this purpose. An LLM back-translation plus MiniLM cosine similarity can miss a single mistranslated argument or condition, which is exactly what breaks BigCodeBench tests. The mean GAME of 90 is reported with no distribution, no per-CMD breakdown, and no human check. Third, there are no error bars; with greedy decoding, differences like 50.9 vs 51.3 are indistinguishable from noise. Fourth, the paper makes specific claims about model training data (e.g., DeepSeek's '13% code-mixed natural language') with no citation; that should be checked or dropped. Fifth, all data and scripts are promised 'upon acceptance,' so the benchmark itself cannot be independently evaluated yet.\n\nThe core idea is sound and the paper is worth a serious referee, but my recommendation would be major revision. The authors need to report results for all three language pairs, quantify uncertainty, validate semantic fidelity more carefully (human samples, task-level checks), and align the claims with what Table 3 actually shows. If those are fixed, CodeMixBench is a reasonable contribution to code LLM evaluation.","headline":"CodeMixBench is a genuinely useful evaluation idea, but the headline claim of consistent degradation is contradicted by the paper's own table and the evidence base is narrower than the abstract suggests.","tokens_in":12536,"tokens_out":3158,"would_cite":false,"duration_ms":28823,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Code-mixed prompts consistently cut code LLM Pass@1 scores, with the largest losses in small models.","keywords":["code-mixed prompts","code generation","large language models","Pass@1","code-mixing degree","multilingual robustness","benchmark evaluation"],"falsifier":"Take a sample of code-mixed prompts, back-translate them to English, and re-run the models: if the Pass@1 drop on back-translated English prompts matches the drop on code-mixed prompts, the degradation is an artifact of translation rather than code-mixing. Alternatively, have bilingual human raters judge semantic equivalence; a substantial fraction of changed-meaning verdicts would break the benchmark's central inference.","tokens_in":11561,"feed_emoji":"🌐","tokens_out":5162,"duration_ms":49187,"temperature":0.7,"pith_summary":"CodeMixBench, a new benchmark built by augmenting an existing 1,140-task execution-based code benchmark, tests whether large language models can generate correct code when the natural-language instructions are code-mixed. The paper's central claim is that code-mixed prompts consistently lower Pass@1 accuracy compared with English-only prompts, and the drop grows as the code-mixing degree increases. The effect is most severe for small models, while large and instruction-tuned models are more resilient. If true, English-only benchmark numbers overstate how well code assistants serve real multilingual developers.","feed_headline":"English-only scores overstate code LLMs for mixed-language users","feed_subtitle":"A new 17-model benchmark finds Pass@1 drops grow with code-mixing level, hitting small models hardest.","key_machinery":"The central instrument is the controllable code-mixing degree (CMD), a number in $[0,1]$ that sets how many eligible content words in a translated prompt are replaced back into English. Replacement choice is driven by a frequency score $s_i = f(eng_i)/f(hi_i)$ computed from a code-mixed corpus; words whose matrix-language form is rare receive an infinite score and are always replaced first. Semantic preservation is verified with GAME, an embedding-based back-translation similarity metric that the paper reports as averaging around 90 out of 100 across CMD levels.","core_discovery":"The paper introduces a code-mixing degree (CMD) in $[0,1]$ that controls how many content words in a translated prompt are switched back to English, and uses it to augment 1,140 tasks at CMD levels 0.6 and 0.9 across Hinglish, Spanish-English, and Chinese Pinyin-English. Evaluating 17 open-source models from 1.5B to 15B parameters, the discovery is that Pass@1 declines consistently as CMD rises; at CMD 0.9, several small models lose more than half of their English-only accuracy. Larger, instruction-tuned models lose less, and robustness appears to track training-data diversity more than model size alone.","pith_inferences":["The degradation mechanism could be isolated by comparing token fragmentation and sequence length on romanized words; if longer sequences explain the drop, a transliteration-normalizing tokenizer should recover much of it.","Extending code-mixing to comments, variable names, and inline annotations, where the prompt structure does not protect meaning, would likely show even earlier and larger accuracy drops.","A human-verification subset of the code-mixed prompts could separate semantic-fidelity failures from robustness failures: models that also fail on back-translated English versions are reacting to mistranslation rather than to code-mixing itself.","Because replacement scores come from a Twitter code-mixed corpus, the benchmark inherits an informal-register bias; mixing in repository documentation or formal technical writing may behave differently."],"forward_implications":["English-only Pass@1 numbers give an optimistic picture of code assistants for users who naturally write prompts in mixed language.","Evaluation dashboards should report performance at multiple CMD levels, since heavy mixing (CMD 0.9) exposes failure modes that light mixing does not.","Small models, which are often deployed on edge devices, are precisely the ones that degrade most under code-mixed prompts.","Adding noisy multilingual or code-mixed data to pretraining appears more effective for code-mixing robustness than instruction tuning alone."],"supporting_citations":[{"why":"Provides the 1,140 task base and the execution harness with library function calls that CodeMixBench augments.","marker":"Zhuo et al. (2025)"},{"why":"Supplies the controlled code-mixing (CMD) generation method and the GAME semantic-fidelity metric.","marker":"Gupta et al. (2024)"},{"why":"Defines the Pass@1 evaluation metric and the English-only code-generation evaluation protocol that the study extends.","marker":"Chen et al. (2021)"},{"why":"Provides the code-mixed corpus frequencies used to compute replacement scores for the controlled code-mixing injection.","marker":"Nayak and Joshi (2022)"}],"fun_headline_variants":["CodeMixBench: Mixed-language prompts hit small LLMs hardest","English-only code benchmarks overstate performance for mixed-language users","CodeMixBench: Small code models lose the most on mixed-input prompts","Code-mixed prompts degrade code LLMs, small ones worst","Pass@1 drops on code-mixed prompts; small models suffer most"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark assumes the LLM-created code-mixed prompts preserve the original task meaning well enough that any Pass@1 drop is caused by code-mixing itself rather than by translation errors or added ambiguity.","fun_headline_variants_meta":{"raw":{"variants":["CodeMixBench: Mixed-language prompts hit small LLMs hardest","English-only code benchmarks overstate performance for mixed-language users","CodeMixBench: Small code models lose the most on mixed-input prompts","Code-mixed prompts degrade code LLMs, small ones worst","Pass@1 drops on code-mixed prompts; small models suffer most"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002102,"raw_usage":{"total_tokens":8149,"prompt_tokens":907,"completion_tokens":7242,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":7153}},"tokens_in":523,"tokens_out":7242,"duration_ms":50107,"temperature":1.0,"reasoning_tokens":7153,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:13:38.964602+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of code-mixed prompts, back-translate them to English, and re-run the models: if the Pass@1 drop on back-translated English prompts matches the drop on code-mixed prompts, the degradation is an artifact of translation rather than code-mixing. Alternatively, have bilingual human raters judge semantic equivalence; a substantial fraction of changed-meaning verdicts would break the benchmark's central inference.","supporting_citations":[{"cited_title":"Bigcodebench: Evaluating language models on tool-augmented code generation","cited_arxiv_id":null,"evidence_quote":"Provides the 1,140 task base and the execution harness with library function calls that CodeMixBench augments."},{"cited_title":"L 3 C ube- H ing C orpus and H ing BERT : A code mixed H indi- E nglish dataset and BERT language models","cited_arxiv_id":null,"evidence_quote":"Provides the code-mixed corpus frequencies used to compute replacement scores for the controlled code-mixing injection."}],"review_version":1}