{"id":"90f01d03-c65e-42f7-bdf8-a4739009b048","arxiv_id":"2507.19904","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CrossPL, a 1,982-task benchmark built from GitHub repositories, shows that LLMs achieve at most 79.74% pass@1 on cross-language IPC code generation and struggle with low-level protocols like Pipe.","lead":"This paper introduces CrossPL, the first benchmark for testing whether large language models can write code that makes two different programming languages talk to each other. The authors find that even the best models only get about 80% of the tasks right, and that a model's size or reasoning mode does not reliably help.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FSM matching is never validated against execution, and because FSM strictness likely varies by protocol and language, the reported rankings in Tables I-IV may be artifacts of the checker rather than of model capability.","rationale":"The reader's weakest assumption is exactly the one I would flag: FSM-based matching is treated as functional correctness without validation. My stress-test adds two details that make the concern more specific. First, the FSMs are not just the scorer; they are the filter that decides which canonical snippets enter the benchmark at all (Algorithm 2, A3). If the FSMs are too permissive, the dataset contains non-interoperating 'reference' code; if too strict, valid snippets are dropped. In both cases the benchmark content itself is contaminated before any model is evaluated. Second, because the FSMs are per-language and per-technique, any mismatch between FSM strictness and true correctness can masquerade as a language or protocol effect. The paper's observation that gRPC is easiest and Pipe hardest is exactly the kind of result that FSM calibration artifacts could produce, since gRPC has a small standardized API surface and Pipe has many valid low-level idioms. This does not refute the conservative qualitative conclusion that models are far from reliable on CPL code: if the checker is permissive, true pass rates are lower, and the 'struggle' headline only strengthens. The reason I keep the verdict unchanged rather than moving to REJECT is that the benchmark is still novel and the direction of the main claim is unlikely to flip; the missing execution validation is a well-scoped condition that the authors can satisfy with a modest sample. A stricter reading might move to CONDITIONAL with the execution check as an acceptance condition, which is consistent with the reader's existing verdict. I do not see an internally fatal contradiction, and the concerns about dataset imbalance, error bars, and the HumanEval-X citation are secondary to the FSM validity issue.","tokens_in":21799,"tokens_out":5520,"duration_ms":66085,"concrete_test":"Stratified execution check: sample about 120 tasks, roughly 5 per language x technique cell, and for each take the generated outputs that passed and failed FSM validation for one or two representative models. Build a real two-process test harness using the task's language peer plus a correct counterpart in the other language, compile and run each output, and record true functional success (processes start, bind/connect succeeds, data round-trips). Compare FSM verdict with execution verdict per cell. If false-positive or false-negative rates differ by more than about 10 percentage points across cells, recompute Tables I and III with execution-based labels and check whether the gRPC/Pipe and C++/Go rankings persist. This directly settles whether the observed rankings are checker artifacts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"CrossPL's central numbers rest on the claim (Section III-C) that FSM-based matching 'validates functional correctness.' This is load-bearing because the same FSMs are used at both ends of the pipeline: Algorithm 2 uses FSM A3 to accept the canonical snippets that become tasks, and Algorithm 1 line 24 uses FSM(σi) to score every generated answer. No experiment in the paper checks the FSM verdict against actual compilation or IPC execution. Static API-sequence matching can pass code with wrong types, wrong ports, missing peer setup, or runtime errors; it can also reject correct code that uses an equivalent but non-enumerated API sequence. The risk is not just uniform inflation: the 156 FSMs are hand-built per technique and language, so acceptance thresholds are unlikely to be calibrated across cells. That makes RQ1 and RQ2 comparisons unsafe. For example, gRPC's highly standardized API may be easy to match while low-level Pipe has many valid variants, which could explain the observed 'gRPC highest, Pipe lowest' pattern in Tables III-IV without any real model-capability difference. The same applies to C++ vs Go in Table I. The Limitations section (Section VI) lists uneven samples, missing FFI, and Qwen3-only RQ3, but does not flag the unvalidated FSM proxy, which is the most consequential threat. Since the headline conclusion that 'even the best models struggle' would likely survive even if all pass rates fell, the concern primarily undermines the benchmark's quantitative claims and its language/technique rankings, which are core to RQ1-RQ2.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"CrossPL is a benchmark of 1,982 tasks intended to test LLMs on generating cross-programming-language (CPL) interoperating code. The tasks are derived from 19,169 multi-language GitHub repositories: 156 hand-crafted FSMs locate IPC interaction points, and an LLM-based pipeline (DeepSeek-V3) extracts snippets, generates instructions, and validates snippets. The authors evaluate 20 LLMs using FSM-based validation and report pass@1/pass@5 across six languages and seven IPC techniques. They find that even the best model (GLM4-plus) reaches only 79.74% pass@1, with most models below 70%, and conclude that current LLMs struggle with CPL/IPC generation, that performance varies by language and technique, and that thinking mode does not reliably help.","tokens_in":22123,"tokens_out":10080,"duration_ms":110602,"significance":"If the validity threats were resolved, CrossPL would be a useful addition: it is, to my knowledge, the first large-scale benchmark targeting IPC code generation in multi-language project contexts, and it compares 20 models on a substantial task set with public data and code. The paper is also honest about some limitations (uneven samples, no FFI, RQ3 scope). However, the benchmark's central numbers depend on two premises that are not established: that the FSM matcher measures functional correctness, and that the tasks are genuinely cross-language rather than single-language IPC exercises. These are not minor caveats; they affect the interpretation of every table and the main contribution claim. The strengths are the scale, the multi-language/multi-technique coverage, and the transparent pipeline.","major_comments":[{"comment":"The paper repeatedly states that FSM-based validation 'validates functional correctness' (Abstract and Section III-C), but the FSMs are static API-call sequence matchers, not execution. No experiment in the manuscript compares FSM verdicts with compilation, runtime behavior, or human judgment. Code that follows the enumerated steps can still have wrong types, wrong ports, or a broken peer setup and pass; correct code that uses an equivalent but non-enumerated API sequence can fail. Because the same family of FSMs is used both to accept extracted snippets during benchmark construction (Algorithm 2, A3) and to score generated code, any FSM blind spot biases both the dataset and the evaluation in the same direction. The pass@1/pass@5 numbers in Tables I-IV are therefore best interpreted as protocol-template match rates, not functional correctness rates, and the claim should be either renamed or backed by a stratified validation study (e.g., executing a sample of accepted and rejected outputs, or human annotation) with agreement statistics.","section":"Section III-C (Algorithm 1, line 24)"},{"comment":"The tasks in CrossPL appear to be single-language IPC implementations. The metadata recorded for each instance is (p_i, τ_i, θ_i, L_i, σ_i, K_i) with a single language L_i, and the example in Fig. 7 asks for a Python gRPC server setup only. The 'cross-language' property is inferred from the multi-language repository in which the snippet was found, not from the task itself, and the model is not required to know or generate any counterpart in another language. This raises a construct-validity issue for the title claim: the benchmark measures the ability to generate boilerplate IPC code in one language, which is related to but distinct from generating code that interoperates with a specific other-language component. The authors should either include cross-language context in the task (e.g., the other side's API or protocol contract) or reframe the contribution as an IPC-code benchmark for multi-language software contexts.","section":"Section III-B and Fig. 7"},{"comment":"Pass@5 is computed with a minimum sample size of N = 5. For k = 5 and n = 5, the unbiased pass@k estimator reduces to 1 whenever at least one of the five samples passes (since C(5-c,5) = 0 for c > 0), so the reported pass@5 is an 'any-of-five' success rate, not the standard pass@5 used in HumanEval and later work. This inflates all pass@5 values and makes comparisons with other benchmarks meaningless. The authors should use n > k (e.g., n = 20) and report the estimator, or explicitly rename the metric and discuss its properties.","section":"Section V-A"},{"comment":"The cross-language and cross-technique comparisons are descriptive only: no confidence intervals, standard errors, or significance tests are reported, despite highly unbalanced subset sizes (e.g., C++ has 51 tasks, UDP has 92, Java has 615). The claimed patterns 'C++ best, Go worst' and 'gRPC best, Pipe worst' could therefore be driven by task difficulty, FSM strictness, or noise rather than model capability. Because the FSM acceptor is hand-built per technique and language, acceptance thresholds are unlikely to be calibrated across cells; the paper should report per-cell sample sizes with confidence intervals and test the key comparisons.","section":"Section V-B, Tables I-IV"},{"comment":"The RQ3 conclusion that 'thinking mode often reduces performance' is not supported by the reported numbers. In Table V, thinking mode improves Qwen3-8b (52.76 to 55.64) and Qwen3-30b-a3b (61.28 to 63.39), while it hurts Qwen3-4b (59.14 to 54.44) and Qwen3-235b (75.02 to 72.92); the remaining results are mixed. No significance tests or effect sizes are given. Similarly, the 'deviation from scaling law' claim is based on descriptive means without addressing variance or task imbalance. These claims should be rephrased as observations or supported with statistical analysis.","section":"Section V-D, Tables V-VI"}],"minor_comments":[{"comment":"The text states that pass@5 'consistently exceed their pass@1 counterparts across all task subcategories,' but Table IV shows Gemma-7b with pass@5 below pass@1 on HTTP (43.31% vs 47.50%) and Message Queue (46.68% vs 50.00%). This contradiction should be fixed.","section":"Section V-B and Table IV"},{"comment":"There are typos in model names and sizes: 'Lamma3-8b-instruct' appears in Tables I-IV, 'Qwen2.5-coder-32b-insrtuct' appears in Fig. 5, and the size for Lamma3-8b in Table II is listed as 70B while Table I says 8B. Please correct these.","section":"Tables I-II"},{"comment":"The manuscript refers to supplementary details via 'in here' placeholders (e.g., prompt templates and additional benchmark details). These links are not resolved in the submitted text, which makes the pipeline difficult to reproduce. Please provide a complete appendix or stable URLs.","section":"Section III-B"},{"comment":"The Limitations section lists uneven samples, missing FFI, and Qwen3-only RQ3, but it does not mention the most consequential threats discussed above: the lack of execution-based validation for the FSM checker and the single-language nature of the tasks. The section should be updated to acknowledge these.","section":"Section VI"},{"comment":"The claim that CrossPL is 'the first benchmark' for CPL code generation would be more convincing with a systematic comparison of existing benchmarks that include IPC or multi-language interaction tasks (e.g., web/API generation benchmarks), rather than only multilingual translation benchmarks.","section":"Section I"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the FSM proxy lands: the central numbers are not yet credible as functional-correctness rates. I nonetheless recommend major revision rather than reject because the dataset and pipeline are potentially salvageable: a stratified execution/human validation study, a corrected pass@5 protocol, and a reframing of the 'CPL' claim would address the main issues. The authors should also consider whether the benchmark should be described as 'IPC code generation in multi-language repositories' rather than 'cross-programming-language code generation.'"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Briefly: CrossPL is the first real attempt at benchmarking LLM generation of cross-language IPC code, and that alone is worth something. The dataset construction is substantial — 156 hand-built FSMs, 1,982 tasks across six languages and seven techniques, 20 models evaluated. The paper correctly identifies a gap that multilingual benchmarks don't fill: translation between languages is not the same as writing code that makes two languages talk to each other. The empirical headline, that even strong models struggle, is plausible and probably robust.\n\nThe soft spots are real. The evaluation is FSM-based matching, not execution. Section III-C calls this 'functional correctness,' but an FSM is a static pattern over API call order. Code that hits the right calls in the right order passes even if it wouldn't compile or run. That's not a trivial quibble: the same FSMs are used to extract the benchmark snippets and to score generated answers, so the numbers in Tables I-IV are only as meaningful as the checker's alignment with reality. The paper never validates the FSM verdicts against compilation or a single IPC exchange, and the Limitations section doesn't even mention this. The dataset imbalance (C++ 51 tasks vs Java 615) and lack of error bars make the language and technique rankings even shakier. The stress-test note's worry that gRPC-high, Pipe-low could be an artifact of FSM strictness is fair.\n\nThat said, the FSM concern doesn't sink the main claim. Even if all pass rates fell a bit, models still aren't near reliable on this kind of code. But it does undermine the quantitative comparisons, which are the core of RQ1 and RQ2.\n\nThis paper deserves a serious referee, but the referee should demand either execution-based validation on a sample or a much more careful statement that the benchmark measures API-sequence compliance, not functional correctness. I'd also want confidence intervals and a corrected citation for HumanEval-X. For a reading group, it's a good discussion piece; I wouldn't cite the raw numbers in my own work until the validation issue is addressed.","headline":"CrossPL builds a genuinely new benchmark for cross-language IPC code generation, but its credibility rests on an unvalidated FSM checker in place of execution.","tokens_in":22661,"tokens_out":1948,"would_cite":false,"duration_ms":22668,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new benchmark finds even the best LLMs fail roughly one in five cross-language IPC coding tasks.","keywords":["cross-programming language","code generation benchmark","inter-process communication","finite state machine validation","large language models","multilingual software systems","pass@k evaluation"],"falsifier":"Take a random sample of CrossPL tasks whose generated code passed FSM validation, compile or run each snippet together with its real counterpart process, send actual messages, and compare the observed success rate with the FSM pass rate; a large drop would show the validator overestimates model capability.","tokens_in":21619,"feed_emoji":"🔗","tokens_out":6109,"duration_ms":67292,"temperature":0.7,"pith_summary":"The paper introduces CrossPL, a benchmark of 1,982 tasks that ask an LLM to generate working inter-process communication (IPC) code connecting components written in different programming languages. Its purpose is to test a capability existing benchmarks ignore: not translating code, but producing code in one language that must interoperate with code in another. Across 20 LLMs and seven IPC techniques, the authors find that even the strongest model passes only 79.74% of tasks at pass@1 and most models stay below 70%, so correct cross-language glue code is not yet reliable. If the benchmark is sound, it gives the field a measurable target for a real-world skill that matters for multi-language software systems.","feed_headline":"Best LLM scores 79.7% on cross-language IPC code tasks","feed_subtitle":"This 1,982-task benchmark tests whether models can write code that lets programs in different languages talk to one another.","key_machinery":"The load-bearing object is the finite state machine (FSM) encoding of an IPC interaction pattern, such as import networking library, create server socket, bind port, listen, accept, exchange data, and close. The 156 FSMs, extended from a coarser eight-FSM toolkit, describe expected API call sequences per language and technique, skip code comments, and carry semantic descriptions of each step. These same FSMs do three jobs: they locate candidate CPL snippets in repositories, validate the extracted snippets during benchmark construction, and score LLM-generated code during evaluation. Because they encode protocol-required call order, they can catch missing steps such as a gRPC server that never binds a port.","core_discovery":"The paper claims to present the first benchmark dedicated to LLM generation of cross-programming-language (CPL) interoperating code. CrossPL contains 1,982 IPC tasks mined from 19,169 multi-language GitHub repositories, covering Java, Python, Go, JavaScript, PHP, and C++ across seven IPC techniques. Correctness is judged by 156 hand-crafted finite state machines that encode the expected API call sequence for each IPC scenario. The central empirical result is that no evaluated model reliably generates such code: the best pass@1 is 79.74% (GLM4-plus), most models fall below 70%, performance varies sharply by language and protocol, and reasoning modes that simulate thinking do not help and sometimes hurt.","pith_inferences":["If the FSM validator is more permissive than real execution, the true success rates are likely lower than the reported pass@1 numbers, making the gap larger rather than smaller.","The FSM descriptions could be reused as structured generation constraints or as a reward signal in RL fine-tuning, a direction the paper does not explore.","Because FFI (foreign function interface) is deliberately excluded, the benchmark covers only one half of cross-language interoperability; models might behave differently on FFI tasks that require type mapping and runtime linking.","The thinking-mode result suggests current reasoning modes are tuned for general problem-solving rather than protocol compliance; a direct test would be to compare error types, such as missing steps versus wrong syntax, between base and thinking modes."],"forward_implications":["A reliable capability gap is now measurable: no evaluated model reaches 80% pass@1 on CPL IPC tasks, and most models sit below 70%.","CrossPL gives the community a fixed 1,982-task testbed for comparing models on cross-language interoperability, broken down by language and by IPC technique.","The per-technique results isolate where models fail most: gRPC is comparatively easy, while Pipe and HTTP tasks drag scores down.","The Qwen3 family results indicate that neither larger scale nor thinking mode is a reliable lever for CPL IPC code generation."],"supporting_citations":[{"why":"Supplies the unbiased pass@k metric used for all reported scores.","marker":"[17]"},{"why":"Introduces the FSM-based interoperation analysis toolkit that CrossPL extends with 156 finer-grained FSMs.","marker":"[38]"},{"why":"Provides the LLM used in the automated pipeline that extracts CPL snippets and generates task instructions.","marker":"[42]"},{"why":"Demonstrates the LLM-based instruction-generation practice that CrossPL adopts for building tasks.","marker":"[33]"},{"why":"Documents one of the top-performing general-purpose models evaluated in the study.","marker":"[44]"},{"why":"Documents the best-performing evaluated model, which achieves 79.74% pass@1.","marker":"[45]"}],"fun_headline_variants":["Best LLM gets 79.7% on new cross-language code benchmark","CrossPL: first benchmark for LLM cross-language code generation","LLMs struggle with cross-language IPC code: top score 79.7%","79.7% top pass rate: LLMs fail at cross-language code tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The FSM check assumes that generating the expected API calls in the right order is sufficient for the code to work, and it never compiles or runs the generated code.","fun_headline_variants_meta":{"raw":{"variants":["Best LLM gets 79.7% on new cross-language code benchmark","CrossPL: first benchmark for LLM cross-language code generation","LLMs struggle with cross-language IPC code: top score 79.7%","79.7% top pass rate: LLMs fail at cross-language code tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000337,"raw_usage":{"total_tokens":1856,"prompt_tokens":926,"completion_tokens":930,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":848}},"tokens_in":542,"tokens_out":930,"duration_ms":9353,"temperature":1.0,"reasoning_tokens":848,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:51:06.911040+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of CrossPL tasks whose generated code passed FSM validation, compile or run each snippet together with its real counterpart process, send actual messages, and compare the observed success rate with the FSM pass rate; a large drop would show the validator overestimates model capability.","supporting_citations":[{"cited_title":"Polyfax: A toolkit for characterizing multi- language software,","cited_arxiv_id":null,"evidence_quote":"Introduces the FSM-based interoperation analysis toolkit that CrossPL extends with 156 finer-grained FSMs."},{"cited_title":"Domaineval: An auto-constructed benchmark for multi-domain code generation,","cited_arxiv_id":null,"evidence_quote":"Demonstrates the LLM-based instruction-generation practice that CrossPL adopts for building tasks."},{"cited_title":"Glm-4 model api documentation,","cited_arxiv_id":null,"evidence_quote":"Documents the best-performing evaluated model, which achieves 79.74% pass@1."}],"review_version":1}