{"id":"1714ec07-b2a6-43e3-943a-1b95700ebce3","arxiv_id":"2505.16107","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Using multiple programming languages as code-style prompts during fine-tuning improves LLM information extraction accuracy over single-language prompting.","lead":"Teaching a language model to extract facts works better when the same input is written as code in three programming languages, not one. The approach beats a model four times its size on average, but the gains are small and the paper omits error bars.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central multi-PL claim is plausible but rests on a single unverified prompt-equivalence assumption; the key missing control is a matched single-PL-in-three-syntaxes experiment.","rationale":"The reader identified the same load-bearing assumption I do: the three templates are assumed semantically equivalent, differing only in syntax, and this is never verified. My concern is a sharper version of that: the Appendix C examples show actual structural differences (Python function-only versus C++/Java with main() wrappers and different output-collection styles), so the assumption is not merely unverified; it is visibly violated in the provided examples. The paper's own controlled Table 5 shows that prompt-format variation with 3x data (76.63) and seed ensembling (76.60) do not reach MPL (77.60), but that comparison does not isolate syntax diversity from PL identity. The missing control is a single-language, three-syntax variant. I additionally checked the manuscript text for self-attested limitations: the Limitation section only mentions training cost; the code-release statements are contradictory (abstract says 'released', Section 7 says 'will release later', Appendix C defers to 'submitted code file'); no test statistics or confidence intervals are shown despite the claimed t-test in Table 1; and Appendix B leaves most SLM reference papers as '?'. These are reporting issues that justify the CONDITIONAL verdict, not rejection. The main result is well supported across backbones and datasets, and the ablation design is otherwise thoughtful, so I do not see a load-bearing flaw that would overturn the central claim; I would keep the reader's CONDITIONAL verdict, pending the equivalence check and the reporting fixes.","tokens_in":18224,"tokens_out":1673,"duration_ms":13121,"concrete_test":"Run an equivalence-control ablation: train LLaMA3-8B on (a) the current Python template, (b) the current C++ template, (c) the current Java template, (d) three syntactically varied Python templates that preserve the same task definition, label set, and execution flow but vary function signatures, variable names, and boilerplate, and (e) MPL. If (d) matches or exceeds (b)/(c) in average Micro-F1, the multi-PL gain is likely due to surface variability rather than PL-specific diversity. Additionally, compute a token-alignment or semantic-equivalence measure among the three templates on a held-out sample to verify the Section 2.3 consistency assumption.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's core interpretation is that training on Python, C++, and Java code-style inputs outperforms Python alone because programming-language diversity adds structural variability (Section 5.2). The load-bearing assumption, stated only as a design guideline in Section 2.3, is that the three templates are semantically equivalent in task definition, label descriptions, and execution logic, differing only in syntax. The Appendix C examples (Figures 4-6) show that while the label descriptions are largely copied, the surrounding code differs in more than syntax: Python uses a function returning EntityList; C++ includes an explicit main() that calls the function and appends entities; Java wraps everything in a class and a main method. These are different execution flows and different amounts of boilerplate, and the paper never verifies that information content is identical. More importantly, the ablation in Table 4 compares Python-only (76.4) against MPL (77.6), but no experiment trains on the Python template plus two other syntactic variants of the same template, nor measures token-level or information-level equivalence across the three templates. Without that control, the +1.2 gain attributed to programming-language diversity could partly reflect incidental content differences, such as C++/Java templates containing additional structural cues or different output presentation. This is not an internal inconsistency, but it is a real soft spot in the causal claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MPL, a supervised fine-tuning framework for information extraction that reformulates each IE instance as code-style input in three programming languages (Python, C++, and Java) and trains an LLM on the shuffled mixture. It also introduces function-prompt with a 'virtual running' step as a more compact alternative to the class-prompt used by prior work. The authors evaluate on 11 IE datasets spanning NER, RE, EAE, and EE, using several backbone LLMs, and report that MPL-8B reaches 77.6 average Micro-F1, exceeding the GoLLIE-34B baseline by 1.1 points and its own Python-only baseline by 1.2 points. Additional experiments cover zero-shot transfer, ablations of single vs. multiple PLs, data-size controls, function-prompt vs. class-prompt efficiency, and complementarity analyses across PLs.","tokens_in":18414,"tokens_out":4502,"duration_ms":42560,"significance":"If the central claim holds, MPL is a useful and low-cost recipe for improving code-style IE: it works across multiple backbones, adds no external pretraining data, and comes with a more token-efficient prompt format. The paper's strengths are its breadth of evaluation (11 datasets, multiple LLMs), the explicit ablation of ensemble and data-size confounds in Section 5.2, the zero-shot assessment, and the release of code. The main weakness is that the causal interpretation—that programming-language diversity, rather than incidental template-content differences, drives the +1.2 F1 gain—is not yet backed by a matched control experiment. The statistical reporting of the claimed p-values is also incomplete.","major_comments":[{"comment":"The central attribution of the +1.2 F1 gain to 'programming language diversity' is not supported by the current controls. The paper never verifies that the Python, C++, and Java templates are semantically equivalent beyond the design guideline stated in Section 2.3. The Appendix C examples show that the templates differ not only in surface syntax but in execution scaffolding: Python returns an EntityList from a function, C++ adds a main() with push_back calls, and Java wraps everything in a class with List.add. These differences could enrich the C++/Java inputs with additional structural cues or alter output presentation. The prompt-variation baselines in Table 5 (label reordering, format variation) are all applied within Python and do not isolate language identity from template content. A decisive control would be to train on the Python template plus two syntactic transliterations of that same template into C++ and Java, preserving the label descriptions and code structure as much as the target language allows. Without this condition, the observed gain could partly reflect content differences rather than PL diversity.","section":"Section 5.2, Table 5"},{"comment":"The claim that results are 'statistically significant with a p-value of less than 0.05' is not verifiable from the paper. Only five-run averages are reported, with no standard deviations, no test statistic, no statement of whether the t-tests are paired or unpaired, and no correction for the multiple comparisons across 11 datasets and multiple model pairs. Since the reader cannot reconstruct the tests, the significance claim should be moved to an appendix with full details, or the averages should be accompanied by per-dataset variances. This is important because several reported differences (e.g., 93.5 vs. 93.1 on CoNLL03) are very small in absolute terms.","section":"Section 4.1, Table 1 caption"},{"comment":"The appendix does not currently provide the faithful per-language exemplars that the method needs. Figure 4 is captioned as the Python input but displays C++ code (vector<Entity*>, const string&, int main); Figure 5 repeats the same C++ block; Figure 6 contains the invalid token 'GPEe'. These artifacts prevent a reader from knowing the actual Python template and undermine the template-equivalence assumption at the point where it could be checked. The authors should correct the figures against the released templates, or acknowledge the duplication and provide the true Python example.","section":"Appendix C, Figures 4-6"}],"minor_comments":[{"comment":"The Abstract states that code has been released, while the Conclusion says the code 'will be released later'; the GitHub link in the footnote should be reconciled with these statements.","section":"Abstract / Conclusion"},{"comment":"Several SLM baseline results are listed with '?' in the reference column; full citations should be provided for all reported baseline numbers.","section":"Appendix B, Table 8"},{"comment":"The table header contains the typo 'Avg.Socre'; also, the 'Union' aggregation is optimistic by construction (a prediction is correct if any PL is correct), so the discussion in Section 5.4 should state more clearly that this is an upper-bound style metric, not a deployable inference strategy.","section":"Table 6"},{"comment":"The text says the comparison is between class-prompt and function-prompt but does not explicitly state that both use the same language (presumably Python-only); this should be stated so the reader can relate it to the multi-PL results.","section":"Figure 3"},{"comment":"The Jaccard similarity computation is not defined precisely: it should specify whether it is over gold or predicted spans, and how partial matches are handled.","section":"Table 7"},{"comment":"The KnowCoder column is missing OntoNotes5* and RAMS values; the footnote explains the alignment issue, but the headline 'competitive with KnowCoder' should be qualified in the text for these datasets.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical study with a plausible hypothesis, and I do not see an internal inconsistency or a fatal error. The main risk is that the paper's headline causal claim—multi-PL diversity helps—is not yet isolated from template-content changes. The missing control (same template content expressed in three syntactic variants) is feasible and would materially strengthen the paper. The statistical reporting also needs a concrete appendix. The appendix figure duplication is a clear correctness issue that must be fixed. If these are addressed, the paper could be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the MPL paper. The headline result is that fine-tuning an LLM on code-style IE inputs written in Python, C++, and Java beats the same training with Python alone: LLaMA3-8B hits 77.6 average Micro-F1 across 11 IE datasets, +1.2 over the Python-only baseline and +1.1 over GoLLIE-34B. The idea is genuinely new—prior code-style IE work (GoLLIE, KnowCoder, CodeIE, Code4Struct) is Python-only—and the function-prompt with virtual running is a nice efficiency win (about 20% shorter inputs). The experimental breadth is impressive: multiple backbones, zero-shot tests, and a careful ablation study in Section 5.2 that attempts to rule out ensembling and prompt-perturbation explanations. That section is a real methodological contribution.\n\nThe main soft spot is the causal claim. The paper attributes the gain to “programming language diversity,” but the three templates are not pure syntactic variants: the C++ and Java versions wrap the logic in a main()/class and use typed lists, while the Python version is top-level. The authors state as a design guideline that the logic and execution flow are consistent, but they never verify information equivalence. A control that trains on, say, the C++ template plus two syntactic variants of that same template (e.g., different type spellings or variable names) would directly test whether structural diversity—not just content differences—is what matters. Without that, a skeptical reader can say the gain comes from extra boilerplate or output-format cues. The single-PL results are close (76.4/76.6/76.4), which mitigates the concern, but the p-value claim in Table 1 makes it worse: they say t-test p<0.05 but report no standard deviations or test statistics anywhere. You cannot check that.\n\nOther reporting issues: the abstract says “we have released our code” while the conclusion says “we will release… later”; Appendix B’s baseline table has question marks instead of citations for most entries; and the appendix figures are mislabeled—Figure 4 is captioned “Python” but shows C++ code, and Figure 5 is also C++, so no Python example appears. These are all fixable.\n\nWho is this for? Anyone working on code-style IE or structured output generation with LLMs will want to know this result. It deserves a serious referee; the empirical foundation is solid enough that the paper should not be desk-rejected. I’d recommend conditional acceptance after the authors provide variance estimates, correct the figure captions and code-release statement, and ideally run the syntactic-variant control. If they show the gain survives that control, the causal story stands.","headline":"Multi-PL training beats Python-only code-style IE by about a point across backbones, and the controls make a credible case that the gain is not just ensembling; the causal story is plausible but unverified, and the paper needs a round of reporting fixes.","tokens_in":19010,"tokens_out":5905,"would_cite":true,"duration_ms":49051,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a language model on the same information-extraction task written in Python, C++, and Java yields higher accuracy than fine-tuning on Python alone, and a new function-prompt with virtual running makes the code-style input…","keywords":["information extraction","code-style prompting","multiple programming languages","supervised fine-tuning","function-prompt","virtual running","structured output generation","large language models"],"falsifier":"A reader could test the language-diversity claim by keeping the docstrings byte-identical across Python, C++, and Java and varying only the syntactic boilerplate; if the advantage over a single-language model disappears under that control, the effect is not caused by programming-language diversity. A second observation would be to train on a Python-only dataset tripled in size under the exact hyperparameters used for MPL; the paper reports 76.1 for that setting, so reproducing that drop would support the claim, while matching the MPL score would undermine it.","tokens_in":17967,"feed_emoji":"⌨️","tokens_out":9534,"duration_ms":77857,"temperature":0.7,"pith_summary":"This paper tries to establish that code-style prompting for information extraction need not be tied to a single programming language. It claims that encoding the same task, label set, and input text as a function in Python, C++, and Java, and fine-tuning a language model on all three, yields better Micro-F1 than fine-tuning on Python alone. The paper also proposes a lighter prompt format, the function-prompt with virtual running, that wraps the task into one function and simulates its execution on the input text; this format cuts tokenized input length by about 20% compared with the class-prompt while slightly improving accuracy. Across eleven NER, relation, and event extraction datasets, an 8B-parameter model trained this way averages 77.6 Micro-F1, 1.2 points above its Python-only baseline and 1.1 points above a 34B-parameter rival. If the claim holds, the syntactic diversity of programming languages is itself a training signal for structured extraction, not merely a formatting convenience.","feed_headline":"Multi-language code training lifts an 8B extraction model","feed_subtitle":"Fine-tuning on Python, C++, and Java prompts adds 1.2 Micro-F1 over Python alone and tops a 34B baseline.","key_machinery":"The functioning engine of the method is the function-prompt with a 'virtual running' step. A named function such as Named_Entity_Recognition(InputText) is defined with the task definition and full label set inside its docstring, and the text to be analyzed is supplied as the argument; generation then proceeds as if the function had executed and returned a list, with every extracted item appended via calls such as EntityList.append(GPE(\"Iraq\")). This replaces the class-prompt's boilerplate constructors and repeated attribute assignments, shrinking tokenized input by roughly 20% while adding 0.4 Micro-F1 on the default setting. Around that core, the framework's distinctive element is the use of three syntactic shells, Python, C++, and Java, that share identical docstring content and execution order, differing in type declarations, container syntax, and how outputs are appended. The three templates are shuffled during fine-tuning, and predictions from all three are aggregated by voting at test time.","core_discovery":"The central discovery is that fine-tuning on the same extraction task expressed in several programming languages produces a measurable, statistically significant improvement over fine-tuning on a single language. The mechanism is the function-prompt: each task becomes a single function whose docstring carries the task definition and label set, and generation proceeds as if the function had been 'virtually run' on the input text, with each extracted item appended to a returned list. The authors generate three semantically matched templates for every input, one in Python, one in C++, and one in Java, following the same design logic and execution flow, and shuffle them during training. Test-time predictions from the three languages are combined by majority voting, reaching 77.6 average Micro-F1 on eleven benchmarks; a Python-only model trained under identical settings scores 76.4, tripling the Python data does not reproduce the gain, and ensembling random seeds or reordering labels gives at most 76.63. The three languages also produce complementary errors, since the union of their predictions scores 79.5, with the largest union-voting gaps on relation extraction and event extraction.","pith_inferences":["If the gap between MPL_sampled, which trains on one randomly chosen language per example and scores 77.1, and full MPL at 77.6 is confirmed at larger scale, then most of the benefit comes from diversity rather than token volume; sampling one language per example could capture most of the gain at a third of the training cost.","The three languages are a convenience sample; measuring the marginal F1 added by a fourth syntactic shell, such as Rust, Go, or SQL, would reveal whether the benefit saturates or continues to grow.","The virtual-running step resembles program-trace prompting; if its 0.4-point gain replicates on other structured-output tasks, the same 'define a function, run it on the input, append to a list' device may transfer to spreadsheet generation, database query construction, or configuration reasoning.","The union-voting gap of up to 16 points on relation extraction suggests the three languages are not redundant; a testable extension is to have the model output per-prediction confidence scores and train a small aggregator on them, which could beat the default voting scheme without changing the fine-tuning data."],"forward_implications":["If MPL's claim holds, code-style IE systems can stop treating Python as the default; training on multiple languages is a cheap way to raise extraction F1 without growing the model.","Function-prompt is shorter and slightly more accurate than class-prompt, so switching prompt format offers a 20% reduction in tokenized input length, and a proportional reduction in per-example training cost, while improving performance.","The union of the three languages' outputs exceeds voting by about 2 points, implying that better aggregation, such as confidence weighting or a learned combiner, could raise results further without retraining.","The zero-shot gains on datasets absent from training suggest that multi-language supervision transfers to new schemas and domains, not just to the eleven seen benchmarks.","The reported 77.6 average comes at more than double the training cost of the single-language setting, so the method is a compute-for-accuracy trade rather than a free lunch."],"supporting_citations":[{"why":"Supplies the class-prompt baseline and the main comparison point; MPL is measured against its 34B version.","marker":"[Sainz et al., 2023]"},{"why":"KnowCoder is the other leading code-style IE system with a schema-understanding phase; MPL aims to match its performance with lighter training.","marker":"[Li et al., 2024c]"},{"why":"QLoRA is the parameter-efficient fine-tuning method used to train all MPL variants.","marker":"[Dettmers et al., 2024]"},{"why":"CodeLLaMA backbones anchor the main results; the 7B and 13B versions are compared against the corresponding baselines.","marker":"[Roziere et al., 2023]"},{"why":"CodeIE establishes the code-style simulation idea for IE that MPL extends from Python to multiple languages.","marker":"[Li et al., 2023d]"},{"why":"Code4UIE uses class-prompting with retrieval augmentation; it is a direct prior for the prompt format MPL replaces.","marker":"[Guo et al., 2023]"}],"fun_headline_variants":["Function-prompt training across PLs adds 1.2 Micro-F1","C++/Java prompts beat Python-only IE by 1.2 F1","Multi-PL SFT: 1.2 F1 gain over single-language","Three PLs beat Python-only for extraction","Multi-language code prompts lift extraction F1 by 1.2"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Python, C++, and Java templates carry the same task and label information and differ only in programming-language syntax; if one template inadvertently includes richer label descriptions, the gains could come from that content rather than from language diversity.","fun_headline_variants_meta":{"raw":{"variants":["Function-prompt training across PLs adds 1.2 Micro-F1","C++/Java prompts beat Python-only IE by 1.2 F1","Multi-PL SFT: 1.2 F1 gain over single-language","Three PLs beat Python-only for extraction","Multi-language code prompts lift extraction F1 by 1.2"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000902,"raw_usage":{"total_tokens":3891,"prompt_tokens":962,"completion_tokens":2929,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":2835}},"tokens_in":578,"tokens_out":2929,"duration_ms":21065,"temperature":1.0,"reasoning_tokens":2835,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:06:19.751588+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could test the language-diversity claim by keeping the docstrings byte-identical across Python, C++, and Java and varying only the syntactic boilerplate; if the advantage over a single-language model disappears under that control, the effect is not caused by programming-language diversity. A second observation would be to train on a Python-only dataset tripled in size under the exact hyperparameters used for MPL; the paper reports 76.1 for that setting, so reproducing that drop would support the claim, while matching the MPL score would undermine it.","supporting_citations":[],"review_version":1}