{"id":"f3210d43-8844-409f-97f7-8d2ff895b13a","arxiv_id":"2501.16524","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Fine-tuning an LLM on synthetic examples with pronounceable nonce inputs yields a state-of-the-art open-source model for single-sound-law induction from examples.","lead":"The authors train small open-source code models to write sound laws, ordered rewrite rules that turn ancestral proto-words into descendant words, from example word pairs. They find that fine-tuning on artificial examples with pronounceable made-up words works best, producing a model that outperforms larger open-source competitors on a small benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed SOTA for SLI is only measured on a single-law benchmark; the cascade/search-ordering component is removed in Section 5, so the headline overstates what PySLICoder actually does.","rationale":"The reader's weakest-assumption identification is the right one. The paper is internally consistent: the synthetic-data comparison is well controlled (equal 2.5k instances, same base model, same prompt), the RP-LI result is replicated across three runs, and the qualitative conclusion that substantive inputs help but substantively constrained programs hurt is plausible and supported by the diversity analysis in Appendix D. The paper also discloses several limitations (small benchmark, Magicoder-only fine-tuning, limited sampling budget), and the public availability of the full prompts is a reproducibility asset. None of these undermine the internal comparison. The load-bearing gap is external validity. The title, abstract, and Section 7 frame the contribution as a model for SLI, but Section 5 narrows the task to single-law induction and explicitly drops search and ordering. Sound laws are defined in the paper's own introduction as ordered cascades; a model that is never required to order laws has not been shown to solve the problem the paper opens with. The proposed test is feasible because the cascade chronology was already manually reconstructed to build the single-law benchmark (Section 5.1), so the raw data exist to construct the end-to-end version. Since the reader already conditioned acceptance on this issue, my read does not change the verdict; it remains CONDITIONAL, with the condition that the SLI claims be qualified to isolated rule induction until cascade evaluation is provided.","tokens_in":22428,"tokens_out":8043,"duration_ms":83423,"concrete_test":"Build an end-to-end cascade benchmark from the same five language pairs: use the original protoform-to-final-reflex word lists with no intermediate supervision, and score generated programs by whether the complete ordered cascade reproduces the attested reflexes. Evaluate PySLICoder-RP-LI-gpt-4o, Codestral-22B, and GPT-4o under the same s=20 sampling budget. If PySLICoder's 6% pass-rate advantage over Codestral disappears or reverses on this benchmark, the 'strongest open-source LLM for SLI' claim holds only for isolated rule induction and must be qualified accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim is that PySLICoder-RP-LI-gpt-4o is the strongest open-source LLM for sound law induction (Section 7). The evaluation, however, deliberately drops the part of SLI that the introduction identifies as central: cascades of ordered sound laws. Section 5 states that end-to-end SLI 'involves a search and ordering component which is not relevant to our study,' and the benchmark is therefore constructed by taking manually reconstructed cascades, executing one law at a time, and keeping the intermediate changed words (plus distractors) as a 'single law' instance. As a result, the model is never asked to infer which law applies at a given stage, to detect feeding/bleeding interactions between laws, or to choose an ordering that explains the observed reflexes. In real SLI these are not peripheral: the same surface forms can be produced by different law orderings, and a high pass rate on isolated rewrites does not imply competence at full cascade induction. The paper's abstract and Section 7 make unqualified 'SLI' and 'strongest open-source LLM for SLI' claims, so the central claim is only as strong as the single-law proxy. This is an explicit scope restriction, not a hidden flaw, but the paper never tests whether single-law performance predicts cascade performance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper frames sound law induction (SLI) as programming by examples (PBE) for LLMs: given protoform/reflex pairs, generate a Python program (a BasicAction) implementing the single sound law that maps inputs to outputs. The authors introduce a structure-versus-substance framework for synthetic fine-tuning data, propose four conditions (RP-RI, RP-LI, RP-PI, IDP-PI) with increasing substantive bias, fine-tune Magicoder-6.7B on 2.5k synthetic instances from each condition, and evaluate on an 85-instance single-law benchmark built from five language pairs. They report that RP-LI (random programs plus LLM-generated pronounceable nonce inputs) yields the highest pass rate, train two PySLICoder variants on RP-LI data generated by Codestral-22B and GPT-4o, and claim that PySLICoder-RP-LI-gpt-4o is the strongest open-source LLM for SLI, outperforming Codestral-22B by 6% in pass rate with a third of the parameters.","tokens_in":22688,"tokens_out":5997,"duration_ms":58130,"significance":"If the results are taken at face value, the paper makes a useful empirical contribution: it proposes a concrete PBE formulation for a downstream linguistic task, demonstrates that fine-tuning on synthetic data with a middle level of input substance (pronounceable nonce words) helps, and shows that fine-tuning a 6.7B model on 2.5k examples can beat larger zero-shot code LLMs on the constructed benchmark. The controlled comparison of four synthetic data conditions is a strength, and the automatic functional evaluation of generated programs (executing the BasicAction and comparing targeted outputs) provides machine-checked correctness signals. The paper also ships verbatim prompts and fine-tuning details, supporting reproducibility. However, the headline 'SOTA for SLI' claim is broader than what the single-law benchmark can support, and several statistical and experimental-design choices need tightening before the quantitative claims can be considered robust.","major_comments":[{"comment":"The evaluation reduces SLI to single-law induction by explicitly removing the search and ordering component of cascades (Section 5: 'this involves a search and ordering component which is not relevant to our study'). Yet the abstract and Section 7 claim an unqualified 'SOTA open-source model for SLI' and 'strongest open source LLM for SLI'. Since the paper's own introduction defines sound laws as ordered cascades, a high pass rate on isolated rewrites does not demonstrate competence at full cascade induction; the same surface forms can arise from different law orderings, and feeding/bleeding interactions are absent from the benchmark. The manuscript never tests whether single-law performance predicts cascade performance. Please qualify all SLI claims to 'single-law induction' or provide evidence on the cascade task.","section":"Section 5 and Section 7"},{"comment":"The best synthetic data condition (RP-LI) is selected using the same 85-instance benchmark on which the final PySLICoder models are then compared with other LLMs. This selection-on-benchmark means the reported +6% pass-rate advantage over Codestral-22B is not an independent estimate: the condition was chosen because it performed well on those very instances. No held-out split or nested evaluation is used. The paper should either report results on a held-out subset for the final comparison or explicitly acknowledge and correct for the selection when interpreting the SOTA claim.","section":"Section 6.1 and Section 6.2"},{"comment":"The Wilcoxon signed-rank tests treat each of the 20 sampled programs per instance as an independent observation, giving 85×3×20 = 5100 values for reward per program and passing programs. These 20 samples are strongly correlated because they share the same prompt, the same instance, and the same model, and the three runs are random-seed variations rather than independent draws. This violates the i.i.d. assumption of the test and inflates the effective sample size, making the reported p<0.0001 values overconfident. Please aggregate at the instance level (e.g., mean or max over the 20 samples) before running statistical tests, and report cluster-aware or permutation-based significance tests.","section":"Section 6.1, Tables 6-7"},{"comment":"The paper's own tests show no statistically significant difference in pass rate between RP-RI and RP-LI (p>0.05 in both poc and ptk comparisons), yet RP-LI is chosen as the best condition and used for the final model. The significance tests support only reward-per-program and passing-programs differences, not the pass-rate ranking that motivates the condition selection. The claim that RP-LI is 'best' should be stated with this caveat, or the condition selection should be based on a metric that shows a significant difference. As written, the main empirical ranking (RP-LI > RP-PI > RP-RI > IDP-PI) is only partially supported by the reported statistics.","section":"Section 6.1, Tables 6-7"}],"minor_comments":[{"comment":"The equation for reward@m sums rewards over the top-m programs but the prose says 'average reward'; the displayed formula is missing a factor 1/m. The values in Table 4 (where R@3 is usually smaller than R@1) indicate that an average was actually computed, so the equation should be corrected to match the text and the implementation.","section":"Section 5.3"},{"comment":"Several baseline models (DeepSeekCoder-7B-Instruct, Qwen2.5-Coder-7B-Instruct, DeepSeekCoder-v2-16B-Instruct) are evaluated in only a single run, while PySLICoder and Codestral are averaged over three runs. This asymmetric evaluation makes the comparisons against the starred models anecdotal; please either run the baselines multiple times or explicitly state that single-run results are not intended for significance testing.","section":"Section 5.2 and Tables 1-2"},{"comment":"The Limitations section discusses the small benchmark size, potential language-pair bias, and inference budget, but it does not mention that the benchmark covers only single-law induction and excludes cascade ordering. Given that the paper's introduction emphasizes cascades, this scope restriction should be listed as a prominent limitation.","section":"Limitations section"},{"comment":"There are several minor presentation issues: 'the task is of finding' in Section 2 should be 'the task is finding'; 'consecutive' is misspelled in the Appendix B.2 prompt; Table 8's heading 'ID-PI-ptk' is inconsistent with the IDP-PI-ptk notation used elsewhere; and the reward@m notation in Section 5.3 uses both 'reward@m' and 'reward@1 = 1' in a way that could be clarified.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid core: a reproducible PBE setup, a thoughtful synthetic-data framework, and large effect sizes for the main trends. The main obstacles to acceptance are the overbroad SOTA claim relative to the single-law benchmark, the selection-on-the-test-set issue, and the statistical tests that treat correlated samples as independent. These are fixable within the manuscript's scope by reframing claims, adding a held-out split or stronger caveats, and using instance-level statistics. I do not see a fundamental correctness error that would justify rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful core here is the controlled comparison of four synthetic fine-tuning distributions for LLM-based programming-by-examples, not the SOTA headline. The PBE framing of sound law induction is new, and the structure/substance axis is a sensible way to organize the space. The main empirical findings have large effect sizes and are plausible: fine-tuning helps a lot over base Magicoder; LLM-generated pronounceable nonce inputs (RP-LI) beat random strings and also beat realistic Index Diachronica programs (IDP-PI); adding real proto-language vocabulary helps less than expected. I'd trust the qualitative ranking of conditions more than any single number.\n\nThe soft spots are addressable. The benchmark is 85 instances, with 43 from Ptk-Huishu, and the best condition is selected on the same benchmark used for the final comparison, so the +6% pass-rate claim over Codestral is optimistic; an unbiased estimate would need a held-out split or nested selection. The Wilcoxon tests over '5100 values' treat the 20 sampled programs per instance as independent, which they are not, and that inflates the p-values. The baselines are zero-shot, so 'strongest open-source model for SLI' really means 'strongest open-source model we fine-tuned and compared to zero-shot code LLMs.' I also didn't see a release link for the benchmark or fine-tuning data, which matters for a paper whose selling point is a recipe.\n\nThe stress-test note is on target. Section 5 explicitly strips out cascade search and ordering, and the model is only ever asked to induce one law at a time. That is an honest scope restriction, but the abstract and Section 7 say 'SLI' without that qualifier. If the authors reword the claims or add a small cascade evaluation, the overstatement goes away; I don't think it undermines the synthetic-data finding.\n\nWho gets value: historical linguists wanting an open-source drafting assistant for single sound laws, and PBE researchers interested in what 'same distribution' means for synthetic fine-tuning data. It deserves a serious referee, but the revision should tame the claims and fix the statistics.","headline":"A thoughtful synthetic-data recipe for single sound laws with an overbroad 'SOTA for SLI' claim; send it to review but push for scope-tamed wording and honest statistics.","tokens_in":23249,"tokens_out":3405,"would_cite":true,"duration_ms":36271,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the best synthetic fine-tuning distribution for teaching LLMs to induce sound laws pairs random rewrite programs with LLM-generated pronounceable nonce words, and that a 6.7B open model trained this way beats a 22B…","keywords":["sound law induction","programming by examples","large language models","synthetic data generation","historical linguistics","structure vs substance","fine-tuning","phonological change"],"falsifier":"Re-run the experiment on a cascade version of the benchmark where several sound laws must be induced and ordered together, scoring the end-to-end reflex prediction; if the RP-LI fine-tuned models lose their advantage over the zero-shot 22B baseline, then the 'strongest open-source model for SLI' claim applies only to single-law induction, not to sound-law induction as historical linguists practice it.","tokens_in":22220,"feed_emoji":"🔤","tokens_out":9828,"duration_ms":87519,"temperature":0.7,"pith_summary":"Sound-law induction—reconstructing the ordered string-rewrite rules that turn ancestral words into attested descendant words—is usually done by hand. This paper tries to show that a large language model can do it when the task is framed as programming by examples, provided the fine-tuning data is generated in the right way. The authors compare four synthetic data generation schemes along a structure-versus-substance spectrum and report that the winner is a middle course: random rewrite programs applied to LLM-generated 'pronounceable' nonce words, rather than purely random strings or genuinely substantive real-world inputs and real sound laws. On an 85-instance single-law benchmark, that recipe produces PySLICoder, an open 6.7B-parameter model whose 68.4% pass rate beats the 22B Codestral-22B by 6 percentage points while still trailing GPT-4o's 82.1%.","feed_headline":"A 6.7B fine-tuned model outdoes a 22B rival at sound-law induction","feed_subtitle":"Winning recipe: random rewrite programs plus pronounceable nonce words, beating Codestral-22B by 6 points.","key_machinery":"The carrying mechanism is the controlled synthetic-data pipeline that separates the input distribution $P_X$ from the program distribution $P_\\rho$. Sound laws are written as executable Python objects of a class called BasicAction—a list of predicates matching a context window, change positions, and mapping functions that perform insertion, deletion, or substitution—so every generated program can be executed on examples and scored by an edit-distance reward function. By holding dataset size, base model, and evaluation fixed across four conditions, the pipeline isolates which distribution shift (more realistic inputs, more substantive programs) changes performance, and thereby identifies RP-LI as the best condition.","core_discovery":"The paper's central claim is that, for programming-by-examples with LLMs, the best synthetic fine-tuning distribution lies at a specific point on the structure–substance spectrum: inputs should be linguistically plausible but not real vocabulary, and programs should be structurally well-formed but substantively random. Among the four conditions—random programs with random inputs (RP-RI), random programs with LLM-generated nonce inputs (RP-LI), random programs with Proto-Oceanic or Proto-Tangkhulic words (RP-PI), and real sound laws from Index Diachronica applied to protolanguage words (IDP-PI)—RP-LI gives the highest average pass rate (59.1%), with RP-PI close behind and IDP-PI worst. The authors attribute the failure of the most substantive condition to reduced program diversity: uniform real-world rules teach narrow biases, whereas random programs force the model to learn general rule structure. Using RP-LI data generated by GPT-4o to fine-tune Magicoder produces PySLICoder-RP-LI-gpt-4o, which the paper reports as the strongest open-source LLM for sound-law induction, with a 6% higher pass rate than Codestral-22B (p<0.01), higher reward per program and more passing programs (p<0.0001), at roughly a third of Codestral's parameters.","pith_inferences":["A testable extension the paper does not pursue: generate nonce inputs with phonotactics matched to the target language family (Polynesian-sounding nonce words for Polynesian sound laws) rather than the English-like wug words used in RP-LI; the structure–substance account predicts this would push the RP-LI condition higher still.","If the 'plausible inputs, diverse programs' recipe transfers, other PBE domains such as data wrangling could see the same pattern: real-looking but not identical examples plus broad program diversity may beat both purely random and fully curated training data, something the paper lists only as future work.","The reported 6% advantage is measured on isolated single laws; on an end-to-end cascade benchmark where laws must be ordered, the ranking could change because the model was never trained to order laws, so the 'strongest open-source model for SLI' label should carry that scope qualification.","Scaling beyond 2.5k synthetic examples is the paper's own next step; since both PySLICoder variants trail the models that generated their training data, a larger RP-LI corpus fine-tuned on the same base model could plausibly approach GPT-4o's 82.1% pass rate."],"forward_implications":["Fine-tuning a 6.7B open code LLM on 2.5k synthetic examples can outperform a 22B zero-shot code LLM on single-law sound-law induction, so parameter size is not the main driver of PBE performance.","For creating PBE training data, inputs should be plausible but not identical to evaluation inputs: pronounceable nonce words beat both random strings and real protolanguage words.","Program diversity matters more than program realism: sampling real sound laws from a database hurt performance, so synthetic program generators should be kept broad rather than constrained to attested rules.","The resulting PySLICoder models give historical linguists a low-cost tool for automated forward reconstruction: open weights, 6.7B parameters, and state-of-the-art open-source pass rates on the single-law benchmark.","The structure-versus-substance spectrum is a reusable recipe for other PBE domains: it tells a practitioner where to look when deciding what synthetic distribution to fine-tune on."],"supporting_citations":[{"why":"Defines Programming by Examples, the task formulation the paper adapts to sound-law induction.","marker":"(Gulwani, 2016)"},{"why":"Shows LLMs struggle at PBE and improve when fine-tuned on data from the same distribution; this motivates the synthetic-data comparison.","marker":"(Li and Ellis, 2024)"},{"why":"Corroborates that LLMs are not effective at example-based code generation by default, supporting the need for fine-tuning.","marker":"(Fu et al., 2024)"},{"why":"Frames the structure versus substance distinction that organizes the four synthetic data conditions.","marker":"(Boye and Engberg-Pedersen, 2015)"},{"why":"Provides the real sound laws used in the IDP-PI condition, the most substantive program distribution.","marker":"(Index Diachronica Contributors, 2016)"},{"why":"Supplies the DiaSim computerized forward reconstruction system used to build the gold cascades that become the single-law evaluation benchmark.","marker":"(Marr and Mortensen, 2020)"},{"why":"Gives the edit-distance reward function used to score generated programs and defines an earlier MCTS approach to sound-law induction.","marker":"(Luo, 2021)"},{"why":"Describes Magicoder, the open 6.7B code LLM that is fine-tuned in every condition and used as the base for PySLICoder.","marker":"(Wei et al., 2024)"},{"why":"Provides GPT-4o, the model that generates the RP-LI training data for the best PySLICoder variant.","marker":"(Hurst et al., 2024)"}],"fun_headline_variants":["Random programs + fake words beats real data for sound laws","Small LLM beats big rival on sound laws via nonce training","Nonce words and random rules: the winning LLM recipe","How a 6.7B model outdoes 22B at sound-law induction","Sound-law induction: synthetic nonce data trumps real vocab"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a benchmark of isolated single sound laws, with the cascade search-and-ordering step deliberately removed, is a valid proxy for real sound-law induction capability; if composing and ordering laws is what makes the task hard, the headline results measure only the easier subproblem.","fun_headline_variants_meta":{"raw":{"variants":["Random programs + fake words beats real data for sound laws","Small LLM beats big rival on sound laws via nonce training","Nonce words and random rules: the winning LLM recipe","How a 6.7B model outdoes 22B at sound-law induction","Sound-law induction: synthetic nonce data trumps real vocab"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000257,"raw_usage":{"total_tokens":1616,"prompt_tokens":1023,"completion_tokens":593,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":639,"completion_tokens_details":{"reasoning_tokens":502}},"tokens_in":639,"tokens_out":593,"duration_ms":6659,"temperature":1.0,"reasoning_tokens":502,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T12:37:33.924387+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the experiment on a cascade version of the benchmark where several sound laws must be induced and ordered together, scoring the end-to-end reflex prediction; if the RP-LI fine-tuned models lose their advantage over the zero-shot 22B baseline, then the 'strongest open-source model for SLI' claim applies only to single-law induction, not to sound-law induction as historical linguists practice it.","supporting_citations":[],"review_version":1}