Pith. sign in

REVIEW 4 major objections 4 minor 2 references

Programming by Examples Meets Historical Linguistics: A Large Language Model Based Approach to Sound Law Induction

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that the best synthetic fine-tuning distribution for teaching LLMs to induce sound laws pairs random rewrite programs with LLM-generated pronounceable nonce words, and that a 6.7B open model trained this way beats a 22B…

desk verdict A thoughtful synthetic-data recipe for single sound laws with an overbroad 'SOTA for SLI' claim; send it to review but push for scope-tamed wording and honest statistics. read the letter →

arxiv 2501.16524 v1 pith:5LV6H4UT submitted 2025-01-27 cs.CL

Atharva Naik , Darsh Agrawal , Hong Sng , Clayton Marr , Kexun Zhang , Nathaniel R Robinson , Kalvin Chang , Rebecca Byrnes
show 3 more authors
Aravind Mysore Carolyn Rose David R Mortensen
This is my paper · ORCID
classification cs.CL
keywords soundlawinductionprogrammingbyexampleslargelanguagemodelssyntheticdatagenerationhistoricallinguisticsstructurevssubstancefine-tuningphonologicalchange
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sound-law induction—reconstructing the ordered string-rewrite rules that turn ancestral words into attested descendant words—is usually done by hand. This paper tries to show that a large language model can do it when the task is framed as programming by examples, provided the fine-tuning data is generated in the right way. The authors compare four synthetic data generation schemes along a structure-versus-substance spectrum and report that the winner is a middle course: random rewrite programs applied to LLM-generated 'pronounceable' nonce words, rather than purely random strings or genuinely substantive real-world inputs and real sound laws. On an 85-instance single-law benchmark, that recipe produces PySLICoder, an open 6.7B-parameter model whose 68.4% pass rate beats the 22B Codestral-22B by 6 percentage points while still trailing GPT-4o's 82.1%.

What carries the argument

The carrying mechanism is the controlled synthetic-data pipeline that separates the input distribution $P_X$ from the program distribution $P_\rho$. Sound laws are written as executable Python objects of a class called BasicAction—a list of predicates matching a context window, change positions, and mapping functions that perform insertion, deletion, or substitution—so every generated program can be executed on examples and scored by an edit-distance reward function. By holding dataset size, base model, and evaluation fixed across four conditions, the pipeline isolates which distribution shift (more realistic inputs, more substantive programs) changes performance, and thereby identifies RP-LI as the best condition.

What would settle it

Re-run the experiment on a cascade version of the benchmark where several sound laws must be induced and ordered together, scoring the end-to-end reflex prediction; if the RP-LI fine-tuned models lose their advantage over the zero-shot 22B baseline, then the 'strongest open-source model for SLI' claim applies only to single-law induction, not to sound-law induction as historical linguists practice it.

Watch

Extended reading notes

Core claim

The paper's central claim is that, for programming-by-examples with LLMs, the best synthetic fine-tuning distribution lies at a specific point on the structure–substance spectrum: inputs should be linguistically plausible but not real vocabulary, and programs should be structurally well-formed but substantively random. Among the four conditions—random programs with random inputs (RP-RI), random programs with LLM-generated nonce inputs (RP-LI), random programs with Proto-Oceanic or Proto-Tangkhulic words (RP-PI), and real sound laws from Index Diachronica applied to protolanguage words (IDP-PI)—RP-LI gives the highest average pass rate (59.1%), with RP-PI close behind and IDP-PI worst. The authors attribute the failure of the most substantive condition to reduced program diversity: uniform real-world rules teach narrow biases, whereas random programs force the model to learn general rule structure. Using RP-LI data generated by GPT-4o to fine-tune Magicoder produces PySLICoder-RP-LI-gpt-4o, which the paper reports as the strongest open-source LLM for sound-law induction, with a 6% higher pass rate than Codestral-22B (p<0.01), higher reward per program and more passing programs (p<0.0001), at roughly a third of Codestral's parameters.

Load-bearing premise

The load-bearing premise is that a benchmark of isolated single sound laws, with the cascade search-and-ordering step deliberately removed, is a valid proxy for real sound-law induction capability; if composing and ordering laws is what makes the task hard, the headline results measure only the easier subproblem.

Editorial extensions

If this is right

  • Fine-tuning a 6.7B open code LLM on 2.5k synthetic examples can outperform a 22B zero-shot code LLM on single-law sound-law induction, so parameter size is not the main driver of PBE performance.
  • For creating PBE training data, inputs should be plausible but not identical to evaluation inputs: pronounceable nonce words beat both random strings and real protolanguage words.
  • Program diversity matters more than program realism: sampling real sound laws from a database hurt performance, so synthetic program generators should be kept broad rather than constrained to attested rules.
  • The resulting PySLICoder models give historical linguists a low-cost tool for automated forward reconstruction: open weights, 6.7B parameters, and state-of-the-art open-source pass rates on the single-law benchmark.
  • The structure-versus-substance spectrum is a reusable recipe for other PBE domains: it tells a practitioner where to look when deciding what synthetic distribution to fine-tune on.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not pursue: generate nonce inputs with phonotactics matched to the target language family (Polynesian-sounding nonce words for Polynesian sound laws) rather than the English-like wug words used in RP-LI; the structure–substance account predicts this would push the RP-LI condition higher still.
  • If the 'plausible inputs, diverse programs' recipe transfers, other PBE domains such as data wrangling could see the same pattern: real-looking but not identical examples plus broad program diversity may beat both purely random and fully curated training data, something the paper lists only as future work.
  • The reported 6% advantage is measured on isolated single laws; on an end-to-end cascade benchmark where laws must be ordered, the ranking could change because the model was never trained to order laws, so the 'strongest open-source model for SLI' label should carry that scope qualification.
  • Scaling beyond 2.5k synthetic examples is the paper's own next step; since both PySLICoder variants trail the models that generated their training data, a larger RP-LI corpus fine-tuned on the same base model could plausibly approach GPT-4o's 82.1% pass rate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper frames sound law induction (SLI) as programming by examples (PBE) for LLMs: given protoform/reflex pairs, generate a Python program (a BasicAction) implementing the single sound law that maps inputs to outputs. The authors introduce a structure-versus-substance framework for synthetic fine-tuning data, propose four conditions (RP-RI, RP-LI, RP-PI, IDP-PI) with increasing substantive bias, fine-tune Magicoder-6.7B on 2.5k synthetic instances from each condition, and evaluate on an 85-instance single-law benchmark built from five language pairs. They report that RP-LI (random programs plus LLM-generated pronounceable nonce inputs) yields the highest pass rate, train two PySLICoder variants on RP-LI data generated by Codestral-22B and GPT-4o, and claim that PySLICoder-RP-LI-gpt-4o is the strongest open-source LLM for SLI, outperforming Codestral-22B by 6% in pass rate with a third of the parameters.

Significance. If the results are taken at face value, the paper makes a useful empirical contribution: it proposes a concrete PBE formulation for a downstream linguistic task, demonstrates that fine-tuning on synthetic data with a middle level of input substance (pronounceable nonce words) helps, and shows that fine-tuning a 6.7B model on 2.5k examples can beat larger zero-shot code LLMs on the constructed benchmark. The controlled comparison of four synthetic data conditions is a strength, and the automatic functional evaluation of generated programs (executing the BasicAction and comparing targeted outputs) provides machine-checked correctness signals. The paper also ships verbatim prompts and fine-tuning details, supporting reproducibility. However, the headline 'SOTA for SLI' claim is broader than what the single-law benchmark can support, and several statistical and experimental-design choices need tightening before the quantitative claims can be considered robust.

major comments (4)
  1. [Section 5 and Section 7] The evaluation reduces SLI to single-law induction by explicitly removing the search and ordering component of cascades (Section 5: 'this involves a search and ordering component which is not relevant to our study'). Yet the abstract and Section 7 claim an unqualified 'SOTA open-source model for SLI' and 'strongest open source LLM for SLI'. Since the paper's own introduction defines sound laws as ordered cascades, a high pass rate on isolated rewrites does not demonstrate competence at full cascade induction; the same surface forms can arise from different law orderings, and feeding/bleeding interactions are absent from the benchmark. The manuscript never tests whether single-law performance predicts cascade performance. Please qualify all SLI claims to 'single-law induction' or provide evidence on the cascade task.
  2. [Section 6.1 and Section 6.2] The best synthetic data condition (RP-LI) is selected using the same 85-instance benchmark on which the final PySLICoder models are then compared with other LLMs. This selection-on-benchmark means the reported +6% pass-rate advantage over Codestral-22B is not an independent estimate: the condition was chosen because it performed well on those very instances. No held-out split or nested evaluation is used. The paper should either report results on a held-out subset for the final comparison or explicitly acknowledge and correct for the selection when interpreting the SOTA claim.
  3. [Section 6.1, Tables 6-7] The Wilcoxon signed-rank tests treat each of the 20 sampled programs per instance as an independent observation, giving 85×3×20 = 5100 values for reward per program and passing programs. These 20 samples are strongly correlated because they share the same prompt, the same instance, and the same model, and the three runs are random-seed variations rather than independent draws. This violates the i.i.d. assumption of the test and inflates the effective sample size, making the reported p<0.0001 values overconfident. Please aggregate at the instance level (e.g., mean or max over the 20 samples) before running statistical tests, and report cluster-aware or permutation-based significance tests.
  4. [Section 6.1, Tables 6-7] The paper's own tests show no statistically significant difference in pass rate between RP-RI and RP-LI (p>0.05 in both poc and ptk comparisons), yet RP-LI is chosen as the best condition and used for the final model. The significance tests support only reward-per-program and passing-programs differences, not the pass-rate ranking that motivates the condition selection. The claim that RP-LI is 'best' should be stated with this caveat, or the condition selection should be based on a metric that shows a significant difference. As written, the main empirical ranking (RP-LI > RP-PI > RP-RI > IDP-PI) is only partially supported by the reported statistics.
minor comments (4)
  1. [Section 5.3] The equation for reward@m sums rewards over the top-m programs but the prose says 'average reward'; the displayed formula is missing a factor 1/m. The values in Table 4 (where R@3 is usually smaller than R@1) indicate that an average was actually computed, so the equation should be corrected to match the text and the implementation.
  2. [Section 5.2 and Tables 1-2] Several baseline models (DeepSeekCoder-7B-Instruct, Qwen2.5-Coder-7B-Instruct, DeepSeekCoder-v2-16B-Instruct) are evaluated in only a single run, while PySLICoder and Codestral are averaged over three runs. This asymmetric evaluation makes the comparisons against the starred models anecdotal; please either run the baselines multiple times or explicitly state that single-run results are not intended for significance testing.
  3. [Limitations section] The Limitations section discusses the small benchmark size, potential language-pair bias, and inference budget, but it does not mention that the benchmark covers only single-law induction and excludes cascade ordering. Given that the paper's introduction emphasizes cascades, this scope restriction should be listed as a prominent limitation.
  4. [Throughout] There are several minor presentation issues: 'the task is of finding' in Section 2 should be 'the task is finding'; 'consecutive' is misspelled in the Appendix B.2 prompt; Table 8's heading 'ID-PI-ptk' is inconsistent with the IDP-PI-ptk notation used elsewhere; and the reward@m notation in Section 5.3 uses both 'reward@m' and 'reward@1 = 1' in a way that could be clarified.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the SOTA claim is an empirically measured pass-rate result on a manually constructed benchmark, not a quantity derived from the training distribution.

full rationale

The paper's derivation chain is empirical rather than definitional. The synthetic-data conditions (RP-RI, RP-LI, RP-PI, IDP-PI) are generated by distinct procedures, fine-tuned on equal-sized data, and then evaluated on a separately constructed 'single law' benchmark derived from real Polynesian and Tangkhulic sound changes (Section 5.1). The claim that RP-LI is the best distribution (Table 1) and that PySLICoder-RP-LI-gpt-4o is the strongest open-source model (Table 2, Section 7) is a measurement against gold programs, not a consequence of how the training data were generated. The reward metric is adapted from Luo (2021) with no per-instance fitted parameters; it is an external evaluation metric. The only self-citations (DiaSim, PanPhon, Chang et al. 2023) are tools or related work, not load-bearing uniqueness theorems; the gold cascades were 'manually deduced' with DiaSim 'assisted' only, and PanPhon is a tokenizer. The explicit removal of cascade search/ordering in Section 5 is a scope restriction for the benchmark, not a circular step: the paper's stated claims are about the single-law PBE version, though the title/abstract's unqualified 'SLI' wording may overstate scope. No equation in the paper reduces to its own input, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper's claims rest on standard ML assumptions plus several domain-specific modeling choices, most notably the expressiveness of the BasicAction representation and the correctness of the manually built gold cascades. No physical or mathematical free parameters are fitted; the empirical comparisons are the evidence.

assumptions (4)
  • domain assumption Sound change is regular and can be modeled by ordered string rewrite rules (Neogrammarian hypothesis).
    Invoked in Section 1. If sound laws are not rule-based, the PBE formulation is invalid.
  • domain assumption BasicAction, with predicates, change_pos, and mapping_fn, is expressive enough to represent the benchmark sound laws.
    Introduced in Section 4.1 and Appendix B.1. If some sound laws fall outside this representation, the benchmark and model are incomplete.
  • ad hoc to paper The manually deduced relative chronologies and gold cascades (end-to-end accuracy >= 75%) are correct ground truth for single-law training pairs.
    Section 5.1. The benchmark is built from these manual solutions with no inter-annotator check, so any error propagates into evaluation.
  • ad hoc to paper The single-law task is a valid proxy for full sound law induction.
    Section 5. The paper removes cascade search and ordering, which are core to the real-world SLI problem.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Programming by Examples Meets Historical Linguistics: A Large Language Model Based Approach to Sound Law Induction." pith.science (2026). https://pith.science/paper/5LV6H4UT

@misc{pith2026250116524,
  author       = {Pith},
  title        = {Pith review of: Programming by Examples Meets Historical Linguistics: A Large Language Model Based Approach to Sound Law Induction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5LV6H4UT}},
  note         = {Machine review of arXiv:2501.16524}
}
read the original abstract

Historical linguists have long written "programs" that convert reconstructed words in an ancestor language into their attested descendants via ordered string rewrite functions (called sound laws) However, writing these programs is time-consuming, motivating the development of automated Sound Law Induction (SLI) which we formulate as Programming by Examples (PBE) with Large Language Models (LLMs) in this paper. While LLMs have been effective for code generation, recent work has shown that PBE is challenging but improvable by fine-tuning, especially with training data drawn from the same distribution as evaluation data. In this paper, we create a conceptual framework of what constitutes a "similar distribution" for SLI and propose four kinds of synthetic data generation methods with varying amounts of inductive bias to investigate what leads to the best performance. Based on the results we create a SOTA open-source model for SLI as PBE (+6% pass rate with a third of the parameters of the second-best LLM) and also highlight exciting future directions for PBE research.

Figures

Figures reproduced from arXiv: 2501.16524 by the authors.

Figure 1
Figure 1. Prompt used for synthesizing single sound laws from the protoforms (inputs) and attested forms (outputs). [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The four fine-tuning conditions introduced in [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Prompt used for synthesizing single sound laws from the protoforms (inputs) and attested forms (outputs). [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Instantiation of the BasicAction class to represent a sound law. This example shows a rule where “a” goes to “e” when it occurs before a “j”. The predicates match to an environment of “a@j” where ‘@’ is the separator, and then the first character of the environment or …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 2 linked inside Pith

  1. [2]

    a”goes to“e

    Optimalneuralprogramsynthesisfrommulti- modalspecifications. In Findings of the Association for Computational Linguistics: EMNLP 2021,pages 1691–1704. Shijia Zhou, Leonie Weissweiler, Taiqi He, Hinrich Schütze,DavidR.Mortensen,andLoriLevin.2024. Constructions are so difficult that Even large lan- guage models get them right for the wrong reasons. In Proce...

    arXiv 2004

  2. [2021]

    arXiv preprint arXiv:2108.07732

    Program synthesis with large language mod- els. arXiv preprint arXiv:2108.07732. Leonard Bloomfield. 1926.A set of postulates for the scienceoflanguage . Language,2(3):153–164. Kasper Boye and Elisabeth Engberg-Pedersen. 2015. Substance and structure in linguistics. InWorkshop at the University of Copenhagen. Karl Brugmann and Hermann Osthoff. 1878. Mor- ...

    arXiv 1926

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.