Transfer of Structural Knowledge from Synthetic Languages

Ivan Yamshchikov; Mikhail Budnikov

REVIEW 3 major objections 6 minor 31 references

Transfer of Structural Knowledge from Synthetic Languages

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pretraining a small language model on flat_shuffle, a synthetic language that blockwise shuffles bracket types, transfers to English better than the balanced-bracket languages used in earlier work.

desk verdict A modest new synthetic language and benchmark, but the headline transfer advantage rests on single runs and one table contradicts it. read the letter →

arxiv 2505.15769 v1 pith:YZ7VKC4B submitted 2025-05-21 cs.CL cs.LG

Mikhail Budnikov , Ivan Yamshchikov This is my paper · ORCID

classification cs.CLcs.LG

keywords flat_shufflesyntheticlanguagestransferlearninglanguagemodelpretrainingfine-tuningperplexityembeddingstructureTiny-ClozeBenchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the choice of synthetic pre-training language changes how well a small language model transfers to English. It compares three synthetic languages—nested, a balanced-bracket language; flat, a variant that allows crossing brackets; and the newly introduced flat_shuffle, which adds blockwise permutations of bracket types—and reports that flat_shuffle consistently produces the lowest fine-tuning perplexity on English and, in the fully fine-tuned variant, the highest average score on a new cloze benchmark. To explain the ordering, the paper proposes measuring language complexity and similarity through asymmetric fine-tuning difficulty: when a model pretrained on A adapts to B more easily than a model pretrained on B adapts to A, A is treated as the more complex language. Embedding analyses suggest that the advantage of flat_shuffle lies not in transferring bracket structure to English tokens but in producing a richer embedding geometry that fine-tuning can reuse. If the claim is right, synthetic data can be deliberately engineered for data-efficient language learning.

What carries the argument

The machinery is the synthetic language itself plus the transfer-complexity protocol. flat_shuffle is built from flat (the shuffle-Dyck language, where brackets can cross) by partitioning bracket-type IDs into contiguous blocks of eight and requiring each 16-token segment to use exactly one block as a permutation; this gives the model a locally certain last-token prediction while still demanding block tracking. The comparison is carried by a transfer-based measure: for two languages A and B, pre-train on A and fine-tune to B with (E) embeddings only, (EL) plus LayerNorm parameters, or (ELT) plus the last transformer layer, then use the fine-tuning perplexity as the difficulty of transfer. Half the sum of the two directional difficulties is read as dissimilarity, and half the difference as relative complexity. Supporting analyses use the singular-value spectrum of the embedding matrix and k-means clustering to compare effective dimensionality, linear probes for part-of-speech, frequency, and whitespace features, and a GPT-4-generated Tiny-Cloze Benchmark with twelve subtasks.

What would settle it

Generate an ablation of flat_shuffle that removes the block constraint—same vocabulary, same length, same sampling except bracket types are drawn uniformly from all open brackets at every position—and pre-train the same 8M model on it under the same token budget; then fine-tune to English at the three frozen-weight levels. If the blockwise-permutation variant does not consistently beat the ablated version by more than 0.2 nats at the embedding-only stage across a few seeds, the structural mechanism proposed for flat_shuffle's advantage is not responsible for it.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a synthetic language can be engineered to transfer better to English by combining two previously separate ideas: balanced-bracket structure and shuffle patterns. The new language, flat_shuffle, splits bracket types into contiguous blocks of eight, and every 16-token segment is a permutation drawn from one such block; this makes the next-token prediction task locally predictable at the end of each segment while requiring the model to track which block is active. Across three frozen-weight fine-tuning regimes—embeddings only, plus LayerNorms, plus the last transformer layer—models pretrained on flat_shuffle reach lower perplexity on English than models pretrained on flat or nested, and the same ordering appears in the reverse direction, which the paper reads as flat_shuffle being more complex than flat and flat more complex than nested. The paper also finds that flat_shuffle embeddings have a singular-value spectrum and clustering behavior closer to models trained on English from scratch, while linear probes find no direct encoding of bracket open/close structure in English tokens after fine-tuning. The authors' proposed explanation is that pretraining on a more complex synthetic language forces the intermediate layers to handle richer embedding structure, which fine-tuning can reuse as extra computational flexibility.

Load-bearing premise

The load-bearing premise is that fine-tuning perplexity with frozen weight subsets measures genuine transfer difficulty, and that a 0.2-nat gap is a meaningful threshold for 'close performance'; if perplexity gaps do not track language complexity or similarity, the ordering nested < flat < flat_shuffle and the explanation for flat_shuffle's advantage lose their foundation.

Editorial extensions

If this is right

Pretraining on flat_shuffle yields lower English fine-tuning perplexity than nested or flat at every frozen-weight level, and the reverse fine-tuning direction is harder, placing English above all three synthetic languages in the proposed complexity ordering.
Tiny-Cloze Benchmark provides a cheap 12-subtask diagnostic that separates 8M-parameter models on language understanding where GLUE and MMLU are too difficult.
The same transfer-based complexity measure could replace coarse grammar-class labels with a quantitative, direction-aware ordering of languages within the same Chomsky class.
Embedding spectra and clustering predict that more complex synthetic pretraining yields a higher effective embedding dimension that remains usable after fine-tuning.
Because no literal bracket open/close structure transfers to English, synthetic languages can be designed to exercise general computational patterns rather than to mimic the target language's surface grammar.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The frozen-weight perplexity measure is only tested with English as the target; applying the same protocol to pairs of natural languages with different word orders would show whether the complexity ordering is language-general or English-specific.
Block size and segment length in flat_shuffle are chosen as eight and sixteen; varying these parameters could reveal whether transfer quality scales with the amount of local permutation structure, a test the paper does not run.
The reservoir-computing reading suggests synthetic pretraining may be a way to install reusable computation rather than knowledge; one test would be to measure performance on unrelated algorithmic probes before and after fine-tuning.
The Tiny-Cloze items are generated by GPT-4 and are only ten questions per subtask, so the benchmark's reliability as a standard diagnostic would need human verification and larger samples.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Thanks for the note. This paper is a modest but real contribution to the synthetic-pretraining literature. It introduces flat_shuffle, a block-wise shuffled variant of the flat Dyck language, and a small 120-question Tiny-Cloze benchmark. The experimental framework follows Papadimitriou and Jurafsky, with three fine-tuning levels and scratch baselines, and the authors share code and data.

The central claim, that pre-training on flat_shuffle transfers better to English than nested or flat, is supported by Table 1 at all three fine-tuning levels (2.4 vs 2.8/2.7, etc.). But the support is thinner than it looks. Each cell is a single run: no seeds, no error bars, no significance tests. For an 8M-parameter Transformer fine-tuned for 12,500 steps, run-to-run variation of 0.3–0.4 nats is entirely plausible. There are no convergence curves either, so the ranking could reflect optimization speed rather than final quality.

The paper also trips over its own Table 2. At the embedding-only level, the average Tiny-Cloze score for flat_shuffle is 0.14, which is lower than nested (0.15) and flat (0.16). The text says 'the familiar pattern nested < flat < flat_shuffle < scratch' appears again, but that pattern only shows up at the ELT level, and even there flat_shuffle beats flat by just 0.01 nats. That's an internal inconsistency that any careful reader will catch.

The Section 4 transfer-complexity measure is a good idea, but it is never computed. The paper defines f(A,B) as fine-tuning perplexity and suggests an asymmetry formula, then moves on without giving numbers. As written, it is a proposal, not a result.

What does hold up: the flat_shuffle construction is a reasonable incremental step, the Tiny-Cloze benchmark fills a small gap, and the probing/SVD analysis is honest exploration even if not deep. The limitations section is candid.

I would send this to peer review with a request for major revision: add repeated seeds and variance estimates, correct the Table 2 claim, and either actually compute the complexity measure or soften the claim. The core idea is worth engaging with, but the evidence as presented is too fragile to accept as solid.

Referee Report

3 major / 6 minor

Summary. The paper investigates whether pre-training an 8M-parameter Transformer on synthetic "languages" improves transfer to English. It introduces a new synthetic language, flat_shuffle, a block-wise shuffled variant of the flat Dyck language, and compares it with the nested and flat bracket languages under three fine-tuning schemes (embeddings only, embeddings plus LayerNorm, and embeddings plus LayerNorm plus last transformer block). The authors report transfer perplexities in both directions (Table 1), evaluate fine-tuned models on a new GPT-4-generated Tiny-Cloze Benchmark (Table 2), analyze embedding geometry via singular value spectra and k-means clustering, and train linear probes for word-level features. The central claim is that pre-training on flat_shuffle transfers better to English than pre-training on nested or flat, and that this is explained by the greater complexity of flat_shuffle.

Significance. If established, the central claim would provide evidence that particular synthetic pre-training structures can serve as useful inductive biases for downstream natural-language learning, with potential implications for data-efficient training. The paper has clear strengths: the code and data are publicly released, the experimental setup follows prior work closely, and the evaluation combines several complementary methods (transfer perplexity, a small cloze benchmark, embedding spectra, clustering, and linear probes). The Tiny-Cloze Benchmark is a potentially useful resource for small models. However, the headline empirical claim currently rests on single-run numbers without variance estimates or significance tests, and one of the reported benchmark tables is internally inconsistent with the claimed ordering. These issues are load-bearing for the paper's main conclusions.

major comments (3)

[§5.1, Table 1] The central claim that flat_shuffle transfers better to English than nested and flat is supported only by what appear to be single runs: the paper reports no random seeds, standard deviations, or significance tests. The reported gaps are 0.3–0.4 nats (e.g., 2.4 vs 2.8 for flat_shuffle→English vs nested→English at level E), which is within plausible run-to-run variation for an 8M-parameter Transformer. In addition, fine-tuning is always stopped after a fixed 12,500 steps and no learning curves are shown, so the ranking could reflect optimization speed rather than final transfer quality. Please provide repeated runs with variance estimates and either convergence-based stopping or evidence that 12,500 steps is sufficient for all compared settings.
[§5.2, Table 2] The statement that Table 2 "again shows the familiar pattern nested < flat < flat_shuffle < scratch" is not supported by the embedding-only rows: the average Tiny-Cloze scores are nested E 0.15, flat E 0.16, and flat_shuffle E 0.14, so flat_shuffle is worst among the synthetic languages at this level. The ordering holds only for the ELT rows, where flat_shuffle (0.23) exceeds flat (0.22) by 0.01 nats. Since the paper claims superiority at all fine-tuning levels, this internal inconsistency must be resolved, either by rerunning with multiple seeds or by narrowing the claim to the settings where the ordering is actually observed.
[§4 and §5.1, Table 1] The proposed complexity measure is not validated, and its application to Table 1 appears to contradict the paper's stated logic. Under the reasoning that transferring from a more complex source language to a simpler target should be easier, row 2 shows flat_shuffle→flat with perplexity 3.8 (L1E) while flat→flat_shuffle has 2.5 (L2E), which is opposite to the predicted direction if flat_shuffle is more complex than flat. The 0.2-nats threshold for "close performance" is introduced without justification, and no control experiments establish that fine-tuning perplexity tracks complexity rather than, for example, token distribution overlap or optimization dynamics. This weakens the theoretical framing in Section 4 and the complexity-based explanation of flat_shuffle's advantage.

minor comments (6)

[Table 3] The header "flat_shufflescratch" is missing a space and should read "flat_shuffle scratch".
[§3.3] The example is described as having "purple and green parts," but the text contains no colors; please specify the blocks by token ranges instead.
[Table 1] The caption should define explicitly what L2E and L1E denote and clarify the direction of each transfer; the current notation is hard to parse even after reading the text.
[§5.1] The 0.2-nats threshold is used as a criterion for "close performance" but is not derived or motivated; if it is a free parameter, its role in the conclusions should be stated.
[§4] The phrase "flat is context-dependent" should be "context-sensitive" if that is the intended Chomsky hierarchy class.
[§5.2 and Appendix] The subtask is called "single - plural" in Table 2 but appears as "single plural" in the Appendix; please use consistent naming.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the flat_shuffle transfer claim is evaluated directly against English and scratch baselines, and self-citations are not load-bearing.

full rationale

The paper's central claim—that pre-training on the new flat_shuffle language transfers better to English than nested or flat—is an empirical result read directly from Table 1 (English rows) and Table 2 (Tiny-Cloze averages), compared with scratch baselines. It does not depend on a fitted parameter later renamed as a prediction, nor on the authors' own prior results. The only self-citations (Budnikov et al., 2024; Surkov and Yamshchikov, 2024) are literature pointers in the introduction and are not load-bearing for the transfer experiments. The complexity measure in Section 4 is explicitly operationalized as fine-tuning perplexity asymmetry; calling flat_shuffle 'more complex' on the basis of Table 1 is a stipulated definition applied to data, not a quantity secretly defined by the same English-transfer outcome it is used to explain. The Tiny-Cloze benchmark is evaluated independently by comparing log-likelihood differences against scratch models. Statistical weaknesses (single runs, no error bars, the 0.2 nats threshold) are reproducibility and correctness risks, not circularity. No step in the paper's derivation chain reduces to its own inputs.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper introduces one new synthetic language and one benchmark, but these are datasets rather than explanatory entities. The main assumptions are experimental: the validity of perplexity-based transfer difficulty, the representativeness of TinyStories, and the adequacy of the GPT-4-generated Tiny-Cloze benchmark.

free parameters (1)

close_performance_threshold = 0.2 nats per token
Hand-chosen threshold in Section 5.1 used to declare two models 'close in performance'; the complexity ordering inferred from Table 1 depends on this heuristic.

assumptions (4)

domain assumption Negative log-likelihood after fine-tuning with frozen weights is a valid operationalization of transfer difficulty.
Section 4: 'our proposed way to operationalize this notion of difficulty is to just use perplexity of the model pre-trained on language A and then fine-tuned to language B, with some of the weights frozen.' No external validation of this measure is provided.
domain assumption TinyStories English is representative of English for the transfer conclusions.
All transfer targets use TinyStories; conclusions about transfer to 'English' are based on this single corpus (Section 5.1).
domain assumption The Tiny-Cloze benchmark, generated by GPT-4, measures natural language understanding for small models.
Section 5.2 introduces the benchmark without human validation or comparison against existing small-model NLU benchmarks.
domain assumption The 8M-parameter TinyStories model and the fixed 12,500 fine-tuning steps are sufficient to reveal transfer differences.
Section 5.1 fixes these experimental choices without ablations on model size or step count.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Transfer of Structural Knowledge from Synthetic Languages." pith.science (2026). https://pith.science/paper/YZ7VKC4B

@misc{pith2026250515769,
  author       = {Pith},
  title        = {Pith review of: Transfer of Structural Knowledge from Synthetic Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YZ7VKC4B}},
  note         = {Machine review of arXiv:2505.15769}
}

read the original abstract

This work explores transfer learning from several synthetic languages to English. We investigate the structure of the embeddings in the fine-tuned models, the information they contain, and the capabilities of the fine-tuned models on simple linguistic tasks. We also introduce a new synthetic language that leads to better transfer to English than the languages used in previous research. Finally, we introduce Tiny-Cloze Benchmark - a new synthetic benchmark for natural language understanding that is more informative for less powerful models. We use Tiny-Cloze Benchmark to evaluate fine-tuned models in several domains demonstrating that fine-tuning on a new synthetic language allows for better performance on a variety of tasks.

Figures

Figures reproduced from arXiv: 2505.15769 by the authors.

**Figure 1.** Perplexity on various types of input (Papadimitriou and Jurafsky, 2020). This work attempts to build up on those observations and make a small further step studying the mechanisms of transfer learning. In this paper, we address the following research questions: • How do different synthetic pre-training datasets influence the complexity and transfer performance of language models on English tasks? • What structural… view at source ↗

**Figure 3.** Clustering of bracket embeddings the impact of embedding space structure. Now that we know something about the structure of the embedding space, a natural question to ask is how this structure is used. In other words, what information about a word can one extract from the embedding of the corresponding token? Preliminary experiments showed that clusters of features correspond to properties like "noun", "3rd person v… view at source ↗

**Figure 2.** Spectrum of bracket embeddings The scratch provides a reference for the embeddings on English. We can clearly see that flat_shuffle embeddings are characteristically different from flat and nested embeddings both in terms of the spectrum and in terms of the clusters they form. 5.4 Linear Probes for Word Features We train all probes on embeddings obtained from embedding-only (E) fine-tuned models to isolate [PITH_F… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 8 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2019. On the cross-lingual transferability of monolingual representations. arXiv preprint arXiv:1910.11856

arXiv 2019
[4]

Mikhail Budnikov, Anna Bykova, and Ivan P Yamshchikov. 2024. Generalization potential of large language models. Neural Computing and Applications, pages 1--25

work page 2024
[5]

Cheng-Han Chiang and Hung-yi Lee. 2022. On the transferability of pre-trained language models: A study from artificial datasets. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10518--10525

work page 2022
[6]

Noam Chomsky. 1956. Three models for the description of language. IRE Transactions on information theory, 2(3):113--124

1956
[7]

Ronen Eldan and Yuanzhi Li. 2023. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759

arXiv 2023
[8]

Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126--1135. PMLR

2017

Show all 31 references

[9]

Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C \'e sar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. 2023. Textbooks are all you need. arXiv preprint arXiv:2306.11644

2023 arXiv
[10]

Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, et al. 2021. Pre-trained models: Past, present and future. AI Open, 2:225--250

2021
[11]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

2020 arXiv
[12]

Sepp Hochreiter and J \"u rgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735--1780

1997
[13]

Philip A Huebner, Elior Sulem, Fisher Cynthia, and Dan Roth. 2021. Babyberta: Learning more grammar with small-scale child-directed language. In Proceedings of the 25th conference on computational natural language learning, pages 624--646

2021
[14]

Kundan Krishna, Jeffrey Bigham, and Zachary C Lipton. 2021. Does pretraining for summarization require knowledge transfer? arXiv preprint arXiv:2109.04953

2021 arXiv
[15]

Matthias Lindemann, Alexander Koller, and Ivan Titov. 2023. Injecting a structural inductive bias into a seq2seq model by simulation. arXiv preprint arXiv:2310.00796

2023 arXiv
[16]

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692

2019 arXiv
[17]

Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. 2021. Pretrained transformers as universal computation engines. arXiv preprint arXiv:2103.05247, 1

2021 arXiv
[18]

R Thomas McCoy, Erin Grant, Paul Smolensky, Thomas L Griffiths, and Tal Linzen. 2020. Universal linguistic inductive biases via meta-learning. arXiv preprint arXiv:2006.16324

2020 arXiv
[19]

Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. 2021. An empirical investigation of the role of pre-training in lifelong learning. arXiv preprint arXiv:2112.09153

2021 arXiv
[20]

Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. 2023. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys, 56(2):1--40

2023
[21]

Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. 2020. What is being transferred in transfer learning? Advances in neural information processing systems, 33:512--523

2020
[22]

OpenAI. 2023. http://arxiv.org/abs/2303.08774 Gpt-4 technical report

2023 arXiv
[23]

Isabel Papadimitriou and Dan Jurafsky. 2020. Learning music helps you read: Using transfer to study linguistic structure in language models. arXiv preprint arXiv:2004.14601

2020 arXiv
[24]

Isabel Papadimitriou and Dan Jurafsky. 2023. http://arxiv.org/abs/2304.13060 Injecting structural hints: Using language models to study inductive biases in language learning

2023 arXiv
[25]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

2019
[26]

Ryokan Ri and Yoshimasa Tsuruoka. 2022. Pretraining with artificial language: Studying transferable knowledge in language models. arXiv preprint arXiv:2203.10326

2022 arXiv
[27]

Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. arXiv preprint arXiv:2104.06644

2021 arXiv
[28]

Maxim K Surkov and Ivan P Yamshchikov. 2024. Vygotsky distance: measure for benchmark task similarity. arXiv preprint arXiv:2402.14890

2024 arXiv
[29]

Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. 2022. Will we run out of data? an analysis of the limits of scaling datasets in machine learning. arXiv preprint arXiv:2211.04325

2022 arXiv
[30]

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461

2018 arXiv
[31]

Yuhuai Wu, Markus N Rabe, Wenda Li, Jimmy Ba, Roger B Grosse, and Christian Szegedy. 2021. Lime: Learning inductive bias for primitives of mathematical reasoning. In International Conference on Machine Learning, pages 11251--11262. PMLR

2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2019. On the cross-lingual transferability of monolingual representations. arXiv preprint arXiv:1910.11856

arXiv 2019

[4] [4]

Mikhail Budnikov, Anna Bykova, and Ivan P Yamshchikov. 2024. Generalization potential of large language models. Neural Computing and Applications, pages 1--25

work page 2024

[5] [5]

Cheng-Han Chiang and Hung-yi Lee. 2022. On the transferability of pre-trained language models: A study from artificial datasets. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10518--10525

work page 2022

[6] [6]

Noam Chomsky. 1956. Three models for the description of language. IRE Transactions on information theory, 2(3):113--124

1956

[7] [7]

Ronen Eldan and Yuanzhi Li. 2023. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759

arXiv 2023

[8] [8]

Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126--1135. PMLR

2017