{"id":"b25980db-73d1-474d-939e-de3da443688c","arxiv_id":"2604.17931","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":3,"one_line_summary":"Injecting 1% targeted synthetic data into GPT-2's pre-training substantially improves performance on 8 of 9 failing BLiMP grammatical paradigms, indicating data scarcity causes formal linguistic failures.","lead":"The paper shows that injecting just 1% targeted synthetic data into a small language model's training set fixes most grammatical failures on the BLiMP benchmark, suggesting data scarcity—not architecture—is the bottleneck for formal linguistic competence. A smart generalist might read this because it implies that fixing AI's grammatical blind spots may require better data curation rather than massively scaling up models.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Synthetic data targeting specific BLiMP paradigms risks train-test format contamination; improvements may reflect surface-pattern recognition rather than genuine linguistic competence acquisition.","rationale":"The reader correctly identified the most load-bearing concern: the gap between 'improved BLiMP accuracy' and 'acquired linguistic competence.' This is not a peripheral issue — it is the hinge on which the entire argument turns. The paper's claim is about data scarcity vs. architectural limitations for formal linguistic competence, but the evidence only directly supports improved BLiMP discrimination. The leap from benchmark improvement to competence acquisition is an inference that requires the synthetic data not to be format-contaminated with the test. The reader's additional concerns (no error bars, non-monotonicity, persistent principle_A_c_command failure) are valid but secondary — they affect confidence in the results rather than the interpretation of what the results mean. The verdict should remain CONDITIONAL because the paper is a well-designed, honestly reported contribution with open-sourced code, but the central interpretive claim has not been tested against the most natural alternative explanation. The paper itself acknowledges limitations (single architecture, modest scale), which is commendable, but does not address the memorization-vs-generalization question that would settle whether the existence proof is about linguistic competence or benchmark pattern recognition.","tokens_in":9695,"tokens_out":1961,"duration_ms":82170,"concrete_test":"Construct a novel set of minimal pairs for each of the 9 targeted paradigms using vocabulary and sentence structures that differ from both the synthetic training data and BLiMP test items (e.g., draw from a different corpus, use different lexical items, vary sentence length and complexity). Evaluate the 1%-intervention models on these novel probes. If accuracy on the novel probes shows comparable improvements to BLiMP (within ~5 percentage points), the competence is genuine. If improvements drop substantially (e.g., only_npi_scope falls back near 20-30%), the gains reflect test-format memorization rather than rule acquisition, and the central claim weakens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that data scarcity, not architecture, is the bottleneck — depends on BLiMP accuracy improvements reflecting genuine acquisition of grammatical rules. But the synthetic data is generated to target specific BLiMP paradigms (Section 4, Appendix C), creating a direct train-test similarity risk. If the synthetic examples share the structural format, vocabulary distribution, or minimal-pair structure of BLiMP test items, the model could improve by recognizing surface patterns specific to the benchmark rather than learning the underlying linguistic rule. This concern is sharpened by the principle_A_reconstruction result: 37.2% → 72.3% with only ~30 synthetic examples (0.01% injection, ~10K tokens). Such a dramatic jump from so few examples is more consistent with the model latching onto a narrow surface cue that happens to align with the test format than with genuine grammatical competence emerging. The non-monotonic responses (only_npi_scope: 20.9 → 41.9 → 31.0 → 69.4) further suggest unstable, cue-driven learning rather than robust rule acquisition. The cross-paradigm transfer effects in Table 5 are bidirectional and noisy (some paradigms improve, others worsen), which is more consistent with incidental distributional shifts than systematic generalization. Without an evaluation on held-out probes with different surface forms, the claim that the model has acquired the linguistic rule — as opposed to the BLiMP test pattern — remains unestablished.","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"This paper investigates whether the heterogeneous performance of language models on BLiMP paradigms stems from data scarcity rather than architectural limitations. The authors pre-train GPT-2 Small (124M) on 100M tokens from FineWeb and inject small fractions (0.01%–1%) of synthetic data targeting the 9 worst-performing BLiMP paradigms. They find that 8 of 9 paradigms improve substantially (e.g., only_npi_scope: 20.9%→69.4%), while principle_A_c_command remains below chance. The paper also reports cross-paradigm transfer effects and an ablation over intervention magnitudes. Code and data are open-sourced.","tokens_in":9832,"tokens_out":2495,"duration_ms":62123,"significance":"The question of whether formal linguistic competence failures are data-driven or architecture-driven is important for understanding language model capabilities and for data-centric AI research. The experimental design—controlled pre-training with targeted synthetic data injection—is a clean intervention. The authors deserve credit for open-sourcing code and data, for including an ablation over intervention magnitudes (Table 3), and for reporting the full 67-paradigm breakdown (Table 5) including cross-paradigm transfer effects. The finding that ~30 synthetic examples can boost principle_A_reconstruction from 37.2% to 72.3% is striking and warrants careful scrutiny.","major_comments":[{"comment":"§4–5, Table 3: The central claim—that improvements reflect acquisition of linguistic rules rather than surface-pattern recognition—depends on the synthetic training data not sharing the structural format of BLiMP test items. BLiMP evaluates via minimal pairs (grammatical vs. minimally ungrammatical sentences). If the synthetic data for, e.g., only_npi_scope contains sentences with the same NPI-scope structures and vocabulary distribution as the BLiMP test items, the model may be learning to recognize the test format rather than the underlying grammatical rule. The principle_A_reconstruction result (37.2%→72.3% with ~30 examples at 0.01% injection) is especially consistent with this concern: such a dramatic jump from so few examples is more typical of cue-latching than robust rule acquisition. The paper does not include any evaluation on held-out probes with different surface forms, nor a","section":null},{"comment":"§5.3, Table 3: Several paradigms show non-monotonic responses to intervention magnitude (only_npi_scope: 20.9→41.9→31.0→69.4; existential_there_quantifiers_2: 23.4→13.2→25.2→51.8). The paper notes these anomalies but does not analyze them. Non-monotonicity is important because it is more consistent with unstable, cue-driven learning than with systematic generalization. If the model were acquiring a robust grammatical rule, one would generally expect monotonic improvement with more evidence. The authors should discuss what these non-monotonic patterns imply for their central claim, or at minimum acknowledge them as a limitation on the strength of the 'data scarcity' conclusion.","section":null},{"comment":"§6: The comparison between GPT-2 Small (124M, 100M tokens + intervention) and Llama-3 70B (15T tokens, no intervention) on specific paradigms is presented as evidence that 'data composition may matter more than data scale.' This comparison is not controlled: Llama-3 was not trained with the same intervention, so the comparison conflates data composition with model size, training data, and training procedure. The claim would be strengthened by applying the same intervention to a larger model, or by framing the comparison more carefully as an existence proof rather than evidence that data composition dominates scale.","section":null}],"minor_comments":[{"comment":"The arXiv abstract and title ('LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agent') do not match the paper content ('Heterogeneity in Formal Linguistic Competence of Language Models'). This appears to be a submission error but should be corrected.","section":null},{"comment":"Appendix C describes genre/subgenre taxonomy for synthetic data diversity but does not specify how the linguistic target structures are embedded in the synthetic text. A brief example of a synthetic document for one paradigm would help readers assess the contamination risk.","section":null},{"comment":"Figures 1–2 (Appendix B) are difficult to read; the axis labels and legend text are very small. Consider increasing font sizes or splitting into more figures.","section":null},{"comment":"Table 5 is large and dense; consider highlighting the targeted paradigms or separating targeted vs. non-targeted results for clarity.","section":null},{"comment":"The paper does not report variance or confidence intervals for the BLiMP accuracy numbers in Tables 3 and 5. Since BLiMP paradigms contain 1000 items each, binomial confidence intervals would be informative, especially for near-chance scores.","section":null}],"recommendation":"major_revision","confidential_remarks":"The mismatch between the arXiv metadata (agentic RL for deep research) and the actual paper content (linguistic competence of LLMs) is unusual and may indicate a submission error. The editor may want to verify that the correct manuscript was uploaded. Separately, the core concern about train-test format similarity is the kind of issue that could be addressed within one revision cycle if the authors can show that their synthetic data does not share the minimal-pair structure of BLiMP items, or by running a held-out evaluation. The paper's contribution is interesting but the evidence chain needs strengthening before the central claim is fully supported."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive report. All three major comments are well-taken. Comment 1 (surface-form overlap between synthetic data and BLiMP test items) identifies a genuine gap in our evaluation: we did not include held-out probes with different surface forms, and we will add such an evaluation. Comment 2 (non-monotonic responses to intervention magnitude) is correct that we noted but did not analyze these patterns; we will add a substantive discussion acknowledging them as a limitation on the strength of the data-scarcity conclusion. Comment 3 (uncontrolled GPT-2 vs. Llama-3 comparison) is also correct; we will reframe the comparison as an existence proof rather than evidence that data composition dominates scale. All three comments lead to revisions.","responses":[{"response":"The referee raises a legitimate and important concern. We acknowledge that our current evaluation does not rule out the possibility that some portion of the improvement reflects surface-form matching between the synthetic training data and the BLiMP minimal-pair test format, rather than robust acquisition of the underlying grammatical rule. The principle_A_reconstruction result—37.2% to 72.3% from approximately 30 examples—is indeed the case most susceptible to this alternative explanation, and we agree it warrants careful scrutiny. To address this, we will add a new evaluation using held-out probes with different surface forms and vocabulary distributions from the BLiMP test items. Specifically, we will generate novel minimal pairs for the targeted paradigms using different lexical items and sentence frames than those in BLiMP, and evaluate whether the improvements transfer. We will also add an explicit discussion of the cue-latching concern, particularly for the principle_A_reconstruction paradigm, and acknowledge that without such held-out probes, our claim that improvements reflect rule acquisition rather than surface-pattern recognition is not fully established. We note that the cross-paradigm transfer effects reported in Table 5 (where targeting one paradigm improves related but non-targeted paradigms) provide partial evidence against pure cue-latching, since the transfer paradigms use different surface forms than the targeted synthetic data—but we agree this is not a substitute for a dedicated held-out evaluation.","revision_made":"yes","referee_comment":"§4–5, Table 3: The central claim depends on synthetic training data not sharing the structural format of BLiMP test items. If synthetic data for e.g. only_npi_scope contains sentences with the same NPI-scope structures and vocabulary distribution as BLiMP test items, the model may be learning to recognize the test format rather than the underlying grammatical rule. The principle_A_reconstruction result (37.2%→72.3% with ~30 examples at 0.01% injection) is especially consistent with cue-latching. The paper does not include evaluation on held-out probes with different surface forms."},{"response":"We agree that the non-monotonic response patterns deserve more than the brief mention they currently receive. The referee is correct that non-monotonicity is more consistent with unstable, cue-driven learning than with systematic generalization, and that this has implications for the strength of our data-scarcity conclusion. We will revise Section 5.3 to include a substantive discussion of these patterns. Specifically, we will acknowledge that the non-monotonic responses in only_npi_scope and existential_there_quantifiers_2 complicate the interpretation that improvements reflect robust rule acquisition, and that they are at least partially consistent with the cue-latching concern raised in the referee's first comment. We will also note that the paradigms showing monotonic improvement (e.g., principle_A_reconstruction, coordinate_structure_constraint_complex_left_branch, left_branch_island_echo_question) provide stronger evidence for the data-scarcity hypothesis than those showing non-monotonic patterns. We will add an explicit statement in the Limitations section that the non-monotonic responses constrain the strength of the conclusion that data scarcity is the sole bottleneck for all paradigms, and that the interaction between targeted synthetic data and existing corpus statistics may produce paradigm-specific dynamics that our framework does not fully explain.","revision_made":"yes","referee_comment":"§5.3, Table 3: Several paradigms show non-monotonic responses to intervention magnitude (only_npi_scope: 20.9→41.9→31.0→69.4; existential_there_quantifiers_2: 23.4→13.2→25.2→51.8). The paper notes these anomalies but does not analyze them. Non-monotonicity is more consistent with unstable, cue-driven learning than with systematic generalization. The authors should discuss what these non-monotonic patterns imply for their central claim, or at minimum acknowledge them as a limitation."},{"response":"The referee is correct that the comparison between GPT-2 Small with intervention and Llama-3 70B without intervention is not controlled, and that presenting it as evidence that 'data composition may matter more than data scale' conflates multiple variables. We cannot run the same intervention on Llama-3 70B, as we do not have access to its pre-training pipeline, so this concern cannot be fully resolved within the scope of the current work. However, we will revise Section 6 to reframe the comparison explicitly as an existence proof—demonstrating that a small model with targeted data intervention can achieve competence on specific paradigms that a much larger model without such intervention does not—rather than as controlled evidence that data composition dominates scale. We will also add a clear statement that this comparison conflates model size, training data, training procedure, and data composition, and that disentangling these factors would require applying the same intervention strategy to larger models, which we identify as a direction for future work. This is also already partially acknowledged in our Limitations section, but we will strengthen the language there to match the more careful framing in Section 6.","revision_made":"yes","referee_comment":"§6: The comparison between GPT-2 Small (124M, 100M tokens + intervention) and Llama-3 70B (15T tokens, no intervention) is not controlled: Llama-3 was not trained with the same intervention, so the comparison conflates data composition with model size, training data, and training procedure. The claim would be strengthened by applying the same intervention to a larger model, or by framing the comparison more carefully as an existence proof rather than evidence that data composition dominates scale."}],"tokens_in":9617,"tokens_out":1391,"duration_ms":79753,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Two things to know upfront: this is a well-designed, honest experiment showing that 1% targeted synthetic data during pre-training fixes 8 of 9 worst BLiMP paradigms in GPT-2 Small, and the central soft spot is whether the improvements reflect genuine linguistic competence or the model latching onto surface cues that happen to align with BLiMP's minimal-pair format. The reader's take and the stress-test concern are largely on target, though I think the paper is a bit better than the CONDITIONAL verdict suggests — the experimental design is clean and the limitations are acknowledged with unusual candor. The code is open-sourced, which matters here. What's genuinely new is the systematic intervention across 9 paradigms with controlled injection at three magnitudes (0.01%, 0.1%, 1%), plus the cross-paradigm transfer table (Table 5). The principle_A_reconstruction result — 37.2% to 72.3% with ~30 synthetic examples — is striking and worth taking seriously. The persistent failure of principle_A_c_command across all conditions is an honest negative result that tempers the strong claim. The paper does not overclaim on this point; the conclusion is appropriately hedged. The stress-test concern about train-test format contamination is the right thing to worry about, and it lands with real force on the principle_A_reconstruction result. A 35-point jump from 30 examples is more consistent with cue-detection than rule-learning. The non-monotonic responses (only_npi_scope: 20.9 → 41.9 → 31.0 → 69.4) also look unstable and cue-driven rather than reflecting robust acquisition. The cross-paradigm transfer effects in Table 5 are bidirectional and noisy — some paradigms improve, others worsen — which is more consistent with incidental distributional shifts than systematic generalization. That said, the concern is not fully damning: the synthetic data generation uses diverse genres and subgenres (Appendix C), which provides some surface-form variation, and the cross-paradigm transfer is at least suggestive of partial generalization. But without a held-out probe with different surface forms, the claim that the model acquired the rule rather than the test pattern remains unestablished. The reader correctly flags the missing error bars and multiple seeds — this is a real gap for a paper making quantitative claims about accuracy differences. Minor relative to the contamination concern, but it should be fixed. This paper is for NLP researchers interested in data composition, the Poverty of Stimulus debate, and sample-efficient pre-training. It deserves a serious referee who can push on the generalization question — specifically, whether the authors can show improvements on held-out probes that don't share BLiMP's surface format. If they can address that, this is a solid contribution. If not, the central claim needs significant narrowing.","headline":"Clean data-intervention experiment on GPT-2/BLiMP; main concern is whether improvements reflect genuine rule acquisition or surface-pattern matching to the benchmark format.","tokens_in":10479,"tokens_out":680,"would_cite":false,"duration_ms":73970,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Tiny Data Injection Fixes Most LLM Grammar Failures","keywords":[],"falsifier":"If a model trained on the synthetic data passes BLiMP minimal pairs but fails on genuinely novel sentences testing the same grammatical rule in different surface forms, the improvement would be memorization rather than competence acquisition.","tokens_in":9724,"feed_emoji":"🔬","tokens_out":1112,"duration_ms":25991,"temperature":0.7,"pith_summary":"This paper asks why language models master some grammatical rules near-perfectly while failing badly on others, even after training on trillions of tokens. The authors test whether these failures come from inherent architectural limitations or simply from insufficient exposure to specific grammatical constructions in the training data. They pre-train a GPT-2 Small model (124M parameters) on 100M tokens sampled from the FineWeb corpus and then inject as little as 1% targeted synthetic data covering nine of the worst-performing BLiMP paradigms — a benchmark that tests whether a model prefers grammatical sentences over minimally different ungrammatical ones. The intervention substantially improves performance on 8 of the 9 targeted paradigms, with one case (only_npi_scope) jumping from 20.9% to 69.4% accuracy. In one striking result, just 0.01% synthetic data (roughly 10K tokens) lifted principle_A_reconstruction from 37.2% to 72.3%. The 124M-parameter model with this tiny intervention even outperformed Llama-3 70B (trained on 15+ trillion tokens) on specific paradigms. One paradigm, principle_A_c_command, resisted improvement entirely. The authors conclude that the heterogeneity in formal linguistic competence is largely a data composition problem, not an architecture problem — standard transformer architectures have the latent capacity to acquire formal syntax if the training data contains sufficient distributional evidence for the relevant constructions.","feed_headline":"1% of Targeted Data Fixes Most LLM Grammar Blind Spots","feed_subtitle":"A 124M model with a tiny data injection beats a 70B model on specific grammatical paradigms — data composition, not architecture, is the bot","key_machinery":"The central mechanism is targeted synthetic data injection: the authors generate synthetic text documents designed to contain specific grammatical constructions (drawn from nine BLiMP paradigms) across diverse genres and subgenres, then mix a small fraction (0.01%–1%) of this synthetic data into a 100M-token pre-training corpus. The BLiMP benchmark itself is the evaluation instrument — it tests formal linguistic competence by presenting minimal pairs (a grammatical sentence and an ungrammatical variant differing in one feature) and checking whether the model assigns higher probability to the grammatical version.","core_discovery":"The central discovery is that injecting as little as 1% targeted synthetic data into pre-training can rescue a small language model's performance on grammatical paradigms where it previously performed below chance, while preserving overall benchmark performance. This holds across 8 of 9 tested BLiMP paradigms, and in one case a mere 0.01% injection was sufficient for a large jump. The one persistent failure (principle_A_c_command) suggests that some constructions may require either much higher data density or genuinely different inductive biases. The finding that a 124M model with targeted data can beat a 70B model on specific paradigms reframes the bottleneck from model scale to data compos","pith_inferences":[],"forward_implications":["Data composition, not raw scale, may be the primary lever for closing specific competence gaps in language models — which would redirect effort from scaling laws toward curated data design.","If the result generalizes beyond GPT-2 Small, billion-parameter models could potentially fix their own grammatical blind spots with very small, targeted data additions rather than retraining from scratch.","The persistent failure of principle_A_c_command suggests a boundary case where either the construction is genuinely hard for transformers to learn from surface statistics, or the synthetic data generation method does not adequately capture the relevant distributional signal.","The finding that targeting one paradigm sometimes improves or degrades other non-targeted paradigms (Appendix D) points to transfer effects — both positive and negative — that could inform a more principled approach to curriculum design in pre-training."],"fun_headline_variants":["Small agent beats Claude 4.5 on deep research benchmarks","4B search agent outperforms commercial models via virtual training world","Virtual search environment enables scalable RL for deep research agents","Tiny search agent hits open-source SOTA on GAIA and Xbench","Lite virtual world training lets small agent surpass large research models"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paper assumes that improved BLiMP accuracy after synthetic data injection reflects genuine acquisition of the underlying grammatical rule, rather than the model learning to recognize the surface patterns of the synthetic data or the minimal-pair format used by the benchmark itself.","fun_headline_variants_meta":{"raw":{"variants":["Small agent beats Claude 4.5 on deep research benchmarks","4B search agent outperforms commercial models via virtual training world","Virtual search environment enables scalable RL for deep research agents","Tiny search agent hits open-source SOTA on GAIA and Xbench","Lite virtual world training lets small agent surpass large research models"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":637,"prompt_tokens":552,"completion_tokens":85,"prompt_tokens_details":null},"tokens_in":552,"tokens_out":85,"duration_ms":54943,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-05T14:57:05.616603+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If a model trained on the synthetic data passes BLiMP minimal pairs but fails on genuinely novel sentences testing the same grammatical rule in different surface forms, the improvement would be memorization rather than competence acquisition.","supporting_citations":[],"review_version":2}