{"id":"2f77f25f-189a-4d64-b760-e8057c227efd","arxiv_id":"2411.15577","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLMs with retrieval from descriptive grammars outperform no-grammar baselines on typological feature classification, and BM25 retrieval rivals top neural rerankers on this domain.","lead":"The paper introduces two new benchmarks to test whether large language models can read descriptive grammar books and correctly answer structured typological questions, such as a language's dominant word order or number of cases. The main finding is that adding retrieval from grammar books improves accuracy over the model's prior knowledge, and that a cheap BM25 retriever performs almost as well as expensive neural rerankers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 4 compares RAG runs against baselines evaluated on smaller subsets; if RAG was run on all 148 grammars, the central 'all RAG outperform baselines' claim is unsupported.","rationale":"The reader's conditional verdict is reasonable, and I would not move it; my most load-bearing concern is different from the reader's weakest assumption. The WALS/Grambank label noise is real and acknowledged in the paper, but it affects the baseline and RAG columns in the same direction and is therefore less decisive for the relative RAG-over-baseline claim. The sharper problem is that Table 4's comparison may not be over the same grammars: the baseline is explicitly computed on reduced subsets because some grammars lack sufficient information, while the RAG columns are not stated to be restricted to those subsets. Appendix I's wording about 'recalculated ... on the same subsets as used for the ablation' makes this more than a stylistic gap. If the RAG runs used all 148 grammars, the model is being asked to answer questions that the grammar text cannot answer, and the comparison is confounded by item selection. The concrete check is cheap: inspect the released code or rerun the RAG configurations on the baseline subsets to confirm the exact N for each cell; if RAG was already on the same subsets, the concern dissolves. The single-run design also means the RAG columns have no error bars, so even on identical subsets the headline should be read as directional. The paper's open-source code and the fully released reranker benchmark are genuine positives, but they do not resolve the subset ambiguity. This keeps the reader's CONDITIONAL verdict unchanged.","tokens_in":22024,"tokens_out":17817,"duration_ms":168256,"concrete_test":"Run the four RAG configurations (BM25, BM25+CoT, Reranker, Reranker+CoT) using exactly the baseline subsets (136/146/121/140), as the released code or a rerun permits; then recompute Table 4 deltas versus the baseline means. If any RAG-minus-baseline delta on identical items falls within the baseline standard deviation or changes sign, the 'all RAG outperform baselines' headline is not supported. Report N per cell in the table.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—'All RAG configurations outperform the baselines' (Section 5.2, Table 4)—rests on a comparison that the paper never states is apples-to-apples. Section 5.2 says the baseline was evaluated only on languages where the grammar text contains enough information to determine the feature: 136 (WALS 81A), 146 (GB 107), 121 (WALS 116A*), and 140 (WALS 49A) out of 148. No sentence in Section 5.2 says the four single-run RAG configurations were restricted to those same subsets. Appendix I then says the ablation 'recalculated the metrics for the four RAG configurations on the same subsets as used for the ablation,' which reads as if the main RAG table had used the full 148. If so, the RAG columns include grammars whose texts do not support any answer, so the model can only rely on prior knowledge of WALS/Grambank values; the baseline columns exclude those same grammars. The reported gains can then be an artifact of different item sets, not of retrieval from grammars. And since each RAG cell is a single run, there is no variance estimate to absorb this confound. The paper must state the N per cell or recompute on identical subsets.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces two benchmarks for evaluating retrieval-augmented classification of typological features from descriptive grammars: a reranker benchmark containing 700 paragraphs from 14 English-language grammars annotated on a six-level relevance scale for WALS 81A, and a RAG benchmark containing 148 grammars per feature for WALS 81A (word order), GB 107 (standard negation), WALS 116A* (seven polar-question strategies), and WALS 49A (number of cases). The proposed pipeline retrieves paragraphs with BM25, optionally reranks the top 50 with an embedding-based reranker, and classifies with GPT-4o. The main empirical claims are that all RAG configurations outperform a no-grammar baseline, that BM25 is competitive with neural rerankers on the grammar domain, and that MTEB retrieval rankings are negatively correlated with rankings on the new reranker benchmark.","tokens_in":22257,"tokens_out":4381,"duration_ms":38545,"significance":"If the empirical results hold, the paper makes a useful contribution: it provides a repeatable evaluation setup for LLM extraction from descriptive grammars, releases an annotated reranker benchmark and code, and gives concrete evidence that MTEB rankings do not transfer to this low-resource linguistic domain. The no-grammar baseline with ten runs and the careful removal of benchmark-language examples from Chain-of-Thought prompts (Appendix G.3) are commendable design choices that reduce circularity concerns. The main limitations—single-annotator labels, single-run RAG scores, and imperfect database ground truth—are openly acknowledged, but they affect the strength of the central comparative claims and need to be addressed before acceptance.","major_comments":[{"comment":"The comparison \"All RAG configurations outperform the baselines\" is not stated to be apples-to-apples. The baseline is computed on subsets of 136, 146, 121, and 140 grammars (WALS 81A, GB 107, WALS 116A, WALS 49A), while no sentence in Section 5.2 restricts the four RAG configurations to those subsets. Appendix I's statement that the ablation \"recalculated the metrics for the four RAG configurations on the same subsets\" suggests that the main Table 4 used the full 148 grammars. If so, the RAG columns include grammars whose texts do not support any answer, allowing GPT-4o to rely on prior knowledge, while the baseline columns exclude those same grammars, so the reported gains could be an artifact of differing item sets. The authors should state the N per cell or recompute all RAG metrics on the same subsets.","section":"5.2, Table 4, Appendix I"},{"comment":"Each of the four non-baseline RAG configurations was executed once, as stated in Section 8, and Table 4 shows no variance for those columns. The central claim that all RAG configurations outperform baselines therefore rests on single-run deltas, some of which are small (e.g., WALS 81A micro F1: 0.6892 vs 0.7027). Reporting multiple runs with confidence intervals, or at least explicitly restricting the claims to these single runs, is necessary before the qualitative conclusions about BM25 versus rerankers and about Chain-of-Thought can be trusted.","section":"Table 4; Section 8"},{"comment":"The reranker benchmark's 700 paragraphs were annotated by a single annotator (the first author), with no inter-annotator agreement reported. The NDCG@20 scores in Table 1 and the Spearman rho = -0.8571 in Table 2 are computed on this six-level relevance scale; if the scale is not reproducible, the ranking and the correlation may not be stable. The authors should report inter-annotator agreement on at least a subset or provide detailed annotation guidelines and examples to mitigate this concern.","section":"4.1, 4.2; Section 8"},{"comment":"Using WALS and Grambank values as ground truth is acknowledged in Section 8 to be imperfect: the authors cite Baylor et al. (2023), who report only 69.04% average agreement between the two databases. Since the measured classification accuracy is defined against these values, errors in these databases propagate directly into all F1 scores, and comparisons between features with different label reliability are affected. A sensitivity analysis (for example, reporting accuracy only on the subset of languages where WALS and Grambank agree, or excluding features with high known disagreement) would strengthen the central empirical claim.","section":"5.1, 8"}],"minor_comments":[{"comment":"The last sentence of the contribution list, \"semi-automating extraction of data from descriptive grammars and,\" appears truncated and should be completed.","section":"Section 1"},{"comment":"The sentence \"The distribution of values for each feature is presented in Appendix ??\" contains an unresolved placeholder and should be filled in.","section":"Section 5.1"},{"comment":"The paper describes WALS 116A* as a multilabel feature but also states \"we do not treat any features as multilabel classifications.\" Please clarify whether micro/macro F1 are computed per label and what this statement means for the seven binary sub-features.","section":"Table 4 and Section 5.2"},{"comment":"The abstract says the benchmarks encompass 248 languages, but the RAG benchmark has 148 grammars per feature and the reranker benchmark has 14 grammars; state explicitly how the 248 figure is derived.","section":"Abstract"},{"comment":"The Spearman correlation is computed on only seven models; the small sample size should be noted when interpreting the negative correlation.","section":"Table 2"},{"comment":"The appendices jump from G to I with no Appendix H; renumber or relabel the appendices for clarity.","section":"Appendix ordering"},{"comment":"The ablation results are referred to as appearing in \"Table ??\" and \"the two columns on the right in Table ??\"; insert the correct table reference.","section":"Appendix I"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful benchmark and an interesting negative result about MTEB transfer, but the main quantitative claim currently rests on a comparison whose item sets are not clearly identical, and the non-baseline RAG columns have no variance estimates. These issues are fixable with recomputation and additional runs, so I would be willing to re-review a revised version. The single-annotator and ground-truth reliability concerns should also be addressed with concrete evidence rather than only acknowledged in the limitations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is the first public benchmark set for LLM-based typological extraction from descriptive grammars, and it is built carefully: a 700-paragraph reranker set with a six-level relevance scale, a 148-grammar RAG set covering four typological features, and a no-grammar baseline that controls for the model's prior knowledge. Second, the headline results—BM25 is competitive with expensive neural rerankers, and MTEB retrieval rankings are negatively correlated with grammar-retrieval rankings (Spearman -0.857)—are genuinely new and practically useful.\n\nThe reranker benchmark is the strongest part. The annotation scheme distinguishes misleading mentions from genuinely informative ones, and the appendix examples make the labels transparent. The authors also scrupulously removed benchmark languages from the CoT prompts, which addresses contamination.\n\nThe soft spots are real. The central claim that all RAG configurations outperform baselines rests on a comparison the paper never states is apples-to-apples. The baseline is evaluated only on the 121–146 grammars per feature that contain enough information; the RAG columns appear to be on all 148, since Appendix I says the ablation 'recalculated' RAG on the same subsets, implying the main table did not. If that is true, the RAG columns include grammars where retrieval cannot help, and the comparison is between different item sets. The paper must state N per cell or recompute on identical subsets. This is the load-bearing issue.\n\nAlso: the reranker labels come from a single annotator with no inter-annotator agreement reported; the four RAG configurations were run once each, so 0.01–0.02 F1 differences are within noise; and WALS/Grambank ground truth is itself imperfect (the 69% agreement figure is cited). The RAG benchmark is only partially released due to copyright, which limits reproducibility.\n\nWho is this for? Anyone building typology databases or doing low-resource NLP with grammars. It deserves a serious referee. I would send it out, with the requirement that the authors report per-cell N, add repeated runs or error bars, and get a second annotator for at least a subset of the reranker labels.","headline":"Useful new benchmarks for grammar-based typology extraction, but the main RAG-vs-baseline claim needs a same-subset comparison and error bars before I'd trust it.","tokens_in":22821,"tokens_out":5450,"would_cite":true,"duration_ms":48022,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Grammar-book retrieval beats an LLM's memory on typology","keywords":["retrieval-augmented generation","typological features","descriptive grammars","low-resource languages","BM25","text embedding benchmarks","WALS","Grambank"],"falsifier":"Build a gold set by hand-verifying typological feature values for the 148 benchmark languages directly from their grammars, independent of WALS and Grambank, and compare RAG predictions against that gold set; if the pipeline's advantage over the no-retrieval baseline disappears on the corrected labels, the central claim fails. A cheaper check is to re-run each RAG configuration many times: the paper reports single runs, so if run-to-run variance exceeds the gap between BM25 and the best reranker, the ranking conclusions are not established.","tokens_in":21786,"feed_emoji":"📚","tokens_out":9178,"duration_ms":75484,"temperature":0.7,"pith_summary":"This paper tries to establish that retrieval-augmented classification over descriptive grammars—splitting a grammar book into paragraphs, retrieving the passages relevant to a typological question, and having an LLM pick an answer from a fixed menu—consistently beats asking the LLM to answer from its own knowledge. Across 148 grammars covering four typological features (basic word order, standard negation, polar-question strategies, and number of cases), every RAG configuration the authors tested outperformed the no-retrieval baseline. The paper also argues that retrieval quality on grammar text is not predicted by the MTEB embedding leaderboard: BM25, a simple term-frequency matcher, trails the best neural reranker by only a small margin on NDCG@20, while MTEB rank and grammar-benchmark rank are strongly negatively correlated. If these claims hold, linguists could semi-automate filling typological databases and NLP systems could serve the many languages that have grammars but almost no digital text.","feed_headline":"Grammar-book retrieval beats an LLM's memory on typology","feed_subtitle":"Feeding retrieved passages from 148 grammars lifts GPT-4o's classification of linguistic features; BM25 nearly matches neural rerankers.","key_machinery":"The load-bearing component is the two-stage retrieval-augmented classification loop: grammars are split into paragraphs; BM25 retrieves the top 50 paragraphs for a query built from the typological feature (optionally reranked to top 20 by SFR-Embedding-Mistral with a task-specific instruction); GPT-4o then outputs one of a closed set of feature values. The retrieval-quality benchmark uses a six-level relevance scale (0–5) for 700 paragraphs from 14 grammars and NDCG@20, which is what lets the paper separate 'does retrieval work?' from 'does the LLM classify well?'","core_discovery":"The central claim, stated on the paper's own terms, is that a RAG pipeline designed for closed-set linguistic classification—retrieve relevant paragraphs from a descriptive grammar, then have GPT-4o choose one value from the feature's answer set—yields higher F1 than prompting the model with feature definitions alone. The claim is tested on 148 grammars sampled to balance language families and world macroareas, with features chosen to represent easy, morphosyntactic, multilabel, and quantitative extraction: WALS 81A word order, Grambank 107 negation, a seven-way polar-question composite, and WALS 49A number of cases. The retrieval component is evaluated separately on 700 annotated paragraphs from 14 grammars with a six-level relevance scale; BM25 achieves NDCG@20 of 0.7494, ranking 6th of 8 and only slightly behind the leading reranker, and the correlation between MTEB retrieval ranking and this grammar benchmark's ranking is Spearman's $\\rho = -0.8571$. The authors conclude that descriptive grammars remain a non-saturated reading task for LLMs and that retrieval—not just generation—deserves attention in this domain.","pith_inferences":["Editorial inference: because each non-baseline RAG configuration was run once, the reported differences between BM25-only and reranker configurations may be within run-to-run noise; repeating runs would show whether the reranker's edge is real.","Editorial inference: the near-parity of BM25 with neural rerankers suggests retrieval on grammars is dominated by exact technical vocabulary; augmenting queries with a small glossary of typological terms could close the gap at near-zero cost.","Editorial inference: the strongly negative MTEB correlation, if replicated, implies that retrieval benchmarks should be built per content domain rather than once for general English text; the same method could apply to legal, medical, or historical documents with similar terminology drift.","Editorial inference: the ablation with human-selected pages shows retrieval error and classification error are separable, and for some easy features human retrieval can even hurt because the LLM already knew the answer; the next testable step is a retrieval model trained on the six-level relevance judgments."],"forward_implications":["If RAG over grammars reliably classifies typological features, the same pipeline can be pointed at new grammars to propose values for underannotated languages, reducing manual annotation effort.","Because BM25 nearly matches neural rerankers on grammar text, a production system for low-resource languages does not need GPU-heavy reranking; a term-frequency retriever plus an instruction-tuned LLM is a viable baseline.","The negative MTEB correlation implies that for specialist domains like descriptive linguistics, generic embedding leaderboards can mislead model selection; domain-specific retrieval benchmarks like the 700-paragraph set are needed.","The six-level relevance annotations show that naive retrieval is easy to fool by paragraphs about ditransitive word order or ordering of clitics; any automated typology-extraction system must distinguish evidence about the monotransitive construction from other order mentions.","Since the paper found that Chain-of-Thought prompting sometimes hurts, building prompts with feature-specific annotation guidelines is not automatically beneficial; prompt choice interacts with retrieval quality."],"supporting_citations":[{"why":"Introduces MTOB, the one-grammar-book machine-translation setup that this paper extends from translation to typological classification.","marker":"Tanzer et al., 2024"},{"why":"Supplies the typological database and feature definitions that provide labels for word order and case-count features.","marker":"Dryer and Haspelmath, 2013"},{"why":"Supplies the typological database and feature definitions that provide labels for negation and polar-question features.","marker":"Skirgård et al., 2023"},{"why":"Defines MTEB, the embedding leaderboard whose retrieval rankings the paper compares against its grammar benchmark.","marker":"Muennighoff et al., 2023"},{"why":"Provides the BM25 formulation used as the base retriever.","marker":"Trotman et al., 2012"},{"why":"Proposes using Wikipedia summaries as retrieval queries over grammars, the query design adopted in the reranker benchmark.","marker":"Kornilov, 2023"},{"why":"Shows that irrelevant context degrades LLM answers, motivating retrieval and reranking instead of feeding entire grammars.","marker":"Shi et al., 2023"},{"why":"Documents typological-database label disagreement, grounding the paper's limitation that database values may be wrong.","marker":"Baylor et al., 2023"},{"why":"Provides the Genus-Macroarea sampling method used to stratify the 148-language benchmark.","marker":"Miestamo et al., 2016"},{"why":"Supplies the implemented sampling procedure that the paper adapts to assemble the grammar collection.","marker":"Cheveleva, 2023"}],"fun_headline_variants":["RAG from grammar books beats LLM's own typology knowledge","Retrieval beats memory for classifying linguistic features","BM25 rivals neural rerankers on grammar retrieval benchmark","Grammar RAG lifts GPT-4o's typology classification accuracy","New benchmark tests LLMs on 248 languages' grammar descriptions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the WALS and Grambank values used as labels are correct for the variety described in each grammar; if a database entry is wrong, refers to a different dialect, or the grammar itself documents a different variety, then the measured classification accuracy and the conclusion that retrieval helps are miscalibrated.","fun_headline_variants_meta":{"raw":{"variants":["RAG from grammar books beats LLM's own typology knowledge","Retrieval beats memory for classifying linguistic features","BM25 rivals neural rerankers on grammar retrieval benchmark","Grammar RAG lifts GPT-4o's typology classification accuracy","New benchmark tests LLMs on 248 languages' grammar descriptions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1641,"prompt_tokens":979,"completion_tokens":662,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":578}},"tokens_in":595,"tokens_out":662,"duration_ms":6325,"temperature":1.0,"reasoning_tokens":578,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:09:07.340219+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a gold set by hand-verifying typological feature values for the 148 benchmark languages directly from their grammars, independent of WALS and Grambank, and compare RAG predictions against that gold set; if the pipeline's advantage over the no-retrieval baseline disappears on the corrected labels, the central claim fails. A cheaper check is to re-run each RAG configuration many times: the paper reports single runs, so if run-to-run variance exceeds the gap between BM25 and the best reranker, the ranking conclusions are not established.","supporting_citations":[],"review_version":1}