Pith. sign in

REVIEW 4 major objections 7 minor 13 references

From MTEB to MTOB: Retrieval-Augmented Classification for Descriptive Grammars

T0 review · 4 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Grammar-book retrieval beats an LLM's memory on typology

desk verdict Useful new benchmarks for grammar-based typology extraction, but the main RAG-vs-baseline claim needs a same-subset comparison and error bars before I'd trust it. read the letter →

arxiv 2411.15577 v2 pith:PIHAU6DB submitted 2024-11-23 cs.CL

Albert Kornilov , Tatiana Shavrina This is my paper · ORCID
classification cs.CL
keywords retrieval-augmentedgenerationtypologicalfeaturesdescriptivegrammarslow-resourcelanguagesBM25textembeddingbenchmarksWALSGrambank
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that retrieval-augmented classification over descriptive grammars—splitting a grammar book into paragraphs, retrieving the passages relevant to a typological question, and having an LLM pick an answer from a fixed menu—consistently beats asking the LLM to answer from its own knowledge. Across 148 grammars covering four typological features (basic word order, standard negation, polar-question strategies, and number of cases), every RAG configuration the authors tested outperformed the no-retrieval baseline. The paper also argues that retrieval quality on grammar text is not predicted by the MTEB embedding leaderboard: BM25, a simple term-frequency matcher, trails the best neural reranker by only a small margin on NDCG@20, while MTEB rank and grammar-benchmark rank are strongly negatively correlated. If these claims hold, linguists could semi-automate filling typological databases and NLP systems could serve the many languages that have grammars but almost no digital text.

What carries the argument

The load-bearing component is the two-stage retrieval-augmented classification loop: grammars are split into paragraphs; BM25 retrieves the top 50 paragraphs for a query built from the typological feature (optionally reranked to top 20 by SFR-Embedding-Mistral with a task-specific instruction); GPT-4o then outputs one of a closed set of feature values. The retrieval-quality benchmark uses a six-level relevance scale (0–5) for 700 paragraphs from 14 grammars and NDCG@20, which is what lets the paper separate 'does retrieval work?' from 'does the LLM classify well?'

What would settle it

Build a gold set by hand-verifying typological feature values for the 148 benchmark languages directly from their grammars, independent of WALS and Grambank, and compare RAG predictions against that gold set; if the pipeline's advantage over the no-retrieval baseline disappears on the corrected labels, the central claim fails. A cheaper check is to re-run each RAG configuration many times: the paper reports single runs, so if run-to-run variance exceeds the gap between BM25 and the best reranker, the ranking conclusions are not established.

Watch

Extended reading notes

Core claim

The central claim, stated on the paper's own terms, is that a RAG pipeline designed for closed-set linguistic classification—retrieve relevant paragraphs from a descriptive grammar, then have GPT-4o choose one value from the feature's answer set—yields higher F1 than prompting the model with feature definitions alone. The claim is tested on 148 grammars sampled to balance language families and world macroareas, with features chosen to represent easy, morphosyntactic, multilabel, and quantitative extraction: WALS 81A word order, Grambank 107 negation, a seven-way polar-question composite, and WALS 49A number of cases. The retrieval component is evaluated separately on 700 annotated paragraphs from 14 grammars with a six-level relevance scale; BM25 achieves NDCG@20 of 0.7494, ranking 6th of 8 and only slightly behind the leading reranker, and the correlation between MTEB retrieval ranking and this grammar benchmark's ranking is Spearman's $\rho = -0.8571$. The authors conclude that descriptive grammars remain a non-saturated reading task for LLMs and that retrieval—not just generation—deserves attention in this domain.

Load-bearing premise

The load-bearing premise is that the WALS and Grambank values used as labels are correct for the variety described in each grammar; if a database entry is wrong, refers to a different dialect, or the grammar itself documents a different variety, then the measured classification accuracy and the conclusion that retrieval helps are miscalibrated.

Editorial extensions

If this is right

  • If RAG over grammars reliably classifies typological features, the same pipeline can be pointed at new grammars to propose values for underannotated languages, reducing manual annotation effort.
  • Because BM25 nearly matches neural rerankers on grammar text, a production system for low-resource languages does not need GPU-heavy reranking; a term-frequency retriever plus an instruction-tuned LLM is a viable baseline.
  • The negative MTEB correlation implies that for specialist domains like descriptive linguistics, generic embedding leaderboards can mislead model selection; domain-specific retrieval benchmarks like the 700-paragraph set are needed.
  • The six-level relevance annotations show that naive retrieval is easy to fool by paragraphs about ditransitive word order or ordering of clitics; any automated typology-extraction system must distinguish evidence about the monotransitive construction from other order mentions.
  • Since the paper found that Chain-of-Thought prompting sometimes hurts, building prompts with feature-specific annotation guidelines is not automatically beneficial; prompt choice interacts with retrieval quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because each non-baseline RAG configuration was run once, the reported differences between BM25-only and reranker configurations may be within run-to-run noise; repeating runs would show whether the reranker's edge is real.
  • Editorial inference: the near-parity of BM25 with neural rerankers suggests retrieval on grammars is dominated by exact technical vocabulary; augmenting queries with a small glossary of typological terms could close the gap at near-zero cost.
  • Editorial inference: the strongly negative MTEB correlation, if replicated, implies that retrieval benchmarks should be built per content domain rather than once for general English text; the same method could apply to legal, medical, or historical documents with similar terminology drift.
  • Editorial inference: the ablation with human-selected pages shows retrieval error and classification error are separable, and for some easy features human retrieval can even hurt because the LLM already knew the answer; the next testable step is a retrieval model trained on the six-level relevance judgments.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces two benchmarks for evaluating retrieval-augmented classification of typological features from descriptive grammars: a reranker benchmark containing 700 paragraphs from 14 English-language grammars annotated on a six-level relevance scale for WALS 81A, and a RAG benchmark containing 148 grammars per feature for WALS 81A (word order), GB 107 (standard negation), WALS 116A* (seven polar-question strategies), and WALS 49A (number of cases). The proposed pipeline retrieves paragraphs with BM25, optionally reranks the top 50 with an embedding-based reranker, and classifies with GPT-4o. The main empirical claims are that all RAG configurations outperform a no-grammar baseline, that BM25 is competitive with neural rerankers on the grammar domain, and that MTEB retrieval rankings are negatively correlated with rankings on the new reranker benchmark.

Significance. If the empirical results hold, the paper makes a useful contribution: it provides a repeatable evaluation setup for LLM extraction from descriptive grammars, releases an annotated reranker benchmark and code, and gives concrete evidence that MTEB rankings do not transfer to this low-resource linguistic domain. The no-grammar baseline with ten runs and the careful removal of benchmark-language examples from Chain-of-Thought prompts (Appendix G.3) are commendable design choices that reduce circularity concerns. The main limitations—single-annotator labels, single-run RAG scores, and imperfect database ground truth—are openly acknowledged, but they affect the strength of the central comparative claims and need to be addressed before acceptance.

major comments (4)
  1. [5.2, Table 4, Appendix I] The comparison "All RAG configurations outperform the baselines" is not stated to be apples-to-apples. The baseline is computed on subsets of 136, 146, 121, and 140 grammars (WALS 81A, GB 107, WALS 116A, WALS 49A), while no sentence in Section 5.2 restricts the four RAG configurations to those subsets. Appendix I's statement that the ablation "recalculated the metrics for the four RAG configurations on the same subsets" suggests that the main Table 4 used the full 148 grammars. If so, the RAG columns include grammars whose texts do not support any answer, allowing GPT-4o to rely on prior knowledge, while the baseline columns exclude those same grammars, so the reported gains could be an artifact of differing item sets. The authors should state the N per cell or recompute all RAG metrics on the same subsets.
  2. [Table 4; Section 8] Each of the four non-baseline RAG configurations was executed once, as stated in Section 8, and Table 4 shows no variance for those columns. The central claim that all RAG configurations outperform baselines therefore rests on single-run deltas, some of which are small (e.g., WALS 81A micro F1: 0.6892 vs 0.7027). Reporting multiple runs with confidence intervals, or at least explicitly restricting the claims to these single runs, is necessary before the qualitative conclusions about BM25 versus rerankers and about Chain-of-Thought can be trusted.
  3. [4.1, 4.2; Section 8] The reranker benchmark's 700 paragraphs were annotated by a single annotator (the first author), with no inter-annotator agreement reported. The NDCG@20 scores in Table 1 and the Spearman rho = -0.8571 in Table 2 are computed on this six-level relevance scale; if the scale is not reproducible, the ranking and the correlation may not be stable. The authors should report inter-annotator agreement on at least a subset or provide detailed annotation guidelines and examples to mitigate this concern.
  4. [5.1, 8] Using WALS and Grambank values as ground truth is acknowledged in Section 8 to be imperfect: the authors cite Baylor et al. (2023), who report only 69.04% average agreement between the two databases. Since the measured classification accuracy is defined against these values, errors in these databases propagate directly into all F1 scores, and comparisons between features with different label reliability are affected. A sensitivity analysis (for example, reporting accuracy only on the subset of languages where WALS and Grambank agree, or excluding features with high known disagreement) would strengthen the central empirical claim.
minor comments (7)
  1. [Section 1] The last sentence of the contribution list, "semi-automating extraction of data from descriptive grammars and," appears truncated and should be completed.
  2. [Section 5.1] The sentence "The distribution of values for each feature is presented in Appendix ??" contains an unresolved placeholder and should be filled in.
  3. [Table 4 and Section 5.2] The paper describes WALS 116A* as a multilabel feature but also states "we do not treat any features as multilabel classifications." Please clarify whether micro/macro F1 are computed per label and what this statement means for the seven binary sub-features.
  4. [Abstract] The abstract says the benchmarks encompass 248 languages, but the RAG benchmark has 148 grammars per feature and the reranker benchmark has 14 grammars; state explicitly how the 248 figure is derived.
  5. [Table 2] The Spearman correlation is computed on only seven models; the small sample size should be noted when interpreting the negative correlation.
  6. [Appendix ordering] The appendices jump from G to I with no Appendix H; renumber or relabel the appendices for clarity.
  7. [Appendix I] The ablation results are referred to as appearing in "Table ??" and "the two columns on the right in Table ??"; insert the correct table reference.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; central RAG claims are empirical and externally anchored, with only a minor non-load-bearing self-citation.

full rationale

No circular step was found in the derivation chain. The RAG-vs-baseline comparison is an empirical evaluation, not a fitted prediction: the baseline omits retrieved paragraphs while keeping the same Appendix F prompts, and the Chain-of-Thought variants strip benchmark-language examples (Appendix G.3), so the model cannot solve the task from the prompt alone. WALS/Grambank values serve as external labels; sourcing definitions and labels from the same databases is standard benchmark construction, and the limitations section itself flags label error risk by citing Baylor et al.'s 69.04% agreement. The reranker benchmark intentionally evaluates reranking on BM25's top-50 candidate pool; this constrains the scope of the 'BM25 is competitive' claim but is an explicit experimental design, not a self-definitional equivalence. The sole self-citation (Kornilov 2023) motivates the Wikipedia-summary query in Section 4.1 and is not load-bearing. The flagged concern that Table 4 may compare RAG runs on 148 grammars to baselines on 136/146/121/140 grammars, given Appendix I's statement that metrics were 'recalculated ... on the same subsets as used for the ablation,' is a benchmark-comparability and variance-reporting risk, not a circularity; it is placed under correctness risk and does not raise the circularity score. The score of 2 reflects the minor, non-load-bearing self-citation; no circularity is present.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central evaluation depends on the correctness of WALS/Grambank labels, the reliability of single-annotator relevance judgments, the representativeness of English-language grammars, and the sufficiency of paragraph-level chunking. No new physical entities or fitted constants are introduced; the only hand-chosen values are top-k cutoffs, temperature, and sample size.

free parameters (4)
  • BM25 top-k = 50
    The pipeline retrieves 50 paragraphs per grammar before reranking; this cutoff is chosen by hand and affects which paragraphs the LLM sees.
  • Reranker top-k = 20
    After reranking, 20 paragraphs are inserted into the prompt; chosen by hand, no sensitivity analysis.
  • GPT-4o temperature = 0.2
    Decoding temperature set to 0.2 to favor deterministic answers; affects output variability.
  • Number of grammars per feature = 148
    Sample size chosen as 'ostensibly arbitrary number (initially 150)' for reliability of accuracy percentages; affects the statistical stability of all reported metrics.
assumptions (5)
  • domain assumption WALS and Grambank feature values used as ground truth are correct for the languages described by the sampled grammars.
    The RAG benchmark labels come from these databases; the paper itself cites 69.04% average agreement between them (Baylor et al., 2023), so label noise is known.
  • domain assumption The 0-5 relevance annotations for the reranker benchmark are reliable.
    All 700 paragraphs were annotated by the first author alone with no inter-annotator agreement measure; the ranking of rerankers depends on these labels.
  • domain assumption English-language grammars are representative of descriptive grammars for typological extraction.
    The sampling restricts to English-written grammars (Section 5.1), which may not represent the variety of descriptive styles or languages.
  • domain assumption Splitting grammars into paragraphs is a sufficient chunking strategy.
    The method uses simple paragraph chunking (Section 3); if relevant information spans paragraphs or sections, retrieval may miss it.
  • domain assumption GPT-4o's responses are reliable enough to score with a single pass.
    Each RAG configuration was run once at temperature 0.2; stochasticity and prompt sensitivity are not quantified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From MTEB to MTOB: Retrieval-Augmented Classification for Descriptive Grammars." pith.science (2026). https://pith.science/paper/PIHAU6DB

@misc{pith2026241115577,
  author       = {Pith},
  title        = {Pith review of: From MTEB to MTOB: Retrieval-Augmented Classification for Descriptive Grammars},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PIHAU6DB}},
  note         = {Machine review of arXiv:2411.15577}
}
read the original abstract

Recent advances in language modeling have demonstrated significant improvements in zero-shot capabilities, including in-context learning, instruction following, and machine translation for extremely under-resourced languages (Tanzer et al., 2024). However, many languages with limited written resources rely primarily on formal descriptions of grammar and vocabulary. In this paper, we introduce a set of benchmarks to evaluate how well models can extract and classify information from the complex descriptions found in linguistic grammars. We present a Retrieval-Augmented Generation (RAG)-based approach that leverages these descriptions for downstream tasks such as machine translation. Our benchmarks encompass linguistic descriptions for 248 languages across 142 language families, focusing on typological features from WALS and Grambank. This set of benchmarks offers the first comprehensive evaluation of language models' in-context ability to accurately interpret and extract linguistic features, providing a critical resource for scaling NLP to low-resource languages. The code and data are publicly available at \url{https://github.com/al-the-eigenvalue/RAG-on-grammars}.

Figures

Figures reproduced from arXiv: 2411.15577 by the authors.

Figure 1
Figure 1. NDCG@k across all values of k for best performing configurations for each model. [PITH_FULL_IMAGE:figures/full_fig_p017_1.png] view at source ↗
Figure 2
Figure 2. Mean NDCG@k for all grammars: Wikipedia Summary vs Term Only. [PITH_FULL_IMAGE:figures/full_fig_p020_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 12 canonical work pages

  1. [1]

    bge-en-icl (Xiao et al., 2023): https:// huggingface.co/BAAI/bge-en-icl License: Apache 2.0

    work page 2023

  2. [2]

    co/dunzhang/stella_en_1.5B_v5 License: MIT

    stella_en_1.5B_v5: https://huggingface. co/dunzhang/stella_en_1.5B_v5 License: MIT

    work page

  3. [3]

    NV-Retriever-v1 (de Souza et al., 2024): https://huggingface.co/nvidia/ NV-Retriever-v1 License: NVIDIA license agreement

    work page 2024

  4. [4]

    gte-Qwen2-7B-instruct (Li et al., 2023): https://huggingface.co/Alibaba-NLP/ gte-Qwen2-7B-instruct License: Apache 2.0

    work page 2023

  5. [5]

    Linq-Embed-Mistral (Kim et al., 2024): https://huggingface.co/ Linq-AI-Research/Linq-Embed-Mistral License: CC-BY-NC-4.0

    work page 2024

  6. [6]

    SFR-Embedding-2_R (Meng* et al., 2024): https://huggingface.co/Salesforce/ SFR-Embedding-2_R License: CC-BY-NC-4.0

    work page 2024

  7. [7]

    SVO”, “OVS

    SFR-Embedding-Mistral (Meng et al., 2024): https://huggingface.co/Salesforce/ SFR-Embedding-Mistral License: CC-BY-NC-4.0 B Benchmark for Rerankers: Examples for Relevance Categories 0 — the paragraph does not mention word order at all. In order to express ‘from’, these demon- strative members must take the ablative- 1 suffix (-ngomay), like all other adv...

    work page 1966

  8. [10]

    Conclusion:

    After the reasoning, output the word "Conclusion:" and the chosen option at the end of your response. Prompt for Standard Negation (partially based on GB107): Please determine if standard negation in the language <...> can be marked by a modification of the verb or an affix/clitic that is phonologically bound to the verb. The term "standard negation" refe...

    work page

Show all 13 references
  1. [12]

    Conclusion:

    After the reasoning, output the word "Conclusion:" and the chosen option at the end of your response. Prompt for Polar Questions (partially based on Grambank chapters related to strategies for marking polar questions): Please determine all possible strategies for forming polar...

  2. [14]

    Conclusion:

    After the reasoning, output the word "Conclusion:" and the chosen option at the end of your response. Prompt for Number of Cases (partially based on W ALS 49A): Please determine the number of cases in the language <...>. The term "cases" in the context of this feature refers t...

  3. [15]

    Provide the reasoning for the chosen option

  4. [16]

    Conclusion:

    After the reasoning, output the word "Conclusion:" and the chosen option at the end of your response. G Benchmark for RAG: Prompt Details G.1 Wikipedia Summaries Feature Wikipedia Title W ALS 81A Word order3 GB 107 Affirmation and negation4 W ALS 49A Grammatical case5 W ALS 11...

    2024

  5. [2014]

    In Proceedings of the 19th Australasian Document Computing Symposium, pages 58–65

    Improvements to bm25 and language models examined. In Proceedings of the 19th Australasian Document Computing Symposium, pages 58–65. Tasaku Tsunoda. 2012. A Grammar of Warrongo. De Gruyter Mouton, Berlin, Boston. Shafqat Mumtaz Virk, Lars Borin, Anju Saxena, and Harald Hammar...

    2012 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.