Pith. sign in

REVIEW 4 major objections 6 minor 12 references

Multilingual LLMs Are Not Multilingual Thinkers: Evidence from Hindi Analogy Evaluation

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that multilingual LLMs answer Hindi analogy questions best when the instructions are in English, with Gemma-2-9B reaching 79.75% accuracy under a grounded English-only prompt.

desk verdict HATS is a genuinely useful new Hindi analogy benchmark, but the title's 'not multilingual thinkers' claim overreaches because the design doesn't isolate reasoning from instruction-following. read the letter →

arxiv 2507.13238 v2 pith:IGU4DV56 submitted 2025-07-17 cs.CL cs.AI

Ashray Gupta , Rohan Joseph , Sunny Rai This is my paper
classification cs.CLcs.AI
keywords HindianalogybenchmarkmultilingualLLMspromptlanguagechain-of-thoughtpromptingreasoningevaluationIndiclanguagesmultiple-choiceexamanalogiesHATS
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces HATS, a 405-question multiple-choice benchmark of semantic Hindi analogies drawn from Indian government exams, and uses it to ask whether multilingual LLMs reason in Hindi as well as they do in English. The central empirical claim is that three current multilingual models (Aya-expanse-8B, Llama-3.1-8B-instruct, Gemma-2-9B-it) answer the Hindi questions more accurately when the task instructions are in English rather than Hindi, across zero-shot, chain-of-thought, grounded, and few-shot prompting. The best score, 79.75%, comes from Gemma-2-9B with an English-only grounded chain-of-thought prompt. If the claim is right, current multilingual LLMs depend on English instruction-following even when the content is Hindi, and evaluating them in Hindi alone understates their performance. The paper also proposes a grounded chain-of-thought method that explicitly separates identifying the A:B relation, applying it to C, and checking the options, and reports that this method improves accuracy over plain zero-shot prompting.

What carries the argument

The central apparatus is the Hindi Analogy Test Set (HATS), a fixed set of 405 multiple-choice proportional analogies of the form $A : B :: C : ?$ taken from national and state-level Indian administrative exams, with each question carrying four original options. The argumentative load is carried by the language-of-instructions contrast: the same questions are presented with system and user prompts in Hindi-only, English-only, or mixed settings, and accuracy is aggregated by cyclic option rotation with majority voting to counter positional bias. The paper's proposed intervention is a grounded chain-of-thought prompt that guides the model through three explicit steps—identify the relationship between the first two terms, apply that relationship to the third term, and examine the options—based on a cognitive account of analogical reasoning as abduction, induction, and adequacy evaluation. This prompt template, together with the English-only setting, is what produces the paper's headline result.

What would settle it

Hold instruction-following constant by running the same HATS questions on models with verified near-perfect adherence to Hindi instruction format (checked independently of the analogy answer), then compare Hindi-prompted versus English-prompted accuracy; if the gap persists, the reasoning interpretation stands, and if it largely disappears, the gap is an artifact of instruction-following difficulty.

Watch

Extended reading notes

Core claim

The paper's claim, stated on its own terms, is that multilingual LLMs are not multilingual thinkers in the sense that their reasoning performance on Hindi analogies is best unlocked by English instructions. On the HATS benchmark, every model's top accuracy occurs in the English-only condition regardless of prompting strategy: for example, Gemma-2-9B improves from 73.08% (Hindi zero-shot) to 78.77% (English zero-shot) and from 75.31% (Hindi grounded 0-shot CoT) to 79.75% (English grounded 0-shot CoT). The paper further claims that its grounded chain-of-thought prompting, which structures the solution as identify the relation between the first pair, transfer it to the third term, then evaluate the options, yields consistent gains over plain zero-shot prompting. HATS is presented as a needed native-language resource because prior Hindi analogy datasets were translated from English and covered only syntactic relations, not the semantic relations that exam questions test.

Load-bearing premise

The load-bearing premise is that the higher accuracy under English prompts reflects better reasoning ability in English, rather than a difference in how well the models follow instructions or produce the required answer format in Hindi; the paper itself notes that some models struggled to follow Hindi instructions and occasionally responded in a language different from the prompt language.

Editorial extensions

If this is right

  • Multilingual LLMs' reported Hindi reasoning performance depends strongly on the language of the instructions, so evaluations that prompt only in Hindi will understate the models' actual ability on the same content.
  • The three-step grounded CoT prompt (relation identification, transfer, option evaluation) provides a reusable template that improves accuracy over plain zero-shot prompting, especially in English-only settings.
  • HATS offers a native, exam-sourced benchmark of semantic Hindi analogies, distinct from translated syntactic datasets, that can support future work on Indic-language reasoning.
  • The translation-based few-shot CoT, which solves the analogy in English and maps back to Hindi, nearly matches the best English-only performance (77.04% vs 79.75% for Gemma), suggesting translation can partially recover performance when Hindi prompting underperforms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The title's phrase 'not multilingual thinkers' is an interpretive leap: the experiment directly establishes an English-prompt advantage, but because instruction-following in Hindi was not controlled, the data do not isolate reasoning ability from format compliance.
  • A control experiment that filters to responses following the requested Hindi output format could separate these factors; if the gap persists after filtering, the reasoning story is supported.
  • The same scrape-and-benchmark recipe can be applied to other low-resource languages to test whether the English-prompt advantage generalizes across scripts and typologically different languages.
  • The grounded CoT benefit is model-specific: it raises Gemma's Hindi score but lowers Aya's, so the proposed mechanism is not a universal prompt fix.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces HATS, a 405-question multiple-choice analogy test set in Hindi sourced from Indian government exams, and uses it to benchmark three multilingual LLMs (Aya-Expanse-8B, Llama-3.1-8B-instruct, Gemma-2-9B-it). The authors evaluate several prompting strategies—zero-shot, zero-shot chain-of-thought, a grounded chain-of-thought method based on cognitive theories of analogy, few-shot grounded CoT, and a translation-based CoT—under Hindi-only, mixed, and English-only instruction settings. The main empirical claim is that English prompts yield the best accuracy regardless of prompting strategy; the paper also claims that the grounded CoT approach improves performance, and the title interprets the results as evidence that multilingual LLMs are 'not multilingual thinkers.' A separate direct-probability task (Task A) is also reported.

Significance. If the empirical pattern holds, HATS is a useful new resource for evaluating Hindi reasoning in LLMs, and the English-prompt advantage across several models is a noteworthy phenomenon for multilingual evaluation. The paper has concrete strengths: the benchmark is externally sourced with fixed gold answers, no model fitting or prompt optimization on the test set is performed, the cyclic rotation of answer options is a sound safeguard against positional bias, and the full prompts and test set are provided publicly. However, the central interpretive claim—that the results show models are 'not multilingual thinkers'—is not established by the reported experiments, because the accuracy metric conflates reasoning with instruction-following and output-format compliance, and because several quantitative claims in the abstract are not supported by the table entries or by significance testing.

major comments (4)
  1. [Title, Abstract, Section 3.6, Table A6] The headline claim that models are 'not multilingual thinkers' is not identified by the experimental design. The En+En versus Hi+Hi accuracy gap in Table 2 is interpreted as a reasoning gap, but Section 3.6 reports that 'some models struggled to follow instructions in Hindi,' and Table A6 shows response language diverging from prompt language in multiple conditions (e.g., Llama-3.1-8B-instruct responds in Hindi under En+En in 0-Shot, and aya-expanse-8B responds in Hindi under Hi+En in several conditions). Because scoring requires a valid final-answer format and majority voting over cyclically rotated variants, a model that solves an analogy but outputs a nonconforming or differently-languaged response is counted as incorrect. The English-prompt advantage could therefore reflect instruction-following or answer-format proficiency rather than analogical reasoning. The authors should report format-compliance rates and response-language conditional accuracies, and should re-analyze failures where the correct option is embedded in a nonconforming output, before making the 'not multilingual thinkers' claim.
  2. [Table 2 caption] The caption states that 'Accuracy is calculated only for valid analogies' but never defines 'valid' and never reports how many items were excluded per cell. If different cells drop different numbers of items, the headline comparisons are not over a fixed item set, and the observed English-prompt advantage could be an artifact of differential exclusion rates. The authors should define validity explicitly, report per-cell valid counts, and verify that the main conclusions hold when all 405 items are scored (with nonconforming outputs treated as incorrect, or with a lenient matching rule).
  3. [Abstract and Section 3.6, Table 2] The abstract's claim that models 'perform best with English prompts, irrespective of the prompting strategy' is contradicted by Table 2: for Llama-3.1-8B-instruct, 0-Shot CoT scores 67.40 in Hi+Hi but 66.91 in En+En. The claim should be qualified to 'usually' or 'on average,' and the counterexample should be discussed. Additionally, the abstract's claim that the grounded CoT approach 'improves model performance' rests on the 0.27-point average in Section 3.6, which is not significance-tested and is negative or zero in seven of the nine model-by-setting cells (e.g., Aya Hi+Hi drops from 62.71 to 60.74, and Llama Hi+Hi drops from 67.40 to 64.93). The authors should report confidence intervals or significance tests for the differences and should temper the improvement claim unless it is robust across models and settings.
  4. [Section 2 and Section 5] Because HATS is built from publicly available exam questions, there is a nontrivial contamination risk: the evaluated models may have memorized these or similar questions during pretraining, which would inflate absolute accuracy and could interact with prompt language if English prompts better retrieve memorized English content. The paper does not report any contamination check or n-gram overlap analysis with training corpora. This is especially relevant for the cross-language comparison, since the English-prompt advantage could in part reflect better retrieval of memorized answers under English instructions. The authors should add a contamination analysis or at least discuss this threat explicitly in the Limitations section.
minor comments (6)
  1. [Section 4] The text refers to 'aya–expanse–9b,' but the model evaluated throughout is Aya-Expanse-8B; please correct the model name.
  2. [Section 5] The conclusion says 'We designed five tasks,' but the paper describes Task A, Task B, and Task C with several prompting variants; please clarify what the five tasks are or revise the wording.
  3. [Section 2] The phrase '405 of in-situ semantic analogies' is awkward and should be rephrased, for example as '405 semantic analogies in their original exam format.'
  4. [Section 2] The citation to 'BeautifulSoup (Richardson, 2024)' should include the software version and a URL or persistent identifier, since it is a software tool rather than a peer-reviewed publication.
  5. [Appendix A.2.2, Table A6] The table reports response languages for some but not all settings (e.g., 'Hi+En' rows lack 0-Shot and 0-Shot CoT entries for all models), and the caption does not explain these missing cells; please state why those conditions were not measured or provide the missing data.
  6. [Table A4] In the En-En setting of Grounded Few-Shot CoT, the examples contain Hindi question text and Hindi option text within an otherwise English prompt; the paper should clarify that 'English-only' refers to the instruction language, not to the language of the analogy content, to avoid confusion about what is being varied.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the benchmark items are external exam questions with fixed gold answers, and no fitted parameter is renamed as a prediction.

full rationale

This paper reports an empirical benchmark evaluation rather than a derivation or predictive model. The HATS items are scraped from national and state-level administrative examinations with predetermined correct options, so accuracy is measured against an external gold standard. No parameter is fitted to a subset of the data and then 'predicted' on a closely related quantity; the prompting strategies, including the proposed grounded Chain-of-Thought, are fixed prompt templates applied to the same held-out exam questions. The only self-references (Joseph et al., 2023; Bogireddy et al., 2023) appear in the introduction as examples of figurative-text processing and are not load-bearing for the paper's claims. The few-shot exemplars in Section 3.5.3 were generated and curated by the authors, but they serve as illustrative in-context examples and are not derived from or fitted to the HATS test responses. The title's inference that models are 'not multilingual thinkers' is vulnerable to a construct-validity concern: Section 3.6 reports that 'some models struggled to follow instructions in Hindi,' and Table A6 shows response-language leakage, so the English-prompt advantage may partly reflect instruction-following rather than reasoning ability. That is a measurement-validity issue, not a circularity issue under the definitions used here. Likewise, the Table 2 caption's reference to 'valid analogies' is under-specified, but this affects reproducibility rather than circularity. The empirical comparison is self-contained against an external benchmark, so the appropriate circularity score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No fitted numbers appear in the paper; the only hand-chosen content is the set of few-shot exemplars embedded in the proposed prompts. The assumptions listed are the domain-level premises the title claim and the benchmark scores rest on. No new entities, forces, or parameters are postulated; the grounded chain-of-thought is a prompting strategy, not an entity.

free parameters (1)
  • hand-curated few-shot exemplars (5 per prompting setting)
    The Grounded FS-CoT and Translation-CoT prompts (Sections 3.5.3, 3.5.4) embed five worked examples generated by Claude-3.7-Sonnet and corrected by a Hindi expert; the examples materially shape the claimed few-shot gains, and the selection criteria are not reported.
assumptions (3)
  • domain assumption The 405 exam-sourced items are valid analogies with correct gold answers.
    The dataset was scraped from national and state exam materials and re-standardized by the authors (Section 2); no verification protocol is reported, and Table 2's caption excludes 'valid analogies' only, without defining validity or reporting exclusion counts.
  • domain assumption Prompt-language accuracy gaps isolate Hindi reasoning ability.
    Needed for the title claim. Section 3.6 reports models 'struggled to follow instructions in Hindi' and Table A6 shows output-language mismatches (e.g., Llama returning Hindi under En+En prompts), so the gap could reflect instruction-following rather than reasoning.
  • domain assumption Cyclically rotated majority voting removes positional bias without distorting accuracy.
    Section 3.3; assumes option order is the only positional effect and that n generated variants are exchangeable. No vote-distribution statistics are reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multilingual LLMs Are Not Multilingual Thinkers: Evidence from Hindi Analogy Evaluation." pith.science (2026). https://pith.science/paper/IGU4DV56

@misc{pith2026250713238,
  author       = {Pith},
  title        = {Pith review of: Multilingual LLMs Are Not Multilingual Thinkers: Evidence from Hindi Analogy Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IGU4DV56}},
  note         = {Machine review of arXiv:2507.13238}
}
read the original abstract

Analogies test a model's ability to infer implicit relationships between concepts, making them a key benchmark for evaluating reasoning capabilities. While large language models (LLMs) are widely evaluated for reasoning in English, their abilities in Indic languages remain understudied, limiting our understanding of whether these models generalize across languages. To address this gap, we introduce a new Hindi Analogy Test Set (HATS), comprising 405 multiple-choice questions sourced from Indian government exams. We benchmark state-of-the-art multilingual LLMs using various prompting strategies and introduce a grounded Chain of Thought approach that leverages cognitive theories of analogical reasoning. This approach improves model performance on Hindi analogy questions. Our experiments show that models perform best with English prompts, irrespective of the prompting strategy. Our test set addresses the lack of a critical resource to evaluate LLM reasoning capabilities in Hindi.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 11 canonical work pages

  1. [3]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou

    Plan-and-solve prompting: Improving zero- shot chain-of-thought reasoning by large language models. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompt- ingelicitsreasoninginlargelanguagemodels. arXiv preprint arXiv: 2201.11903. Thilini Wijesiriwardene, Ruwan Wickram...

    arXiv 2023

  2. [6]

    For each step, think carefully and consider multiple possible relationships before deciding

    Finally, examine each of the given options and select the one that best completes the analogy based on the relationship you identified. For each step, think carefully and consider multiple possible relationships before deciding. Be ex- plicitinyourreasoning. Presentyourfinalanswerintheformat: ###FinalAnswer: (X)option_text Now solve the following analogy ...

    work page

  3. [9]

    For each step, think carefully and consider multiple possible relationships before deciding

    Finally, examine each of the given options and select the one that best completes the analogy based on the relationship you identified. For each step, think carefully and consider multiple possible relationships before deciding. Be ex- plicitinyourreasoning. Presentyourfinalanswerintheformat: ###FinalAnswer: (X)option_text Now solve the following analogy ...

    work page

  4. [10]

    सबसेपहले,पहलेदोशब्दों( Aऔर B)के बीचके ɟवɡशȲसंबंधकोपहचानें।यहसमझेंɟक￿का￿सेक्या संबंधहै।

    work page

  5. [11]

    ɟफर,उसीसंबंधकोतीसरेशब्द( C)परलागूकरेंऔरदेखेंɟकचौथाशब्दक्याहोनाचाɟहए।

    work page

  6. [12]

    गंगा"और"नदʍ

    अंतमें,ɞदएगएɟवकल्पोंमेंसेउसɟवकल्पकाचयनकरेंजोआपके पहचानेगएसंबंधके आधारपरसमानता कोसहीतरीके सेपूराकरताहै। प्रत्येकचरणमेंसावधानीपूवर्कसोचेंऔरअंɟतमɟनणर्यलेनेसेपहलेकईसंभाɟवतसंबंधोंपरɟवचारकरें।अपनेतकर् कोȺȲरूपसेप्रस्तुतकरें।अपनेअंɟतमउȉरकोइसप्रारूपमेंदें: अंɟतमउȉर:( X)ɟवकल्पयहांकुछउदाहरणɞदएगएहैं:उदाहरण1: ɟनम्नɡलɤखतसमानताकोपूराकरें:गंगा:नदʍ::ɟहमालय:? (A)पवर्त (B)द...

    work page

  7. [15]

    गंगा" (Ganga) and

    Finally, examine each of the given options and select the one that best completes the analogy based on the relationship you identified. For each step, think carefully and consider multiple possible relationships before deciding. Be explicit in your reasoning. Present your final answer in the format: ###Final Answer: (X) option_text Here are some examples:...

    work page

  8. [16]

    Think about how A relates to B

    First, identify the specific relationship between the first two terms (A and B). Think about how A relates to B

    work page

Show all 12 references
  1. [17]

    Next, applythissamerelationshiptothethirdterm(C)todeterminewhatthefourthtermshould be

  2. [18]

    गंगा" (Ganga) and

    Finally, examine each of the given options and select the one that best completes the analogy based on the relationship you identified. For each step, think carefully and consider multiple possible relationships before deciding. Be explicit in your reasoning. Present your fina...

  3. [2022]

    arXiv preprint arXiv:2210.04186

    Analogy generation by prompting large lan- guage models: A case study of instructgpt. arXiv preprint arXiv:2210.04186. Neha Reddy Bogireddy, Smriti Suresh, and Sunny Rai

    arXiv

  4. [2023]

    In Companion Proceedings of the ACM Web Confer- ence 2023, pages 1004–1013

    I’m out of breath from laughing! i think? a dataset of covid-19 humor and its toxic variants. In Companion Proceedings of the ACM Web Confer- ence 2023, pages 1004–1013. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakanta...

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.