{"id":"d879e21c-3c14-4905-a2de-4c034b39c7a7","arxiv_id":"2507.13238","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A new 405-question Hindi analogy benchmark shows three multilingual LLMs scoring higher under English prompts than Hindi prompts, with the proposed grounded chain-of-thought prompt adding only 0.27 points on average.","lead":"The authors release HATS, a 405-question set of Hindi analogy problems taken from Indian government exams, and test three open-weights language models on it. They find the models answer more accurately when the instructions are in English than when they are in Hindi, even though the questions themselves stay in Hindi.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central 'not multilingual thinkers' inference is confounded with Hindi instruction-following and answer-format compliance, so the reported English-prompt advantage has not been shown to be a reasoning advantage.","rationale":"The reader's weakest_assumption is exactly the load-bearing concern I identify: the accuracy gap between English and Hindi prompts is interpreted as a difference in reasoning, but the experiment never isolates reasoning from instruction-following or output-format proficiency in Hindi. The reader's evidence is independently supported by the paper itself: Section 3.6 states models struggled with Hindi instructions, and Table A6 documents response-language mismatches. The additional 'valid analogies' language in the Table 2 caption is a further measurement risk because it can change the item set across cells, but the central interpretive problem is the instruction-following confound. I do not see an internal inconsistency that would overturn the resource or the raw per-cell pattern; the pattern that English prompts match or exceed Hindi prompts is likely reproducible. The fault is in the leap from that pattern to 'not multilingual thinkers.' Because the reader already conditions acceptance on this exact issue, my read does not change the verdict: CONDITIONAL remains appropriate, with the condition being that the title and abstract be revised to the supported claim, or the confound be removed by a format-controlled experiment. I therefore recommend UNCHANGED.","tokens_in":21890,"tokens_out":3052,"duration_ms":41709,"concrete_test":"Re-run the generation tasks with a language-neutral output protocol: (1) use all 405 HATS items with no invalid-item exclusions; (2) require the model to output only the option letter (A-D) under both Hi+Hi and En+En and parse with a strict regex; (3) have a bilingual annotator independently extract the intended option from free-form responses; and (4) compute accuracy as correct/405 with 95% bootstrap confidence intervals for every cell in Table 2. If the En+En minus Hi+Hi gap persists under this format-controlled, all-items protocol, the English-prompt superiority claim is robust to the instruction-following confound. If the gap shrinks or reverses, the title claim is unsupported and the paper should report only the narrower empirical pattern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim—that models are 'not multilingual thinkers'—requires the En+En versus Hi+Hi accuracy gap in Table 2 to measure Hindi analogical reasoning. The design does not establish this. Section 3.6 reports that 'some models struggled to follow instructions in Hindi,' and Table A6 shows response languages diverging from prompt languages: Llama-3.1-8B-instruct answers in Hindi even under En+En in the 0-Shot condition, while aya-expanse-8B stays Hindi under Hi+En. The evaluation pipeline scores answers only when they conform to an expected final-answer format and aggregates by majority voting; hence a model that solves the analogy but outputs a nonconforming or differently-languaged response is counted as wrong. The observed English advantage could therefore be an advantage in instruction-following and answer formatting rather than in relation-mapping. This is not a minor caveat: the paper's own evidence shows the key independent variable (prompt language) leaking into the response-language channel, which the accuracy metric cannot separate from reasoning. Additionally, the Table 2 caption says accuracy is 'calculated only for valid analogies' without defining validity or reporting per-cell exclusion counts, so the headline comparison may not even be over a fixed item set. The title claim rests on an assumption the paper's own data contradict rather than test.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces HATS, a 405-question multiple-choice analogy test set in Hindi sourced from Indian government exams, and uses it to benchmark three multilingual LLMs (Aya-Expanse-8B, Llama-3.1-8B-instruct, Gemma-2-9B-it). The authors evaluate several prompting strategies—zero-shot, zero-shot chain-of-thought, a grounded chain-of-thought method based on cognitive theories of analogy, few-shot grounded CoT, and a translation-based CoT—under Hindi-only, mixed, and English-only instruction settings. The main empirical claim is that English prompts yield the best accuracy regardless of prompting strategy; the paper also claims that the grounded CoT approach improves performance, and the title interprets the results as evidence that multilingual LLMs are 'not multilingual thinkers.' A separate direct-probability task (Task A) is also reported.","tokens_in":21968,"tokens_out":6267,"duration_ms":66704,"significance":"If the empirical pattern holds, HATS is a useful new resource for evaluating Hindi reasoning in LLMs, and the English-prompt advantage across several models is a noteworthy phenomenon for multilingual evaluation. The paper has concrete strengths: the benchmark is externally sourced with fixed gold answers, no model fitting or prompt optimization on the test set is performed, the cyclic rotation of answer options is a sound safeguard against positional bias, and the full prompts and test set are provided publicly. However, the central interpretive claim—that the results show models are 'not multilingual thinkers'—is not established by the reported experiments, because the accuracy metric conflates reasoning with instruction-following and output-format compliance, and because several quantitative claims in the abstract are not supported by the table entries or by significance testing.","major_comments":[{"comment":"The headline claim that models are 'not multilingual thinkers' is not identified by the experimental design. The En+En versus Hi+Hi accuracy gap in Table 2 is interpreted as a reasoning gap, but Section 3.6 reports that 'some models struggled to follow instructions in Hindi,' and Table A6 shows response language diverging from prompt language in multiple conditions (e.g., Llama-3.1-8B-instruct responds in Hindi under En+En in 0-Shot, and aya-expanse-8B responds in Hindi under Hi+En in several conditions). Because scoring requires a valid final-answer format and majority voting over cyclically rotated variants, a model that solves an analogy but outputs a nonconforming or differently-languaged response is counted as incorrect. The English-prompt advantage could therefore reflect instruction-following or answer-format proficiency rather than analogical reasoning. The authors should report format-compliance rates and response-language conditional accuracies, and should re-analyze failures where the correct option is embedded in a nonconforming output, before making the 'not multilingual thinkers' claim.","section":"Title, Abstract, Section 3.6, Table A6"},{"comment":"The caption states that 'Accuracy is calculated only for valid analogies' but never defines 'valid' and never reports how many items were excluded per cell. If different cells drop different numbers of items, the headline comparisons are not over a fixed item set, and the observed English-prompt advantage could be an artifact of differential exclusion rates. The authors should define validity explicitly, report per-cell valid counts, and verify that the main conclusions hold when all 405 items are scored (with nonconforming outputs treated as incorrect, or with a lenient matching rule).","section":"Table 2 caption"},{"comment":"The abstract's claim that models 'perform best with English prompts, irrespective of the prompting strategy' is contradicted by Table 2: for Llama-3.1-8B-instruct, 0-Shot CoT scores 67.40 in Hi+Hi but 66.91 in En+En. The claim should be qualified to 'usually' or 'on average,' and the counterexample should be discussed. Additionally, the abstract's claim that the grounded CoT approach 'improves model performance' rests on the 0.27-point average in Section 3.6, which is not significance-tested and is negative or zero in seven of the nine model-by-setting cells (e.g., Aya Hi+Hi drops from 62.71 to 60.74, and Llama Hi+Hi drops from 67.40 to 64.93). The authors should report confidence intervals or significance tests for the differences and should temper the improvement claim unless it is robust across models and settings.","section":"Abstract and Section 3.6, Table 2"},{"comment":"Because HATS is built from publicly available exam questions, there is a nontrivial contamination risk: the evaluated models may have memorized these or similar questions during pretraining, which would inflate absolute accuracy and could interact with prompt language if English prompts better retrieve memorized English content. The paper does not report any contamination check or n-gram overlap analysis with training corpora. This is especially relevant for the cross-language comparison, since the English-prompt advantage could in part reflect better retrieval of memorized answers under English instructions. The authors should add a contamination analysis or at least discuss this threat explicitly in the Limitations section.","section":"Section 2 and Section 5"}],"minor_comments":[{"comment":"The text refers to 'aya–expanse–9b,' but the model evaluated throughout is Aya-Expanse-8B; please correct the model name.","section":"Section 4"},{"comment":"The conclusion says 'We designed five tasks,' but the paper describes Task A, Task B, and Task C with several prompting variants; please clarify what the five tasks are or revise the wording.","section":"Section 5"},{"comment":"The phrase '405 of in-situ semantic analogies' is awkward and should be rephrased, for example as '405 semantic analogies in their original exam format.'","section":"Section 2"},{"comment":"The citation to 'BeautifulSoup (Richardson, 2024)' should include the software version and a URL or persistent identifier, since it is a software tool rather than a peer-reviewed publication.","section":"Section 2"},{"comment":"The table reports response languages for some but not all settings (e.g., 'Hi+En' rows lack 0-Shot and 0-Shot CoT entries for all models), and the caption does not explain these missing cells; please state why those conditions were not measured or provide the missing data.","section":"Appendix A.2.2, Table A6"},{"comment":"In the En-En setting of Grounded Few-Shot CoT, the examples contain Hindi question text and Hindi option text within an otherwise English prompt; the paper should clarify that 'English-only' refers to the instruction language, not to the language of the analogy content, to avoid confusion about what is being varied.","section":"Table A4"}],"recommendation":"major_revision","confidential_remarks":"The resource contribution is genuine and the benchmark is likely to be useful to the community. However, the paper's headline claims go beyond what the current experimental design can support. The confound between reasoning and instruction-following is particularly serious because the paper's own Table A6 shows the prompt-language manipulation leaking into the response-language channel. I would be willing to reconsider after the authors provide per-cell validity counts, response-language conditional accuracies, a contamination check, and significance testing for the main comparisons, and after the title and abstract are revised to match the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the HATS resource is real and worth having; the English-prompt advantage is visible and probably reproducible; the title claim and the abstract's grounded-CoT claim are both overstatements. If they fix the framing and a few reporting wobbles, this is a solid benchmark paper.\n\nWhat's new: HATS, 405 in-situ Hindi semantic analogies from Indian government exams. That's a genuine gap: existing Hindi analogy sets are translated and syntax-focused. I checked the examples; they're proper semantic analogies with fixed gold answers, so the benchmark is externally grounded, not a model-generated artifact. That alone justifies a look.\n\nThe paper also does something quietly useful: it reports the response-language divergence in Table A6. That's the kind of diagnostic that tells you the accuracy gap may not be a reasoning gap. The authors saw that some models answered in Hindi even under English prompts, and they didn't hide it. Credit where due.\n\nNow the soft spots.\n\nThe title says 'not multilingual thinkers.' The experiment doesn't establish that. The En-vs-Hi accuracy gap is plausibly an instruction-following and output-format gap. The paper's own Section 3.6 says some models struggled to follow Hindi instructions, and Table A6 shows Llama-3.1-8B-instruct answering in Hindi under En+En zero-shot. When the response language doesn't track the prompt language, the accuracy metric conflates reasoning with compliance. The title needs to be softened.\n\nThe abstract's grounded-CoT improvement claim rests on a 0.27-point average with no significance test, and Table 2 shows it going negative for Aya and Llama in the Hindi-only condition. That claim should either be dropped or supported with proper statistics.\n\nAlso, the Table 2 caption says accuracy is computed 'only for valid analogies' without defining validity or reporting per-cell exclusions. That needs clarification; otherwise the headline comparison might not be over a fixed item set.\n\nMinor but real: the paper says the mixed En+Hi setting was 'not evaluated separately' yet Table 2 reports En+Hi rows; and it refers to 'aya-expanse-9b' in Section 4 when the model is 8B. These are fixable.\n\nWho's this for? People working on multilingual evaluation and Indic NLP. They'll get value from HATS and from the prompt-language effect, even if the interpretation needs work.\n\nFor peer review: yes, a serious referee should engage with it. The resource is valuable, the main empirical pattern is visible, and the flaws are in framing and reporting, not in the core data. My verdict: revise and resubmit with the claims scaled back.","headline":"HATS is a genuinely useful new Hindi analogy benchmark, but the title's 'not multilingual thinkers' claim overreaches because the design doesn't isolate reasoning from instruction-following.","tokens_in":22683,"tokens_out":2554,"would_cite":true,"duration_ms":27488,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that multilingual LLMs answer Hindi analogy questions best when the instructions are in English, with Gemma-2-9B reaching 79.75% accuracy under a grounded English-only prompt.","keywords":["Hindi analogy benchmark","multilingual LLMs","prompt language","chain-of-thought prompting","reasoning evaluation","Indic languages","multiple-choice exam analogies","HATS"],"falsifier":"Hold instruction-following constant by running the same HATS questions on models with verified near-perfect adherence to Hindi instruction format (checked independently of the analogy answer), then compare Hindi-prompted versus English-prompted accuracy; if the gap persists, the reasoning interpretation stands, and if it largely disappears, the gap is an artifact of instruction-following difficulty.","tokens_in":21510,"feed_emoji":"🧠","tokens_out":10314,"duration_ms":95336,"temperature":0.7,"pith_summary":"The paper introduces HATS, a 405-question multiple-choice benchmark of semantic Hindi analogies drawn from Indian government exams, and uses it to ask whether multilingual LLMs reason in Hindi as well as they do in English. The central empirical claim is that three current multilingual models (Aya-expanse-8B, Llama-3.1-8B-instruct, Gemma-2-9B-it) answer the Hindi questions more accurately when the task instructions are in English rather than Hindi, across zero-shot, chain-of-thought, grounded, and few-shot prompting. The best score, 79.75%, comes from Gemma-2-9B with an English-only grounded chain-of-thought prompt. If the claim is right, current multilingual LLMs depend on English instruction-following even when the content is Hindi, and evaluating them in Hindi alone understates their performance. The paper also proposes a grounded chain-of-thought method that explicitly separates identifying the A:B relation, applying it to C, and checking the options, and reports that this method improves accuracy over plain zero-shot prompting.","feed_headline":"LLMs score higher on Hindi analogies when prompted in English","feed_subtitle":"A new 405-question exam-based benchmark shows the gap across zero-shot, CoT, grounded, and few-shot prompts.","key_machinery":"The central apparatus is the Hindi Analogy Test Set (HATS), a fixed set of 405 multiple-choice proportional analogies of the form $A : B :: C : ?$ taken from national and state-level Indian administrative exams, with each question carrying four original options. The argumentative load is carried by the language-of-instructions contrast: the same questions are presented with system and user prompts in Hindi-only, English-only, or mixed settings, and accuracy is aggregated by cyclic option rotation with majority voting to counter positional bias. The paper's proposed intervention is a grounded chain-of-thought prompt that guides the model through three explicit steps—identify the relationship between the first two terms, apply that relationship to the third term, and examine the options—based on a cognitive account of analogical reasoning as abduction, induction, and adequacy evaluation. This prompt template, together with the English-only setting, is what produces the paper's headline result.","core_discovery":"The paper's claim, stated on its own terms, is that multilingual LLMs are not multilingual thinkers in the sense that their reasoning performance on Hindi analogies is best unlocked by English instructions. On the HATS benchmark, every model's top accuracy occurs in the English-only condition regardless of prompting strategy: for example, Gemma-2-9B improves from 73.08% (Hindi zero-shot) to 78.77% (English zero-shot) and from 75.31% (Hindi grounded 0-shot CoT) to 79.75% (English grounded 0-shot CoT). The paper further claims that its grounded chain-of-thought prompting, which structures the solution as identify the relation between the first pair, transfer it to the third term, then evaluate the options, yields consistent gains over plain zero-shot prompting. HATS is presented as a needed native-language resource because prior Hindi analogy datasets were translated from English and covered only syntactic relations, not the semantic relations that exam questions test.","pith_inferences":["The title's phrase 'not multilingual thinkers' is an interpretive leap: the experiment directly establishes an English-prompt advantage, but because instruction-following in Hindi was not controlled, the data do not isolate reasoning ability from format compliance.","A control experiment that filters to responses following the requested Hindi output format could separate these factors; if the gap persists after filtering, the reasoning story is supported.","The same scrape-and-benchmark recipe can be applied to other low-resource languages to test whether the English-prompt advantage generalizes across scripts and typologically different languages.","The grounded CoT benefit is model-specific: it raises Gemma's Hindi score but lowers Aya's, so the proposed mechanism is not a universal prompt fix."],"forward_implications":["Multilingual LLMs' reported Hindi reasoning performance depends strongly on the language of the instructions, so evaluations that prompt only in Hindi will understate the models' actual ability on the same content.","The three-step grounded CoT prompt (relation identification, transfer, option evaluation) provides a reusable template that improves accuracy over plain zero-shot prompting, especially in English-only settings.","HATS offers a native, exam-sourced benchmark of semantic Hindi analogies, distinct from translated syntactic datasets, that can support future work on Indic-language reasoning.","The translation-based few-shot CoT, which solves the analogy in English and maps back to Hindi, nearly matches the best English-only performance (77.04% vs 79.75% for Gemma), suggesting translation can partially recover performance when Hindi prompting underperforms."],"supporting_citations":[{"why":"Supplies the abduction–induction–adequacy framework used to build the grounded CoT steps.","marker":"Minnameier, 2010"},{"why":"Provides the Plan-and-Solve prompting approach that the grounded CoT extends with analogy-specific steps.","marker":"Wang et al., 2023"},{"why":"Defines the zero-shot chain-of-thought baseline tested in Task C.","marker":"Kojima et al., 2023"},{"why":"One of the three multilingual models benchmarked on HATS (Aya-expanse-8B).","marker":"Dang et al., 2024"},{"why":"One of the three multilingual models benchmarked on HATS (Llama-3.1-8B).","marker":"Grattafiori et al., 2024"},{"why":"One of the three multilingual models benchmarked on HATS (Gemma-2-9B), which produced the top score.","marker":"Team et al., 2024"},{"why":"Establishes the few-shot learning paradigm used in the few-shot CoT settings.","marker":"Brown et al., 2020"},{"why":"Presents the prior multilingual analogy dataset with syntactic relations that HATS contrasts with.","marker":"Abdou et al., 2018"}],"fun_headline_variants":["Hindi analogies? Prompt in English for best LLM scores","LLMs think in English even for Hindi analogies","New Hindi benchmark shows LLMs favor English prompts","Multilingual LLMs best at Hindi analogies in English","LLM reasoning on Hindi analogies peaks with English prompts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the higher accuracy under English prompts reflects better reasoning ability in English, rather than a difference in how well the models follow instructions or produce the required answer format in Hindi; the paper itself notes that some models struggled to follow Hindi instructions and occasionally responded in a language different from the prompt language.","fun_headline_variants_meta":{"raw":{"variants":["Hindi analogies? Prompt in English for best LLM scores","LLMs think in English even for Hindi analogies","New Hindi benchmark shows LLMs favor English prompts","Multilingual LLMs best at Hindi analogies in English","LLM reasoning on Hindi analogies peaks with English prompts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000579,"raw_usage":{"total_tokens":2703,"prompt_tokens":891,"completion_tokens":1812,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":1733}},"tokens_in":507,"tokens_out":1812,"duration_ms":13849,"temperature":1.0,"reasoning_tokens":1733,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:27:44.689561+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold instruction-following constant by running the same HATS questions on models with verified near-perfect adherence to Hindi instruction format (checked independently of the analogy answer), then compare Hindi-prompted versus English-prompted accuracy; if the gap persists, the reasoning interpretation stands, and if it largely disappears, the gap is an artifact of instruction-following difficulty.","supporting_citations":[],"review_version":1}