{"id":"b12501ef-41ba-41a2-aedc-b3d71d44bb7a","arxiv_id":"2508.16265","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A 97-language table QA benchmark built by LLM translation of 50 source tables claims median back-translation BLEU 60.19 and reports that synthetic QA data boosts low-resource model performance.","lead":"This paper builds a table question-answering benchmark that covers 97 languages by translating 50 English and Chinese tables with a six-step LLM pipeline, then annotates 2,916 question-answer pairs across four tasks. A generalist reader might care because multilingual table reasoning is barely tested, and the authors report that synthetic, unlabeled questions improve LLM performance in low-resource languages.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Median back-translation BLEU alone does not validate per-language semantic fidelity of translated tables and QA annotations.","rationale":"The reader's weakest assumption is precisely that LLM translation yields task-valid table-QA instances in all 97 languages, with only back-translation BLEU as support. My stress-test converges on the same point: the central claim of a new multilingual benchmark standard depends on per-language semantic correctness of tables and QA pairs, and the abstract's sole evidence is a median round-trip BLEU score, which is insufficient. This is not an internal inconsistency but a genuine validation gap. Since the full text was not available, no stronger verdict than UNVERDICTED is possible; the concern strengthens the case for withholding acceptance until per-language human evaluation is provided. I propose a concrete, feasible test that would settle the concern. No ad hominem or theatrical language is intended; the critique targets the argued connection between back-translation BLEU and benchmark validity, not the authors' integrity.","tokens_in":822,"tokens_out":2407,"duration_ms":25390,"concrete_test":"Select a stratified sample of at least 20 QA pairs per language (or all pairs for languages with fewer), across all 97 languages. For each sample, have two independent native-speaking annotators perform two checks: (1) Does the translated question preserve the source question's intent? (2) Given only the translated table, is the provided answer correct? Measure per-language error rates and inter-annotator agreement (e.g., Cohen's kappa). Pre-register a threshold (e.g., combined error rate ≤5% per language). If any language exceeds the threshold, the benchmark's claim of universal validity is not established. As a control, have annotators answer the translated questions directly from the translated tables without seeing source answers, and compare to the benchmark's annotated answers; low agreement in a language would indicate annotation–translation mismatch.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that m3TQA-Instruct provides a valid 97-language table-QA benchmark. For that to hold, each translated table must preserve the information needed to answer each question, and each translated QA pair must remain semantically consistent with the translated table. The only reported evidence in the abstract is a single median back-translation BLEU of 60.19. That metric measures round-trip surface similarity, not whether a question is answerable from the translated table or whether the answer's truth value is preserved. A median over 97 languages can hide severe per-language failures, especially for low-resource languages where LLM translation errors are more likely. If any language's translated tables or QA pairs contain semantic inconsistencies, model rankings on that language would be artifacts of translation quality rather than measurements of table reasoning. The load-bearing assumption—that the six-step pipeline yields task-valid instances in all 97 languages—is therefore not established by the reported evidence. This is not an attack on the pipeline; it is a request for per-language, task-level validation before the benchmark can be treated as a standard.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces m3TQA-Instruct, a massively multilingual table question answering benchmark covering 97 languages, constructed from 50 real-world Chinese/English tables via a six-step LLM-based translation pipeline (DeepSeek and GPT-4o). The benchmark contains 2,916 human-annotated QA pairs across four table reasoning tasks. The authors report a median back-translation BLEU of 60.19 as evidence of translation fidelity and use the benchmark to evaluate state-of-the-art LLMs, finding that synthetic unannotated QA data improves performance, especially for low-resource languages. The paper claims to establish a new standard for multilingual table understanding.","tokens_in":1081,"tokens_out":1326,"duration_ms":15838,"significance":"If the benchmark is validated, it would fill a clear gap: existing multilingual table QA benchmarks are geolinguistically imbalanced and limited in scale. A 97-language benchmark with per-language coverage of low-resource languages would be a valuable community resource, and the reported finding that synthetic QA data transfers across languages is potentially useful. The strength of the proposal is its ambition and the explicit attention to language diversity. However, the evidentiary basis presented in the abstract is too thin to support the central validity claim; the contribution is therefore conditional on substantial additional validation.","major_comments":[{"comment":"The central load-bearing claim is that m3TQA-Instruct is a 'validated' benchmark across 97 languages, but the only evidence provided is a single median back-translation BLEU of 60.19. Back-translation BLEU is a surface-level, round-trip similarity metric; it does not establish that each translated table preserves the information needed to answer its associated questions, nor that the translated QA pairs remain semantically consistent with the translated table. A median over 97 languages can conceal systematic failures in individual languages. The paper must provide per-language, task-level validation (e.g., human evaluation of a sample of QA pairs per language, or a consistency check that the golden answer is still derivable from the translated table) before the benchmark can be treated as a reliable evaluation platform.","section":"Abstract"},{"comment":"The scale of manual annotation appears disproportionate to the benchmark's stated coverage: 2,916 QA pairs across 97 languages yields roughly 30 pairs per language. Even ignoring translation fidelity, this sample size is too small to support reliable per-language evaluation of LLM table reasoning, particularly for low-resource languages where variance is high. The paper should report the per-language distribution and discuss the statistical power of comparisons across languages; if the intended use is cross-lingual analysis, the current density may be insufficient.","section":"Abstract"},{"comment":"The abstract conflates the benchmark name 'm3TQA-Instruct' with 'M3T-Bench' in the final claim ('M3T-Bench establishes a new standard'). It is unclear whether these are the same resource or different components. Clarify the relationship and ensure the naming is consistent, as this affects reproducibility and attribution.","section":"Abstract"}],"minor_comments":[{"comment":"The phrase 'high translation fidelity' is asserted before presenting the median BLEU score; BLEU is not a fidelity measure. Rephrase to 'surface similarity as measured by BLEU'.","section":"Abstract"},{"comment":"The six-step translation pipeline is not described at all. Since the method is a key contribution, a citation or a brief enumeration of the steps is needed even in the abstract.","section":"Abstract"},{"comment":"The abstract reports results on 'state-of-the-art LLMs' without naming them or giving the experimental setup. This makes the empirical insights difficult to assess.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The central idea is timely, but the current abstract-only evidence is insufficient to support the strong 'new standard' claim. The major revision should focus on adding per-language, task-level validation and clearly reporting the annotation distribution. If the full paper already contains such validation, the abstract must be rewritten to represent it accurately. I would not reject the paper outright, as the benchmark construction methodology could be sound, but the validation gap is load-bearing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is an abstract-only read, so take everything below as provisional. The paper's contribution is a 97-language table QA benchmark, m3TQA-Instruct, built by taking 50 English/Chinese tables and running them through a six-step LLM translation pipeline, then adding 2,916 professionally annotated QA pairs across four tasks. The headline finding is that synthetic unannotated QA data helps low-resource table QA. That's genuinely new in scope: no existing multilingual table benchmark covers 97 languages, and the translation methodology is a concrete artifact others can reuse. The empirical result, if it holds, is interesting.\n\nThe soft spots are in the validation. The abstract reports a single median back-translation BLEU of 60.19. That number does not tell you whether each translated table still contains the information needed to answer each question, or whether each QA pair's answer remains correct in the target language. A median across 97 languages can hide serious per-language failures. And 2,916 pairs across 97 languages is roughly 30 per language, which is thin for drawing per-language conclusions. The paper should release per-language, per-task translation quality checks, ideally including human evaluation or at least task-consistency checks where the answer is verified against the translated table.\n\nOne more minor concern: the abstract refers to both 'm3TQA' and 'M3T-Bench' as the benchmark name, which suggests a last-minute rename. Not a substantive issue, but the camera-ready should clean it up.\n\nOverall, I think this paper deserves peer review. The benchmark artifact is potentially valuable, the translation pipeline is a reusable methodology, and the empirical claim is testable. But the reviewers should push for per-language evidence before any 'standard' language is used.","headline":"A promising 97-language table QA benchmark whose translation validity is not yet established by the reported back-translation BLEU alone.","tokens_in":1567,"tokens_out":2027,"would_cite":false,"duration_ms":20904,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces a 97-language table question-answering benchmark built by translating 50 tables, claiming it sets a new standard for multilingual table understanding.","keywords":["table question answering","multilingual benchmark","language model evaluation","cross-lingual generalization","machine translation","low-resource languages","dataset construction"],"falsifier":"Take a random sample of, say, 20 QA pairs in each of several low-resource languages, have native speakers judge whether the translated question and table still support the annotated answer, and compare agreement with the source-language labels; if agreement drops well below what the back-translation BLEU of 60.19 suggests, the benchmark rankings are artifacts of translation fidelity.","tokens_in":760,"feed_emoji":"📊","tokens_out":4382,"duration_ms":41684,"temperature":0.7,"pith_summary":"This paper claims that multilingual table question answering can be tested at a scale and language coverage no prior benchmark offers. It introduces m3TQA-Instruct, a benchmark built by curating 50 real tables in English and Chinese and translating them into 97 languages with a six-step LLM translation pipeline. The benchmark contains 2,916 annotated question-answer pairs across four table-reasoning tasks. The authors report that such translated tables, combined with synthetic unannotated QA data, noticeably improve model performance in low-resource languages. If the benchmark holds up, the field gains a shared platform for measuring whether language models truly understand tables in nearly a hundred languages.","feed_headline":"97-language benchmark tests table reasoning at scale","feed_subtitle":"Built from 50 English and Chinese tables, it offers 2,916 QA pairs across four reasoning tasks.","key_machinery":"The load-bearing mechanism is m3TQA-Instruct, a benchmark generated by a six-step LLM-based translation pipeline. The pipeline takes 50 real tables in English and Chinese, translates them into 97 languages, and checks translation fidelity with back-translation (median BLEU 60.19). On top of the translated tables, 2,916 annotated QA pairs cover four table-reasoning tasks. The benchmark's role is to make cross-lingual table reasoning measurable by holding the tables constant and varying only language.","core_discovery":"The central claim is that a small set of source tables can be transformed into a massively multilingual evaluation set without losing task validity. The paper's discovery is a construction recipe: take 50 curated English and Chinese tables, run them through a six-step LLM translation pipeline, validate fidelity by back-translation (reporting a median BLEU of 60.19), and annotate 2,916 question-answer pairs over four tasks. Using this benchmark, the authors find that synthetically generated, unannotated QA data boosts performance on translated tables, with the largest gains in low-resource languages. They present m3TQA both as a challenging evaluation platform and as a scalable methodology fo","pith_inferences":["A stronger validation would replace round-trip BLEU with human judgment of meaning preservation in each target language; without that, scores on low-resource languages conflate translation quality with reasoning ability.","The same translation-based construction could be applied to neighboring structured-data tasks such as table-to-text generation or fact verification, giving them instant multilingual coverage.","Languages from different families may behave differently: those close to English or Chinese could inherit source-table biases, while distant languages provide a cleaner test of cross-lingual transfer."],"forward_implications":["Model rankings on m3TQA can now be compared across 97 languages, showing which LLMs generalize table reasoning beyond English and Chinese.","Synthetic unannotated QA data is a viable and scalable way to improve low-resource-language performance on table tasks.","The six-step translation pipeline can be reused to extend other table understanding tasks to many languages.","The benchmark's four-task design makes it possible to separate different reasoning abilities, such as lookup versus aggregation, across languages."],"supporting_citations":[],"fun_headline_variants":["97-language table QA benchmark from just 50 source tables","Synthetic QA data improves multilingual table reasoning","Table QA benchmark scales to 97 languages with 2,916 pairs","50 tables become 97-language QA benchmark","Multilingual table reasoning benchmark spans 97 languages"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The benchmark's validity depends on the assumption that translating 50 tables into 97 languages preserves the correctness of the 2,916 question-answer pairs, so a model's score reflects table reasoning and not translation errors.","fun_headline_variants_meta":{"raw":{"variants":["97-language table QA benchmark from just 50 source tables","Synthetic QA data improves multilingual table reasoning","Table QA benchmark scales to 97 languages with 2,916 pairs","50 tables become 97-language QA benchmark","Multilingual table reasoning benchmark spans 97 languages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000564,"raw_usage":{"total_tokens":2521,"prompt_tokens":765,"completion_tokens":1756,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":1680}},"tokens_in":509,"tokens_out":1756,"duration_ms":13520,"temperature":1.0,"reasoning_tokens":1680,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T17:24:20.287339+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 20 QA pairs in each of several low-resource languages, have native speakers judge whether the translated question and table still support the annotated answer, and compare agreement with the source-language labels; if agreement drops well below what the back-translation BLEU of 60.19 suggests, the benchmark rankings are artifacts of translation fidelity.","supporting_citations":[],"review_version":1}