{"id":"f785de42-86af-4b23-9357-24091a8d4ee9","arxiv_id":"2412.11711","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"MiMoTable is a real-world spreadsheet benchmark with 1,719 bilingual question-answer pairs and a meta-operation difficulty criterion on which the best LLM scores 77.4%.","lead":"This paper introduces MiMoTable, a bilingual benchmark of 1,719 spreadsheet question-answer pairs built from 428 real-world Excel files with varied headers, sheets, files, and tables. It also proposes a six-category meta operation difficulty score and reports that current LLMs, best Claude-3.5-Sonnet at 77.4%, still have room to improve.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The meta-operation criterion's effectiveness is not established: the monotonic accuracy drop across three datasets is confounded by dataset differences and GPT-4o-generated labels, and no within-benchmark control for table difficulty is reported.","rationale":"I agree with the reader that the difficulty labels lack human validation and are a weak point, but I see the more fundamental issue as the absence of any controlled test of the criterion's incremental validity. The three-point between-dataset correlation in Figure 10 is confounded by answer format, language, and table structure, and the within-benchmark question-difficulty analysis in Table 6 is not stratified by table difficulty. The proposed regression directly tests whether the meta-operation difficulty score predicts per-question accuracy once structural table difficulty and language are held constant; if the effect disappears, the criterion is not 'proven effective' as claimed. This is a falsifiable check that the authors can run on data they have already collected. The benchmark itself may still be a useful resource, and the experimental results indicating that current LLMs struggle with it are plausible, so I do not recommend rejection. A conditional acceptance requiring this validation, or a revised claim, is appropriate. Since the reader already assigned CONDITIONAL and my concern does not change that verdict, I mark the verdict as unchanged.","tokens_in":15475,"tokens_out":5022,"duration_ms":51215,"concrete_test":"Using the released per-sample predictions, fit a mixed-effects logistic regression: correctness ~ qsi + table_difficulty (simple/medium/hard) + language + model, with a random intercept per spreadsheet. If the coefficient on qsi is not significantly negative (p < 0.05) after controlling for table difficulty and language, then the meta-operation difficulty score adds no incremental predictive validity beyond structural table difficulty, and the abstract's 'proving effectiveness' claim should be downgraded. This test uses only the MiMoTable benchmark itself and avoids the cross-dataset confounds in Figure 10.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims that the meta-operation difficulty criterion is 'proven effective' because LLM accuracy declines as benchmark difficulty increases (WikiSQL 1.5, WikiTableQuestion 2.0, MiMoTable-Simple 2.2; Figure 10). This claim rests on two insecure supports. First, the difficulty scores for WikiSQL and WikiTableQuestion are produced by GPT-4o classifying each question into meta operations using the prompt in Table 8, with no reported human validation; MiMoTable-Simple's score uses the same GPT-4o-generated labels from construction. If the labels are noisy or systematically biased toward the generating model, the x-axis values in Figure 10 are not reliable measures of difficulty. Second, the comparison is a dataset-level correlation with only three points, and the datasets differ in many uncontrolled ways: answer format (WikiSQL's short SQL-derived answers vs. WikiTableQuestion's free-form vs. MiMoTable's mixed open-ended answers), language, table size, and header complexity. The observed monotonic drop could therefore reflect these confounds rather than the criterion's validity. The paper also reports within-MiMoTable accuracy by question-difficulty bins in Table 6, but these bins are not stratified by table difficulty; since hard tables likely contain harder questions, the drop from 89.0% to 63.3% for Claude-3.5-Sonnet could be driven by table structure rather than meta-operation complexity. Without a controlled test, the central effectiveness claim is unsupported, and the benchmark's value as a resource does not rescue the criterion's validation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MiMoTable, a bilingual (Chinese/English) spreadsheet reasoning benchmark built from 428 real-world spreadsheets and 1,719 (spreadsheet, question, answer) triplets. The benchmark covers four table-reasoning task types and a wide range of table structures, including simple and complex headers, multiple sheets, multiple files, and multiple tables per sheet. The authors also propose a new difficulty criterion based on six meta operations (Lookup, Edit, Calculate, Compare, Visualize, Reasoning), assign each operation a grade, and derive question-level and dataset-level difficulty scores. They evaluate 16 LLMs and report that Claude-3.5-Sonnet achieves the best accuracy of 77.4%. Finally, they grade WikiSQL and WikiTableQuestion with the same criterion and claim that the resulting difficulty ordering is validated by the observed drop in LLM accuracy across the three datasets.","tokens_in":15744,"tokens_out":7272,"duration_ms":65385,"significance":"If the benchmark and the difficulty criterion are validated, MiMoTable is a useful resource: it targets real-world spreadsheet complexity that most existing table benchmarks lack, it spans multiple task types and languages, and it is open-sourced. The construction pipeline is described in detail, answers were human-annotated with a reported Cohen's Kappa of 0.83, and the best-model accuracy of 77.4% indicates headroom for current LLMs. The meta-operation difficulty criterion is an interesting and potentially generalizable analytical tool. However, the evidence for the criterion's effectiveness is currently much weaker than the abstract's 'proven effectiveness' claim: the external validation rests on three dataset-level points with model-generated difficulty labels, and the within-benchmark difficulty results are not controlled for table difficulty. These issues are addressable and do not undermine the value of the dataset itself.","major_comments":[{"comment":"The claim that the meta-operation difficulty criterion is 'proven effective' rests on Figure 10, which shows a monotonic accuracy drop across only three datasets: WikiSQL (difficulty 1.5), WikiTableQuestion (2.0), and MiMoTable-Simple (2.2). The x-axis values are computed by applying the GPT-4o classifier in Table 8 to existing benchmarks, with no reported human validation of those classifications. Because the same model family is later evaluated, the difficulty labels and model accuracies are not independent. Moreover, the three datasets differ in answer format, table provenance, and question distribution, so the observed drop could reflect these confounds rather than the criterion's validity. Please provide human-validated difficulty labels and a controlled comparison, for example by scoring questions from a single dataset or by matching questions across table-difficulty strata.","section":"§3.2, Figure 10"},{"comment":"Table 6 reports accuracy by question-difficulty bins without stratifying by table difficulty. For Claude-3.5-Sonnet, accuracy falls from 89.0% on questions with difficulty [1,2) to 63.3% on [3,4], while accuracy on hard tables is 72.1%. If hard tables are overrepresented among high-difficulty questions, the drop could be explained by table structure rather than by meta-operation complexity. Please report accuracy within table-difficulty strata, or a regression with table difficulty as a covariate, to support the claim that question difficulty as defined by meta operations is the driving factor.","section":"§3.2, Table 6"},{"comment":"All accuracy numbers in Table 6 depend on GPT-4o judging whether candidate answers are correct. The prompt in Table 10 allows a 0-1 score for open-ended questions, but the paper does not state how these scores are converted into binary accuracy, nor does it report agreement between the GPT-4o judge and human annotators. Since GPT-4o also generated the questions and is one of the evaluated models, this is a potential source of bias. Please specify the conversion rule and report a human-validated sample of judge decisions, for example at least 100 items across models and difficulty levels.","section":"§3.1, Table 10"},{"comment":"The reported 77.4% accuracy is computed only on questions without the Edit and Visualize meta operations, because most evaluated models cannot produce edited files or charts. The paper does not state how many of the 1,719 questions are excluded. The headline result therefore does not characterize performance on the full benchmark. Please report the size of the evaluated subset, the accuracy of GPT-4o-CI on the full set, and, if possible, an evaluation of the full benchmark using format-appropriate metrics for file and chart outputs.","section":"§3.2, Table 6"}],"minor_comments":[{"comment":"The phrase 'proving the effectiveness' overstates the evidence presented; consider 'providing evidence for' or 'supporting' until the validation concerns are addressed.","section":"Abstract and §3.2"},{"comment":"Equation (3) uses M_msqi before its values are specified in Eq. (5); reorder the presentation and state the assumption that each meta operation can appear at most once in a question before introducing the formula.","section":"§2.2, Eq. (3)"},{"comment":"For every model, the accuracy under 'Question Difficulty [1,2)' is identical to the accuracy under 'Lookup'; this suggests the two subsets coincide or that a column is mislabeled, so please clarify the subset definitions.","section":"Table 6"},{"comment":"The multi-file construction is described as 'randomly sampled' groups of 2-5 files, but the paper does not report how many such groups were created or whether files are reused; please add this statistic for reproducibility and to rule out cross-question leakage.","section":"§2.3"},{"comment":"The term 'multi-scale' in the benchmark name is not defined; please state explicitly that it refers to variation in sheets, files, tables per sheet, and header complexity.","section":"§1"},{"comment":"The radar chart overlays many models and is difficult to read in print; consider separate panels or a table for the per-meta-operation accuracies.","section":"Figure 7"}],"recommendation":"major_revision","confidential_remarks":"The dataset itself is a solid resource and the paper is within scope for the journal. The main revision target should be the validation of the meta-operation difficulty criterion: the current evidence is too weak for the abstract's 'proven effectiveness' claim. I see no grounds for rejection, but the authors should be asked to provide controlled, human-validated evidence rather than merely softening the wording."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Zheng,\n\nMiMoTable is worth knowing about. It is a new bilingual spreadsheet reasoning benchmark: 428 real-world Excel files across seven domains, with simple and complex headers, single and multiple sheets, multiple files, and multiple tables per sheet. It spans four task families (TableQA, Table2Text, Table Manipulation, Advanced Data Analysis), which no cited benchmark combines. The construction pipeline is described in enough detail to be reproducible: GPT-4o generates questions and candidate answers, code is debugged, answers are voted, and experts annotate with a reported Cohen's Kappa of 0.83. The best LLM (Claude-3.5-Sonnet) gets 77.4%, so the dataset is genuinely challenging. If the data and code ship as promised, this is a useful evaluation resource.\n\nThe soft spot is the second contribution, the meta-operation difficulty criterion. The paper claims the criterion is 'proven effective' because accuracy on WikiSQL (difficulty 1.5), WikiTableQuestion (2.0), and MiMoTable-Simple (2.2) declines monotonically. That evidence is weaker than the claim. The difficulty labels for the existing benchmarks are assigned by GPT-4o using the prompt in Table 8, with no reported human validation of those classifications. The three datasets differ in many ways — answer format, language, table construction — so the monotonic drop could reflect those confounds rather than the criterion. Within MiMoTable, the question-difficulty bins in Table 6 are not stratified by table difficulty, so the drop from 89.0% to 63.3% could be driven by harder tables containing harder questions. No error bars, no significance tests. The criterion may be a useful heuristic — it is simple and face-valid — but 'proven' is too strong.\n\nNone of this breaks the benchmark. The resource stands on its own. I'd advise a serious referee to ask for (a) human validation of a sample of the meta-operation labels, (b) a controlled within-benchmark test that holds table difficulty fixed while varying meta-operation difficulty, and (c) toned-down claims. With those changes, this is a solid dataset paper.\n\nBring it to reading group if you care about tabular reasoning evaluation. I'd cite it once the data is out.","headline":"Useful new bilingual spreadsheet benchmark; the meta-operation difficulty criterion is a good idea but the 'proven effective' claim overreaches its evidence.","tokens_in":16284,"tokens_out":2089,"would_cite":true,"duration_ms":18982,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper builds a real-world spreadsheet benchmark, MiMoTable, and claims that the best LLM gets 77.4% accuracy, with a new meta-operation difficulty score that explains why harder benchmarks yield lower accuracy.","keywords":["spreadsheet reasoning","table reasoning benchmark","large language models","meta operations","difficulty scoring","real-world spreadsheets","question answering","table manipulation"],"falsifier":"Hand-label a random sample of 200 questions from WikiSQL and WikiTableQuestions into the six meta operations and recompute the dataset difficulty scores; if the scores no longer sit at 1.5 and 2.0, or the accuracy drop in Figure 10 disappears with human-verified labels, the criterion's predictive claim fails.","tokens_in":15239,"feed_emoji":"📊","tokens_out":6723,"duration_ms":52498,"temperature":0.7,"pith_summary":"This paper aims to close the gap between table-reasoning benchmarks and real-world spreadsheet use by introducing MiMoTable, a dataset of 428 genuine Excel files and 1,719 question-answer pairs spanning seven domains, both languages, and four table-reasoning tasks. It also proposes a new way to measure question difficulty: each question is tagged with one or more of six meta operations (Lookup, Edit, Compare, Calculate, Visualize, Reasoning), and a formula converts the tags into a difficulty score. The paper's key evidence is that the best tested LLM, Claude-3.5-Sonnet, scores only 77.4% accuracy on MiMoTable, and that on existing benchmarks, model accuracy falls as the meta-operation difficulty score rises (WikiSQL 1.5, WikiTableQuestion 2.0, MiMoTable-Simple 2.2). If this holds, the meta-operation score becomes a practical tool for predicting LLM performance on any table dataset before running a single model, and MiMoTable stands as a harder, more realistic test bed.","feed_headline":"Real-world spreadsheets still stump LLMs: best score 77%","feed_subtitle":"A new benchmark built from real Excel files shows even the strongest models fail a quarter of table tasks.","key_machinery":"The central object is the taxonomy of six meta operations, each with a difficulty grade: Lookup and Edit are grade 1, Compare, Calculate, and Visualize are grade 2, and Reasoning is grade 3. A question's difficulty score is computed as $q_{si} = m_{s_i} + \\frac{\\sum_{k=1}^{K_i} s_k - m_{s_i}}{M_{m_{s_i}}}$, where $m_{s_i}$ is the maximum grade among the question's operations and $M_{m_{s_i}}$ caps the sum of the remaining grades (1 for grade 1, 6 for grade 2, 8 for grade 3), giving a score in [1,4]. A dataset's difficulty is the average of its question scores. This score is what lets the paper grade existing benchmarks like WikiSQL and WikiTableQuestions on the same scale and compare them against measured LLM accuracy.","core_discovery":"The paper proposes that current table-reasoning benchmarks miss real-world spreadsheet complexity, and that a new meta-operation difficulty criterion can place benchmarks from different tasks on a common scale. It validates this by showing that LLM accuracy declines monotonically as the meta-operation difficulty score rises from WikiSQL (1.5) to WikiTableQuestions (2.0) to MiMoTable-Simple (2.2), and that the best model, Claude-3.5-Sonnet, reaches only 77.4% on MiMoTable, with performance dropping to 63.3% on Reasoning questions and 72.1% on hard tables.","pith_inferences":["A human-annotation study of the meta-operation labels on WikiSQL and WikiTableQuestions would test whether the 1.5 and 2.0 difficulty scores are stable; if labels shift, the claimed monotonic trend may be a GPT-4o artifact.","The meta-operation taxonomy could be used to build difficulty-controlled subsets of a benchmark, letting researchers evaluate models on a curriculum from Lookup-only to Reasoning-heavy questions.","The same scoring scheme could be extended to other structured-data formats, such as databases or forms, to create comparable difficulty metrics across data modalities.","The paper's limitation section notes no supervised fine-tuning was performed; fine-tuning models per meta operation would reveal which operations are learnable and whether 77.4% is a data ceiling or a model ceiling."],"forward_implications":["MiMoTable provides a more realistic stress test for LLMs, since it includes real spreadsheets with complex headers, multiple sheets, multiple files, and multiple tables per sheet.","The meta-operation difficulty score gives a common scale for comparing benchmarks across different table tasks, so a score of 2.0 on one dataset can be interpreted as equally hard as the same score on another.","Model rankings on WikiSQL and WikiTableQuestions can be explained by difficulty rather than by dataset-specific quirks, and the same reasoning can be applied to any new table benchmark.","The 77.4% ceiling quantifies remaining headroom, and the per-operation scores identify which capabilities need the most work, such as Reasoning at 63.3% for the best model.","For simple and medium tables, a code-interpreter approach beats a text-based approach on Calculate and Compare, while text-based is better on hard tables and on Lookup and Reasoning, suggesting future systems should route by table difficulty and operation type."],"supporting_citations":[{"why":"Supplies the WikiTableQuestions baseline whose higher difficulty (2.0) explains lower accuracy than WikiSQL.","marker":"(Pasupat and Liang, 2015)"},{"why":"Supplies the WikiSQL baseline whose low difficulty (1.5) anchors the accuracy-difficulty trend.","marker":"(Zhong et al., 2017)"},{"why":"Provides Tablellama, a tabular-model baseline showing the benchmark is hard for table-specialized models.","marker":"(Zhang et al., 2023)"},{"why":"The GPT-4 model family used for question generation, answer generation, meta-operation labeling, and answer judging.","marker":"(OpenAI, 2023)"}],"fun_headline_variants":["LLMs hit 77% ceiling on real spreadsheet tests","Real spreadsheet tasks expose LLM limits: 77% top score","New benchmark: even Claude bests only 77% on real tables","Meta-operations rank table tasks: LLMs slide with difficulty","Spreadsheet smarts: LLMs trail at 77% on real-world data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evidence that the difficulty criterion works assumes that GPT-4o's meta-operation labels for existing benchmarks are correct, and the paper reports no human check of those labels.","fun_headline_variants_meta":{"raw":{"variants":["LLMs hit 77% ceiling on real spreadsheet tests","Real spreadsheet tasks expose LLM limits: 77% top score","New benchmark: even Claude bests only 77% on real tables","Meta-operations rank table tasks: LLMs slide with difficulty","Spreadsheet smarts: LLMs trail at 77% on real-world data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000641,"raw_usage":{"total_tokens":2931,"prompt_tokens":908,"completion_tokens":2023,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":1931}},"tokens_in":524,"tokens_out":2023,"duration_ms":13724,"temperature":1.0,"reasoning_tokens":1931,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:39:11.598878+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hand-label a random sample of 200 questions from WikiSQL and WikiTableQuestions into the six meta operations and recompute the dataset difficulty scores; if the scores no longer sit at 1.5 and 2.0, or the accuracy drop in Figure 10 disappears with human-verified labels, the criterion's predictive claim fails.","supporting_citations":[],"review_version":1}