Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

OmniTQA compiles natural-language table questions into mixed relational-and-LLM operator graphs so latent attributes hidden in free-form text can be recovered during execution, lifting hybrid accuracy by 14 points on average.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 13:48 UTC pith:3BDSMIKO

load-bearing objection Clean operator-DAG system for hybrid table QA that really narrows the structured/semi-structured gap; the 14-point lift holds under judge consolidation, with planning quality as the real soft spot. the 3 major comments →

arxiv 2604.02444 v2 pith:3BDSMIKO submitted 2026-04-02 cs.DB

From Textual Columns to Query Plans: A Unified Relational-Semantic Execution Framework for Hybrid Query Processing

Nima Shahbazi , Seiji Maekawa , Nikita Bhutani , Estevam Hruschka This is my paper
classification cs.DB
keywords table question answeringsemi-structured datahybrid query processingsemantic operatorsText-to-SQLquery plan diversificationLLM database systemsdual-engine execution
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Real tables mix clean columns with free-form text that silently holds the attributes, predicates, or join keys a question needs. Pure Text-to-SQL systems cannot see those latent pieces; pure LLM methods can read the text but collapse on long or multi-table data. OmniTQA treats semantic recovery as ordinary operators inside a relational plan: it builds directed acyclic graphs of SQL-style steps and LLM steps, diversifies those plans to handle ambiguity, optimizes them to push cheap filters first, and runs them on a dual engine. The result is a single framework that works on both fully structured and semi-structured tables, closing most of the accuracy gap that appears when schema information is buried in text.

Core claim

The paper establishes that hybrid table question answering—where some query-relevant structure lives only inside free-form columns—can be solved by compiling each natural-language question into a DAG of atomic relational operators and LLM-based semantic operators, then diversifying, cost-optimizing, and dual-engine-executing those graphs. On semi-structured benchmarks this yields a 14-point average accuracy gain over the strongest baseline (27 points on the hardest multi-table long subset) while remaining competitive on fully structured data.

What carries the argument

Relational-semantic DAGs: each node is either a classical relational operator (scan, filter, join, aggregate, …) or a semantic operator (LLM map, filter, join, aggregate) that recovers latent structure from text; plan diversification generates K alternative groundings, a cost-aware optimizer reorders and defers expensive LLM steps, and a dual-engine runtime routes nodes to SQL or batched LLM execution.

Load-bearing premise

That an LLM planner given only a compact, query-aware data preview can reliably invent a diverse set of correct operator DAGs that recover the right latent schema elements.

What would settle it

On the multi-table long semi-structured subset of REPAIRTQA, if the full OmniTQA pipeline (K=6 diversified plans + LLM judge) fails to outperform the strongest baseline by a double-digit accuracy margin, or if removing diversification and the hybrid operator vocabulary leaves accuracy essentially unchanged, the central claim is false.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes OmniTQA, a unified hybrid table QA framework that compiles natural-language questions into DAGs mixing classical relational operators (ΩR) with LLM-based semantic operators (ΩS). Planning includes schema grounding from a query-aware data preview, parallel plan diversification along schema-mapping/risk/operator/intent axes, and cost-aware rewriting (selection pushdown, projection pruning, join reordering, adaptive semantic deferral). A dual-engine runtime routes relational steps to SQL and semantic steps to batched LLM execution (map/filter, block nested-loop join, recursive aggregate), then consolidates K candidate answers via LLM-as-a-Judge or majority vote. On RepairTQA structured/semi-structured pairs, plus FeTaQA, HybridQA, TAT-QA and conventional WikiSQL/WikiTQ, OmniTQA reports large gains on hybrid settings (about +14 accuracy points on average and +27 on multi-table long under the reported judge setting) while remaining competitive on fully structured data, with ablations of preview, operators, optimization, diversification, and pruning, plus token-cost measurements.

Significance. If the empirical claims hold under realistic single-answer consolidation, this is a meaningful systems contribution: it reframes latent-attribute recovery as first-class operators inside a relational execution model rather than a coarse pipeline, and it brings classical optimization ideas (pushdown, reordering, deferral) into hybrid LLM–SQL plans. The controlled RepairTQA structured vs. semi-structured design, multi-LLM checks, operator ablations, and dual-engine batching are genuine strengths for the hybrid TQA literature. The work is more engineering/systems than theoretical, but the operator DAG abstraction and diversification strategy are reusable design patterns for semi-structured query processing.

major comments (3)
  1. §4 Settings and §4.1 / Figure 3: The abstract’s +14 / +27 headline is the paper’s central claim, yet ACC@6 is explicitly an upper bound under ideal consolidation while LLM-as-a-Judge is the realistic selector. The multi-table long 77% vs 50% sentence sits in the same paragraph as the judge discussion but does not state which metric it uses, and the main text does not quantify the ACC@6–judge–ACC@1–majority-vote gaps on that subset (Appendix C.2 has related plots but not a clear isolation of the +27 case). Because diversification is load-bearing (Table 2; Limitations), please report, in the main results, judge accuracy, ACC@1, and majority vote side-by-side with ACC@6 for the multi-table long semi-structured slice, and restate the abstract numbers only for the single-answer consolidation mode you intend as the primary metric.
  2. §4.5 Figure 7 and Limitations: Planning (incomplete projections, misaligned semantic literals, wrong operator choice) dominates system-attributable errors, and the paper itself calls diverse high-quality plan generation a bottleneck. The weakest load-bearing assumption is that an LLM planner with only a compact preview (semantic top-k + random samples) plus a fixed operator vocabulary can recover latent schema structure reliably enough for the dual-engine story. Please either (i) strengthen the main-text analysis of when diversification actually supplies a correct plan that the judge selects, or (ii) temper the abstract claim of robust latent-structure recovery to match the planning-error distribution, and discuss failure modes of the preview-based grounding more systematically (e.g., long-table multi-hop predicates).
  3. §4.1 baselines and Table 1: Hybrid baselines H-STAR and WEAVER are described as not supporting multi-table / semi-structured latent attributes natively, so the largest multi-table long gap is partly against systems outside their design envelope; Direct-LLM is the fairer peer there. Please make this explicit when interpreting the +27 point subset (e.g., primary comparison to Direct-LLM and Text-to-SQL, with hybrid pipelines as secondary), and avoid implying that all hybrid pipelines fail for the same reason OmniTQA succeeds.
minor comments (6)
  1. Figure 2 is rendered as a dense block of placeholder glyphs in the manuscript text; ensure the architecture figure is legible in the camera-ready PDF with readable operator labels for the UEFA example.
  2. §3.2 / Appendix A.3: The cost model (weights w_sys, w_llm, threshold ε, cardinality estimates) is presented as a decision aid; a short empirical check that Adaptive Semantic Deferral actually reduces tokens without harming accuracy on a held-out slice would strengthen §4.3–4.4 beyond the ‘w/o Opt’ ablation.
  3. Table 2 / Table 3: ‘Composional’ appears to be a typo for ‘Compositional’; fix consistently.
  4. §2 and Table 5: Operator names in the main text (MAP/FILTER/JOIN) vs. prompts (LLM_DERIVE/LLM_FILTER) should be aligned once for readers implementing the system.
  5. Evaluation uses an LLM judge for free-form answers (§4); briefly state judge model, prompt stability, and agreement with exact match on structured subsets to reduce evaluator-bias concerns.
  6. Hyperparameters K=6 and β=100 are free parameters with sensitivity in Appendix C.3; mention default selection criteria in the main experimental settings paragraph.

Circularity Check

0 steps flagged

No circularity: empirical systems paper whose accuracy claims are measured against held-out gold labels, not forced by definition or self-citation.

full rationale

OmniTQA is an engineering framework (relational-semantic DAGs, plan diversification, dual-engine execution, heuristic optimizer). Its central claims are empirical accuracy and cost numbers on REPAIRTQA and external semi-structured/structured TQA benchmarks (FETAQA, HYBRIDQA, TAT-QA, WikiSQL, WikiTableQuestions). Those numbers are produced by executing plans and comparing outputs to gold annotations via an LLM judge; they are not obtained by fitting a parameter that is then renamed a prediction, nor by defining the metric in terms of the method. The cost model (Appendix A.3) and Adaptive Semantic Deferral heuristics are approximate decision aids for operator placement; they do not underwrite the accuracy headline. Overlap of authors with REPAIRTQA (Zhang, Maekawa, Bhutani) makes that dataset a self-cited benchmark, but gold labels remain external evaluation targets and the paper also reports external datasets—so the citation is not load-bearing for a uniqueness or derivation claim. ACC@6 is explicitly labeled an upper bound under ideal consolidation; optimistic multi-plan scoring is an evaluation-methodology concern, not circularity by construction. No self-definitional loop, fitted-input-as-prediction, uniqueness theorem imported from the authors, or ansatz smuggled via citation appears in the derivation chain.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The central empirical claim rests on standard LLM-as-planner/executor assumptions, a fixed operator vocabulary, hand-chosen diversification factor and batch size, and the RepairTQA-style construction that attributes can be moved into free-text columns while gold answers remain well-defined. No new physical entities are postulated; free parameters are system hyperparameters.

free parameters (4)
  • plan diversification factor K = 6
    Default K=6; accuracy rises from K=1 to K=2 then plateaus; cost scales linearly. Chosen by sensitivity analysis, not derived.
  • execution block size β = 100
    Default β=100; larger β reduces cost but can hurt accuracy via longer contexts. Empirically tuned.
  • cost-model weights and deferral threshold ε
    wsys, wllm, ccall, α, ε used for adaptive semantic deferral; calibrated on synthetic/API measurements rather than proved optimal.
  • preview sample sizes k1, k2
    Semantic top-k plus random samples that ground the planner; exact sizes are implementation choices affecting long-table accuracy.
axioms (4)
  • domain assumption LLMs can act as reliable atomic semantic operators (MAP/FILTER/JOIN/AGGREGATE) when given short instructions and batched rows, and as planners that emit valid operator DAGs from schema+preview.
    Core of the dual-engine design (§3); failure modes in planning dominate error analysis.
  • domain assumption Hybrid schemas can be partitioned into structured attributes As and unstructured Au such that latent query-relevant structure is recoverable from Au at query time.
    Data model in §2; underpins the problem formalization and RepairTQA evaluation.
  • domain assumption Classic relational rewrites (selection pushdown, projection pruning, join reordering) plus adaptive semantic deferral preserve answer semantics while reducing LLM cost.
    Plan optimization invariants (§3.2, Algorithm 1); ablation shows cost drop without accuracy change.
  • standard math Standard relational algebra and SQL execution semantics for ΩR operators.
    Assumed background for dual-engine routing.
invented entities (1)
  • OmniTQA relational-semantic operator DAG (ΩR ∪ ΩS with diversification dimensions) no independent evidence
    purpose: Unified intermediate representation that interleaves symbolic and LLM operators and enables diversification/optimization/dual-engine execution.
    The paper's main architectural invention; independent evidence is the empirical accuracy/cost results, not an external physical measurement.

pith-pipeline@v1.1.0-grok45 · 25696 in / 3027 out tokens · 26697 ms · 2026-07-13T13:48:32.843136+00:00 · methodology

0 comments
read the original abstract

Real-world table question answering often involves hybrid schemas in which some query-relevant information is explicit in relational columns, while other attributes, predicates, or join conditions are only implicit in free-form text. Existing systems struggle with this setting: Text-to-SQL methods scale to large and multi-table databases but require fully structured schemas, whereas direct LLM-based methods can interpret textual content but are costly and unreliable when applied to large databases. We present OmniTQA, a unified framework for semi-structured table question answering that treats semantic reasoning as a first-class operation within relational query execution. OmniTQA compiles natural-language questions into directed acyclic graphs of relational and LLM-based semantic operators. This enables ambiguity-aware plan diversification, cost-aware optimization, and dual-engine execution over structured and textual data. Across structured and semi-structured benchmarks, OmniTQA consistently improves performance in hybrid settings, outperforming the strongest baselines by 14 accuracy points on average and by 27 points on the most challenging subset, while maintaining competitive accuracy on fully structured datasets.

Figures

Figures reproduced from arXiv: 2604.02444 by Estevam Hruschka, Nikita Bhutani, Nima Shahbazi, Seiji Maekawa.

Figure 1
Figure 1. Figure 1: Two instances of NCAA Soccer database with distinct schema representations: (left) structured; (right) semi-structured. attributes are well-defined and apply semantic operators only to interpret or match values within individual columns. They are less suited for scenarios where a single textual field conflates multiple latent attributes and relational references that would otherwise be represented as separ… view at source ↗
Figure 2
Figure 2. Figure 2: End-to-end overview of the OmniTQA framework [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of the OmniTQA planning phase for the UEFA Soccer database and the query “Did the player who achieved the UEFA Men’s Player of the Year 2021 win the UCL championship?”. OmniTQA first constructs a query-aware data preview 𝑅ˆQ (shown in blue) to ground natural-language intents to schema attributes. The planner then generates and optimizes multiple candidate logical plans to resolve the schema am… view at source ↗
Figure 4
Figure 4. Figure 4: Accuracy comparison of OmniTQA vs. baselines evaluated with Gemini-3-Flash. RepairTQA-S4 RepairTQA-S3 Dataset 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy % [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Accuracy compar￾ison: GPT-5-Mini. RepairTQA-S4 RepairTQA-S3 Dataset 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy % [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 10
Figure 10. Figure 10: Varying no. of plans 𝐾 vs. accuracy. 10 100 1000 Batch Size ( ) 0.5 0.6 0.7 0.8 0.9 1.0 Accuracy@K RepairTQA-S4 RepairTQA-S5 RepairTQA-M2 [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Large Databases Need Small, Open-Weight Language Models

    cs.AI 2026-06 unverdicted novelty 4.0

    Quantized open-weight LMs on consumer hardware match closed-source API accuracy for LM-enhanced relational operators while delivering 390x lower cost and 3.8x lower latency in the BlendSQL framework.

Reference graph

Works this paper leans on

49 extracted references · 3 linked inside Pith · cited by 1 Pith paper

  1. [1]

    In 2021 IEEE 37th International Conference on Data Engineering (ICDE), pages 984–995

    Memory-efficient key/foreign-key join size estimation via multiplicity and intersection size. In 2021 IEEE 37th International Conference on Data Engineering (ICDE), pages 984–995. IEEE. Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, and 1 others. 2022. Fetaqa: Free- form table question answering.Transactions of the Association for C...

    2021

  2. [2]

    P Griffiths Selinger, Morton M Astrahan, Donald D Chamberlin, Raymond A Lorie, and Thomas G Price

    Semantic operators and their optimization: Enabling llm-based data processing with accuracy guarantees in lotus.Proceedings of the VLDB En- dowment, 18(11):4171–4184. P Griffiths Selinger, Morton M Astrahan, Donald D Chamberlin, Raymond A Lorie, and Thomas G Price

  3. [3]

    InProceedings of the 1979 ACM SIGMOD international conference on Manage- ment of data, pages 23–34

    Access path selection in a relational database management system. InProceedings of the 1979 ACM SIGMOD international conference on Manage- ment of data, pages 23–34. Yunxiang Su, Tianjing Zeng, Zhongjun Ding, Yin Lin, Rong Zhu, Zhewei Wei, Bolin Ding, and Jingren Zhou. 2026. Large language model-enhanced rela- tional operators: Taxonomy, benchmark, and an...

    Pith/arXiv arXiv 1979

  4. [4]

    Peiying Yu, Guoxin Chen, and Jingjing Wang

    Qwen3 technical report.arXiv preprint arXiv:2505.09388. Peiying Yu, Guoxin Chen, and Jingjing Wang. 2025. Table-critic: A multi-agent framework for collabora- tive criticism and refinement in table reasoning. In Proceedings of the 63rd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 17432–17451. Weixu Zhang...

    Pith/arXiv arXiv 2025

  5. [5]

    CoRR, abs/1709.00103

    Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103. Wei Zhou, Bolei Ma, Annemarie Friedrich, and Mohsen Mesgar. 2025. Table question answering in the era of large language models: A comprehensive survey of tasks, methods, and evaluation.arXiv preprint arXiv:2510.09671. Fengbin Zhu, Wenqiang Lei,...

    Pith/arXiv arXiv 2025

  6. [6]

    Return rows from�

    to account for the asymmetry between inex- pensive relational operations and expensive seman- tic reasoning in a deterministic manner. Specifi- cally, we adopt predicate migration as acost-aware semantic placementstrategy. While semantic op- erators are deferred to later stages to minimize un- necessary LLM calls, the optimizer evaluates the cardinality i...

    2017

  7. [7]

    If a column is a Primary Key or Foreign Key, KEEP IT

  8. [8]

    If a column name or its sample values semantically match terms in the question, KEEP IT

  9. [9]

    recent",

    If the question implies a time frame (e.g., "recent", "trend", "when"), KEEP date/timestamp columns

  10. [10]

    If you are 50/50 split on whether a column is relevant, KEEP IT

  11. [11]

    Only exclude a column if you are certain it is noise. ### TABLE ### {table} ### QUESTION ### {question} ### Available Columns ### (Name (Type): [Samples]): {context_str} Task: Return a JSON list of strings containing the columns relevant to the question according to the High-Recall Protocol. Output strictly valid JSON. Planning Prompt: Decomposition: Syst...

  12. [12]

    Return rows from [Table_Name]

    SCAN: "Return rows from [Table_Name]"

  13. [13]

    Return rows from [Previous_Step_ID] where [Column_Name_1] [Operator] [Value/Column_Name_2]

    FILTER: "Return rows from [Previous_Step_ID] where [Column_Name_1] [Operator] [Value/Column_Name_2]"

  14. [14]

    Return [Column_Names] of [Previous_Step_ID], calculating [Expression] if needed

    PROJECT: "Return [Column_Names] of [Previous_Step_ID], calculating [Expression] if needed"

  15. [15]

    Return [Agg_Func] of [Target_Column] grouped by [Grouping_Column] from [Previous_Step_ID]

    AGGREGATE: "Return [Agg_Func] of [Target_Column] grouped by [Grouping_Column] from [Previous_Step_ID]"

  16. [16]

    Return [Previous_Step_ID] sorted by [Column_Name] [ASC/DESC]

    SORT: "Return [Previous_Step_ID] sorted by [Column_Name] [ASC/DESC]"

  17. [17]

    Return the top [N] rows from [Previous_Step_ID]

    LIMIT: "Return the top [N] rows from [Previous_Step_ID]"

  18. [18]

    Return combined rows from [Previous_Step_ID_1] and [ Previous_Step_ID_2] where [Join_Condition] matches

    JOIN: "Return combined rows from [Previous_Step_ID_1] and [ Previous_Step_ID_2] where [Join_Condition] matches"

  19. [19]

    Return the [Union/Intersection/Difference] of [Previous_Step_ID_1] and [Previous_Step_ID_2]

    SET_OP: "Return the [Union/Intersection/Difference] of [Previous_Step_ID_1] and [Previous_Step_ID_2]"

  20. [20]

    Return unique rows from [Previous_Step_ID] based on [Column_Names]

    DISTINCT: "Return unique rows from [Previous_Step_ID] based on [Column_Names]" --- II. Semantic Operators ---

  21. [21]

    Return [Previous_Step_ID] with new column [New_Column_Name] derived from [Input_Columns] by [Instruction]

    LLM_DERIVE: "Return [Previous_Step_ID] with new column [New_Column_Name] derived from [Input_Columns] by [Instruction]"

  22. [22]

    Return rows from [Previous_Step_ID] satisfying the semantic condition : [Instruction]

    LLM_FILTER: "Return rows from [Previous_Step_ID] satisfying the semantic condition : [Instruction]"

  23. [23]

    Return combined rows from [Previous_Step_ID_1] and [ Previous_Step_ID_2] using semantic matching logic: [Instruction]

    LLM_JOIN: "Return combined rows from [Previous_Step_ID_1] and [ Previous_Step_ID_2] using semantic matching logic: [Instruction]"

  24. [24]

    Return a summary of [Target_Column] grouped by [Grouping_Column] from [Previous_Step_ID] using instruction: [Instruction]

    LLM_AGGREGATE: "Return a summary of [Target_Column] grouped by [Grouping_Column] from [Previous_Step_ID] using instruction: [Instruction]" LEGEND: [Table_Name]: Exact name from Schema [Column_Name]: Exact column from Schema [Previous_Step_ID]: The ’id’ of a step generated earlier [Agg_Func]: max, min, count, sum, avg [Operator]: !=, =, >, <, >=, <=, conta...

  25. [25]

    **Atomic Decomposition:** Each step must correspond to exactly one atomic operation from the list

  26. [26]

    **Schema Fidelity:** You must use the EXACT column and table names provided in the schema

  27. [27]

    Semantically cross-reference user terms with values in <data_preview>

    **Value Inspection:** Do not rely solely on column names. Semantically cross-reference user terms with values in <data_preview>

  28. [28]

    **Dependency Graph:** The ‘parent‘ field must list the IDs of immediate predecessors

  29. [29]

    plans": [ {{

    **Output Format:** Return ONLY a raw JSON object containing a list of plans. ### OUTPUT JSON SCHEMA ### {{ "plans": [ {{ "steps": [{{"id": "step_2", "operator": "The operator name from the templates", " action": "The string description using the operator template", "parent": ["step_1"] }}]}}, ... (up to {k} plans)] }} Planning Prompt: Decomposition: User ...

  30. [30]

    The available tables are: {tables}

  31. [31]

    One step --> one SQL

  32. [32]

    so it forms a result table

    If you create an output scalar or boolean, still return it as a SELECT ... so it forms a result table

  33. [33]

    Name any new column via AS. ### INPUT ### ### TABLE SCHEMAS ### {schema} ### TABLE PREVIEWS ### {preview_rows} ### ATOMIC STEP ### {step} Execution Prompt: Semantic Executor for FILTER, MAP, and AGGREGATE: System Prompt Semantic Executor for FILTER & MAP & AGGREGATE: System Prompt You are an expert data-transformation and relational reasoning engine speci...

  34. [34]

    Execute semantic data transformations on structured tabular data

  35. [35]

    Return results in STRICT JSON format only

  36. [36]

    rows": [ {

    Preserve data integrity and handle edge cases gracefully ### CRITICAL OUTPUT FORMAT RULES ### Your response must contain ONLY valid JSON with NO additional text, explanation, or Markdown. For MAP and JOIN operations: { "rows": [ { "col1": value1, "col2": value2, ... }, { "col1": value3, "col2": value4, ... } ] } For FILTER operations: [0, 2, 5, ...] // 0-...

  37. [37]

    Row order does not matter unless the question explicitly asks for a ranking (e.g., "top 10")

    Order Sensitivity: Treat the results as SETS. Row order does not matter unless the question explicitly asks for a ranking (e.g., "top 10")

  38. [38]

    1,000" vs

    Formatting: Ignore differences in formatting (e.g., "1,000" vs "1000", "$50" vs "50", "2023-01-01" vs "Jan 1, 2023")

    2023

  39. [39]

    Data Types: JSON objects, lists of tuples, and CSV strings should be compared based on content, not syntax

  40. [40]

    The answer is 50

    Conversational Filler: If the prediction contains extra text (e.g., "The answer is 50"), extract the value "50" and compare it

  41. [41]

    ### GROUND-TRUTH ### {ground_truth} ### PREDICTION ### {prediction} ### GUIDELINES ### Think step-by-step:

    Column Names: Ignore column name differences (aliases) unless the user specifically asked for a specific column name. ### GROUND-TRUTH ### {ground_truth} ### PREDICTION ### {prediction} ### GUIDELINES ### Think step-by-step:

  42. [42]

    Analyze the content of both the Ground Truth and Prediction

  43. [43]

    Identify if there are differences in ordering, formatting, or wrapper text

  44. [44]

    reasoning

    Determine if they convey the exact same data/information. ### OUTPUT ### - "reasoning": A brief string explaining why they match or differ. - "verdict": "CORRECT" or "INCORRECT". Consolidation Prompt: Majority V ote Majority Vote: Prompt You are an expert aggregator. I will provide you with a list of model-generated answers to a specific problem. Some ans...

  45. [45]

    Group the predictions that represent the same semantic answer/conclusion

  46. [46]

    Identify which group has the most members (the plurality)

  47. [47]

    If a group has empty or unclear answers, ignore those

  48. [48]

    If there is a tie for the largest group, choose any one of them

  49. [49]

    No explanation

    Return ONLY the final answer from that majority group. No explanation. Consolidation Prompt: LLM-as-a-Judge V ote LLM-as-a-Judge Vote: Prompt You are an expert judge evaluating different reasoning paths for the following question about a database. Your task is to select the plan that demonstrates the most logical, accurate, and complete reasoning: ### Tas...