Pith. sign in

REVIEW 3 major objections 4 minor 29 references

ExpliCIT-QA: Explainable Code-Based Image Table Question Answering

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that visual table question answering can be made fully auditable by pairing every answer with a human-readable reasoning trace and the runnable code that computed it.

desk verdict Honest system paper with a clear code-based TableVQA pipeline, but the transparency payoff is unmeasured and the paper itself concedes the explanation layer is not strictly tied to the executed code. read the letter →

arxiv 2507.11694 v1 pith:BGMHZGDM submitted 2025-07-15 cs.CL cs.AI

Maximiliano Hormazábal Lagos , Álvaro Bueno Sáez , Pedro Alonso Doval , Jorge Alcalde Vesteiro , Héctor Cerezo-Costas This is my paper
classification cs.CLcs.AI
keywords visualquestionansweringtableimageschain-of-thoughtreasoningcode-aidedexplainabilityauditabilityPandascodegeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents ExpliCIT-QA, a five-stage pipeline for answering questions about tables that appear as images rather than as text. The pipeline first turns the table image into a structured, CSV-like table using vision-language reasoning, then produces a human-readable step-by-step reasoning trace, converts that trace into executable Python/Pandas code, runs the code deterministically, and finally writes a natural-language explanation based on the code that was executed. The central claim is that this design makes answers auditable: every intermediate artifact, including the parsed table, the reasoning steps, the generated code, and the final explanation, is available for inspection. If the claim is right, users in finance, healthcare, and other regulated settings could verify how a table-image answer was computed instead of accepting a black-box model's output. The paper evaluates the pipeline on a four-domain table-image QA benchmark and reports accuracy comparable to small-model baselines, with interpretability as the main benefit.

What carries the argument

The central mechanism is code-as-specification: the natural-language reasoning trace is converted into executable Python/Pandas operations, and the deterministic execution of that code, not the model's next-token prediction, produces the answer. The pipeline is held together by a vision-language model that first reconstructs a normalized table from the image, fuzzy-match helper functions that correct slight categorical mismatches, and an error-feedback loop that lets the code generator revise failed scripts up to three times against the runtime error message. This substitutes a formal, auditable computation for the hidden internal reasoning of end-to-end models.

What would settle it

Run the pipeline on a sample of benchmark questions, translate each final explanation into a pseudo-code checklist, and compare that checklist against the operations actually present in the executed Python script; if any nontrivial fraction of explanations mentions filters, sums, or lookups that the code never performs, the promised auditability is not delivered.

Watch

Extended reading notes

Core claim

The discovery is that a table-image question can be answered through a chain of inspectable artifacts rather than by one opaque model. In this design the answer is not a language model's guess: the question and extracted table drive a step-by-step reasoning trace, the trace is translated into Pandas code, a Python interpreter computes the answer, and the final explanation is generated from the executed code so that it describes the actual computation. The paper argues that this closes the explainability gap in end-to-end visual question answering over table images, because a human auditor can follow every link from image to answer. The main cost is accuracy: the pipeline trails state-of-the-art vision-to-text pipelines on the benchmark, and the authors identify table-image understanding, not code generation, as the current bottleneck.

Load-bearing premise

The transparency guarantee depends on the natural-language explanation genuinely describing the operations in the code that computed the answer; the paper notes that complete linkage between the code and the explanation is not strictly enforced.

Editorial extensions

If this is right

  • Every answer can be audited end to end: the parsed table, reasoning steps, generated code, and final explanation are all exposed for inspection.
  • Arithmetic and logical mistakes shift to the Python interpreter, so the reported answer is the output of a deterministic computation rather than a sampled guess.
  • The reasoning and code-generation modules are swappable, so the pipeline can be run with smaller models on limited hardware or larger models when accuracy matters more.
  • The same code-based explanation format transfers to regulated domains such as finance and healthcare, where documenting how an answer was derived is part of the requirement.
  • The accuracy bottleneck is the table-understanding stage: complex layouts with merged cells and multi-row headers, and fact-verification questions, are where the pipeline currently loses the most.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, one could build an automated auditor that regenerates the code for each question, re-executes it, and flags any mismatch between the explanation and the code; the paper claims inspectability, not automatic verification.
  • The code-based answer representation suggests a natural port to SQL or spreadsheet formulas, which would make explanations legible to business users who do not read Python.
  • On the fact-verification subset, where the paper reports near-failing accuracy, an extension might combine code computation with a separate semantic-alignment step, since pure deterministic operations appear ill-suited to that task.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. ExpliCIT-QA is a five-stage pipeline for visual table question answering: it uses Qwen-2.5-VL to convert a table image into a CSV-like structure, Qwen-3 to produce a chain-of-thought reasoning trace and Python/Pandas code, executes the code with error feedback, and then uses a third-party code-explainer model to generate a natural-language explanation. The paper reports accuracy on the four TableVQA-Bench subsets for three Qwen-3 sizes and compares them with GPT-4V+GPT-4 and Gemini-Pro baselines, concluding that the system trades some accuracy for interpretability and traceability.

Significance. The contribution is a concrete, modular, code-available system that produces inspectable intermediate artifacts for every answer, and the paper shows that a small open-weights model stack can operate on table images. The authors are explicit about the accuracy/interpretability trade-off and make no overclaim about state-of-the-art accuracy. However, the paper's central value proposition—that the system 'provides human-readable CoT explanations and runnable code for every answer, addressing the ambiguity of end-to-end model explanations'—is not measured. There is no fidelity check, user study, or any quantitative evidence that the final natural-language explanation corresponds to the executed computation. Because the paper itself declares that full code-explanation linkage is unnecessary and defers fidelity measurement to future work, the interpretability claim remains an assertion rather than a demonstrated result.

major comments (3)
  1. [§3.5, §6 (with §1)] The central contribution stated in §1 is that ExpliCIT-QA 'provides human-readable CoT explanations and runnable code for every answer, addressing the ambiguity of end-to-end model explanations.' This requires that the natural-language explanation faithfully describe the operations that actually produced the answer. However, §3.5 states that 'it is not strictly necessary that the code is always fully linked to the explanation,' and §6 lists 'the internal coherence of the CoT and the fidelity of the generated explanations' as a future-work priority. The final explanation is produced by a separate pretrained code-explainer, so it can in principle omit, add, or misdescribe executed operations. Since no experiment or metric in the paper verifies explanation-code fidelity, the transparency guarantee that motivates the system is not established. Please either add a fidelity evaluation (e.g., human annotation or automatic comparison of explanation statements against AST-level code operations) or restrict the claim to auditability of the code, which is a distinct and valid contribution.
  2. [§4, Table 2] The evaluation reports accuracy for each configuration as a single number with no error bars, no number of runs, and no information about sampling temperature or seed. The pipeline includes stochastic components (LLM reasoning generation, code generation with up to three retries), so the reported differences across Qwen-3 sizes—for instance, 53.02% vs. 49.66% on VWTQ-Syn between the 8B and 14B configurations—may be within stochastic variation. Without repeated runs and variance estimates, the claim that 'the accuracy values ... remain comparable, showing only marginal variation' is not fully supported. Please report confidence intervals or multiple seeds, and state the decoding parameters used.
  3. [§5, VTabFact discussion] The VTabFact results (2.68%, 2.23%, 4.91%) are far below the random baseline for a binary fact-verification task, which normally has 50% chance accuracy. The paper attributes this to table-understanding failures, but the magnitude suggests a systematic issue rather than a simple difficulty with complex layouts. If any subset of the benchmark yields near-zero performance, the claim that the system 'handles complex table images' and is a general TableVQA pipeline needs qualification. Please report the accuracy of the table-understanding stage alone on VTabFact (e.g., cell-level extraction accuracy) to clarify where the failure occurs, and discuss whether the code-based reasoning module is appropriate for semantic fact-checking at all.
minor comments (4)
  1. [§3.5.3] The printed Python code sample has formatting errors: the 'if not filtered_df.empty' line appears without a colon in the listing, and the indentation of the 'pd.api.types.is_numeric_dtype' continuation is broken, making the code non-executable as printed.
  2. [Throughout] The text contains typographical and formatting errors, including 'propmts' in the §3 introduction, 'an stock' in §3.3, 'T able' in the title, and the Table 2 header 'VT abFactFinT abNetQAA vg.' with missing spaces.
  3. [§3.5.2] Please clarify whether the numbered CoT list is a verbatim model output or a cleaned paraphrase, since the paper emphasizes auditability of intermediate outputs.
  4. [§3.4, §4] The paper does not specify the exact model versions (e.g., Qwen-3-4B/8B/14B checkpoints and Qwen-2.5-VL-8B), the decoding temperature, or the number of trials; these details are needed for reproducibility, and the code repository would benefit from a pinned commit hash.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the pipeline is evaluated against an external benchmark, and the transparency claims rest on generated, executed code rather than on fitted inputs or self-citation chains.

full rationale

The paper's contribution is a modular software pipeline, not a mathematical derivation: table image to VLM-based extraction, LLM reasoning, automatic Python/Pandas code generation, deterministic execution, and a natural-language explanation. No parameter is fitted to the TableVQA-Bench results, and no 'prediction' is statistically forced by construction. The only self-citation, MRT [9], is used descriptively ('extends our previous MRT approach') and is not load-bearing for the benchmark outcome or for the auditability claims. The accuracy comparison is made against external baselines reported in the TableVQA-Bench paper, so the evaluation is self-contained. The paper does flag a limitation in Section 3.5: 'it is not strictly necessary that the code is always fully linked to the explanation,' and Section 6 lists measuring 'the fidelity of the generated explanations' as future work. That is a legitimate concern about whether the final explanation accurately reflects the executed computation, and it is an unverified empirical premise for the transparency guarantee, but it is not circularity: the explanation is produced after the answer is computed and is not an input to that computation. No circular step can be quoted or exhibited, so the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No invented physical or theoretical objects. All components are existing models and code libraries. The central claim rests on engineering assumptions about extraction fidelity, explanation fidelity, and evaluation metrics rather than on new mathematical entities.

free parameters (3)
  • max_tries = 3
    Number of code execution retries for error feedback, set in Section 3.4. Chosen by hand and not swept or justified.
  • Model scale choices = Qwen2.5-VL 8B; Qwen3 4B/8B/14B
    Model selection for the VLM and LLM modules, described in Section 4. The choice is justified only by hardware constraints, not by systematic comparison.
  • Prompt template designs = Not fully specified in the paper
    The CoT extraction prompt, reasoning prompt, and code-generation prompt are engineered ad hoc. Prompts are not fully reproduced, and they determine all reported results.
assumptions (3)
  • domain assumption The VLM-extracted CSV faithfully represents the table image content needed to answer the question.
    Section 3.1 relies on Qwen2.5-VL's extraction, and Section 5 attributes VTabFact failures to cell identification errors. The whole pipeline inherits this assumption.
  • domain assumption Generated code, when executed, computes the answer the LLM reasoning intended, and the final explanation describes that execution.
    Sections 3.3 to 3.5 depend on this. The paper weakens it in Section 3.5 by stating that code and explanation may not be fully linked.
  • domain assumption Exact-match accuracy on TableVQA-Bench is the right primary evaluation for comparing explainable QA systems.
    Section 4 uses it, while Section 5 criticizes relieved accuracy and proposes ANLS without applying it. The reported comparison is therefore under a contested metric.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ExpliCIT-QA: Explainable Code-Based Image Table Question Answering." pith.science (2026). https://pith.science/paper/BGMHZGDM

@misc{pith2026250711694,
  author       = {Pith},
  title        = {Pith review of: ExpliCIT-QA: Explainable Code-Based Image Table Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BGMHZGDM}},
  note         = {Machine review of arXiv:2507.11694}
}
read the original abstract

We present ExpliCIT-QA, a system that extends our previous MRT approach for tabular question answering into a multimodal pipeline capable of handling complex table images and providing explainable answers. ExpliCIT-QA follows a modular design, consisting of: (1) Multimodal Table Understanding, which uses a Chain-of-Thought approach to extract and transform content from table images; (2) Language-based Reasoning, where a step-by-step explanation in natural language is generated to solve the problem; (3) Automatic Code Generation, where Python/Pandas scripts are created based on the reasoning steps, with feedback for handling errors; (4) Code Execution to compute the final answer; and (5) Natural Language Explanation that describes how the answer was computed. The system is built for transparency and auditability: all intermediate outputs, parsed tables, reasoning steps, generated code, and final answers are available for inspection. This strategy works towards closing the explainability gap in end-to-end TableVQA systems. We evaluated ExpliCIT-QA on the TableVQA-Bench benchmark, comparing it with existing baselines. We demonstrated improvements in interpretability and transparency, which open the door for applications in sensitive domains like finance and healthcare where auditing results are critical.

Figures

Figures reproduced from arXiv: 2507.11694 by the authors.

Figure 1
Figure 1. ExpliCIT-QA: (1) Multimodal understanding: a vision model generates a vi￾sual reasoning to extract and structure the tabular content of a complex layout table image; (2) Natural language reasoning: an LLM produces a human-readable step-by￾step describing the logic of the answer; (3) Automatic code generation: the steps are translated into executable code; (4) Code execution: the script is run deterministically on th… view at source ↗
Figure 2
Figure 2. Example of image which belongs to the FinTabNetQA subdataset on the TableVQA-Bench [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 9 canonical work pages

  1. [1]

    5-vl technical report

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2. 5-vl technical report. arXiv pre print arXiv:2502.13923 (2025)

    arXiv 2025

  2. [2]

    ICDAR 2019 Competition on Scene Text Visual Question Answering

    Biten, A.F., Tito, R., Mafla, A., Gómez, L., Rusiñol, M., Ma thew, M., Jawahar, C.V., Valveny, E., Karatzas, D.: ICDAR 2019 com petition on scene text visual question answering. CoRR abs/1907.00490 (2019), http://arxiv.org/abs/1907.00490

    work page Pith review arXiv 2019

  3. [3]

    In: Integrating Artificial Intelligence and Visualization for Visual Knowledge Discov- ery, pp

    Braşoveanu, A.M., Andonie, R.: Visualizing and explaini ng language models. In: Integrating Artificial Intelligence and Visualization for Visual Knowledge Discov- ery, pp. 213–237. Springer (2022)

    work page 2022

  4. [4]

    CoRR abs/1909.02164 (2019), http://arxiv.org/abs/1909.02164

    Chen, W., Wang, H., Chen, J., Zhang, Y., Wang, H., Li, S., Zh ou, X., Wang, W.Y.: Tabfact: A large-scale dataset for table-based fact v erification. CoRR abs/1909.02164 (2019), http://arxiv.org/abs/1909.02164

    arXiv 2019

  5. [5]

    Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., C allan, J., Neubig, G.: Pal: Program-aided language models (2023), https://arxiv.org/abs/2211.10435

    arXiv 2023

  6. [6]

    arXiv preprint arXiv:2501.12948 (2025)

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zh u, Q., Ma, S., Wang, P., Bi, X., et al.: Deepseek-r1: Incentivizing reason ing capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)

    arXiv 2025

  7. [7]

    Hao, S., Sukhbaatar, S., Su, D., Li, X., Hu, Z., Weston, J., Tian, Y.: Train- ing large language models to reason in a continuous latent sp ace (2024), https://arxiv.org/abs/2412.06769

    arXiv 2024

  8. [8]

    CoRR abs/2004.02349 (2020), https://arxiv.org/abs/2004.02349

    Herzig, J., Nowak, P.K., Müller, T., Piccinno, F., Eisens chlos, J.M.: TAPAS: weakly supervised table parsing via pre-training. CoRR abs/2004.02349 (2020), https://arxiv.org/abs/2004.02349

    arXiv 2020

Show all 29 references
  1. [9]

    In: Proceedings of the 19th Internatio nal Workshop on Seman- tic Evaluation (SemEval-2025)

    Hormázabal-Lagos, M., Álvaro Bueno Saez, Cerezo-Costas , H., Doval, P.A., Vesteiro, J.A.: MRT at SemEval-2025 Task 8: Maximizing Reco very from Tables with Multiple Steps. In: Proceedings of the 19th Internatio nal Workshop on Seman- tic Evaluation (SemEval-2025). Association ...

    2025

  2. [10]

    Hu, A., Xu, H., Zhang, L., Ye, J., Yan, M., Zhang, J., Jin, Q ., Huang, F., Zhou, J.: mplug-docowl2: High-resolution compressing for ocr-f ree multi-page document understanding (2024), https://arxiv.org/abs/2409.03420

    2024 arXiv

  3. [11]

    In: 2025 IEEE/CVF Winter Conference on Applicat ions of Com- puter Vision (W ACV)

    Jirachanchaisiri, P., Ly, N.T., Takasu, A.: TRH2TQA: Ta ble Recognition with Hierarchical Relationships to Table Question-Answer ing on Business Ta- ble Images . In: 2025 IEEE/CVF Winter Conference on Applicat ions of Com- puter Vision (W ACV). pp. 8844–8852. IEEE Computer Soc...

    2025

  4. [12]

    CoRR abs/2111.15664 (2021), https://arxiv.org/abs/2111.15664

    Kim, G., Hong, T., Yim, M., Park, J., Yim, J., Hwang, W., Yu n, S., Han, D., Park, S.: Donut: Document understanding transformer wi thout OCR. CoRR abs/2111.15664 (2021), https://arxiv.org/abs/2111.15664

    2021 arXiv

  5. [13]

    Kim, Y., Yim, M., Song, K.Y.: Tablevqa-bench: A visual qu estion answering bench- mark on multiple table domains (2024), https://arxiv.org/abs/2404.19205

    2024 arXiv

  6. [14]

    In: Proceedings of the Fourth ACM Internat ional Conference on AI in Finance

    Lakkaraju, K., Jones, S.E., Vuruma, S.K.R., Pallagani, V., Muppasani, B.C., Sri- vastava, B.: Llms for financial advisement: A fairness and effi cacy study in personal decision making. In: Proceedings of the Fourth ACM Internat ional Conference on AI in Finance. pp. 100–107 (2023...

    2023

  7. [15]

    In: Proceedings of the 31st ACM International Conference on Multimedia

    Lan, Y., Li, X., Liu, X., Li, Y., Qin, W., Qian, W.: Improvi ng zero-shot visual question answering via large language models with reasonin g question prompts. In: Proceedings of the 31st ACM International Conference on Multimedia. pp. 4389–4400 (2023)

    2023

  8. [16]

    ACM Transactions on Software Engineering and Meth odology 34(2), 1–23 (2025)

    Li, J., Li, G., Li, Y., Jin, Z.: Structured chain-of-thou ght prompting for code gen- eration. ACM Transactions on Software Engineering and Meth odology 34(2), 1–23 (2025)

    2025

  9. [17]

    In: Proceedings of t he IEEE/CVF Con- ference on Computer Vision and Pattern Recognition

    Mitra, C., Huang, B., Darrell, T., Herzig, R.: Compositi onal chain-of-thought prompting for large multimodal models. In: Proceedings of t he IEEE/CVF Con- ference on Computer Vision and Pattern Recognition. pp. 144 20–14431 (2024)

    2024

  10. [18]

    In: Informatics

    Nazi, Z.A., Peng, W.: Large language models in healthcar e and medical domain: A review. In: Informatics. vol. 11, p. 57. MDPI (2024)

    2024

  11. [19]

    Nguyen, G., Brugere, I., Sharma, S., Kariyappa, S., Nguy en, A.T., Lecue, F.: Interpretable llm-based table question answeri ng (2025), https://arxiv.org/abs/2412.12386

    2025 arXiv

  12. [20]

    In: Proceedings of the IEEE conference on compute r vision and pattern recognition

    Park, D.H., Hendricks, L.A., Akata, Z., Rohrbach, A., Sc hiele, B., Darrell, T., Rohrbach, M.: Multimodal explanations: Justifying decisi ons and pointing to the evidence. In: Proceedings of the IEEE conference on compute r vision and pattern recognition. pp. 8779–8788 (2018)

    2018

  13. [21]

    CoRR abs/1508.00305 (2015), http://arxiv.org/abs/1508.00305

    Pasupat, P., Liang, P.: Compositional semantic parsing on semi-structured tables. CoRR abs/1508.00305 (2015), http://arxiv.org/abs/1508.00305

    2015 arXiv

  14. [22]

    Team, O.: Gpt-4 technical report (2024), https://arxiv.org/abs/2303.08774

    2024 arXiv

  15. [23]

    Wang, Z., Zhang, H., Li, C.L., Eisenschlos, J.M., Perot, V., Wang, Z., Miculicich, L., Fujii, Y., Shang, J., Lee, C.Y., Pfister, T.: Chain-of-table: Evolving tables in the reasoning chain for table understand ing (2024), https://arxiv.org/abs/2401.04398

    2024 arXiv

  16. [24]

    Advances in neural information processing systems 35, 24824–24837 (2022)

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi , E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning i n large language models. Advances in neural information processing systems 35, 24824–24837 (2022)

    2022

  17. [25]

    Wu, Z., Feng, Y.: Protrix: Building models for planning a nd reasoning over tables with sentence context (2025), https://arxiv.org/abs/2403.02177

    2025 arXiv

  18. [27]

    arXiv preprint ar Xiv:2505.09388 (2025)

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Y u, B., Gao, C., Huang, C., Lv, C., et al.: Qwen3 technical report. arXiv preprint ar Xiv:2505.09388 (2025)

    2025 arXiv

  19. [28]

    CoRR abs/2005.08314 (2020), https://arxiv.org/abs/2005.08314

    Yin, P., Neubig, G., Yih, W., Riedel, S.: Tabert: Pretrai ning for joint un- derstanding of textual and tabular data. CoRR abs/2005.08314 (2020), https://arxiv.org/abs/2005.08314

    2020 arXiv

  20. [29]

    CoRR abs/1709.00103 (2017), http://arxiv.org/abs/1709.00103

    Zhong, V., Xiong, C., Socher, R.: Seq2sql: Generating st ructured queries from natural language using reinforcement learning. CoRR abs/1709.00103 (2017), http://arxiv.org/abs/1709.00103

    2017 arXiv

  21. [30]

    CoRR abs/1911.10683 (2019), http://arxiv.org/abs/1911.10683

    Zhong, X., ShafieiBavani, E., Jimeno-Yepes, A.: Image-b ased table recog- nition: data, model, and evaluation. CoRR abs/1911.10683 (2019), http://arxiv.org/abs/1911.10683

    2019 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.