Pith. sign in

REVIEW 2 major objections 1 minor 7 references

SAFE-Cascade routes chart questions through OCR and text reasoning first, escalating to a vision model only when the router decides it is needed.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-26 18:44 UTC pith:HLBR6ALM

load-bearing objection The paper delivers a working router that cuts VLM calls by 27% on chart QA with accuracy within noise of the full-VLM baseline, but the router features, training details, and statistical test for equivalence are missing. the 2 major comments →

arxiv 2606.19646 v1 pith:HLBR6ALM submitted 2026-06-17 cs.IR cs.CV

SAFE-Cascade: Cost-Adaptive Vision-Language Routing for Chart Question Answering

Ayush Dwivedi , Qixin Wang , Ashvi Soni , Ruoteng Wang , Han Li , Animesh Mahapatra , Neeraj Agrawal , Xintao Wu This is my paper
classification cs.IR cs.CV
keywords chart question answeringvision-language routingcost-adaptive systemsOCR text extractionRandom Forest routermultimodal question answering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces SAFE-Cascade as a system that extracts chart text with OCR, obtains a provisional answer from a text-only language model, and then applies a learned router to accept that answer or escalate to a vision-language model. On a held-out ChartQA split of 375 examples, the system reaches 69.1 percent unified accuracy while invoking the vision model for 73.1 percent of queries, compared with 67.7 percent accuracy at 100 percent invocation for the full vision-language baseline. The authors interpret the 1.4-point difference as within uncertainty and therefore treat the approaches as equivalent in performance while cutting vision-model calls by 26.9 percent and estimated cost by 9.3 percent. The demo interface displays each step, including routing probability and cost estimates, and lets users adjust the escalation threshold to explore accuracy-cost trade-offs.

Core claim

SAFE-Cascade first extracts chart text via OCR, generates a provisional answer with a text-only model, and uses a Random Forest router on inference-time features to decide whether to accept the text answer or escalate to the vision-language model, achieving 69.1 percent accuracy with 73.1 percent vision-model invocations versus 67.7 percent accuracy at full invocation on held-out ChartQA data and thereby matching performance at reduced cost.

What carries the argument

The Random Forest router trained on inference-time features from OCR and text-only reasoning that outputs an escalation probability and decides whether the text answer suffices or the vision-language model must be called.

Load-bearing premise

The router trained on the 2500-example set generalizes to the 375 held-out examples without overfitting, and the observed 1.4-point accuracy gap falls within statistical uncertainty so that equivalence can be claimed.

What would settle it

A statistically significant accuracy drop on a larger or differently distributed held-out ChartQA set would falsify the claim that selective routing matches full vision-language performance.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Reduces vision-language model invocations by 26.9 percent while preserving unified accuracy on the tested split.
  • Exposes OCR evidence, text answer, routing probability, and cost estimates side-by-side for inspection.
  • Allows users to vary the escalation threshold and observe the resulting accuracy-cost frontier directly.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same text-first routing pattern could apply to other multimodal tasks where many queries are answerable without vision.
  • Retraining the router on larger or more varied chart distributions might tighten the accuracy-cost trade-off further.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper presents SAFE-Cascade, an interactive demo system for chart question answering that extracts text via OCR, obtains a provisional answer from a text-only LM (gpt-5-mini), and uses a Random Forest router on inference-time features to decide whether to accept that answer or escalate to a VLM (gemini-2.5-flash-image). On a held-out ChartQA split of 375 examples drawn from a 2,500-example experiment, it reports 69.1% unified accuracy at 73.1% VLM invocation versus 67.7% accuracy at 100% invocation for the full-VLM baseline, interpreting the +1.4 pp gap as statistically uncertain and therefore claiming performance parity together with a 26.9% reduction in VLM calls and 9.3% estimated cost savings. The system exposes routing probability, evidence, and cost/latency estimates to users and allows threshold adjustment.

Significance. If the router generalizes and the equivalence interpretation is statistically supported, the work illustrates a practical route to cost-aware multimodal routing that preserves accuracy while lowering VLM usage; the transparent demo interface additionally contributes to interpretability of modality selection in chart QA.

major comments (2)
  1. [Abstract] Abstract: the central claim that the observed +1.4 pp accuracy difference supports performance equivalence because it is 'statistically uncertain' is not accompanied by any confidence interval, p-value, non-inferiority margin, or test statistic. With n=375 the binomial standard error on the difference is approximately 3.4 pp, so the gap is consistent with noise, but without the actual quantification the equivalence interpretation lacks evidential grounding.
  2. [Abstract] Abstract: the Random Forest router is described only as 'trained on inference-time features' from the 2,500-example experiment, with no enumeration of the feature set, training/validation split details, hyperparameters, or explicit confirmation that the 375 held-out examples are disjoint and free of label or feature leakage. These omissions make it impossible to assess whether the reported 26.9% reduction in VLM calls is reproducible or generalizable.
minor comments (1)
  1. [Abstract] The model names 'gpt-5-mini' and 'gemini-2.5-flash-image' appear non-standard; clarify whether these are internal aliases, future releases, or specific deployment identifiers.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments on the abstract. We address each major comment below. We agree that both points identify areas where the current description is insufficient and will revise the manuscript to add the requested statistical details and router specifications.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that the observed +1.4 pp accuracy difference supports performance equivalence because it is 'statistically uncertain' is not accompanied by any confidence interval, p-value, non-inferiority margin, or test statistic. With n=375 the binomial standard error on the difference is approximately 3.4 pp, so the gap is consistent with noise, but without the actual quantification the equivalence interpretation lacks evidential grounding.

    Authors: We agree that the abstract's reference to statistical uncertainty would be strengthened by explicit quantification. In the revised manuscript we will report a 95% confidence interval on the accuracy difference (using the normal approximation to the binomial) and a two-sided p-value for the test of equal proportions. This will replace the informal phrasing while preserving the interpretation that the observed gap is consistent with sampling variability. revision: yes

  2. Referee: [Abstract] Abstract: the Random Forest router is described only as 'trained on inference-time features' from the 2,500-example experiment, with no enumeration of the feature set, training/validation split details, hyperparameters, or explicit confirmation that the 375 held-out examples are disjoint and free of label or feature leakage. These omissions make it impossible to assess whether the reported 26.9% reduction in VLM calls is reproducible or generalizable.

    Authors: We acknowledge that the current abstract and manuscript text provide insufficient detail on the router. The revised version will enumerate the full set of inference-time features, describe the train/validation split within the 2,500 examples, list the Random Forest hyperparameters, and explicitly confirm that the 375 held-out examples were never used for router training or feature construction, thereby ruling out leakage. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical router evaluation on held-out split is independent of inputs

full rationale

The paper reports measured accuracy (69.1% vs 67.7%) and VLM invocation rates (73.1% vs 100%) on a held-out 375-example test split after training a Random Forest router on a separate 2,500-example experiment. No equations, derivations, or self-citations are present that reduce these metrics to fitted parameters by construction or import uniqueness from prior author work. The evaluation uses disjoint data and direct measurement, satisfying the criteria for a self-contained empirical result with no load-bearing circular steps.

Axiom & Free-Parameter Ledger

2 free parameters · 1 axioms · 0 invented entities

The router is a trained model whose internal parameters are free; the evaluation assumes a unified accuracy metric across text-only and VLM outputs; no new physical entities are postulated.

free parameters (2)
  • Random Forest router parameters
    The router is explicitly trained on inference-time features from the 2,500-example set, so its internal parameters are fitted to data.
  • escalation threshold
    Users can adjust the threshold that controls VLM invocation rate, implying it is a tunable free parameter chosen to trace the accuracy-cost frontier.
axioms (1)
  • domain assumption A single unified accuracy metric can be computed across answers produced by the text-only model and the VLM.
    The comparison of 69.1% versus 67.7% accuracy presupposes that answers from the two pathways are scored on the same scale.

pith-pipeline@v0.9.1-grok · 5912 in / 1458 out tokens · 33416 ms · 2026-06-26T18:44:35.105163+00:00 · methodology

0 comments
read the original abstract

Vision-language models (VLMs) are powerful for chart question answering, but invoking a VLM for every query can be unnecessarily expensive when many questions are answerable from OCR text and lightweight language reasoning. We demonstrate SAFE-Cascade, an interactive system for cost-adaptive chart question answering. Given a chart image and a natural-language question, SAFE-Cascade first extracts chart text with OCR, obtains a provisional answer from a text-only language model, and then uses a learned router to decide whether to accept the text answer or escalate to a VLM. The demo exposes this decision process to users: OCR evidence, text-only answer, routing probability, escalation decision, final answer, estimated cost, and estimated latency are shown side by side. SAFE-Cascade is designed as a transparent interface for understanding when visual grounding is actually needed. Users can upload or select charts, ask questions, inspect the evidence used by each pathway, compare text-only and VLM answers, and adjust the escalation threshold to explore the accuracy-cost frontier. The system is implemented with Azure Document Intelligence for OCR, gpt-5-mini as the text-only model, gemini-2.5-flash-image as the VLM, and a Random Forest router trained on inference-time features. On a held-out ChartQA test split of 375 examples from a 2,500-example experiment, SAFE-Cascade achieves 69.1% unified accuracy with 73.1% VLM invocation, compared with 67.7% accuracy and 100% VLM invocation for the full-VLM baseline. The observed +1.4 percentage-point difference is statistically uncertain, so we interpret SAFE-Cascade as matching full-VLM performance while reducing VLM calls by 26.9% and estimated cost by 9.3%. The demonstration shows how selective modality routing can make multimodal knowledge systems more transparent, tunable, and cost-aware.

Figures

Figures reproduced from arXiv: 2606.19646 by Animesh Mahapatra, Ashvi Soni, Ayush Dwivedi, Han Li, Neeraj Agrawal, Qixin Wang, Ruoteng Wang, Xintao Wu.

Figure 1
Figure 1. Figure 1: SAFE-Cascade system architecture and processing workflow. The system exposes the same stages to users: [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Interactive operating frontier. SAFE-Cascade exposes the escalation threshold [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

7 extracted references · 3 canonical work pages · 2 internal anchors

  1. [1]

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning. InFindings of the Association for Computational Linguistics: ACL 2022, 2263–2279. DOI: 10.18653/v1/2022.findings-acl.177

    work page doi:10.18653/v1/2022.findings-acl.177 2022

  2. [2]

    Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking. InProceed- ings of ACM Multimedia

    2022

  3. [3]

    Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeongyeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. 2022. OCR-free Document Understanding Transformer. InProceedings of ECCV

    2022

  4. [4]

    Yonatan Geifman and Ran El-Yaniv. 2019. SelectiveNet: A Deep Neural Network with an Integrated Reject Option. InProceedings of ICML

    2019

  5. [5]

    Lingjiao Chen, Matei Zaharia, and James Zou. 2023. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv:2305.05176

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  6. [6]

    RouteLLM: Learning to Route LLMs with Preference Data

    Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. 2024. RouteLLM: Learning to Route LLMs with Preference Data.arXiv:2406.18665

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  7. [7]

    Ashish Vaswani et al. 2017. Attention Is All You Need. InProceedings of NeurIPS

    2017