{"id":"444dae18-0243-4b15-aa26-f6afb13f4fea","arxiv_id":"2606.19646","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":2,"one_line_summary":"SAFE-Cascade uses OCR plus text LM with a Random Forest router to selectively invoke VLMs on ChartQA, reporting 69.1% accuracy at 73.1% VLM use versus 67.7% at 100% VLM use.","lead":"SAFE-Cascade routes chart questions first through OCR and a text-only language model, then uses a learned router to decide whether to escalate to a vision-language model. A smart generalist might read it to see a concrete example of making multimodal systems cheaper and more transparent by selective routing.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Router generalization to held-out 375 examples and statistical basis for equivalence claim lack verification details","rationale":"The reader's weakest_assumption directly identifies the two load-bearing evaluation risks (router overfitting and statistical support for equivalence). No stronger internal inconsistency appears in the reported numbers or abstract description; the concern is precisely the unverifiable assumptions needed for the claim to hold.","tokens_in":1933,"tokens_out":388,"duration_ms":22348,"concrete_test":"Re-run the 375-example evaluation after (a) confirming the router was fit only on the complementary 2,125 examples and (b) reporting the 95% CI on the paired accuracy difference (or McNemar test p-value); if the CI lower bound is below -3 pp or the router shows >10 pp train-test accuracy gap, the equivalence and generalization claims weaken.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim interprets the +1.4 pp accuracy gap (69.1% vs 67.7%) as supporting performance equivalence because it is 'statistically uncertain,' while the Random Forest router (trained on inference-time features from the 2,500-example experiment) reduces VLM calls by 26.9%. This rests on two unverified conditions: (1) the router was trained exclusively on a proper subset disjoint from the 375 held-out examples with no leakage via feature computation or labeling, and (2) the uncertainty quantification (unspecified test, CI, or equivalence margin) is sufficient to treat the systems as matching. With n=375 the binomial SE on the difference is ~3.4 pp, so the observed gap is consistent with noise, but without the actual CI, p-value, or non-inferiority analysis the equivalence interpretation is unsupported. Full methods for router training, feature set, and statistical procedure are absent from the supplied abstract.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper presents SAFE-Cascade, an interactive demo system for chart question answering that extracts text via OCR, obtains a provisional answer from a text-only LM (gpt-5-mini), and uses a Random Forest router on inference-time features to decide whether to accept that answer or escalate to a VLM (gemini-2.5-flash-image). On a held-out ChartQA split of 375 examples drawn from a 2,500-example experiment, it reports 69.1% unified accuracy at 73.1% VLM invocation versus 67.7% accuracy at 100% invocation for the full-VLM baseline, interpreting the +1.4 pp gap as statistically uncertain and therefore claiming performance parity together with a 26.9% reduction in VLM calls and 9.3% estimated cost savings. The system exposes routing probability, evidence, and cost/latency estimates to users and allows threshold adjustment.","tokens_in":2149,"tokens_out":585,"duration_ms":25528,"significance":"If the router generalizes and the equivalence interpretation is statistically supported, the work illustrates a practical route to cost-aware multimodal routing that preserves accuracy while lowering VLM usage; the transparent demo interface additionally contributes to interpretability of modality selection in chart QA.","major_comments":[{"comment":"Abstract: the central claim that the observed +1.4 pp accuracy difference supports performance equivalence because it is 'statistically uncertain' is not accompanied by any confidence interval, p-value, non-inferiority margin, or test statistic. With n=375 the binomial standard error on the difference is approximately 3.4 pp, so the gap is consistent with noise, but without the actual quantification the equivalence interpretation lacks evidential grounding.","section":"Abstract"},{"comment":"Abstract: the Random Forest router is described only as 'trained on inference-time features' from the 2,500-example experiment, with no enumeration of the feature set, training/validation split details, hyperparameters, or explicit confirmation that the 375 held-out examples are disjoint and free of label or feature leakage. These omissions make it impossible to assess whether the reported 26.9% reduction in VLM calls is reproducible or generalizable.","section":"Abstract"}],"minor_comments":[{"comment":"The model names 'gpt-5-mini' and 'gemini-2.5-flash-image' appear non-standard; clarify whether these are internal aliases, future releases, or specific deployment identifiers.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads primarily as a system demonstration rather than a full empirical research article; the journal's scope in cs.IR may favor submissions with more extensive ablation studies, statistical reporting, and comparison against additional routing baselines."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments on the abstract. We address each major comment below. We agree that both points identify areas where the current description is insufficient and will revise the manuscript to add the requested statistical details and router specifications.","responses":[{"response":"We agree that the abstract's reference to statistical uncertainty would be strengthened by explicit quantification. In the revised manuscript we will report a 95% confidence interval on the accuracy difference (using the normal approximation to the binomial) and a two-sided p-value for the test of equal proportions. This will replace the informal phrasing while preserving the interpretation that the observed gap is consistent with sampling variability.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the central claim that the observed +1.4 pp accuracy difference supports performance equivalence because it is 'statistically uncertain' is not accompanied by any confidence interval, p-value, non-inferiority margin, or test statistic. With n=375 the binomial standard error on the difference is approximately 3.4 pp, so the gap is consistent with noise, but without the actual quantification the equivalence interpretation lacks evidential grounding."},{"response":"We acknowledge that the current abstract and manuscript text provide insufficient detail on the router. The revised version will enumerate the full set of inference-time features, describe the train/validation split within the 2,500 examples, list the Random Forest hyperparameters, and explicitly confirm that the 375 held-out examples were never used for router training or feature construction, thereby ruling out leakage.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the Random Forest router is described only as 'trained on inference-time features' from the 2,500-example experiment, with no enumeration of the feature set, training/validation split details, hyperparameters, or explicit confirmation that the 375 held-out examples are disjoint and free of label or feature leakage. These omissions make it impossible to assess whether the reported 26.9% reduction in VLM calls is reproducible or generalizable."}],"tokens_in":1631,"tokens_out":445,"duration_ms":24235,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that SAFE-Cascade routes chart questions through OCR and a text model first, then uses a Random Forest to decide whether to call the VLM. On the 375 held-out ChartQA examples it reaches 69.1% accuracy at 73% VLM use versus 67.7% at 100% use, and the authors treat the small gap as evidence that performance matches while cost drops.\n\nWhat is new is the end-to-end numbers for this exact pipeline on this split, plus the interactive demo that surfaces the routing decision, OCR evidence, and cost estimate. The engineering is straightforward and the transparency angle is a reasonable addition to existing routing work.\n\nThe soft spots sit in the evidence for the central claim. No feature list or training procedure for the router is given, so it is impossible to check for leakage between the 2500-example training set and the held-out 375. The accuracy difference is called statistically uncertain without a reported test, confidence interval, or non-inferiority margin; with n=375 the binomial standard error on the difference is roughly 3.4 points, so the result is consistent with noise but does not yet support a strong equivalence statement.\n\nThis paper is for engineers and applied researchers who need a concrete cost-saving example in chart QA. A reader looking for a new theoretical framework or broad claims will find little, but someone who wants to see modality routing implemented and measured will get a usable data point.\n\nIt deserves peer review. The empirical configuration is fresh and the system is actually built, so referees can request the missing router and statistics details without starting from zero.","headline":"The paper delivers a working router that cuts VLM calls by 27% on chart QA with accuracy within noise of the full-VLM baseline, but the router features, training details, and statistical test for equivalence are missing.","tokens_in":2662,"tokens_out":424,"would_cite":false,"duration_ms":21168,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"SAFE-Cascade routes chart questions through OCR and text reasoning first, escalating to a vision model only when the router decides it is needed.","keywords":["chart question answering","vision-language routing","cost-adaptive systems","OCR text extraction","Random Forest router","multimodal question answering"],"falsifier":"A statistically significant accuracy drop on a larger or differently distributed held-out ChartQA set would falsify the claim that selective routing matches full vision-language performance.","tokens_in":2850,"feed_emoji":"📊","tokens_out":693,"duration_ms":20983,"temperature":0.7,"pith_summary":"The paper introduces SAFE-Cascade as a system that extracts chart text with OCR, obtains a provisional answer from a text-only language model, and then applies a learned router to accept that answer or escalate to a vision-language model. On a held-out ChartQA split of 375 examples, the system reaches 69.1 percent unified accuracy while invoking the vision model for 73.1 percent of queries, compared with 67.7 percent accuracy at 100 percent invocation for the full vision-language baseline. The authors interpret the 1.4-point difference as within uncertainty and therefore treat the approaches as equivalent in performance while cutting vision-model calls by 26.9 percent and estimated cost by 9.3 percent. The demo interface displays each step, including routing probability and cost estimates, and lets users adjust the escalation threshold to explore accuracy-cost trade-offs.","feed_headline":"SAFE-Cascade matches VLM accuracy with 27% fewer calls on charts","feed_subtitle":"Text-first routing on ChartQA yields 69.1% accuracy at 73.1% vision invocations versus 67.7% at full use","key_machinery":"The Random Forest router trained on inference-time features from OCR and text-only reasoning that outputs an escalation probability and decides whether the text answer suffices or the vision-language model must be called.","core_discovery":"SAFE-Cascade first extracts chart text via OCR, generates a provisional answer with a text-only model, and uses a Random Forest router on inference-time features to decide whether to accept the text answer or escalate to the vision-language model, achieving 69.1 percent accuracy with 73.1 percent vision-model invocations versus 67.7 percent accuracy at full invocation on held-out ChartQA data and thereby matching performance at reduced cost.","pith_inferences":["The same text-first routing pattern could apply to other multimodal tasks where many queries are answerable without vision.","Retraining the router on larger or more varied chart distributions might tighten the accuracy-cost trade-off further."],"forward_implications":["Reduces vision-language model invocations by 26.9 percent while preserving unified accuracy on the tested split.","Exposes OCR evidence, text answer, routing probability, and cost estimates side-by-side for inspection.","Allows users to vary the escalation threshold and observe the resulting accuracy-cost frontier directly."],"fun_headline_variants":["SAFE-Cascade matches VLM accuracy with 27% fewer calls","Text routing cuts VLM calls 27% on ChartQA","SAFE-Cascade reduces VLM invocations 27% on charts"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The router trained on the 2500-example set generalizes to the 375 held-out examples without overfitting, and the observed 1.4-point accuracy gap falls within statistical uncertainty so that equivalence can be claimed.","fun_headline_variants_meta":{"raw":{"variants":["SAFE-Cascade matches VLM accuracy with 27% fewer calls","Text routing cuts VLM calls 27% on ChartQA","SAFE-Cascade reduces VLM invocations 27% on charts"]},"model":"grok-4.3","cost_usd":0.01256,"raw_usage":{"total_tokens":5545,"prompt_tokens":827,"num_sources_used":0,"completion_tokens":55,"cost_in_usd_ticks":125599500,"prompt_tokens_details":{"text_tokens":827,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":4663,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":827,"tokens_out":55,"duration_ms":31595,"temperature":1.0,"reasoning_tokens":4663,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T18:44:35.105163+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A statistically significant accuracy drop on a larger or differently distributed held-out ChartQA set would falsify the claim that selective routing matches full vision-language performance.","supporting_citations":[],"review_version":1}