{"id":"26d0936a-ba98-4628-a890-01331f4e0d46","arxiv_id":"2608.07861","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Preprocessing effects on cloud VLM VQA vary strongly by model, API paradigm, and provider token accounting, so no single preprocessing strategy is universally best.","lead":"This paper benchmarks 12 image preprocessing techniques on four commercial cloud vision-language models used for mobile visual question answering, measuring accuracy, latency, payload, and token cost across 95,168 API calls. It finds that preprocessing effects are strongly model, provider, and task dependent, so developers must select input preprocessing per deployment scenario rather than assume one technique works everywhere.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported API-call count (95,168) is inconsistent with the stated full-dataset protocol, leaving per-condition sample sizes and the statistical significance of accuracy deltas unverified.","rationale":"I selected the API-call-count/sample-size inconsistency as the most load-bearing concern because it directly affects the empirical foundation of the central claim. The abstract's total of 95,168 calls cannot be reconciled with the stated protocol of 12 techniques × 4 models × full datasets, implying either undisclosed subsampling or a counting error. Without knowing the true per-condition N, the paper's one-decimal accuracy and latency means cannot be assessed for statistical significance, and the central qualitative claim—that preprocessing effects are model-dependent rather than intrinsic—could conceivably be an artifact of small samples, especially for the smaller accuracy gaps. The LLM-as-judge validity issue raised by the reader is real but secondary: the main claim is already supported by VQA-MHUG, which uses human-reference soft accuracy, and the open-ended datasets are case studies whose tier structure matches the human-grounded results. I therefore disagree with the reader's choice of weakest assumption, though I agree with the CONDITIONAL verdict overall. The concern does not invalidate the paper's direction; it demands a concrete accounting of the experimental scale and appropriate interval estimates. The public artifact makes this check feasible, and the authors can resolve it by releasing per-cell counts. I set verdict_should_be to UNCHANGED because the reader already issued CONDITIONAL, and my concern reinforces rather than overturns that verdict.","tokens_in":36274,"tokens_out":8410,"duration_ms":90139,"concrete_test":"Inspect the public GitHub artifact (https://github.com/Hvanhuynegem/VQABench) for per-request logs or result CSVs; count actual API calls per (dataset, technique, model) tuple and compare the total to 95,168 and to the full-protocol expectation of ≥191,520. If logs are absent, request per-cell call counts and any subsampling decisions from the authors. Then recompute the Table V accuracy deltas with 95% confidence intervals (e.g., Wilson intervals) at the recovered N; if Downsampler's −23.4% vs −2.4% gap is no longer significant, the model-dependence claim requires the disclosed N to be confirmed before acceptance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper states it evaluated 12 preprocessing techniques on three datasets (VQA-MHUG N=3,990, VOILA-A N=500, DriVQA N=24) across four commercial VLMs, totaling 95,168 API calls. If every technique ran on every model and every sample, the main evaluation alone would require at least 12×4×3,990 = 191,520 calls for VQA-MHUG, already double the reported total; adding DriVQA and VOILA-A only increases the gap. Reconciling 95,168 requires subsampling VQA-MHUG to roughly 1,800–2,000 images or omitting technique/model combinations, yet no subsampling is disclosed anywhere in the paper. The paper reports accuracy and latency means to one decimal place without confidence intervals or significance tests. The central claim—that preprocessing effectiveness is not intrinsic and depends on the model–provider stack—relies on accuracy deltas such as Downsampler (−6.6% on Realtime-mini vs −23.4% on GPT-5.4) and GazeROI (−3.1% vs −10.2%). If the actual per-cell N is much smaller than the full dataset, these deltas may fall within sampling error, and the model-dependence conclusion would rest on an unverified quantitative basis even if its qualitative direction is correct.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VQABench, a benchmark that treats client-side image preprocessing as a controlled variable for cloud-VLM-based VQA. It evaluates 12 preprocessing techniques (compression, downsampling, grayscale, and gaze-, saliency-, and YOLO-based ROI methods with and without thumbnails) on three datasets (VQA-MHUG, DriVQA, VOILA-A) across four commercial VLMs from three providers, claiming 95,168 API calls. The central claim is that preprocessing effectiveness is not intrinsic to a technique but depends on the model–provider stack, API paradigm, provider token-accounting rule, and task formulation. The paper reports latency decompositions, token-usage breakdowns, answer-quality metrics, Pareto analyses, and scenario-specific deployment rankings, plus controlled token-accounting experiments in the appendices.","tokens_in":36442,"tokens_out":6380,"duration_ms":68869,"significance":"If the empirical results hold, the paper fills a real gap: prior VQA benchmarks focus almost exclusively on accuracy, and no existing benchmark treats preprocessing as a controlled variable across commercial cloud APIs. The controlled token-accounting experiments in Appendices H–J, which decouple decoded pixel geometry from transmitted payload size, are a strong and reusable contribution, and the public code and measurement artifacts support reproducibility. The observed provider-specific accounting rules (e.g., Gemini's flat per-image budget, Realtime-mini's square/non-square token buckets) are concrete, falsifiable findings of practical use. However, the central quantitative claims are currently weakened by an unexplained discrepancy in the reported API-call count, missing per-condition sample sizes, and the absence of confidence intervals or significance tests for the accuracy and latency deltas that support the model-dependence conclusion.","major_comments":[{"comment":"The reported total of 95,168 API calls is inconsistent with the stated full-dataset protocol. The abstract and Section III state that 12 preprocessing techniques are evaluated on VQA-MHUG (N=3,990), DriVQA (N=24), and VOILA-A (N=500) across four VLMs. Under that protocol, VQA-MHUG alone requires 12 × 4 × 3,990 = 191,520 calls, already more than double the reported total; even if DriVQA and VOILA-A used only one model, at least 12 × 4 × 1,852 ≈ 88,896 calls would be needed for VQA-MHUG, implying a subsample of about 1,850 pairs rather than the stated 3,990. No subsampling is disclosed anywhere in the manuscript. Since Tables II–V report means without per-cell sample sizes, confidence intervals, or significance tests, the accuracy deltas that drive the central claim (e.g., Downsampler −6.6% on Realtime-mini vs. −23.4% on GPT-5.4 in Table V, or the latency ranking in Figure 6) cannot be verified against sampling error. Please disclose the exact per-condition sample sizes, clarify how the 95,168 total is obtained, and report confidence intervals or significance tests for the main accuracy and latency comparisons.","section":"Section VI.B, Section VI.C, Eq. (2)"},{"comment":"The open-ended quality conclusions for DriVQA and VOILA-A rest entirely on a GPT-4.1 LLM judge with hand-set rubric weights (0.50/0.25/0.15/0.10 for DriVQA in Section III.D and 0.40/0.30/0.20/0.10 in Eq. (2) for VOILA-A), yet the paper reports no human-agreement validation, no judge-stability analysis, and no alternative-judge sensitivity check. With only 24 DriVQA items, the observed differences (e.g., baseline 0.52 vs. GazeROI 0.50 in Figure 9) are within a plausible range of judge or sampling noise, and a different rubricing could change which techniques are considered to preserve quality. Because these results are used to claim that preprocessing effects depend on task formulation, please add a human-validation study or, at minimum, a judge-variance/sensitivity analysis and report uncertainties around the per-technique open-ended scores.","section":"Section VI.B, Section VI.C, Eq. (2)"}],"minor_comments":[{"comment":"The header 'Reqest→ First response' contains a typo and should read 'Request→ First response'; the same table also uses inconsistent capitalization for the latency components.","section":"Table III"},{"comment":"The statement that Base64/data-URL wrapping inflates the payload by 'roughly 33%' is correct for pure binary-to-Base64 expansion but may be slightly different after JSON escaping and provider headers; a brief definition or formula would help reproducibility.","section":"Section III.B"},{"comment":"Appendix D reports that identical client-side preprocessing techniques have 2–3× higher local preprocessing latency under the Realtime WebSocket API than under REST (Table XI). Since preprocessing is described as running locally before the request is sent, it is unclear how the API interface could affect it; please clarify whether the measured interval includes any connection/session overhead or whether the code path differs between the two interfaces.","section":"Section IV.A, Appendix D"},{"comment":"The stacked accuracy distributions in Figure 4 are informative, but the 'Weighted Accuracy' labels on the right are not defined in the caption; please add a short explanation or a table with the same values.","section":"Figure 4"},{"comment":"The Pareto analysis uses a normalized per-query token cost as the third objective, but the normalization formula is only described in the figure caption; please state it in the main text so the trade-off plots are reproducible.","section":"Section VII.A, Figures 11 and 20-22"}],"recommendation":"major_revision","confidential_remarks":"The API-call count discrepancy is the main risk: if the full-data description is inaccurate, the paper overstates its scale and leaves the per-condition statistics unverifiable. I would ask the authors to provide a per-condition enumeration of API calls (perhaps as supplementary material) and to add confidence intervals or significance tests before reconsidering. The qualitative direction of the findings is plausible and the controlled token-accounting experiments are valuable, so I do not see grounds for rejection if the statistical grounding is fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead VQABench. The useful core: first systematic look at client-side image preprocessing as a controlled variable on commercial cloud VLMs, measuring accuracy, latency, payload, and tokens together. The token-accounting findings are genuinely new and well evidenced: compression-only methods change payload but not tokens; Realtime-mini buckets all square images at 194 tokens regardless of pixel count (aspect ratio moves the bucket); Gemini Flash bills per image part. The controlled resolution/JPEG-quality experiments in Appendices H and I are the strongest part. The artifact is public and the API calls are real. Credit where due.\n\nBut there are two soft spots. First, the 95,168 call count cannot be reconciled with the stated protocol. The full dataset is 3,990 + 500 + 24; 12 techniques x 4 models x 3,990 samples alone is 191,520. To land near 95k they must have subsampled VQA-MHUG or varied technique/model coverage. No subsampling is disclosed. The paper reports means to one decimal without confidence intervals or significance tests. Given that, the model-dependence deltas (Downsampler -6.6% on Realtime-mini vs -23.4% on GPT-5.4; GazeROI -3.1% vs -10.2%) are plausible but not quantitatively grounded. The direction is probably right — the token and latency patterns are consistent across datasets — but the accuracy-tier claims need error bars or at least per-cell sample sizes.\n\nSecond, the LLM-as-judge weights on DriVQA and VOILA-A are hand-set without human-agreement validation. With only 24 DriVQA items and a GPT-4.1 judge, the ordering of preprocessing tiers on open-ended quality could shift. Minor relative to the token/latency core, but it does weaken the accuracy conclusions.\n\nThe claim that preprocessing effectiveness is not intrinsic is correct in direction. This paper deserves serious referee time: the benchmark plus the token-accounting probes are a real contribution to mobile systems practice. A referee should demand a reconciled call count, confidence intervals, and a short human-judge spot check. Not a desk reject. I'd bring it to reading group, and I'd cite the token-accounting result.","headline":"Solid systems benchmark with real measurement work; the headline claim is probably right, but the run-count arithmetic doesn't add up and the accuracy deltas lack error bars.","tokens_in":37045,"tokens_out":1561,"would_cite":true,"duration_ms":16662,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Client-side image preprocessing for cloud VQA is not intrinsically helpful: its effect on accuracy, latency, payload, and token cost depends on the target model, API paradigm, and task.","keywords":["cloud-based VQA","client-side preprocessing","token accounting","latency measurement","vision-language models","gaze-guided cropping","cost-quality trade-off"],"falsifier":"Conduct a human-rater study on the same DriVQA and VOILA-A outputs and compare per-technique rankings: if GazeROI no longer beats Downsampler or compression methods under human scores, the paper's accuracy conclusions for open-ended tasks fail; separately, submit the same source image at identical resolution encoded at JPEG quality 1 and quality 100 and check that provider-reported image tokens are identical, which would confirm the decoded-pixel-geometry accounting mechanism.","tokens_in":35986,"feed_emoji":"🖼️","tokens_out":8503,"duration_ms":82437,"temperature":0.7,"pith_summary":"VQABench is a benchmark that treats client-side image preprocessing as the controlled variable in cloud-VLM-based visual question answering, asking what visual input a mobile assistant should offload when accuracy, latency, payload, and token cost all matter. Across 12 preprocessing techniques, 3 datasets, and 4 commercial VLMs (95,168 API calls), the paper finds that preprocessing effectiveness is not intrinsic to the technique: the same method can speed up one model–provider stack, barely affect another, and slow down a third. It also finds that smaller payloads do not mean fewer tokens, because providers bill image tokens on decoded pixel geometry rather than transmitted bytes, so compression-only methods save no tokens on pixel-based accounting. The practical upshot is that preprocessing strategy must be chosen jointly with model selection and task type, and that gaze-guided cropping without a thumbnail is the strongest cost-aware candidate on models whose token accounting rewards pixel reduction.","feed_headline":"Squeezing images before cloud VQA can backfire","feed_subtitle":"A 95,168-call benchmark shows preprocessing gains hinge on the model's token rules, not the technique.","key_machinery":"The controlled comparison matrix is the central object: 12 preprocessing variants (baseline, five global-reduction methods, three region-selective methods, and three thumbnail-augmented variants) crossed with three datasets and four commercial VLMs, with each request measured on accuracy, payload, tokens, and decomposed latency. The mechanism that explains the results is provider token accounting on decoded pixel geometry rather than transmitted bytes: pixel-sensitive models (tile-based or area-proportional charges) reward cropping and downsampling, flat-rate per-image models charge the same for any single image and multiply charges for each additional image part, and at least one hybrid endpoint assigns images to two token buckets based on aspect ratio.","core_discovery":"On its own terms, the paper establishes that client-side preprocessing reshapes the full cost-quality trade-off of a cloud-VLM VQA request across four dimensions—answer accuracy, payload size, token usage, and latency—and that the direction and size of the effect are governed by the target model, the API paradigm, the provider's token-accounting rule, and the task formulation. The evidence shows three regimes: compression-only methods preserve accuracy and shrink payload but save no tokens; cropping and downsampling save tokens only on providers that account per decoded pixel, while multi-image submissions (thumbnail-augmented or multi-crop) inflate tokens on fixed-rate and hybrid providers; and accuracy loss tracks what information a method discards—color and spatial detail cost less than removing the task-relevant region, with saliency-based and thumbnail-only inputs causing the largest drops and more hallucinated claims. A poorly chosen strategy can simultaneously raise cost and latency while lowering accuracy, and the paper recommends gaze-guided cropping as the knee point on the Pareto frontier for pixel-sensitive models.","pith_inferences":["Editorial inference: the aspect-ratio token-bucket finding implies a content-independent optimization—padding any non-square image to square before submission can cut image tokens by about 40% on that hybrid endpoint, a lever the benchmark's main analysis does not exploit.","Editorial inference: since provider token-accounting rules are undocumented and may change, the quantitative rankings are snapshots; the durable insight is the mechanism (decoded-pixel geometry vs. per-image charges), which should be re-tested when endpoints update.","Editorial inference: the accuracy reversals across models suggest a future system could route each user query to the model–preprocessing pair that maximizes expected quality per unit cost, treating preprocessing selection as part of model routing rather than a separate step.","Editorial inference: the saliency-crop failures imply that attention signals tied to user intent (gaze) are more reliable than bottom-up visual saliency for deciding what to keep, which suggests intent-conditioned cropping as a research direction."],"forward_implications":["Developers should co-select preprocessing with model and provider rather than applying a fixed reduction recipe, because the same technique moves along different accuracy-latency-cost frontiers on different stacks.","Lossy compression at quality 85 is the safe default: it preserves near-baseline accuracy across models and cuts payload, while saving no tokens on pixel-accounting providers.","Multi-image methods such as thumbnail-augmented or multi-crop ROI submissions should be avoided on flat-rate and hybrid providers, where they can inflate token usage by up to 416%.","Gaze-guided cropping without a thumbnail is the recommended knee point on pixel-sensitive models, roughly halving tokens while holding accuracy close to baseline.","On fixed-budget providers such as the tested flash-tier model, the unprocessed baseline can dominate every preprocessing variant, so preprocessing should be skipped."],"supporting_citations":[{"why":"Supplies the VQA-MHUG dataset of images, questions, and gaze recordings used for the main accuracy, latency, and token benchmarks.","marker":"[40]"},{"why":"Supplies the DriVQA driving-scenario dataset with 66–68 human responses per open-ended question.","marker":"[12]"},{"why":"Supplies the VOILA-A open-ended gaze-conditioned dataset and the prior gaze-alignment result it builds on.","marker":"[69]"},{"why":"Provides the gaze-guided ROI preprocessing method and the earlier claim that gaze reduces visual tokens by up to 93% on open VLMs.","marker":"[41]"},{"why":"Defines the VQA soft-accuracy metric used to score VQA-MHUG short answers.","marker":"[47]"},{"why":"Provides the LLM-as-judge protocol that the paper adapts for grading open-ended DriVQA and VOILA-A answers.","marker":"[78]"},{"why":"Supplies the G-Eval style rubric used in the judge-based answer-quality scoring.","marker":"[79]"},{"why":"Documents the provider image-tokenization and tiling rule used to explain token-accounting behavior.","marker":"[27]"},{"why":"Documents the area-proportional image-token charge used to explain token savings on one provider.","marker":"[65]"},{"why":"Documents the image-understanding and token-handling behavior of the fixed-budget provider.","marker":"[66]"}],"fun_headline_variants":["VQA preprocessing: not a free lunch","Cloud VQA: preprocessing can hike cost and hurt accuracy","VQA benchmark: preprocessing gains depend on token rules","Preprocessing images for cloud VQA: know your token rules","Why squeezing images for cloud VQA can backfire"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the LLM-as-judge rubric scores for DriVQA and VOILA-A reflect genuine answer quality; if the judge mis-ranks answers on these open-ended tasks (the rubric weights are hand-set and no human-agreement check is reported), the accuracy-tier conclusions weaken even though the latency and token measurements stand.","fun_headline_variants_meta":{"raw":{"variants":["VQA preprocessing: not a free lunch","Cloud VQA: preprocessing can hike cost and hurt accuracy","VQA benchmark: preprocessing gains depend on token rules","Preprocessing images for cloud VQA: know your token rules","Why squeezing images for cloud VQA can backfire"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000581,"raw_usage":{"total_tokens":2783,"prompt_tokens":1037,"completion_tokens":1746,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":1683}},"tokens_in":653,"tokens_out":1746,"duration_ms":12027,"temperature":1.0,"reasoning_tokens":1683,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:45:26.362727+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Conduct a human-rater study on the same DriVQA and VOILA-A outputs and compare per-technique rankings: if GazeROI no longer beats Downsampler or compression methods under human scores, the paper's accuracy conclusions for open-ended tasks fail; separately, submit the same source image at identical resolution encoded at JPEG quality 1 and quality 100 and check that provider-reported image tokens are identical, which would confirm the decoded-pixel-geometry accounting mechanism.","supporting_citations":[{"cited_title":"VQA-MHUG: A gaze dataset to study multimodal neural attention in visual question answering,","cited_arxiv_id":null,"evidence_quote":"Supplies the VQA-MHUG dataset of images, questions, and gaze recordings used for the main accuracy, latency, and token benchmarks."},{"cited_title":"V oila-a: Aligning vision-language models with user's gaze attention,","cited_arxiv_id":null,"evidence_quote":"Supplies the VOILA-A open-ended gaze-conditioned dataset and the prior gaze-alignment result it builds on."},{"cited_title":"Eye gaze tells you where to compute: Gaze-driven efficient vlms,","cited_arxiv_id":null,"evidence_quote":"Provides the gaze-guided ROI preprocessing method and the earlier claim that gaze reduces visual tokens by up to 93% on open VLMs."},{"cited_title":"Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering,","cited_arxiv_id":null,"evidence_quote":"Defines the VQA soft-accuracy metric used to score VQA-MHUG short answers."},{"cited_title":"Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,","cited_arxiv_id":null,"evidence_quote":"Provides the LLM-as-judge protocol that the paper adapts for grading open-ended DriVQA and VOILA-A answers."},{"cited_title":"G-Eval: NLG evaluation using GPT-4 with better human alignment,","cited_arxiv_id":null,"evidence_quote":"Supplies the G-Eval style rubric used in the judge-based answer-quality scoring."},{"cited_title":"Images and vision,","cited_arxiv_id":null,"evidence_quote":"Documents the provider image-tokenization and tiling rule used to explain token-accounting behavior."},{"cited_title":"Vision - claude api docs,","cited_arxiv_id":null,"evidence_quote":"Documents the area-proportional image-token charge used to explain token savings on one provider."},{"cited_title":"Image understanding — gemini api,","cited_arxiv_id":null,"evidence_quote":"Documents the image-understanding and token-handling behavior of the fixed-budget provider."}],"review_version":1}