Pith. sign in

REVIEW 1 major objections 6 minor 14 references

Rendering code as compact images cuts reported input tokens by 76–87 percent across three API providers, but the savings arrive only after a provider-specific break-even point.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 08:29 UTC pith:NFXPQEUB

load-bearing objection A careful, reproducible measurement snapshot of input-token accounting for code-as-images; headline ratios are solid, Gemini's crossover and page-boundary findings are single-shot and need replication. the 1 major comments →

arxiv 2607.21672 v1 pith:NFXPQEUB submitted 2026-07-23 cs.SE cs.AIcs.CV

Pixels for Programs? A Cross-Provider Case Study of Input-Token Accounting for Source Code as Text and Images

Ronak Bhalgami This is my paper
classification cs.SE cs.AIcs.CV
keywords vision-language modelssource codetoken accountingvisual compressionmultimodal APIsinput tokensimage-to-text ratioempirical software engineering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that when source code is sent to commercial vision-language APIs as compact rendered images instead of raw text, the APIs report far fewer input tokens—86.5% fewer for one provider, 80.6% for another, 75.8% for a third—but the aggregate numbers hide a regime where images cost more. The evidence comes from paired requests: five pinned source files, nine line counts from 20 to 2,000, 15 model aliases, and 675 complete text/image observations. A sympathetic reader should care because reported input tokens drive quotas and billing, so the size-stratified break-even curves give systems builders a concrete routing rule to test. The paper explicitly does not claim that the images preserve the code's information, improve task accuracy, lower latency, or reduce dollar cost; it measures only provider-reported token accounting.

Core claim

The central discovery is a size-stratified accounting surface: Anthropic and OpenAI report fewer input tokens for compact images at every tested size, while Gemini reports 6.95 times as many tokens for a 20-line image as for the same text and first reports fewer tokens only at 200 lines. A targeted audit also reproduces a non-monotonic jump in Gemini's reported image tokens at a page boundary, showing that image accounting is not a smooth function of characters, pixels, or pages. Aggregated over all 675 pairs, the weighted image-to-text ratios are 0.135 for Anthropic, 0.194 for OpenAI, and 0.242 for Gemini. The paper treats these figures as black-box provider-reported metadata, not as eviden

What carries the argument

The central object is a paired measurement pipeline: nested line-prefixes from five pinned open-source files, a compact-image transform that replaces four-space indentation with '>' markers and renders the result as PNG pages, and identical summarization-instruction requests sent as raw text and as images to the same provider model alias. The load-bearing identity is the weighted ratio R_G = sum(I_i) / sum(T_i), which weights every reported token equally and lets long contexts dominate, in contrast to an unweighted per-cell mean. Stratifying that ratio by source length exposes minimum-image-token overheads, crossover points, and the page-boundary discontinuity.

Load-bearing premise

The load-bearing premise is that the usage fields returned by the three providers during the July 2026 live collection window are stable and comparable representations of each provider's input-token accounting; if Gemini's page-boundary rule or Anthropic/OpenAI image minimums shifted mid-collection or differ by region, the break-even sizes and the non-monotonic anomaly would describe only that window.

What would settle it

Re-run the same pinned corpus and renderer against the same provider aliases now and check three values: whether the 20-line Gemini ratio is still near 6.95, whether Anthropic and OpenAI images are still below text at every tested size, and whether the Gemini 800-to-1,200-line page transition still drops reported image tokens sharply (2,322 to 516 in the paper's Python audit). A stable divergence in any of these would falsify the study's snapshot as a description of current provider accounting.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A coding harness that routes short requests as text and long read-mostly context as images can cut reported input tokens substantially, but the routing rule must be provider- and size-specific rather than a global 'send code as images' policy.
  • The reproduced non-monotonic page boundary means token-accounting studies should report results by page count and source size, not only as a single compression percentage.
  • Aggregate 'percent saved' figures are ambiguous: under unweighted cell averaging, Gemini's direction flips to a 54.8% increase, so any fair comparison must state its estimand.
  • The near-identical token signatures across many model aliases suggest providers share accounting rules within families, so alias-level replication is not independent evidence.
  • The measurement boundary implies that the next experiment must pair these token counts with exact-transcription fidelity, task quality, latency, and cost before deployment claims can be made.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A compact-text control arm—indent markers applied but no rasterization—would separate how much of the reduction comes from removing indentation spaces versus from vision-token pricing, a split the paper leaves open.
  • The page-boundary anomaly hints that providers rescale or repartition images internally; varying image dimensions, resolution, and page count in a follow-up could reveal the threshold and make token accounting predictable.
  • Because providers can change accounting policies at any time, the paper's own reporting contract—timestamps, request IDs, image hashes, and version identifiers—is what would make future drift visible and re-calibration possible.
  • The measurements imply that cross-provider 'cost per token' comparisons are unsafe, since the usage fields may not denote the same internal unit; the next study should price the actual requests and report cached versus uncached cost.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 6 minor

Summary. This paper reports a paired black-box measurement of provider-reported input-token usage for source code sent as raw text versus a compact-image pipeline (indentation markers plus rendered PNG pages) to Anthropic, OpenAI, and Google Vertex AI. The design uses five revision-pinned open-source files, nine nested prefix lengths, and 15 model aliases, yielding 675 complete text/image pairs. The main empirical findings are weighted image/text ratios of 0.135 (Anthropic), 0.194 (OpenAI), and 0.242 (Gemini). Size-stratified results show Anthropic and OpenAI below parity at all tested sizes, while Gemini is above parity through 100 lines and crosses below only at 200 lines. A targeted audit reports a non-monotonic Gemini page-boundary transition. The authors are careful to scope the result to one compact rendering pipeline and to reported token accounting, not to task quality, latency, or cost.

Significance. The contribution is a reproducible measurement surface rather than a new model or a task-quality result. Its main strengths are artifact discipline: raw JSON records, validators, revision-pinned corpus, deterministic table/figure regeneration, and an explicit reporting contract. The authors avoid inferential statistics given the nested-prefix/one-file-per-language design, and they report both weighted and unweighted estimands, which is important because the two give opposite signs for Gemini. The live-service temporal-stability threat is identified in Section 6 and is partly mitigated by the targeted audit and open raw data; the snapshot nature of the measurements is a limitation of scope, not an internal inconsistency. If the result is taken as a July 2026 snapshot, it is a useful basis for coding-harness routing studies.

major comments (1)
  1. [§4.5, Table 3, abstract] The non-monotonic Gemini page-boundary anomaly is a headline contribution, but the evidence is a single repeated request within the same collection window. The manuscript cannot rule out a mid-window provider-side change or regional routing artifact, and the abstract's word 'reproduces' overstates the support. Please add request timestamps/order and either provide a temporal stability check (e.g., split-half comparison or a later rerun) or rephrase the claim as 'reconfirmed within the same collection window' and explicitly state that the 200-line crossover and the page-boundary discontinuity are observations from that window only.
minor comments (6)
  1. [Abstract and §4.5] The word 'reproduces' should be tempered to 'reconfirms' or 'is consistent with' because the audit repeats the same requests in the same live-service period; it does not establish stability over time.
  2. [Table 3 and §4.3] The 800-line Gemini reduction of 80.7% corresponds to a ratio of 0.193, while the text says 0.194; reconcile the rounding so the table and prose agree.
  3. [Figure 2] The log-scale y-axis tick labels (e.g., '2 × 10−1', '3 × 10−1') are awkward; decimal labels (0.2, 0.3, 0.4, 0.6) would be clearer. Also, mixing '1.2k'/'1.6k' with '1,200'/'1,600' in Table 3 is inconsistent.
  4. [§3.4 and §7.1] The API region(s) used for collection are not reported. Since provider accounting may vary by region, add the region, SDK versions, and concrete model versions to the artifact metadata and reporting contract.
  5. [§2.3] The phrase 'Prompt caching attacks repeated work instead' should read 'addresses repeated work' or 'reduces repeated work'.
  6. [§5.3 and §6] The absence of a compact-text baseline is acknowledged in §6, but it is central enough to be listed explicitly as a required condition in the next-experiment design; otherwise the title's 'pixels' framing invites readers to attribute the reduction to modality alone.

Circularity Check

0 steps flagged

No material circularity: the paper reports arithmetic summaries of direct provider-reported measurements; no fitted parameter, self-citation, or definitional prediction is load-bearing.

full rationale

The paper's central claims are descriptive arithmetic summaries of 675 paired provider-reported usage fields. In Section 3.5, the primary ratio is defined as r_i = I_i / T_i, and the aggregate ratio is R_G = Sum(I_i) / Sum(T_i), with reduction stated as 100(1-R_G)%. These are quotients and algebraic restatements of the same recorded numbers, not derivations that presuppose the outcomes. The paper explicitly scopes the meaning of 'reduction': 'Here, "reduction" means only that the API returned a smaller input-token count' (Section 1) and 'it is not a monetary or compute saving' (Table 2 note). No parameter is fitted to a subset and then used to predict a closely related quantity; no uniqueness theorem is imported from the author's prior work; there are no self-citations at all; and the methodology does not rename an existing result. The acknowledged threats (live service window, undocumented routing, absence of a compact-text arm, nested-prefix dependence) are validity limitations, not circular reasoning. The only 'by construction' element is the definitional equivalence between image/text ratio and percentage reduction, which is a reporting transformation of identical inputs rather than a claim that an input defines its own output as evidence. The study is self-contained against its stated object of measurement: provider-reported token counts for a fixed pipeline and corpus.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The central claims are empirical aggregates, so the ledger contains no fitted constants and no invented entities. What the paper leans on: the fidelity of provider-reported usage fields, the pairing design, the workload choice, and the stability of live service behavior during the collection window. These are explicit or openly disclosed in Sections 3 and 6.

axioms (4)
  • domain assumption Provider-reported input-token usage fields are accurate records of the tokens each provider accounts for a request.
    Section 3.4 normalizes input_tokens / input-usage / prompt-usage fields and treats them as ground truth for the study; systematic underreporting or rounding by the APIs would shift every ratio.
  • domain assumption The paired design — identical pinned file prefix and identical instruction in both arms — isolates the effect of representation on the reported usage fields.
    Section 3.3 and 3.4 rely on this pairing; the authors note the image arm also changes indentation representation, so the isolation is partial.
  • domain assumption The five chosen files and nested prefixes are a meaningful workload for studying size-scaling, despite not being a random sample.
    Section 3.2 states one file per language, nested overlapping prefixes, and no inferential statistics; the claims are explicitly descriptive for these workloads.
  • domain assumption Live provider behavior during the July 2026 collection window is stable enough that single observations per cell are usable measurements.
    Section 6: 'Provider calls were made during a live service period rather than against frozen local weights' and 'undocumented service routing or updates could alter counts'.

pith-pipeline@v1.3.0-alltime-deepseek · 9534 in / 13543 out tokens · 129149 ms · 2026-08-01T08:29:33.137407+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Pixels for Programs? A Cross-Provider Case Study of Input-Token Accounting for Source Code as Text and Images." pith.science (2026). https://pith.science/paper/NFXPQEUB

@misc{pith2026260721672,
  author       = {Pith},
  title        = {Pith review of: Pixels for Programs? A Cross-Provider Case Study of Input-Token Accounting for Source Code as Text and Images},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NFXPQEUB}},
  note         = {Machine review of arXiv:2607.21672}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Long source-code contexts consume many text tokens, motivating the proposal to render code as images for vision-language models. Recent work asks whether models can still solve code tasks after this transformation. We examine a different systems question: how commercial APIs count the resulting requests. We present a reproducible measurement case study of provider-reported input tokens for raw source text and a compact rendered-image representation. The benchmark pairs requests across five programming languages, nine source lengths from 20 to 2,000 lines, and 15 available model aliases exposed by Anthropic, OpenAI, and Google Vertex AI. These aliases collapse to approximately five distinct accounting signatures and are not independent model replications. Across 675 complete text/image pairs, aggregate image-to-text ratios are 0.135, 0.194, and 0.242, corresponding to reported input-token reductions of 86.5\%, 80.6\%, and 75.8\%, respectively. These totals conceal materially different break-even behavior: Anthropic and OpenAI images receive lower counts at every tested size, while Gemini images require 6.95 times as many tokens at 20 lines and cross below text only at 200 lines in the aggregate. A targeted audit also reproduces non-monotonic Gemini image accounting across a page boundary. This study measures black-box request accounting for one compact rendering pipeline. It does not measure semantic fidelity, task accuracy, latency, monetary cost, or coding-agent efficiency. We release the scripts, revision-pinned corpus specification, raw usage records, validators, and deterministic analysis needed to reproduce and extend the study.

Figures

Figures reproduced from arXiv: 2607.21672 by Ronak Bhalgami.

Figure 1
Figure 1. Figure 1: Paired measurement pipeline. Both arms share the revision-pinned source prefix and task instruction, but the [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Weighted image/text input-token ratio by source length. Lower is better; the dashed line marks parity. Anthropic and [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 8 linked inside Pith

  1. [1]

    Jiale Cheng, Yusen Liu, Xinyu Zhang, Yulin Fei, Wenyi Hong, Ruiliang Lyu, Weihan Wang, Zhe Su, Xiaotao Gu, Xiao Liu, Yushi Bai, Jie Tang, Hongning Wang, and Minlie Huang. 2025. Glyph: Scaling Context Windows via Visual-Text Compression. (2025). arXiv:2510.17800 https://arxiv.org/abs/2510.17800

    arXiv 2025

  2. [2]

    In Gim, Guojun Chen, Seung seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. 2023. Prompt Cache: Modular Attention Reuse for Low-Latency Inference. (2023). arXiv:2311.04934 https://arxiv.org/abs/2311.04934

    Pith/arXiv arXiv 2023

  3. [3]

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 13358–13376. https://doi.org/10.18653/v1/2023.emnlp-main.825

    doi:10.18653/v1/2023.emnlp-main.825 2023

  4. [4]

    Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 1...

    doi:10.18653/v1/2024.acl-long.91 2024

  5. [5]

    Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeongyeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Se- unghyun Park. 2021. OCR-free Document Understanding Transformer. (2021). arXiv:2111.15664 https://arxiv.org/abs/2111.15664

    Pith/arXiv arXiv 2021

  6. [6]

    Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2022. Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding. (2022). arXiv:2210.03347 https://arxiv.org/abs/2210.03347

    Pith/arXiv arXiv 2022

  7. [7]

    Lotz, Elizabeth Salesky, Phillip Rust, and Desmond Elliott

    Jonas F. Lotz, Elizabeth Salesky, Phillip Rust, and Desmond Elliott. 2023. Text Rendering Strategies for Pixel Language Models. (2023). arXiv:2311.00522 https://arxiv.org/abs/2311.00522

    Pith/arXiv arXiv 2023

  8. [8]

    Paul Ralph, Nauman bin Ali, Sebastian Baltes, Domenico Bianculli, Jessica Diaz, Yvonne Dittrich, Neil Ernst, Michael Felderer, Robert Feldt, Antonio Filieri, Breno Bernard Nicolau de França, Carlo Alberto Furia, Greg Gay, Nicolas Gold, Daniel Graziotin, Pinjia He, Rashina Hoda, Natalia Juristo, Barbara Kitchen- ham, Valentina Lenarduzzi, Jorge Martínez, J...

    arXiv 2020

  9. [9]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cris- tian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, ...

    Pith/arXiv arXiv 2023

  10. [10]

    Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, and Desmond Elliott

    Phillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, and Desmond Elliott. 2022. Language Modelling with Pixels. (2022). arXiv:2207.06991 https://arxiv.org/abs/2207.06991

    Pith/arXiv arXiv 2022

  11. [11]

    Yuling Shi, Chaoxiang Xie, Zhensu Sun, Yeheng Chen, Chenxu Zhang, Longfei Yun, Chengcheng Wan, Hongyu Zhang, David Lo, and Xiaodong Gu. 2026. CodeOCR: On the Effectiveness of Vision Language Models in Code Under- standing. (2026). arXiv:2602.01785 https://arxiv.org/abs/2602.01785

    Pith/arXiv arXiv 2026

  12. [12]

    Haoran Wei, Yaofeng Sun, and Yukun Li. 2025. DeepSeek-OCR: Contexts Optical Compression. (2025). arXiv:2510.18234 https://arxiv.org/abs/2510.18234

    Pith/arXiv arXiv 2025

  13. [13]

    Hongbo Zhao, Meng Wang, Fei Zhu, Wenzhuo Liu, Bolin Ni, Fanhu Zeng, Gaofeng Meng, and Zhaoxiang Zhang. 2025. VTCBench: Can Vision-Language Models Understand Long Context with Vision-Text Compression? (2025). arXiv:2512.15649 https://arxiv.org/abs/2512.15649

    arXiv 2025

  14. [14]

    Jianping Zhong, Guochang Li, Chen Zhi, Junxiao Han, Zhen Qin, Xinkui Zhao, Nan Wang, Shuiguang Deng, and Jianwei Yin. 2026. Can Vision-Language Models Handle Long-Context Code? An Empirical Study on Visual Compression. (2026). arXiv:2602.00746 https://arxiv.org/abs/2602.00746 7

    arXiv 2026