Pith. sign in

REVIEW 3 major objections 6 minor 43 references

ChatImage makes LLM answers clickable images

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-07 19:27 UTC pith:GHQD74VN

load-bearing objection Engineering pipeline for interactive LLM answers; missing the baseline comparison that would justify its central design choice. the 3 major comments →

arxiv 2607.05290 v1 pith:GHQD74VN submitted 2026-07-06 cs.CV

ChatImage: Navigating Long-Form LLM Answers through Interactive Images

Wencan Jiang , Jiangning Zhang , Yong Liu This is my paper
classification cs.CV
keywords interactive visual answersvisual groundingLLM output formattingtext-to-image generationhuman-computer interactionreferring expression localizationimage segmentationconversational AI
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ChatImage is a system that transforms a long-form text answer from a large language model into an interactive image with clickable hotspots. Each hotspot opens a detail panel and a scoped follow-up dialogue thread tied to a specific visual region. The core technical idea is a two-pass pipeline: first render the answer as a structured image, then run a visual grounding pass to locate where each answer module actually landed in the generated image. Hotspots are placed on these discovered regions rather than on the pre-generation layout plan, because image generators do not reliably place content at requested coordinates. The paper frames this as an interaction-design contribution rather than a new generative model, and releases a reference implementation plus a 30-question benchmark spanning infographic, map, and scene formats.

Core claim

The paper introduces the principle that interaction geometry for generated visual content should be grounded after rendering, not copied from the pre-generation layout plan. By decoupling content specification from spatial realization, ChatImage ties clickable regions to what is actually visible in the rendered image. The system uses any referring-expression localization model to recover the rendered position of each answer module, applies optional SAM-family mask refinement for tighter previews, and falls back through a graded hierarchy (general VLM grounding, OCR, planned bounds) when primary grounding fails. Evaluation on a 30-question benchmark shows 100% pipeline completion and a 70.8%严

What carries the argument

The generate-then-ground loop: (1) normalize the LLM answer into structured visual modules with titles, details, and locator phrases; (2) plan an approximate layout as normalized bounds; (3) generate the image; (4) run a visual grounding model to localize each module in the rendered image; (5) optionally refine with SAM-family segmentation; (6) overlay transparent clickable hotspots at the grounded coordinates. Each hotspot owns an isolated follow-up thread with region-scoped context.

Load-bearing premise

The system assumes that existing visual grounding models can reliably find answer modules in newly generated images across diverse visual styles, but the paper's own evaluation shows nearly 30% of checked hotspots fail the strict alignment gate and 45.8% of segmentation masks are incomplete.

What would settle it

If visual grounding models cannot localize rendered answer modules above chance in generated images, the hotspot layer would misalign with visible content, making the interactive image no better than random clickable boxes.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • As visual grounding models improve, the fidelity of region-interactive LLM answers improves automatically without changes to the ChatImage pipeline itself.
  • The generate-then-ground pattern could extend to other modalities where a generative model produces spatially structured output but does not reliably report where specific content landed, such as 3D scene generation or video summarization.
  • Per-region follow-up threads create a new interaction primitive: the user navigates an answer by clicking visible regions rather than scrolling text, which could change how long-form LLM responses are consumed in educational and technical-documentation contexts.
  • The alignment-source tagging system (grounding model, VLM fallback, OCR, planned fallback) provides a transparency mechanism that lets users know how much to trust each hotspot, which could become a standard feature in interactive AI outputs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the 29.2% strict-gate rejection rate persists across grounding-model improvements, the system may need an active re-generation loop that rejects and re-renders images when grounding fails, rather than falling back to planned bounds.
  • The rectangular hit-testing limitation suggests that polygon-based interaction would be a natural next step, but only once segmentation models reliably produce masks without holes or cavities — the paper reports 45.8% of masks are currently incomplete.
  • The benchmark's 30 questions with 5.5 average hotspots is small; a larger evaluation would reveal whether grounding quality degrades with answer complexity, module count, or visual density.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. ChatImage proposes converting long-form LLM answers into interactive visual images. The pipeline first normalizes a textual answer into structured visual modules, plans a layout, and generates an image. A second visual grounding pass then localizes rendered modules in the generated image and places clickable hotspots over them, enabling region-level inspection and follow-up dialogue. The central design principle is that pre-generation layout coordinates should serve as priors, not as final interaction geometry, because image generators do not reliably place content at requested positions. The authors release a reference implementation and a 30-question benchmark spanning infographic, map, and scene formats, reporting interaction-loop completion, a strict visual-alignment gate, and a SAM-based mask-completeness diagnostic.

Significance. The paper addresses an underexplored interaction design problem: making LLM-generated answers navigable at the region level through a visual interface. The generate-then-ground compositional pattern is a sensible engineering contribution that decouples content specification from spatial realization. The authors provide a reference implementation with code release, a benchmark, and per-hotspot alignment-source attribution, which is a commendable level of transparency. The system is model-agnostic in its grounding backend, which increases potential reproducibility and extensibility. However, the evaluation does not yet establish the empirical superiority of the central design principle over the simpler planned-bounds baseline it argues against, which limits the current significance of the contribution.

major comments (3)
  1. §1, §5 (central design principle) vs. §4 (evaluation): The paper's core claim is that post-rendering grounding improves hotspot alignment over using planned bounds directly, yet the evaluation never compares grounded hotspot placement against the planned-bounds baseline. Table 2 reports that 70.8% of hotspots pass a strict alignment gate and 12.2% fall back to planned bounds, but there is no measurement of whether the grounded 87.8% are actually better aligned than their planned counterparts would have been. If image generators place content roughly where requested, planned bounds might perform comparably, undermining the motivation for the second grounding pass. A direct comparison (e.g., IoU between planned bounds and ground truth vs. IoU between grounded bounds and ground truth) is needed to justify the central design principle.
  2. §4.1 and Table 2: The strict visual-alignment gate serves dual roles — it curates the public showcase (§4.1: 'retained only after strict visual-alignment screening') and functions as the primary evaluation metric (Table 2: 'Strict-gate passes 70.8%'). This creates a potential circularity: the gate both selects demo examples and measures their quality, with no independent ground truth (human annotation or alternative metric) to validate it. The paper should either introduce an independent evaluation criterion or explicitly acknowledge this limitation and discuss its impact on the reported numbers.
  3. §4.2, Table 2: The evaluation sample is extremely small — 24 checked hotspots from a 30-question benchmark (Table 1 reports 5.5 avg hotspots, implying ~165 total hotspots). The 70.8% strict-gate pass rate on 24 hotspots has a wide confidence interval, and it is unclear how the 24 were selected from the larger pool. The paper should clarify the sampling procedure and either expand the checked subset or provide justification for why this subset is representative.
minor comments (6)
  1. §3.2: The layout family selection rules and the semantic-slot catalog for maps are described qualitatively but not specified in detail. Providing the full rule set or referencing it in the supplementary material would aid reproducibility.
  2. §3.3: The grounding confidence thresholds and the criteria for rejecting groundings that 'resemble header strips or cross-panel artifacts' are not quantified. Specifying these thresholds would help readers assess the robustness of the fallback hierarchy.
  3. Figure 2: The qualitative results show only successful examples from the public demo gallery. Including failure cases (e.g., misaligned hotspots, fallback to planned bounds) would give readers a more balanced picture.
  4. §4.3, Table 3: The alignment-source distribution is reported for 90 hotspots, but Table 2 uses 24. The relationship between these two samples should be stated explicitly — are the 24 a subset of the 90, or are they drawn from different runs?
  5. §3.4: The claim that rectangular hit-testing 'preserves stable interaction behavior even when a mask is visually useful' is reasonable but could benefit from a brief user study or quantitative comparison showing that rectangular hotspots do not cause noticeable mis-clicks for irregularly shaped regions.
  6. The paper would benefit from a user study or qualitative feedback on whether the interactive visual image format actually improves user experience (navigation, comprehension, follow-up efficiency) compared to plain text. The current evaluation focuses entirely on pipeline diagnostics, not user-facing utility.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee correctly identifies that our evaluation does not yet directly justify the central design principle (post-rendering grounding vs. planned bounds), that the strict visual-alignment gate plays a dual role creating potential circularity, and that the checked subset is small. We agree with all three major comments and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: §1, §5 (central design principle) vs. §4 (evaluation): The paper's core claim is that post-rendering grounding improves hotspot alignment over using planned bounds directly, yet the evaluation never compares grounded hotspot placement against the planned-bounds baseline. A direct comparison (e.g., IoU between planned bounds and ground truth vs. IoU between grounded bounds and ground truth) is needed.

    Authors: The referee is correct. Our central design principle—that post-rendering grounding improves hotspot alignment over planned bounds—is not empirically validated by a direct comparison in the current manuscript. Table 2 reports grounding outcomes but does not measure the counterfactual: how well would planned bounds have performed on the same hotspots. We will add a direct comparison in the revision. Specifically, we will annotate ground-truth regions for the checked subset and report IoU between planned bounds and ground truth alongside IoU between grounded bounds and ground truth. This will directly test whether the second grounding pass provides measurable alignment improvement over the planned-bounds baseline. If image generators place content roughly where requested, this comparison will reveal it, and we will report the result transparently regardless of outcome. revision: yes

  2. Referee: §4.1 and Table 2: The strict visual-alignment gate serves dual roles — it curates the public showcase and functions as the primary evaluation metric — creating potential circularity with no independent ground truth.

    Authors: We agree that the dual role of the strict visual-alignment gate creates a circularity concern. The gate both selects demo examples and measures their quality, which is not a sound evaluation methodology. In the revision, we will introduce an independent evaluation criterion: human-annotated ground-truth bounding boxes for the checked subset, against which both planned and grounded bounds will be measured using IoU. The strict gate will be reported separately as a system-internal filtering mechanism, not as the primary evaluation metric. We will also explicitly acknowledge the circularity limitation in the current version and discuss how the independent annotation addresses it. revision: yes

  3. Referee: §4.2, Table 2: The evaluation sample is extremely small — 24 checked hotspots from ~165 total. The 70.8% pass rate has a wide confidence interval, and the selection procedure is unclear.

    Authors: The referee is correct on both counts: the sample is too small for confident conclusions, and the manuscript does not explain how the 24 hotspots were selected. In the revision, we will (1) clarify the sampling procedure explicitly, (2) expand the checked subset substantially to reduce the confidence interval, and (3) report confidence intervals on all reported rates. If practical constraints prevent expanding to the full ~165-hotspot pool, we will provide explicit justification for why the expanded subset is representative and will report the sampling protocol in full detail so readers can assess generalizability. revision: yes

Circularity Check

0 steps flagged

No significant circularity; the paper is an engineering pipeline, not a derivation chain

full rationale

ChatImage is a systems paper that proposes a generate-then-ground pipeline for converting LLM answers into interactive images. It does not claim first-principles derivations, fitted-parameter predictions, or mathematical uniqueness theorems. The central claim—that post-rendering visual grounding improves hotspot alignment over planned bounds—is an engineering hypothesis supported by a benchmark evaluation (Tables 2–3), not a derivation that reduces to its inputs by construction. The two self-citations ([13] SPIKE, [37] UniICL) involve overlapping authors but are cited as complementary related work on agent efficiency and multimodal in-context learning; neither is load-bearing for the paper's central design or evaluation. The skeptic's concern that the strict visual-alignment gate is used both to curate the public showcase (§4.1) and as an evaluation metric (Table 2) is a legitimate methodological concern about selection bias, but it is not circularity in the sense of a derivation being equivalent to its inputs: the gate is an external model-based check, not a fitted parameter renamed as a prediction, and the paper transparently reports rejection rates (29.2%) rather than claiming universal success. The evaluation is admittedly limited (no planned-bounds baseline comparison), but that is a correctness/completeness risk, not a circular reduction. Score 2 reflects the minor self-citations that are non-load-bearing.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The paper introduces no new physical entities, particles, or forces. The 'invented entities' are system artifacts and design patterns, which are standard for a systems paper. The free parameters are engineering thresholds (layout selection rules, keyword mappings, minimum area, confidence thresholds) that are hand-tuned rather than learned or derived. The axioms are domain assumptions about image generation and visual grounding reliability, most of which are supported by cited literature or the paper's own evaluation.

free parameters (4)
  • Layout family selection rules
    The mapping from visual mode and relation type to layout family (grid, flow, compare, hub, timeline, matrix, freeform) is described as selected by heuristics (§3.2) but the exact rules are not specified, implying hand-tuned thresholds.
  • Semantic-slot catalog for maps
    Keyword-to-position mappings (e.g., 'entrance' to bottom-center, 'core area' to center) are hand-defined (§3.2) without derivation or optimization.
  • Minimum area and safe margin for hotspot repair
    Grounded boxes are 'repaired for clickability by enforcing a minimum area and safe margin' (§3.3) but the specific thresholds are not stated.
  • Grounding confidence thresholds
    The system selects 'the most plausible box' and re-grounds 'when the initial candidate is uncertain' (§3.3), but the confidence thresholds for these decisions are not specified.
axioms (4)
  • domain assumption Image generators do not reliably place content at requested coordinates.
    Invoked in §1 and §3.1 to motivate the generate-then-ground design. Supported by cited references [6, 8]. This is a reasonable domain assumption.
  • domain assumption Existing vision grounding models can localize answer modules in generated images.
    The entire pipeline depends on this (§3.3). The paper's own data shows 70.8% strict-gate pass, partially supporting but also challenging this assumption.
  • domain assumption LLM answers can be meaningfully decomposed into structured visual modules.
    The normalization step (§3.2) assumes answers have internal structure amenable to visual module decomposition. Not all LLM answers have this property.
  • ad hoc to paper Rectangular hotspots are sufficient for region-level interaction.
    Click hit-testing remains rectangular even with mask-based previews (§3.3, §3.4). The paper acknowledges this as a limitation (§5).
invented entities (2)
  • Interactive visual image (as a defined artifact type) independent evidence
    purpose: The output artifact: a generated image with transparent clickable hotspots, detail panels, and per-region follow-up threads.
    The artifact is defined by construction and the reference implementation produces it. The 30-question benchmark and evaluation metrics provide falsifiable handles on whether the artifact can be reliably produced.
  • Generate-then-ground alignment pattern independent evidence
    purpose: The compositional pattern of rendering content first, then grounding interaction targets to visible regions.
    The pattern is instantiated in code and evaluated. Its effectiveness is measurable via the strict-gate pass rate and alignment-source distribution.

pith-pipeline@v1.1.0-glm · 12892 in / 2826 out tokens · 141265 ms · 2026-07-07T19:27:15.398839+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) can produce detailed answers to complex queries, but these answers are typically presented as dense linear text, which makes fine-grained inspection, navigation, and return visits difficult. We present ChatImage, a system that converts long-form LLM answers into interactive visual images. Given a textual answer, ChatImage first normalizes its content into structured visual modules, plans a visual layout, and renders a coherent image. It then applies a second grounding pass to the rendered image with vision grounding models such as LocateAnything and MiMo-Vision, with optional SAM-style mask refinement, to identify the visible regions that should support interaction. From these grounded regions, ChatImage overlays transparent clickable hotspots on the image. Each hotspot opens a detail panel and a region-scoped follow-up thread, allowing the user to inspect and query a specific part of the answer without re-reading the full response. Instead of treating planned coordinates as the final interaction geometry, ChatImage uses them as priors and grounds the interaction targets after rendering, which improves consistency between visual content and clickable regions. We release a reference implementation and introduce a 30-question benchmark covering infographic, map, and scene-based answer formats. Evaluation with configured external models reports interaction-loop completion, a strict visual-alignment gate, and a SAM-based mask-completeness diagnostic.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 43 canonical work pages · 12 internal anchors

  1. [1]

    The Claude model family.Anthropic Technical Report, 2024

    Anthropic. The Claude model family.Anthropic Technical Report, 2024

    work page 2024

  2. [2]

    VQA: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. InIEEE/CVF International Conference on Computer Vision, 2015

    work page 2015

  3. [3]

    Reranking individuals: The effect of fair classification within-groups

    Weixuan Chen, Bohan Liu, Daohan Zeng, Shitian Li, Liangli Huang, Chang Xu, and Chunjie Huang. Training-free layout-to-image generation with latent diffusion models.arXiv preprint arXiv:2401.13391, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  4. [4]

    InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    work page 2024

  5. [5]

    YOLO-World: Real-time open-vocabulary object detection.IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    Tianheng Cheng, Liming Song, Yixiao Ge, Wenhao Liu, Xinggang Wang, and Ying Shan. YOLO-World: Real-time open-vocabulary object detection.IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    work page 2024

  6. [6]

    Concatenating Binomial Codes with the Planar Code

    Yifu Cheng, Peng Zhou, Jiafei Cai, Zhongjie Wei, Wei Sun, Robby T Tan, and Gang Hua. Intelligent IG: Instruction-driven visual infographic generation and editing.arXiv preprint arXiv:2312.14390, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  7. [7]

    TransVG: Visual grounding with transformers

    Zhengyuan Deng, Jun Wang, Zihang Zhang, Yufan He, and Jianhua Feng. TransVG: Visual grounding with transformers. In IEEE/CVF International Conference on Computer Vision, 2021

    work page 2021

  8. [8]

    LayoutGPT: Compositional visual planning and generation with large language models

    Weixi Feng, Wanrong Zhu, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and Michael S Bern- stein. LayoutGPT: Compositional visual planning and generation with large language models. InAdvances in Neural Information Processing Systems, 2023

    work page 2023

  9. [9]

    Retrieval-Augmented Generation for Large Language Models: A Survey

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  10. [10]

    Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering.arXiv preprint arXiv:1612.00837, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  11. [11]

    Interactive poster visualization with PhD Online.IEEE Computer Graphics and Applications, 2004

    Harry Hochheiser and Ben Shneiderman. Interactive poster visualization with PhD Online.IEEE Computer Graphics and Applications, 2004

    work page 2004

  12. [12]

    Cryptography: Classical versus Post-Quantum

    Yiyang Hu, Liang Pang, Yanyan Lan, Jiafeng Xu, and Xueqi Cheng. Can large language models generate visual mind maps from text?arXiv preprint arXiv:2402.10988, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  13. [13]

    SPIKE: An Adaptive Dual Controller Framework for Cost-Efficient Long-Horizon Game Agents

    Wencan Jiang, Jiangning Zhang, Jianbiao Mei, Jinzhuo Liu, Yu Yang, Xiaobin Hu, Zhucun Xue, Yong Liu, and Dacheng Tao. SPIKE: An adaptive dual controller framework for cost-efficient long-horizon game agents.arXiv preprint arXiv:2605.18636, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  14. [14]

    MDETR: Modulated detection for end-to-end multi-modal understanding

    Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Caron. MDETR: Modulated detection for end-to-end multi-modal understanding. InIEEE/CVF International Conference on Computer Vision, 2021

    work page 2021

  15. [15]

    Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. InIEEE/CVF International Conference on Computer Vision, 2023

    work page 2023

  16. [16]

    Retrieval-augmented generation for knowledge-intensive NLP tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. InAdvances in Neural Information Processing Systems, 2020

    work page 2020

  17. [17]

    GLIGEN: Open-set grounded text-to-image generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. GLIGEN: Open-set grounded text-to-image generation. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023

    work page 2023

  18. [18]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InAdvances in Neural Information Processing Systems, 2023

    work page 2023

  19. [19]

    Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. InEuropean Conference on Computer Vision, 2024

    work page 2024

  20. [20]

    Counting Divisors in the Outputs of a Binary Quadratic Form

    Simeng Lu, Michele Mancini, Andrew Zisserman, and Mario Fritz. InfographicNLP: Expanding the scope of infographic analysis. arXiv preprint arXiv:2310.13632, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  21. [21]

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. InFindings of the Association for Computational Linguistics: ACL 2022, 2022

    work page 2022

  22. [22]

    Minesh Mathew, Dimosthenis Karatzas, and C.V. Jawahar. DocVQA: A dataset for VQA on document images. InIEEE/CVF Winter Conference on Applications of Computer Vision, 2021

    work page 2021

  23. [23]

    CRC Press, 2014

    Tamara Munzner.Visualization Analysis and Design. CRC Press, 2014

    work page 2014

  24. [24]

    LocateAnything-3B: A visual grounding model

    NVIDIA. LocateAnything-3B: A visual grounding model. Hugging Face model card,https://huggingface.co/nvidia/LocateAnything-3B,

    work page

  25. [25]

    Chart-to-Text: Generating textual descriptions of charts

    Hoque Obeid and Enamul Hoque. Chart-to-Text: Generating textual descriptions of charts. InFindings of the Association for Computational Linguistics: EMNLP 2020, 2020

    work page 2020

  26. [26]

    GPT-4 Technical Report

    OpenAI. GPT-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  27. [27]

    SDXL: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. InInternational Conference on Learning Representations, 2024

    work page 2024

  28. [28]

    Hierarchical Text-Conditional Image Generation with CLIP Latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents.arXiv preprint arXiv:2204.06125, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  29. [29]

    SAM 2: Segment Anything in Images and Videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. SAM 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  30. [30]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022

    work page 2022

  31. [31]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, R Konuk Sarrafpour, et al. Photorealistic text-to-image diffusion models with deep language understanding. InAdvances in Neural Information Processing Systems, 2022

    work page 2022

  32. [32]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. InAdvances in Neural Information Processing Systems, 2023

    work page 2023

  33. [33]

    ChartLlama: A Multimodal LLM for Chart Understanding and Generation

    Yifan Tang, Ao Zhang, Hengzhu Wang, Juan Li, and Zhijiang Hu. ChartLlama: A multimodal LLM for chart understanding and generation.arXiv preprint arXiv:2311.16483, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  34. [34]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InAdvances in Neural Information Processing Systems, 2022

    work page 2022

  35. [35]

    MiMo-VL: Xiaomi MiMo vision-language model

    Xiaomi LLM-Core Team. MiMo-VL: Xiaomi MiMo vision-language model. Model release,https://github.com/XiaomiMiMo/MiMo-VL,

    work page

  36. [36]

    Used in this work via themimo-v2.5chat-completion vision endpoint

    work page

  37. [37]

    LLM-oriented token- adaptive knowledge distillation

    Xurong Xie, Zhucun Xue, Jiafu Wu, Jian Li, Yabiao Wang, Xiaobin Hu, Yong Liu, and Jiangning Zhang. LLM-oriented token- adaptive knowledge distillation. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 16676–16684, 9

    work page

  38. [38]

    doi: 10.1609/aaai.v40i16.40701

    work page doi:10.1609/aaai.v40i16.40701

  39. [39]

    UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy

    Yicheng Xu, Jiangning Zhang, Zhucun Xue, Teng Hu, Ran Yi, Xiaobin Hu, Yong Liu, and Dacheng Tao. UniICL: Systematizing unified multimodal in-context learning through a capability-oriented taxonomy.arXiv preprint arXiv:2603.24690, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  40. [40]

    UNINEXT: Universal instance perception as object-in-context prompting

    Bin Yan, Yi Zhu, Jiangshan Wang, Zhihang Li, Yichao Zhang, Wu Liu, Erjin Zhang, and Jian Sun. UNINEXT: Universal instance perception as object-in-context prompting. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023

    work page 2023

  41. [41]

    ReAct: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations, 2023

    work page 2023

  42. [42]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. InIEEE/CVF International Conference on Computer Vision, 2023

    work page 2023

  43. [43]

    Mindmap: A creative visual thinking tool for education.Journal of Educational Technology Systems, 2014

    Yong Zhang. Mindmap: A creative visual thinking tool for education.Journal of Educational Technology Systems, 2014. 10

    work page 2014