Pith. sign in

REVIEW 4 major objections 5 minor 26 references

CHaystack, a new Chinese DocumentVQA benchmark spanning papers, advertisements, web pages, and photographed documents, shows that open-source retrieval models lag far behind Qwen3-VL (71.91 vs 14.40 Recall@1), pointing to Chinese textual en

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

CHaystack, a Chinese document retrieval-and-VQA benchmark, shows Qwen3-VL reaching 71.91 Recall@1 versus 14.40 for the best non-Qwen model, and its VLM filter improves recall.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection The dataset is a genuine new resource, but the headline retrieval numbers only hold because the retriever is handed the gold category, so the open-domain claim doesn't stand. the 4 major comments →

arxiv 2607.24760 v1 pith:AF67DNJG submitted 2026-05-30 cs.IR

CHaystack: Benchmarking Chinese Document Retrieval and VQA

Hanxi Li This is my paper
classification cs.IR
keywords Chinese DocumentVQAbenchmarkdocument retrievalmultimodal RAGCDocRAGVLM relevance filterQwen3-VLanchor-based QA generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CHaystack is a new benchmark for large-scale Chinese document retrieval and question answering, built from four document categories: academic papers, advertisements, web pages, and photographed documents. The paper's central claim is that the benchmark is a valid testbed, and that current open-source multimodal systems perform poorly on it, especially on text-rich documents. Retrieval results show Qwen3-VL reaching 71.91 Recall@1 while the best non-Qwen model reaches only 14.40, which the authors attribute to the core challenge of Chinese textual encoding rather than visual understanding. The paper also introduces CDocRAG, a retrieval-augmented system whose VLM-based relevance filter improves retrieval recall before generation. If the benchmark is sound, it gives the Chinese DocumentVQA community a much-needed evaluation surface and pinpoints where model improvements should focus.

Core claim

CHaystack is constructed through an anchor-based pipeline: for text-rich documents (papers, webpages), DeepSeek-V3 plus OCR extract distinctive entities and values; for vision-rich documents (advertisements, photographed documents), Qwen3-VL extracts visual anchors. These anchors are converted into question-answer pairs, then filtered by an LLM and human reviewers to remove answer leakage, privacy leaks, over-general questions, and vague references. On the resulting 4,543 evaluation samples over a 17,488-image pool, Qwen3-VL-Embedding scores 71.91 Recall@1 overall, while the best non-Qwen model, Chinese-CLIP, scores 14.40; category-level results show Qwen models strong on papers and webpages

What carries the argument

The central mechanism is the 'anchor': a distinctive entity, phrase, number, or visual cue extracted from a document (by OCR plus LLM for text-rich pages, by Qwen3-VL for vision-rich pages) that ties a question to exactly one evidence document. The anchor is what makes query and answer generation document-specific and unambiguous. A second mechanism is the VLM relevance filter in CDocRAG, which makes a binary YES/NO judgment on each retrieved image before generation, removing distractors at the cost of occasional false rejections on dense technical pages.

Load-bearing premise

The load-bearing premise is that the anchor-extraction and filtering pipeline yields unambiguous, evidence-grounded QA pairs, and that the category-specific retrieval index does not leak the gold category to the retriever; if either fails, the benchmark rankings may not reflect Chinese DocumentVQA ability.

What would settle it

Re-run the retrieval experiment with a single open index over all 17,488 document images instead of four category-specific indexes; if non-Qwen Recall@1 rises materially above 14.40, the published gap overstates model weakness because the category was leaked to the retriever.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • CHaystack provides a new evaluation surface for Chinese large-scale document retrieval and VQA, where systems can be compared on both retrieval and generation.
  • The retrieval gap between Qwen3-VL and other embeddings implies that improving Chinese textual encoding in multimodal embeddings should yield large retrieval gains, especially on text-dense papers and webpages.
  • The VLM relevance filter raises Recall@1 from 71.82 to 78.32, suggesting that a light verification step between retrieval and generation is a cost-effective improvement.
  • Even with the gold evidence image supplied, the best generator reaches only 35.44 exact match, so document VQA generation over Chinese text remains a bottleneck independent of retrieval.
  • Category-wise results indicate that academic papers are the hardest category for generation and that advertisement retrieval responds well to visual cues, guiding where future benchmarks and models should concentrate effort.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because retrieval experiments use category-specific indexes, the gold category is effectively known to the retriever; an open-pool variant over all 17,488 images would test whether the reported Recall@1 numbers overstate open-domain retrieval skill.
  • The anchor-based pipeline inherits the quality of the generating models (DeepSeek-V3 and Qwen3-VL); if anchors are occasionally ambiguous or answer-leaky despite human filtering, part of the measured gap could reflect annotation artifact rather than model capability.
  • The photographed-document category has only 102 evaluation samples, so category-level conclusions there are less statistically stable than for the other three categories; expanding that category would strengthen the benchmark.
  • The results suggest a testable hypothesis: fine-tuning embeddings on Chinese text-heavy document triples should close much of the 71.91 versus 14.40 gap, a prediction that CHaystack itself can be used to verify.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces CHaystack, a Chinese DocumentVQA benchmark with 4,543 evaluation samples over a 17,488-image pool spanning academic papers, web pages, advertisements, and photographed documents. Questions are generated from document-specific anchors extracted by DeepSeek-V3 (text-rich documents) and Qwen3-VL (vision-rich documents), then filtered by LLM and human review. The paper also proposes CDocRAG, which inserts a VLM relevance filter between retrieval and generation. Experiments compare five open-source retrievers and three generators. Headline results are that Qwen3-VL reaches 71.91 Recall@1 while the best non-Qwen retriever (Chinese-CLIP) reaches 14.40, and that Qwen2.5-VL achieves the best gold-image generation scores. The paper concludes that the core challenge is Chinese textual encoding and that current open-source multimodal systems have substantial room for improvement.

Significance. If the benchmark construction and evaluation protocol are sound, CHaystack would be a useful new resource for Chinese DocumentVQA, filling a gap left by DuReader and MosaicDoc. The paper also makes a practical contribution in CDocRAG and provides a reproducible comparison point across five retrievers and three generators. However, the central retrieval claim is weakened by the category-specific retrieval index, and the benchmark-validity evidence is currently thin. The headline gap between Qwen and non-Qwen models may still hold after fixing the protocol, but the manuscript as written does not support the open-domain retrieval conclusion. The resource itself is valuable, but the evaluation needs to be re-run under a single unified index and the construction pipeline needs validation before the benchmark can be trusted.

major comments (4)
  1. [§5.1, Table 2] The implementation details state: 'we build category-specific image indexes and use the original question as the retrieval query.' Under this protocol, each question is matched only against the images of its gold category. In fact, Table 2's overall R@K is exactly the sample-weighted average of the per-category rows (e.g., Qwen3-VL overall R@1 = (2105×64.13 + 1319×85.75 + 1017×68.93 + 102×83.33)/4543 ≈ 71.91). This is only possible if the gold category is known in advance. The headline result therefore measures four category-constrained search tasks, not retrieval over the full 17,488-image pool, and the conclusion that 'the core challenge of CHaystack lies in Chinese textual encoding' is not supported as stated. Please rerun retrieval with a single unified index over all images and report the resulting recall numbers, or explicitly frame the category-constrained setting as a separate ta
  2. [§3, Benchmark construction pipeline] The benchmark's validity rests on the claim that the automatically generated anchors, followed by LLM and human filtering, produce unambiguous, evidence-grounded QA pairs. No inter-annotator agreement is reported for the human filtering stage, no residual-error analysis is given after filtering, and the number of samples rejected at each filtering step is not reported. Without this, it is hard to rule out leaked, ambiguous, or too-easy items, especially since the same Qwen3-VL used to construct visual anchors is also used as the top-performing retriever (a selection-bias risk). Please provide filtering statistics, a sample of rejected items, and inter-annotator agreement or an independent verification study on a subset of the final 4,543 samples.
  3. [§4 and §5.3, CDocRAG ablation] The CDocRAG method is a central contribution, but its benefit is only measured through retrieval Recall@K after filtering. No end-to-end experiment is reported that plugs the filter into the full retrieval-and-generation pipeline and measures EM/F1 against the no-filter baseline. Since the stated purpose is to 'improve answer generation,' the current ablation does not show that CDocRAG improves the actual DocumentVQA task. Please add an end-to-end comparison with and without the relevance filter.
  4. [§5.2, Tables 2 and 3] All reported numbers are point estimates without error bars, confidence intervals, or significance tests. The photographed-document category contains only 102 samples, so category-level differences such as the 67.65 EM for Qwen2.5-VL versus 66.67 for InternVL2.5 are likely within noise. Given the emphasis on category-wise strengths, the paper should report uncertainty or at least run multiple retrieval seeds and show variance.
minor comments (5)
  1. [Abstract] Typo: 'dateset' should be 'dataset'.
  2. [Table 1] The row 'Query Docs' is unclear; it would be clearer to label the columns as 'Queries' and 'Documents' and specify whether the numbers are in thousands.
  3. [§5.1, Models] The retrieval models are all under 3B parameters, but the Qwen3-VL embedding model is not clearly identified with its parameter count; please state the exact checkpoint used.
  4. [§5.3, Figure 4] The per-category bars are useful, but the figure would benefit from error bars and a table with the exact filter/no-filter numbers, since the current text quotes only selected values.
  5. [§2, Related work] The comparison with DuReader and MosaicDoc is clear, but the paper could cite recent English large-scale document retrieval benchmarks (e.g., Document Haystacks is mentioned, but no explicit comparison of the evaluation protocol is given).

Circularity Check

0 steps flagged

No significant circularity: empirical benchmark results and procedural system description, with no derived quantity equal to its own input.

full rationale

The paper makes no mathematical derivation chain; its central claims are benchmark measurements and a procedural description of CHaystack and CDocRAG. Question-answer pairs are generated from anchors extracted from source documents (DeepSeek-V3 + PaddleOCR for text-rich documents, Qwen3-VL for vision-rich documents) and then filtered by LLM and human review, so the gold evidence document is grounded in the source document rather than derived from the evaluation model. Using Qwen-family models both in construction and evaluation is a possible source-selection/contamination concern, but it is not circular: no parameter is fitted to a target and then reported as a prediction, and the retrieval/generation scores are independent measurements over the constructed pool. The category-specific image indexes in §5.1 are an evaluation-protocol concern (they prevent truly open-domain retrieval over the full 17,488-image pool), but this restricts the search space rather than defining the answer; it does not make any reported Recall@K equal to an input by construction. DuReader_vis is cited as an existing Chinese benchmark and is not load-bearing as self-citation; no uniqueness theorem, ansatz, or equation-level reduction is invoked. Therefore, the appropriate circularity score is 0.

Axiom & Free-Parameter Ledger

2 free parameters · 6 axioms · 0 invented entities

The benchmark and system rest on standard source datasets, LLM-based anchor extraction, and heuristic filtering. No physical or mathematical entities are invented. The main load-bearing assumptions are the quality of the generated anchors/filtering and the category-specific retrieval setup.

free parameters (2)
  • number of retrieved candidates before filtering = 15
    Hand-set in §5.1 without ablation; CDocRAG's measured improvement may depend on this budget.
  • number of filtered candidates passed to the generator = up to 10
    Hand-set in §5.1; the answer generation budget is not swept.
axioms (6)
  • domain assumption Source datasets (CDLA, DuReader, XFUND, CC-OCR, MTWI) provide accurate categories and document images.
    The benchmark is assembled from these sources; any mislabeling or noise propagates into queries and answers.
  • ad hoc to paper DeepSeek-V3 and Qwen3-VL extract document-specific anchors that uniquely identify the evidence document.
    This is the core construction premise of §3; no validation is provided that anchors are unique or distinctive.
  • ad hoc to paper LLM-based and human filtering removes all leaked, ambiguous, or unanswerable QA pairs.
    Filtering is qualitative; no inter-annotator agreement or residual-error analysis is reported.
  • domain assumption Retrieval is done within category-specific indexes, so the candidate pool is homogeneous per category.
    Section 5.1 states 'we build category-specific image indexes'; this assumes the category is known/leak-free and removes cross-category distractors.
  • domain assumption Exact match and character-level F1 after normalization adequately measure Chinese answer quality.
    Used without comparison to human judgment or alternative metrics; common but not validated here.
  • domain assumption A VLM's YES/NO relevance judgment is a valid proxy for whether an image contains sufficient evidence (§4).
    The filter's binary decision is treated as ground truth; errors are only partially visible in the ablation.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of CHaystack: Benchmarking Chinese Document Retrieval and VQA." pith.science (2026). https://pith.science/paper/AF67DNJG

@misc{pith2026260724760,
  author       = {Pith},
  title        = {Pith review of: CHaystack: Benchmarking Chinese Document Retrieval and VQA},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AF67DNJG}},
  note         = {Machine review of arXiv:2607.24760}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Retrieval-augmented generation (RAG) has made substantial progress in extending the memory of large language models (LLMs), and recent advances have further pushed RAG from pure text settings toward multimodal scenarios. In the document understanding domain, document visual question answering (DocumentVQA) has evolved from question answering over a single document to retrieval-and-generation pipelines over large-scale document collections. However, a benchmark specifically designed for Chinese large-scale document retrieval and question answering is still lacking. To bridge this gap, we introduce CHaystack, a new Chinese DocumentVQA benchmark that covers four document categories, namely academic papers, advertisements, web pages, and real-world photographed documents, enabling a more comprehensive evaluation of DocumentVQA systems. In addition, we present CDocRAG, a Chinese DocumentVQA system that uses a VLM-based relevance filter to verify retrieved document images before answer generation. We evaluate representative open-source embedding and generation models on CHaystack. The results reveal a clear contrast in category-wise strengths: Qwen-family models perform best on text-rich documents such as webpages and papers, whereas other models only achieve competitive results on visually rich categories such as advertisements and degrade sharply on text-dense documents. For retrieval, Qwen3-VL reaches 71.91 Recall@1 while the best non-Qwen model achieves only 14.40. These results indicate that the core challenge of CHaystack lies in Chinese textual encoding, and that Chinese large-scale DocumentVQA still leaves substantial room for improvement. Our code and dateset is available at https://github.com/hanxi19/CHaystack.

Figures

Figures reproduced from arXiv: 2607.24760 by Hanxi Li.

Figure 1
Figure 1. Figure 1: Comparison between previous and our benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Benchmark construction pipeline. values, or local facts that can uniquely identify the source document. For docu￾ments with rich visual content, including advertisements and photographed doc￾uments, we use Qwen3-VL [2] to extract visual anchors from the image content, such as salient objects, layout cues, visual attributes, and associations between text and images that are difficult to capture using OCR al… view at source ↗
Figure 3
Figure 3. Figure 3: Dataset profile of CHaystack. We report the distribution of evaluation samples and document images across four document categories. to the main content. In this way, CHaystack converts noisy OCR outputs into compact document-specific evidence for large-scale retrieval. Visual-anchor construction. For visually rich documents, we extract an￾chors that cannot be reliably captured by plain OCR alone. This part… view at source ↗
Figure 4
Figure 4. Figure 4: Effect of the VLM relevance filter on retrieval recall. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 2 canonical work pages

  1. [1]

    arXiv preprint arXiv:2502.13923 (2025)

    Bai, S., Chen, K., Liu, X., et al.: Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    Pith/arXiv arXiv 2025

  2. [2]

    Bai, S., et al.: Qwen3-vl technical report (2025) 12 H. Li

    2025

  3. [3]

    buptlihang: Cdla.https://github.com/buptlihang/CDLA(2026)

    2026

  4. [4]

    Chen, J., Xu, D., Fei, J., et al.: Document haystacks: Vision-language reasoning over piles of 1000+ documents (2024)

    2024

  5. [5]

    In: Proceedings of AAAI

    Chen, K., Chen, Y., Xue, Y.: Mosaicdoc: A large-scale bilingual benchmark for visually rich document understanding. In: Proceedings of AAAI. pp. 2913–2921 (2026).https://doi.org/10.1609/aaai.v40i4.37282

    work page doi:10.1609/aaai.v40i4.37282 2026

  6. [6]

    arXiv preprint arXiv:2412.05271 (2024)

    Chen, Z., Wang, W., Cao, Y., et al.: Internvl2.5: Expanding performance bound- aries of open-source multimodal models. arXiv preprint arXiv:2412.05271 (2024)

    Pith/arXiv arXiv 2024

  7. [7]

    In: Findings of ACL

    Chen, Z., Liu, G., Zhang, B.W., et al.: Altclip: Altering the language encoder in clip. In: Findings of ACL. pp. 8666–8682 (2023)

    2023

  8. [8]

    Cui, C., et al.: Paddleocr 3.0 technical report (2025)

    2025

  9. [9]

    DeepSeek-AI: Deepseek-v3 technical report (2024)

    2024

  10. [10]

    In: Proceedings of CVPR

    Goyal, Y., Khot, T., Summers-Stay, D., et al.: Making the v in vqa matter. In: Proceedings of CVPR. pp. 6904–6913 (2017)

    2017

  11. [11]

    In: Proceedings of ICPR

    He, M., Liu, Y., Yang, Z., et al.: Icpr2018 contest on robust reading for multi-type web images. In: Proceedings of ICPR. pp. 7–12 (2018)

    2018

  12. [12]

    Hu, Z., Iscen, A., Sun, C., et al.: Reveal: Retrieval-augmented visual-language pre- training with multimodal knowledge memory (2023)

    2023

  13. [13]

    Jeong, S., Kim, K., Baek, J., Hwang, S.J.: Videorag: Retrieval-augmented genera- tion over video corpus (2025)

    2025

  14. [14]

    Advances in Neural Information Processing Sys- tems33, 9459–9474 (2020)

    Lewis, P., Perez, E., Piktus, A., et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Sys- tems33, 9459–9474 (2020)

    2020

  15. [15]

    Trans- actions on Machine Learning Research2025(2025)

    Li, B., Zhang, Y., Guo, D., et al.: Llava-onevision: Easy visual task transfer. Trans- actions on Machine Learning Research2025(2025)

    2025

  16. [16]

    In: Proceedings of the International Conference on Learning Representations (2024)

    Lu, P., Bansal, H., Xia, T., et al.: Mathvista: Evaluating mathematical reason- ing of foundation models in visual contexts. In: Proceedings of the International Conference on Learning Representations (2024)

    2024

  17. [17]

    In: Findings of ACL

    Masry, A., Long, D.X., Tan, J.Q., et al.: Chartqa: A benchmark for question an- swering about charts. In: Findings of ACL. pp. 2263–2279 (2022)

    2022

  18. [18]

    Proceedings of WACV pp

    Mathew, M., Karatzas, D., Jawahar, C.: Docvqa: A dataset for vqa on document images. Proceedings of WACV pp. 2200–2209 (2021)

    2021

  19. [19]

    In: Proceedings of IJCAI

    Penamakuri, A.S., Gupta, M., Gupta, M.D., Mishra, A.: Answer mining from a pool of images: Towards retrieval-based visual question answering. In: Proceedings of IJCAI. pp. 1312–1321 (2023).https://doi.org/10.24963/ijcai.2023/146

    work page doi:10.24963/ijcai.2023/146 2023

  20. [20]

    In: Findings of ACL

    Qi, L., Lv, S., Li, H., et al.: Dureader_vis: A chinese dataset for open-domain document visual question answering. In: Findings of ACL. pp. 1338–1351 (2022)

    2022

  21. [21]

    In: Proceedings of ICML

    Radford, A., Kim, J.W., Hallacy, C., et al.: Learning transferable visual models from natural language supervision. In: Proceedings of ICML. pp. 8748–8763 (2021)

    2021

  22. [22]

    Xu, Y., Lv, T., Cui, L., et al.: Layoutxlm: Multimodal pre-training for multilingual visually-rich document understanding (2021)

    2021

  23. [23]

    arXiv preprint arXiv:2211.01335 (2022)

    Yang, A., Pan, J., Lin, J., et al.: Chinese clip: Contrastive vision-language pre- training in chinese. arXiv preprint arXiv:2211.01335 (2022)

    Pith/arXiv arXiv 2022

  24. [24]

    Yang, Z., Tang, J., Li, Z., et al.: Cc-ocr: A comprehensive and challenging ocr benchmark (2024)

    2024

  25. [25]

    In: Proceedings of ICCV

    Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language image pre-training. In: Proceedings of ICCV. pp. 11975–11986 (2023)

    2023

  26. [26]

    TechRxiv (2024)

    Zhang, R., Liu, C., Su, Y., et al.: A comprehensive survey on multimodal rag. TechRxiv (2024)

    2024

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.