{"id":"d8b7afe4-517c-4324-a603-e686a17995e1","arxiv_id":"2501.17887","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"This technical report introduces the open-source Docling document conversion toolkit and benchmarks its conversion speed against Marker, MinerU, and Unstructured.io on CPU, Apple silicon, and GPU.","lead":"Docling is an open-source toolkit that converts PDFs, Office files, and images into one structured document format, aiming for fast local conversion. The paper benchmarks its speed against three open-source alternatives and reports competitive per-page times on commodity hardware.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Speed comparison may be workload- and configuration-biased: test set overlaps Docling's training data and Table 1 shows differing table-model settings, so the reported efficiency ranking could fail to generalize.","rationale":"The paper is a well-engineered open-source system description with a concrete speed benchmark. The central claim is about efficiency on commodity hardware, and the only empirical evidence is the Section 5 speed comparison. That evidence has two load-bearing weaknesses: (1) the test set is drawn largely from DocLayNet, the training distribution of Docling's own layout model, giving it an in-domain advantage that may affect the number and cost of downstream model invocations; (2) the configuration choices in Table 1 are not commensurable across tools (fast vs rapid vs hi-res table models) and the list of disabled features is omitted, so the comparison may not reflect equal output functionality. The reader's weakest_assumption identified the same cluster of concerns. The lack of a quality evaluation is also real, but it is an absence of evidence about accuracy rather than a direct threat to the stated speed numbers. A conditional acceptance requiring an independent benchmark and released configurations is appropriate; the concern reinforces the conditionality but does not move the verdict. The concrete test would settle whether the speed ranking generalizes and whether configuration choices drive the result.","tokens_in":10297,"tokens_out":6445,"duration_ms":59338,"concrete_test":"Contact the authors to obtain the exact list of disabled features and the 89-file benchmark set; then re-run all four tools on an independent, publicly available corpus (e.g., OmniDocBench) with identical table-recognition settings (or with quality-controlled outputs) and compare per-page median and mean times. A stronger test: rerun Docling with its full TableFormer model instead of the 'fast' variant and Unstructured with its 'hi res' table option; if Docling's speed advantage shrinks or reverses, the reported ranking is an artifact of configuration choices.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 says the benchmark set is 'based to a large extend on our DocLayNet dataset', and §4.1 says the layout model was 're-trained on DocLayNet'. This makes the test set in-domain for Docling but not for Marker, MinerU, or Unstructured. Since layout detection determines which regions trigger TableFormer and OCR, an in-domain advantage could change the distribution of model invocations and thus wall-clock time. More concretely, Table 1 shows Docling used 'TableFormer (fast)' while Unstructured used 'hi res' and MinerU used 'rapid table'; Section 5.3 states non-essential features were 'selectively disabled' but never lists them. Without quality measurements, a speed lead could simply reflect a faster-but-less-accurate table model or a reduced feature set. The paper also reports only Docling's median (0.79 s/page) while comparing competitors on mean times (Figure 5), and the 89-file set is not released, so the comparison cannot be independently audited. These issues threaten the central claim that Docling is efficient on arbitrary commodity-hardware workloads.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Docling, an MIT-licensed open-source document conversion toolkit developed at IBM Research. It describes Docling's architecture (the DoclingDocument data model, parser backends, and the PDF/simple pipelines), its AI models (a layout analysis model retrained on DocLayNet and TableFormer for table structure recognition), and its ecosystem integrations. The central performance claim is supported by a benchmark of 89 PDF files (4008 pages) comparing Docling with Marker, MinerU, and Unstructured on three system configurations: x86 CPU, Apple M3 Max, and NVIDIA L4 GPU. The paper reports per-page conversion times for Docling (median 0.79 s on x86 CPU, 0.32 s on M3 Max, 114 ms on L4 GPU) and average per-page times for all tools, with Docling leading on CPU and M3 Max and MinerU leading on GPU. The paper also emphasizes community adoption, GitHub stars, and integrations with LangChain, LlamaIndex, and other frameworks.","tokens_in":10471,"tokens_out":4127,"duration_ms":39216,"significance":"If the performance and quality claims are taken at face value, Docling is a practically useful open-source asset for local, privacy-preserving document conversion, particularly for RAG and data preparation workloads. The paper's strengths include a concrete speed benchmark with per-model runtime breakdowns, three well-specified hardware configurations, a clearly described modular architecture, and publicly available code and model weights. The speed benchmark is internally consistent and not circular in the sense that conversion time does not reduce to fitting model parameters. However, the paper's broader claims of state-of-the-art quality and accuracy are not measured at all, and the cross-tool speed comparison has several fairness and reproducibility gaps. These issues are load-bearing because the paper's contribution is framed as both efficient and highly accurate; the current evidence supports only the former, and only under in-domain and partially unspecified conditions.","major_comments":[{"comment":"The benchmark dataset is 'based to a large extend on our DocLayNet dataset' (§5.1), while §4.1 states that the layout analysis model was 're-trained on DocLayNet'. This makes the test set in-domain for Docling's layout model but out-of-domain for Marker, MinerU, and Unstructured. Since layout detection determines which pages trigger OCR and table structure recognition, an in-domain advantage could change the distribution of model invocations and therefore wall-clock time. The paper should either use an independent benchmark set, report per-category results that separate DocLayNet-derived material from CCpdf material, or provide evidence that layout accuracy differences do not materially affect runtime. Without this, the reported speed ranking may not generalize to arbitrary enterprise documents.","section":"§5.1 and §4.1"},{"comment":"The per-page statistics reported for Docling are medians (0.79 s/page on x86 CPU, 0.32 s/page on M3 Max, 114 ms/page on L4 GPU), but the comparison with other tools in Figure 5 and the accompanying text uses average times (3.1 s/page on x86 CPU, 1.27 s/page on M3 Max, 0.49 s/page on L4 GPU). Comparing Docling's median against competitors' means is not an apples-to-apples comparison and can bias the ranking, especially for long-tailed runtime distributions. The paper should report the same summary statistic for all tools, ideally both mean and median, and should make the per-document or per-page distributions available.","section":"§5.4"},{"comment":"The benchmarking methodology states that 'non-essential functionalities were selectively disabled' but never lists which functionalities were disabled for each asset. Table 1 also shows different table-structure settings across tools: Docling uses 'TableFormer (fast)', Unstructured uses 'hi res', and MinerU uses 'rapid table'. Without a full disclosure of disabled features and without any output-quality measurements, a speed advantage could simply reflect a faster-but-less-accurate table model or a reduced feature set. The authors should either align the feature sets across tools, fully specify all disabled options, or augment the speed comparison with quality metrics that rule out a speed-quality tradeoff.","section":"§5.3 and Table 1"},{"comment":"Section 8 explicitly defers quality evaluation to future work ('we will focus on building an open-source quality evaluation framework...'), yet the abstract and Section 2 describe Docling as powered by 'state-of-the-art' models and 'highly accurate'. Since the paper's central claim includes accuracy and state-of-the-art quality, the absence of any quality measurement is a load-bearing gap. The authors should either add a quality evaluation (e.g., layout, table structure, reading order, OCR accuracy on the same or an independent benchmark) or revise the claims to be strictly about speed, efficiency, and feature availability.","section":"§8 and Abstract"},{"comment":"The 89-file benchmark dataset is not released, and the paper does not provide exact commit hashes or pinned dependency versions for the tested tools, only version numbers. This prevents independent reproduction or auditing of the central speed comparison. The authors should publish the dataset (or a reproducible sampler with seeds), the exact configuration files, and per-document timing results so that the benchmark can be independently verified.","section":"§5.1 and §5.3"}],"minor_comments":[{"comment":"There is a typo: 'based to a large extend' should be 'based to a large extent'.","section":"§5.1"},{"comment":"The name 'Mineru' is used in Figure 5 and Table 1 while the text uses 'MinerU'; the capitalization should be consistent.","section":"Figure 5 and Table 1"},{"comment":"The vendor name is inconsistently spelled as 'Nvidia' in the text and 'NVIDIA' in the CUDA description; please standardize.","section":"§5.2"},{"comment":"The LlamaIndex reference contains a space inside the URL ('https://github.com/jerryjliu/llama index'), which should be corrected.","section":"References"},{"comment":"The phrase 'docling-ibm models' appears without proper formatting; it should be rendered as a package name, e.g., 'docling-ibm-models'.","section":"§4.1"},{"comment":"The claim that 'the open-source community has fully engaged' is subjective and not supported by evidence in the paper; consider reporting concrete contribution metrics instead.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"This is a system/tool paper whose main empirical contribution is a speed benchmark. The benchmark is described in detail but has three interrelated problems: in-domain test data, mismatched summary statistics, and undocumented feature disabling. None of these are fatal to the toolkit's practical value, but they are load-bearing for the cross-tool comparison. The paper also makes quality claims that are explicitly deferred to future work. The authors also lean heavily on their own datasets and models (DocLayNet, TableFormer), which is acceptable if the limitations are acknowledged, but the current framing overstates the evidence. If the venue publishes system reports, major revision is appropriate; if it requires rigorous comparative evaluation, the authors should be asked to add an independent quality benchmark or substantially narrow the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nDocling v2 is a real piece of engineering, and this paper is a fair technical report for what it is: a description of an open-source toolkit, a unified document model, and a speed benchmark. What's genuinely new is the integration itself—the custom qpdf-based parser, the DoclingDocument schema, and the modular pipeline—plus a concrete speed benchmark across three hardware configurations with per-model breakdowns. The authors cite their own earlier models (DocLayNet, TableFormer) properly, and the code is shipped, MIT-licensed, and demonstrably adopted with 10k GitHub stars and integrations into LangChain, LlamaIndex, and spaCy. That is real, reproducible evidence and deserves credit.\n\nThe soft spot is the benchmark's ability to support the 'efficient' claim for arbitrary workloads. The test set is largely drawn from DocLayNet, on which their layout model was re-trained, so it is in-domain for Docling but not for the competitors. Table 1 shows Docling used TableFormer (fast) while Unstructured got 'hi res' and MinerU 'rapid table'; Section 5.3 says non-essential features were selectively disabled but never lists them. That is a real concern, not a manufactured one. Also, Figure 5 compares means while the text highlights Docling's median (0.79 s/page), and the benchmark set and harness are not released, so the comparison cannot be independently audited. None of this makes the speed claim false, but it makes the ranking fragile. And there is no quality measurement anywhere, so 'state-of-the-art' is borrowed from the component models' earlier papers, not demonstrated end-to-end.\n\nWho is this for? Someone choosing a local document conversion toolkit for RAG will find the architecture and integration pointers genuinely useful. A researcher benchmarking converters should treat the speed numbers as hypotheses, not as a controlled comparison. It deserves a referee if the venue wants engineering reports; for a scientific venue, the missing quality evaluation and benchmark release are substantial but fixable.\n\nMy call: send it to peer review, but expect referees to demand the exact benchmark set and a public quality benchmark before the efficiency claim is fully credible.","headline":"A solid engineering report with a useful speed benchmark, but the self-referential test set and missing quality evaluation mean the efficiency claim is only partially supported.","tokens_in":11106,"tokens_out":1847,"would_cite":true,"duration_ms":18588,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Docling is an MIT-licensed, fully local document conversion toolkit that parses PDFs, Office files, images, and HTML into one structured format at sub-second page speeds.","keywords":["document conversion","PDF parsing","layout analysis","table structure recognition","OCR","retrieval-augmented generation","local inference","unified document representation"],"falsifier":"A reader could rerun the three system configurations on an independent corpus of, say, 100 business and legal PDFs never seen by DocLayNet, using default settings for all tools, and check whether Docling still achieves median sub-second page times and a CPU lead; a reversal in the ranking would show the speed claim does not generalize.","tokens_in":10090,"feed_emoji":"📄","tokens_out":7698,"duration_ms":60199,"temperature":0.7,"pith_summary":"Docling sets out to show that high-quality document conversion no longer requires a commercial cloud service or a large GPU cluster. The paper presents a single Python toolkit, released under the MIT license, that parses PDFs, scanned images, Office documents, HTML, and Markdown into a unified structure called DoclingDocument, with specialized local AI models for page layout and table recognition. Its headline evidence is a conversion-speed benchmark: median page conversion of 0.79 seconds on an x86 CPU, 0.32 seconds on an M3 Max SoC, and 114 milliseconds on an L4 GPU. The authors argue this combination of permissive licensing, fully local execution, and speed makes Docling a practical backbone for retrieval-augmented generation, fine-tuning data preparation, and information extraction.","feed_headline":"Docling parses a PDF page in 0.79 seconds on a CPU","feed_subtitle":"An MIT-licensed open-source toolkit brings layout, table, and OCR models to laptops and air-gapped environments.","key_machinery":"The carrying mechanism is the DoclingDocument data model, a Pydantic-based unified representation that holds text, tables, pictures, captions, section hierarchy, page provenance, and bounding boxes; every parser and pipeline writes into this one structure, which makes inspection, export, chunking, and framework integration uniform. Around it sits the StandardPdfPipeline, which chains a layout-analysis object detector derived from a real-time detection transformer and trained on DocLayNet with the TableFormer vision transformer for table structure, calling OCR only on scanned or bitmap-bearing pages. This modular separation means heavy models run only when needed and table text is matched back to existing PDF tokens instead of being re-transcribed, which keeps the pipeline fast and language-agnostic.","core_discovery":"The paper's central claim is that a modular, non-generative pipeline can deliver document conversion that is both faithful and fast enough for everyday local use. The core discovery is the architecture: parser backends ingest low-level formats (PDF, images) and markup formats, while a standard PDF pipeline applies a layout-analysis detector and the TableFormer table-structure model, engaging OCR only when bitmap content appears. Because all text comes from the PDF's programmatic tokens or from OCR rather than from a language model, the conversion cannot fabricate content. The benchmark reports that this pipeline converts a 4,008-page test set at median speeds of 0.79 sec/page on x86 CPU, 0.32 sec/page on M3 Max, and 114 ms/page on L4 GPU, and that Docling leads the compared open-source converters on CPU while running entirely locally. The paper also documents that disabling OCR and table recognition saves about 75% of runtime across all configurations.","pith_inferences":["The benchmark measures conversion speed only; no quality comparison against the same competitors is reported, so the paper's utility claim rests on an implicit assumption that speed parity implies similar output quality.","Because the test set draws heavily on DocLayNet, the dataset used to train Docling's layout model, the speed ranking could shift on document genres far from DocLayNet; an independent corpus would test this.","The large runtime share of OCR suggests that swapping OCR backends, or skipping OCR for born-digital PDFs, could yield near-interactive conversion for many office documents.","The authors list an open-source quality evaluation framework as future work; building it would let the community compare reading order, table structure, and transcription on public benchmarks, turning the speed advantage into an overall-utility claim."],"forward_implications":["At the reported median speeds, a 100-page document converts in roughly a minute and a half on a commodity CPU, making local conversion practical on laptops and edge servers.","Because text is extracted programmatically or via OCR rather than generated, Docling output cannot hallucinate content the way vision-language converters can, which matters for legal and medical transcription.","The unified representation with per-item bounding boxes and page provenance means downstream RAG can ground answers in exact page and location metadata.","The MIT license and local execution allow integration into commercial products, sensitive-data workflows, and air-gapped environments without licensing fees or data leaving the premises."],"supporting_citations":[{"why":"Supplies DocLayNet, the human-annotated layout dataset used to train Docling's layout model and to compose most of the benchmark corpus.","marker":"Pfitzmann et al. 2022"},{"why":"Introduces TableFormer, the vision-transformer model used in the PDF pipeline to recover table row and column structure.","marker":"Nassar et al. 2022"},{"why":"Refines TableFormer with a custom structure-token language for handling spans, hierarchy, and borderless tables.","marker":"Lysak et al. 2023"},{"why":"Provides the RT-DETR architecture from which the layout-analysis model is derived.","marker":"Zhao et al. 2023"},{"why":"Supplies the qpdf low-level library on which the custom docling-parse PDF backend is built.","marker":"Berkenbilt 2024"},{"why":"Augments the benchmark dataset with CCpdf samples to broaden genre variety beyond DocLayNet.","marker":"Turski et al. 2023"},{"why":"Marker is one of the three open-source conversion tools compared in the speed benchmark.","marker":"Paruchuri 2024"},{"why":"MinerU is a compared converter that takes the lead on the L4 GPU acceleration scenario.","marker":"Wang et al. 2024"},{"why":"Unstructured is the third local converter included in the speed comparison.","marker":"Unstructured.io Team 2024"}],"fun_headline_variants":["Docling: MIT-licensed AI toolkit for fast local PDF conversion","Docling converts PDFs to structured data in under a second","Docling's non-generative pipeline prevents document hallucination","Docling: 0.79s per page on CPU, 114ms on GPU, all local","Docling: the open-source converter that hit 10k stars in a month"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's representativeness rests on the assumption that the test set, built largely from DocLayNet pages, and the chosen versions and settings for the comparison tools reflect how hard real enterprise documents are for every converter equally.","fun_headline_variants_meta":{"raw":{"variants":["Docling: MIT-licensed AI toolkit for fast local PDF conversion","Docling converts PDFs to structured data in under a second","Docling's non-generative pipeline prevents document hallucination","Docling: 0.79s per page on CPU, 114ms on GPU, all local","Docling: the open-source converter that hit 10k stars in a month"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000302,"raw_usage":{"total_tokens":1737,"prompt_tokens":937,"completion_tokens":800,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":701}},"tokens_in":553,"tokens_out":800,"duration_ms":7740,"temperature":1.0,"reasoning_tokens":701,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:06:26.570955+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could rerun the three system configurations on an independent corpus of, say, 100 business and legal PDFs never seen by DocLayNet, using default settings for all tools, and check whether Docling still achieves median sub-second page times and a CPU lead; a reversal in the ranking would show the speed claim does not generalize.","supporting_citations":[{"cited_title":"S.; and Staar, P","cited_arxiv_id":null,"evidence_quote":"Supplies DocLayNet, the human-annotated layout dataset used to train Docling's layout model and to compose most of the benchmark corpus."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces TableFormer, the vision-transformer model used in the PDF pipeline to recover table row and column structure."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Refines TableFormer with a custom structure-token language for handling spans, hierarchy, and borderless tables."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the qpdf low-level library on which the custom docling-parse PDF backend is built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Augments the benchmark dataset with CCpdf samples to broaden genre variety beyond DocLayNet."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Marker is one of the three open-source conversion tools compared in the speed benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Unstructured is the third local converter included in the speed comparison."}],"review_version":1}