Tabular PDF Information Extraction with Local LLMs and Layout-Aware Parsing: A Reliability Evaluation

Azhar Al Afghani; Kurnia Adi Cahyanto; Muhammad Anis Al Hilmi; Musfi Yuliadi; Neelansh Khare; Noel Framil Iglesias

REVIEW 3 major objections 2 minor 16 references

Tabular PDF Information Extraction with Local LLMs and Layout-Aware Parsing: A Reliability Evaluation

T0 review · 3 major / 2 minor · reviewed 2026-05-25 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A Camelot pipeline with LLM fallback extracts structured data from tabular academic PDFs at 0.99-1.00 accuracy and under one second per document on local CPUs.

desk verdict Applied eval of Camelot+LLM on narrow Indonesian KRS corpus; high automated scores but no human validation or broader testing. read the letter →

arxiv 2604.00003 v2 pith:ENVICIYC submitted 2026-03-09 cs.CL cs.AIcs.IR

Muhammad Anis Al Hilmi , Neelansh Khare , Noel Framil Iglesias , Kurnia Adi Cahyanto , Azhar Al Afghani , Musfi Yuliadi This is my paper · ORCID

classification cs.CLcs.AIcs.IR

keywords tabularPDFextractionlocalLLMsCamelotinformationacademicdocumentslayoutparsingreliabilityevaluationOllama

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests methods for extracting course and metadata from Indonesian university PDF registration forms that mix text and tables. It compares running LLMs alone, combining regex with LLMs, and using Camelot for tables plus LLM for the rest. Tests on 140 to 860 real documents from four programs measure exact matches and string similarity after local inference on consumer hardware. The Camelot pipeline with fallback gives the highest scores and fastest times, with Qwen 2.5 performing most steadily. The work shows how mixing rule-based parsing and local models can handle PDF variability without cloud resources or heavy compute.

What carries the argument

Camelot-based pipeline with LLM fallback, which applies layout-aware table parsing and delegates remaining metadata to local 12-14B LLMs when needed.

What would settle it

Running the pipelines on PDFs from additional institutions or with different table structures and encodings and finding exact match or Levenshtein scores below 0.9 would disprove the reliability results.

Watch

Extended reading notes

Core claim

The Camelot based pipeline with LLM fallback produced the best combination of accuracy (EM and LS up to 0.99 - 1.00) and computational efficiency (less than 1 second per PDF in most cases). The Qwen 2.5:14b model demonstrated the most consistent performance across all scenarios. Although not applicable to all models, the hybrid approach can improve efficiency compared to LLM only, especially for deterministic metadata.

Load-bearing premise

The 140 and 860 selected documents from four study programs, combined with exact match and Levenshtein similarity at 0.7 threshold, adequately represent extraction reliability without human validation or testing on more varied PDF layouts and encodings.

Editorial extensions

If this is right

Hybrid deterministic-LLM methods can improve efficiency for deterministic metadata extraction compared to LLM only.
Local 12-14B models via Ollama on CPU can reach near-perfect EM and LS scores on these document types.
Qwen 2.5:14b shows the most consistent performance across all tested scenarios and programs.
The Camelot pipeline with fallback scales to 860 documents while keeping per-PDF time under one second in most cases.
Integrating deterministic and LLM-based methods forms a reliable strategy for tabular text-based PDF extraction in constrained environments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same layout-aware plus LLM fallback pattern could extend to tabular PDFs in administrative or financial settings beyond education.
Local-only setups may reduce dependence on cloud LLM services for institutions with limited connectivity or budgets.
Human review on a sample of outputs could check whether the automated EM and LS metrics align with actual data usability.
Testing on PDFs with more diverse encodings or non-standard table layouts would reveal where the current accuracy drops.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

The core takeaway is that this paper runs a practical comparison of three extraction strategies on Indonesian course registration PDFs and finds the Camelot pipeline with LLM fallback fastest and most accurate on their metrics, with Qwen 2.5:14b the steadiest model. It tests everything locally on CPU with Ollama, which matches real constrained environments. That part is useful and straightforward. The work is new only in the choice of corpus; the methods are standard tools applied to 140-860 documents from four study programs. It does a reasonable job showing efficiency gains from the hybrid approach on deterministic metadata and reporting runtime under one second per file. The evaluation uses exact match and Levenshtein similarity at 0.7, which is fine for a quick benchmark. The soft spots are the evaluation itself. No human ground truth or inter-annotator agreement is described, so the 0.99-1.00 scores rest on whatever reference they used, and the narrow four-program set leaves open whether the results hold for other layouts or encodings. No error analysis or selection criteria appear in the abstract, which makes the reliability claim harder to assess. This is the kind of paper that helps people who actually process similar regional academic documents and need numbers on local LLMs. It is not for readers looking for new algorithms or general advances. I would bring it to a reading group focused on applied document work. I would not cite it in my own papers. It deserves peer review as a solid case study, though the referees will likely ask for validation details and wider testing.

Referee Report

3 major / 2 minor

Summary. The paper evaluates three information extraction strategies for tabular academic PDFs (Indonesian KRS course registration documents): LLM-only, hybrid deterministic-regex plus LLM, and a Camelot-based pipeline with LLM fallback. Experiments use three local 12-14B models (Gemma 3, Phi 4, Qwen 2.5) via Ollama on CPU, testing 140 documents for LLM variants and 860 for the Camelot pipeline across four study programs. Metrics are exact match (EM) and Levenshtein similarity (LS) at 0.7 threshold. The central claim is that the Camelot+LLM-fallback pipeline yields the best accuracy-efficiency trade-off (EM/LS up to 0.99-1.00, <1s per PDF) and that Qwen 2.5:14b is most consistent.

Significance. If the empirical results hold under more rigorous validation, the work provides a concrete, reproducible demonstration that layout-aware deterministic tools combined with local LLMs can deliver high-accuracy tabular extraction at low latency on consumer hardware. This is useful for domains with constrained compute and homogeneous document classes, and the explicit comparison of three model families plus hybrid strategies supplies practical guidance.

major comments (3)

[Evaluation / Abstract] Evaluation methodology (implicit in Experiments and Abstract): no details are given on how EM and LS@0.7 were computed against reference extractions, whether Unicode artifacts were normalized before comparison, or any per-field error breakdown; without this the reported 0.99-1.00 scores cannot be interpreted as evidence of semantic correctness.
[Experiments] Corpus construction (Experiments section): the 140/860 documents are drawn exclusively from four Indonesian study programs with no stated selection criteria, diversity analysis of layouts/encodings, or human ground-truth labeling; automated metrics alone on this narrow set risk overestimating reliability for the claimed use case.
[Results] Absence of statistical validation (Results): the paper reports raw EM/LS numbers and a qualitative statement that Qwen 2.5 is “most consistent” but provides no confidence intervals, significance tests, or inter-annotator agreement to support the comparative claims.

minor comments (2)

[Abstract] The abstract states “Although not applicable to all models” without clarifying which models or why; this phrasing should be expanded or removed.
No mention of code or data release; adding a reproducibility statement would strengthen the empirical contribution.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback, which highlights important areas for improving methodological transparency. We address each major comment below and will incorporate revisions to enhance clarity without altering the core empirical findings.

read point-by-point responses

Referee: [Evaluation / Abstract] Evaluation methodology (implicit in Experiments and Abstract): no details are given on how EM and LS@0.7 were computed against reference extractions, whether Unicode artifacts were normalized before comparison, or any per-field error breakdown; without this the reported 0.99-1.00 scores cannot be interpreted as evidence of semantic correctness.

Authors: We agree that explicit details on metric computation are necessary for interpretability. In the revised manuscript we will add a dedicated subsection in Experiments describing: (1) the exact procedure for computing EM (field-level string equality after extraction) and LS@0.7 (normalized Levenshtein distance with 0.7 threshold), (2) the preprocessing steps applied to both extracted and reference strings, and (3) a per-field error breakdown table. These additions will allow readers to assess whether the high scores reflect semantic correctness. revision: yes
Referee: [Experiments] Corpus construction (Experiments section): the 140/860 documents are drawn exclusively from four Indonesian study programs with no stated selection criteria, diversity analysis of layouts/encodings, or human ground-truth labeling; automated metrics alone on this narrow set risk overestimating reliability for the claimed use case.

Authors: We will revise the Experiments section to explicitly state the document selection criteria (all KRS documents obtainable from the four programs during the collection period) and provide a short analysis of layout and encoding homogeneity across the programs. We will also clarify that reference extractions were produced via manual annotation by the authors and note the domain-specific scope as a limitation on generalizability. revision: yes
Referee: [Results] Absence of statistical validation (Results): the paper reports raw EM/LS numbers and a qualitative statement that Qwen 2.5 is “most consistent” but provides no confidence intervals, significance tests, or inter-annotator agreement to support the comparative claims.

Authors: We acknowledge the benefit of quantitative support for comparative claims. In revision we will add bootstrap-derived confidence intervals for the reported EM and LS scores and include per-program variance to substantiate the consistency statement for Qwen 2.5:14b. Inter-annotator agreement is not reported because ground-truth labeling was performed by the research team rather than multiple independent annotators; we will make this process explicit. revision: partial

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: purely empirical evaluation on fixed corpus with standard metrics

full rationale

The manuscript performs an empirical comparison of three extraction pipelines (LLM-only, hybrid regex+LLM, Camelot+LLM fallback) on 140/860 KRS PDFs from four Indonesian programs. Accuracy is measured by exact match and Levenshtein similarity (threshold 0.7) against the selected reference documents. No equations, parameter fitting, predictions derived from fitted inputs, or load-bearing self-citations appear. All reported results are direct measurements on the chosen test set; the derivation chain contains no self-referential reductions.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Empirical evaluation study with no mathematical derivations, free parameters, or postulated entities.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Tabular PDF Information Extraction with Local LLMs and Layout-Aware Parsing: A Reliability Evaluation." pith.science (2026). https://pith.science/paper/ENVICIYC

@misc{pith2026260400003,
  author       = {Pith},
  title        = {Pith review of: Tabular PDF Information Extraction with Local LLMs and Layout-Aware Parsing: A Reliability Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ENVICIYC}},
  note         = {Machine review of arXiv:2604.00003}
}

read the original abstract

Extracting structured information from academic PDF documents is non trivial: a single page typically combines free text metadata with tabular regions, exhibits cross program variation, and is susceptible to Unicode encoding artifacts that interfere with downstream parsing. This study evaluates the reliability of information extraction approaches for tabular PDF documents, using academic course registration documents (Kartu Rencana Studi or KRS) from Indonesian higher education as a case study. Three strategies are compared: LLM only, Hybrid Deterministic - LLM (regex & LLM), and a Camelot based pipeline with LLM fallback. Experiments were conducted on 140 documents for the LLM based test and 860 documents for the Camelot based pipeline evaluation, covering four study programs with varying data in tables and metadata. Three 12 - 14B LLM models (Gemma 3, Phi 4, and Qwen 2.5) were run locally using Ollama and a consumer grade CPU without a GPU. Evaluations used exact match (EM) and Levenshtein similarity (LS) metrics with a threshold of 0.7. Although not applicable to all models, the results show that the hybrid approach can improve efficiency compared to LLM only, especially for deterministic metadata. The Camelot based pipeline with LLM fallback produced the best combination of accuracy (EM and LS up to 0.99 - 1.00) and computational efficiency (less than 1 second per PDF in most cases). The Qwen 2.5:14b model demonstrated the most consistent performance across all scenarios. These findings confirm that integrating deterministic and LLM based methods is a reliable and efficient strategy for information extraction from tabular text based PDF documents in computationally constrained environments.

Figures

Figures reproduced from arXiv: 2604.00003 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages

[1]

Pengembangan Sistem Manajemen Kartu Rencana Studi (KRS) berbasis Web (Studi Kasus: Sekolah Tinggi Teologi Tawangmangu),

J. N. Tunliu and T. A. Kurniawan, “Pengembangan Sistem Manajemen Kartu Rencana Studi (KRS) berbasis Web (Studi Kasus: Sekolah Tinggi Teologi Tawangmangu),” 2022. [Online]. Available: http://j - ptiik.ub.ac.id

work page 2022
[2]

Hubungan Penerapan Sistem Blok UNIMED terhadap Efektivitas Pembelajaran Statistik Pendidikan Mahasiswa

M. Rahma, N. Sufina Muslim Hutagalung, and P. Guru Sekolah Dasar, “Hubungan Penerapan Sistem Blok UNIMED terhadap Efektivitas Pembelajaran Statistik Pendidikan Mahasiswa”, [Online]. Available: https://jicnusantara.com/index.php/jiic

work page
[3]

Persepsi Mahasiswa Keperawatan Tentang Sistem Pembelajaran Blok dan Non -blok,

B. Permatasari, V . Elita, and W. N. Dewi, “Persepsi Mahasiswa Keperawatan Tentang Sistem Pembelajaran Blok dan Non -blok,” Cerdika: Jurnal Ilmiah Indonesia , vol. 2, no. 6, pp. 646 –655, Jun. 2022, doi: 10.59141/cerdika.v2i6.417

work page doi:10.59141/cerdika.v2i6.417 2022
[4]

Perancangan Sistem Informasi Krs Dan Khs Untuk Pengelolaan Jadwal Dan Nilai Perkuliahan,

M. S. Atillah, E. Sabrina, A. I. Saylendra, R. BASATHA, and N. D. W. I. SEPTIYANTI, “Perancangan Sistem Informasi Krs Dan Khs Untuk Pengelolaan Jadwal Dan Nilai Perkuliahan,” Jurnal Teknologi Pendidikan Dan Pembelajaran| E -ISSN: 3026-6629, vol. 3, no. 2, pp. 787–795, 2025

work page 2025
[5]

API Development Learning Platform,

D. Gala, G. Sutar, P. Chaurasia, and A. Joshi, “API Development Learning Platform,” in International Conference on Information and Communication Technology for Intelligent Systems , 2025, pp. 91 – 102

work page 2025
[6]

AI -driven web API testing,

A. Martin -Lopez, “AI -driven web API testing,” in Proceedings of the ACM/IEEE 42nd international conference on software engineering: companion proceedings, 2020, pp. 202–205

work page 2020
[7]

Mitigating turnover with code review recommendation: Balancing expertise, workload, and knowledge distribution,

E. Mirsaeedi and P. C. Rigby, “Mitigating turnover with code review recommendation: Balancing expertise, workload, and knowledge distribution,” in Proceedings of the ACM/IEEE 42nd international conference on software engineering, 2020, pp. 1183– 1195

work page 2020
[8]

A Comparative Study of PDF Parsing Tools Across Diverse Document Categories,

N. S. Adhikari and S. Agarwal, “A Comparative Study of PDF Parsing Tools Across Diverse Document Categories,” Apr. 2025, [Online]. Available: http://arxiv.org/abs/2410.09871

work page arXiv 2025

Show all 16 references

[9]

LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing,

Z. Ma, A. R. Chen, D. J. Kim, T. H. P. Chen, and S. Wang, “LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing,” in Proceedings - International Conference on Software Engineering, IEEE Computer Society, May 2024, pp. 1209–1221. doi: 10.1145/3597503.3639150

2024 doi
[10]

Exploring the Use of a Large Language Model for Data Extraction in Systematic Reviews: a Rapid Feasibility Study,

L. Schmidt et al. , “Exploring the Use of a Large Language Model for Data Extraction in Systematic Reviews: a Rapid Feasibility Study,” 2024. [Online]. Available: https://www.eshackathon.org/

2024
[11]

Evaluating the reliability of large language models for clinical data extraction in bladder cancer prognosis,

D. Sun et al. , “Evaluating the reliability of large language models for clinical data extraction in bladder cancer prognosis,” Sci. Rep., Nov. 2025, doi: 10.1038/s41598-025-27593-7

2025 doi
[12]

A comparative performance analysis of regular expressions and a large language model-based approach to extract the BI -RADS score from radiological reports,

F. Dennstädt et al. , “A comparative performance analysis of regular expressions and a large language model-based approach to extract the BI -RADS score from radiological reports,” JAMIA Open, vol. 8, no. 6, Dec. 2025, doi: 10.1093/jamiaopen/ooaf128

2025 doi
[13]

A Chinese document parsing and code recognition system using Regex and SVM,

J. Zhang and H. Yao, “A Chinese document parsing and code recognition system using Regex and SVM,” in 2021 IEEE 5th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC) , IEEE, Mar. 2021, pp. 1860 –

2021
[14]

doi: 10.1109/IAEAC50856.2021.9390975

2021 doi
[15]

Parsing Bibliography Descriptions using Few-Shot Prompting with LLM

K. Kassab and N. Teslya, “Parsing Bibliography Descriptions using Few-Shot Prompting with LLM.”
[16]

A Benchmark of PDF Information Extraction Tools Using a Multi-task and Multi- domain Evaluation Framework for Academic Documents,

N. Meuschke, A. Jagdale, T. Spinde, J. Mitrović, and B. Gipp, “A Benchmark of PDF Information Extraction Tools Using a Multi-task and Multi- domain Evaluation Framework for Academic Documents,” 2023, pp. 383 –405. doi: 10.1007/978 - 3-031-28032-0_31

2023 doi

Pith tools

Reviewed May 25, 2026 · model on record in the stance chip above.

[1] [1]

Pengembangan Sistem Manajemen Kartu Rencana Studi (KRS) berbasis Web (Studi Kasus: Sekolah Tinggi Teologi Tawangmangu),

J. N. Tunliu and T. A. Kurniawan, “Pengembangan Sistem Manajemen Kartu Rencana Studi (KRS) berbasis Web (Studi Kasus: Sekolah Tinggi Teologi Tawangmangu),” 2022. [Online]. Available: http://j - ptiik.ub.ac.id

work page 2022

[2] [2]

Hubungan Penerapan Sistem Blok UNIMED terhadap Efektivitas Pembelajaran Statistik Pendidikan Mahasiswa

M. Rahma, N. Sufina Muslim Hutagalung, and P. Guru Sekolah Dasar, “Hubungan Penerapan Sistem Blok UNIMED terhadap Efektivitas Pembelajaran Statistik Pendidikan Mahasiswa”, [Online]. Available: https://jicnusantara.com/index.php/jiic

work page

[3] [3]

Persepsi Mahasiswa Keperawatan Tentang Sistem Pembelajaran Blok dan Non -blok,

B. Permatasari, V . Elita, and W. N. Dewi, “Persepsi Mahasiswa Keperawatan Tentang Sistem Pembelajaran Blok dan Non -blok,” Cerdika: Jurnal Ilmiah Indonesia , vol. 2, no. 6, pp. 646 –655, Jun. 2022, doi: 10.59141/cerdika.v2i6.417

work page doi:10.59141/cerdika.v2i6.417 2022

[4] [4]

Perancangan Sistem Informasi Krs Dan Khs Untuk Pengelolaan Jadwal Dan Nilai Perkuliahan,

M. S. Atillah, E. Sabrina, A. I. Saylendra, R. BASATHA, and N. D. W. I. SEPTIYANTI, “Perancangan Sistem Informasi Krs Dan Khs Untuk Pengelolaan Jadwal Dan Nilai Perkuliahan,” Jurnal Teknologi Pendidikan Dan Pembelajaran| E -ISSN: 3026-6629, vol. 3, no. 2, pp. 787–795, 2025

work page 2025

[5] [5]

API Development Learning Platform,

D. Gala, G. Sutar, P. Chaurasia, and A. Joshi, “API Development Learning Platform,” in International Conference on Information and Communication Technology for Intelligent Systems , 2025, pp. 91 – 102

work page 2025

[6] [6]

AI -driven web API testing,

A. Martin -Lopez, “AI -driven web API testing,” in Proceedings of the ACM/IEEE 42nd international conference on software engineering: companion proceedings, 2020, pp. 202–205

work page 2020

[7] [7]

Mitigating turnover with code review recommendation: Balancing expertise, workload, and knowledge distribution,

E. Mirsaeedi and P. C. Rigby, “Mitigating turnover with code review recommendation: Balancing expertise, workload, and knowledge distribution,” in Proceedings of the ACM/IEEE 42nd international conference on software engineering, 2020, pp. 1183– 1195

work page 2020

[8] [8]

A Comparative Study of PDF Parsing Tools Across Diverse Document Categories,

N. S. Adhikari and S. Agarwal, “A Comparative Study of PDF Parsing Tools Across Diverse Document Categories,” Apr. 2025, [Online]. Available: http://arxiv.org/abs/2410.09871

work page arXiv 2025