{"id":"80dfee66-5cbb-4501-ae81-447479c4074b","arxiv_id":"2604.00003","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":3.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Camelot pipeline with LLM fallback achieves highest accuracy (EM/LS up to 0.99-1.00) and efficiency (<1s per PDF) compared to LLM-only or hybrid regex-LLM on 860 documents using local 12-14B models.","lead":"The paper evaluates three strategies for extracting structured data from tabular academic PDFs using local LLMs on Indonesian university registration forms. A smart generalist might read it to learn practical trade-offs in accuracy, speed, and hardware needs for real-world document processing without cloud services.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"High EM/LS scores rest on unvalidated automated metrics over narrow 4-program corpus","rationale":"The reader's weakest_assumption directly identifies the missing external validation step; this is the single load-bearing assumption for any claim of 'reliability.' The abstract supplies no counter-evidence (e.g., human review or cross-layout testing), so the concern stands. A positive result on the concrete test would strengthen the claim; a negative result would falsify it.","tokens_in":1856,"tokens_out":339,"duration_ms":14612,"concrete_test":"Randomly sample 50 documents from the 860 set; have two domain-expert annotators independently transcribe the target fields (course code, name, credits, etc.); compute precision/recall/F1 of the Camelot pipeline output against this human gold standard. If macro-F1 drops below 0.90 or disagreement exceeds 10%, the reliability claim is unsupported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline claim (Camelot+LLM fallback reaches EM/LS 0.99-1.00 with <1s latency; Qwen 2.5:14b most consistent) requires that exact-match and Levenshtein@0.7 against the chosen reference truly measure extraction correctness. The evaluation uses 140/860 KRS documents from only four Indonesian study programs; no human ground-truth labeling, no inter-annotator agreement, and no explicit handling of Unicode artifacts or layout variants outside this set are described. Without that, the metrics could simply reflect pattern-matching success on a homogeneous collection rather than reliable information extraction.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper evaluates three information extraction strategies for tabular academic PDFs (Indonesian KRS course registration documents): LLM-only, hybrid deterministic-regex plus LLM, and a Camelot-based pipeline with LLM fallback. Experiments use three local 12-14B models (Gemma 3, Phi 4, Qwen 2.5) via Ollama on CPU, testing 140 documents for LLM variants and 860 for the Camelot pipeline across four study programs. Metrics are exact match (EM) and Levenshtein similarity (LS) at 0.7 threshold. The central claim is that the Camelot+LLM-fallback pipeline yields the best accuracy-efficiency trade-off (EM/LS up to 0.99-1.00, <1s per PDF) and that Qwen 2.5:14b is most consistent.","tokens_in":1957,"tokens_out":589,"duration_ms":22735,"significance":"If the empirical results hold under more rigorous validation, the work provides a concrete, reproducible demonstration that layout-aware deterministic tools combined with local LLMs can deliver high-accuracy tabular extraction at low latency on consumer hardware. This is useful for domains with constrained compute and homogeneous document classes, and the explicit comparison of three model families plus hybrid strategies supplies practical guidance.","major_comments":[{"comment":"Evaluation methodology (implicit in Experiments and Abstract): no details are given on how EM and LS@0.7 were computed against reference extractions, whether Unicode artifacts were normalized before comparison, or any per-field error breakdown; without this the reported 0.99-1.00 scores cannot be interpreted as evidence of semantic correctness.","section":"Evaluation / Abstract"},{"comment":"Corpus construction (Experiments section): the 140/860 documents are drawn exclusively from four Indonesian study programs with no stated selection criteria, diversity analysis of layouts/encodings, or human ground-truth labeling; automated metrics alone on this narrow set risk overestimating reliability for the claimed use case.","section":"Experiments"},{"comment":"Absence of statistical validation (Results): the paper reports raw EM/LS numbers and a qualitative statement that Qwen 2.5 is “most consistent” but provides no confidence intervals, significance tests, or inter-annotator agreement to support the comparative claims.","section":"Results"}],"minor_comments":[{"comment":"The abstract states “Although not applicable to all models” without clarifying which models or why; this phrasing should be expanded or removed.","section":"Abstract"},{"comment":"No mention of code or data release; adding a reproducibility statement would strengthen the empirical contribution.","section":null}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback, which highlights important areas for improving methodological transparency. We address each major comment below and will incorporate revisions to enhance clarity without altering the core empirical findings.","responses":[{"response":"We agree that explicit details on metric computation are necessary for interpretability. In the revised manuscript we will add a dedicated subsection in Experiments describing: (1) the exact procedure for computing EM (field-level string equality after extraction) and LS@0.7 (normalized Levenshtein distance with 0.7 threshold), (2) the preprocessing steps applied to both extracted and reference strings, and (3) a per-field error breakdown table. These additions will allow readers to assess whether the high scores reflect semantic correctness.","revision_made":"yes","referee_comment":"[Evaluation / Abstract] Evaluation methodology (implicit in Experiments and Abstract): no details are given on how EM and LS@0.7 were computed against reference extractions, whether Unicode artifacts were normalized before comparison, or any per-field error breakdown; without this the reported 0.99-1.00 scores cannot be interpreted as evidence of semantic correctness."},{"response":"We will revise the Experiments section to explicitly state the document selection criteria (all KRS documents obtainable from the four programs during the collection period) and provide a short analysis of layout and encoding homogeneity across the programs. We will also clarify that reference extractions were produced via manual annotation by the authors and note the domain-specific scope as a limitation on generalizability.","revision_made":"yes","referee_comment":"[Experiments] Corpus construction (Experiments section): the 140/860 documents are drawn exclusively from four Indonesian study programs with no stated selection criteria, diversity analysis of layouts/encodings, or human ground-truth labeling; automated metrics alone on this narrow set risk overestimating reliability for the claimed use case."},{"response":"We acknowledge the benefit of quantitative support for comparative claims. In revision we will add bootstrap-derived confidence intervals for the reported EM and LS scores and include per-program variance to substantiate the consistency statement for Qwen 2.5:14b. Inter-annotator agreement is not reported because ground-truth labeling was performed by the research team rather than multiple independent annotators; we will make this process explicit.","revision_made":"partial","referee_comment":"[Results] Absence of statistical validation (Results): the paper reports raw EM/LS numbers and a qualitative statement that Qwen 2.5 is “most consistent” but provides no confidence intervals, significance tests, or inter-annotator agreement to support the comparative claims."}],"tokens_in":1604,"tokens_out":571,"duration_ms":42517,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core takeaway is that this paper runs a practical comparison of three extraction strategies on Indonesian course registration PDFs and finds the Camelot pipeline with LLM fallback fastest and most accurate on their metrics, with Qwen 2.5:14b the steadiest model. It tests everything locally on CPU with Ollama, which matches real constrained environments. That part is useful and straightforward. The work is new only in the choice of corpus; the methods are standard tools applied to 140-860 documents from four study programs. It does a reasonable job showing efficiency gains from the hybrid approach on deterministic metadata and reporting runtime under one second per file. The evaluation uses exact match and Levenshtein similarity at 0.7, which is fine for a quick benchmark. The soft spots are the evaluation itself. No human ground truth or inter-annotator agreement is described, so the 0.99-1.00 scores rest on whatever reference they used, and the narrow four-program set leaves open whether the results hold for other layouts or encodings. No error analysis or selection criteria appear in the abstract, which makes the reliability claim harder to assess. This is the kind of paper that helps people who actually process similar regional academic documents and need numbers on local LLMs. It is not for readers looking for new algorithms or general advances. I would bring it to a reading group focused on applied document work. I would not cite it in my own papers. It deserves peer review as a solid case study, though the referees will likely ask for validation details and wider testing.","headline":"Applied eval of Camelot+LLM on narrow Indonesian KRS corpus; high automated scores but no human validation or broader testing.","tokens_in":2449,"tokens_out":385,"would_cite":false,"duration_ms":17661,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"PDF table extraction pipeline (Camelot+LLM fallback) has no overlap with RS forcing chain","alignment":"orthogonal","rationale":"The paper evaluates hybrid deterministic-LLM pipelines for KRS PDF metadata/table extraction using regex, Camelot lattice/stream, EM/LS@0.7 metrics on a 4-program Indonesian corpus. Its machinery (layout-aware parsing, few-shot prompts, throughput on CPU-only) lies entirely in the domain of document engineering and has zero structural relation to any RS element (J-cost, φ-ladder, 8-tick period, distinction-to-spacetime forcing, Alexander duality for D=3, etc.).","tokens_in":48000,"confidence":"high","tokens_out":155,"duration_ms":5882,"cache_read_input_tokens":38528,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A Camelot pipeline with LLM fallback extracts structured data from tabular academic PDFs at 0.99-1.00 accuracy and under one second per document on local CPUs.","keywords":["tabular PDF extraction","local LLMs","Camelot","information extraction","academic documents","layout parsing","reliability evaluation","Ollama"],"falsifier":"Running the pipelines on PDFs from additional institutions or with different table structures and encodings and finding exact match or Levenshtein scores below 0.9 would disprove the reliability results.","tokens_in":2774,"feed_emoji":"📄","tokens_out":744,"duration_ms":40416,"temperature":0.7,"pith_summary":"This paper tests methods for extracting course and metadata from Indonesian university PDF registration forms that mix text and tables. It compares running LLMs alone, combining regex with LLMs, and using Camelot for tables plus LLM for the rest. Tests on 140 to 860 real documents from four programs measure exact matches and string similarity after local inference on consumer hardware. The Camelot pipeline with fallback gives the highest scores and fastest times, with Qwen 2.5 performing most steadily. The work shows how mixing rule-based parsing and local models can handle PDF variability without cloud resources or heavy compute.","feed_headline":"Camelot pipeline reaches 99 percent accuracy on PDF table extraction","feed_subtitle":"LLM fallback on consumer CPU processes each academic form in under one second with top consistency from Qwen model","key_machinery":"Camelot-based pipeline with LLM fallback, which applies layout-aware table parsing and delegates remaining metadata to local 12-14B LLMs when needed.","core_discovery":"The Camelot based pipeline with LLM fallback produced the best combination of accuracy (EM and LS up to 0.99 - 1.00) and computational efficiency (less than 1 second per PDF in most cases). The Qwen 2.5:14b model demonstrated the most consistent performance across all scenarios. Although not applicable to all models, the hybrid approach can improve efficiency compared to LLM only, especially for deterministic metadata.","pith_inferences":["The same layout-aware plus LLM fallback pattern could extend to tabular PDFs in administrative or financial settings beyond education.","Local-only setups may reduce dependence on cloud LLM services for institutions with limited connectivity or budgets.","Human review on a sample of outputs could check whether the automated EM and LS metrics align with actual data usability.","Testing on PDFs with more diverse encodings or non-standard table layouts would reveal where the current accuracy drops."],"forward_implications":["Hybrid deterministic-LLM methods can improve efficiency for deterministic metadata extraction compared to LLM only.","Local 12-14B models via Ollama on CPU can reach near-perfect EM and LS scores on these document types.","Qwen 2.5:14b shows the most consistent performance across all tested scenarios and programs.","The Camelot pipeline with fallback scales to 860 documents while keeping per-PDF time under one second in most cases.","Integrating deterministic and LLM-based methods forms a reliable strategy for tabular text-based PDF extraction in constrained environments."],"fun_headline_variants":["Camelot pipeline with LLM fallback on PDF table extraction","Qwen 2.5:14b model consistent across PDF scenarios","Hybrid method improves efficiency over LLM only approach","Consumer CPU LLM tests on 860 academic PDF documents"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The 140 and 860 selected documents from four study programs, combined with exact match and Levenshtein similarity at 0.7 threshold, adequately represent extraction reliability without human validation or testing on more varied PDF layouts and encodings.","fun_headline_variants_meta":{"raw":{"variants":["Camelot pipeline with LLM fallback on PDF table extraction","Qwen 2.5:14b model consistent across PDF scenarios","Hybrid method improves efficiency over LLM only approach","Consumer CPU LLM tests on 860 academic PDF documents"]},"model":"grok-4.3","cost_usd":0.008561,"raw_usage":{"total_tokens":3920,"prompt_tokens":776,"num_sources_used":0,"completion_tokens":63,"cost_in_usd_ticks":85612000,"prompt_tokens_details":{"text_tokens":776,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3081,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":776,"tokens_out":63,"duration_ms":28593,"temperature":1.0,"reasoning_tokens":3081,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-25T06:59:19.098614+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Running the pipelines on PDFs from additional institutions or with different table structures and encodings and finding exact match or Levenshtein scores below 0.9 would disprove the reliability results.","supporting_citations":[],"review_version":1}