{"id":"551846e4-23d1-4890-af42-643fb972328c","arxiv_id":"2509.00909","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new gold-standard benchmark for deep hierarchical PDF segmentation of 49 legal textbooks, with a TOC-based parser and an LLM-refined pipeline that improves heading precision and boundary quality.","lead":"This paper presents HiPS, a benchmark of 49 legal textbooks with 9,812 manually curated headings, plus two PDF segmentation pipelines, one using the table of contents and one using an LLM with OCR and XML features. The result is evidence that TOC-based parsing is strong when metadata is complete, while LLM-refined parsing helps when it is not.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground truth is seeded by TOC metadata and validated with the TOC parser's own matching logic, creating a bias that threatens the central TOC-vs-LLM comparison; independent re-annotation is needed.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the ground truth is not independent of the TOC-based method's assumptions. I agree that this is the most serious threat to the central comparative claim. The paper itself discloses the single-annotator limitation (Section 5.4) and the use of the TOC-based PageParser's matching logic in ground truth construction (Section 3), so the concern is not manufactured. However, the concern does not warrant rejection: the authors provide a valuable dataset and a plausible pipeline, and the bias may not change the overall ranking. The proposed re-annotation test would provide direct evidence. Since the reader already recommends conditional acceptance, I do not adjust the verdict. I also note that the paper's code and data are promised open-source, which would make the re-annotation test feasible.","tokens_in":13802,"tokens_out":5694,"duration_ms":67239,"concrete_test":"Randomly select 10 of the 49 books; have a second annotator (or an LLM-based annotation protocol) independently mark headings, levels, and boundaries from the rendered PDFs without seeing the PDF outline metadata and without using Algorithm 1. Compute inter-annotator agreement (e.g., token-level F1 and level allocation accuracy) between the original GT and the independent GT. Then re-run the three main evaluations (P_ED/R_ED, edit tree distance, Pk/WindowDiff) for TOC-based PageParser, XML-OCR-GPT5, and one open-source baseline (e.g., Marker) against the independent GT. If the rank order of TOC-based vs LLM pipelines changes on the subset, or if agreement is low (<0.8) on levels 4–5, the original benchmark is not an unbiased target and the conditional central claim needs to be re-qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison in Section 5 is only as valid as the ground truth, and the ground truth is constructed in a way that favors the TOC-based method. Section 3 states that the outline metadata of the PDFs was used as the basis for the ground truth, and that after manual correction 'we used the core logic from the TOC-Based PageParser (see Algorithm 1) to detect all ground truth TOC data inside the textbook’s fulltext' to ensure every GT heading is detectable. This filtering step guarantees that every GT heading is reachable by the TOC parser's matching strategies (normalized exact/substring/fuzzy with threshold 80). Consequently, TOC-free methods are evaluated against a target that was partly defined by the TOC parser's own notion of a 'detectable' heading; legitimate headings that are visually obvious but fail the TOC parser's string matching (e.g., very short headings, headings with severe OCR splitting) are excluded from GT, so a TOC-free method that finds them is penalized as a false positive. This directly affects the central claim: the reported superiority of TOC-based heading detection and the 'deep-level recovery' advantage of the LLM pipeline could change if the GT were built independently of TOC metadata and of Algorithm 1's matching logic. Section 5.4's disclosure that a single annotator created the GT adds variance but does not address this systematic bias.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents HiPS, a benchmark and two segmentation pipelines for recovering hierarchical section structure from deeply nested legal textbooks. The authors release 49 open-access law books with 9,812 manually curated headings, levels, and page anchors, and compare a Table-of-Contents-based parser with LLM-refined parsers and existing tools on three subtasks: title detection, hierarchy allocation, and boundary assignment. The headline findings are that the TOC-based pipeline is strongest when PDF outline metadata is complete, while the LLM-refined pipeline (especially XML-OCR-GPT5) improves heading precision, deep-level recovery, and boundary quality when metadata is missing or noisy.","tokens_in":14018,"tokens_out":3959,"duration_ms":48510,"significance":"The benchmark and code are potentially valuable: deep-book hierarchical segmentation has little public gold-standard data, and the paper includes a broad comparison with open-source tools and multiple LLMs, with code release for replication. The TOC-based parser is simple and transparent, and the LLM-refined pipeline's integration of OCR whitespace cues and XML typography is a reasonable design. If the ground truth is valid, the results would support the proposed hybrid approach. However, the paper's central comparison is threatened by the way the gold standard was constructed from PDF outline metadata and then filtered with the TOC parser's own matching logic. Because this concern is load-bearing for the reported TOC-versus-LLM ranking, the resource is promising but the evidence is currently conditional.","major_comments":[{"comment":"","section":"Section 3 (Dataset)"},{"comment":"","section":"Section 5.4 (Threats to Validity)"},{"comment":"","section":"Sections 5.2 and 5.3"}],"minor_comments":[{"comment":"Typos and wording issues: 'Thiapproachess', 'meaningul', 'candin-dates', 'ann existing', and 'theses methods' should be corrected. The prose would benefit from a careful proofreading pass.","section":"Throughout"},{"comment":"The matching function MatchHeading is used in Algorithm 1 but not formally defined in the pseudocode. The normalization steps and threshold of 80 appear in the text; please make them part of the algorithm or provide a precise definition in a table.","section":"Section 4.2 and Algorithm 1"},{"comment":"The metrics P_ED and R_ED allow an edit distance of 2 for a true positive. Please specify whether the edit distance is computed on normalized strings, and whether insertions/deletions of punctuation or whitespace count toward the tolerance. This affects the comparability of the reported numbers.","section":"Section 5.1"},{"comment":"The edit-tree-distance plots use different y-axis scales across subfigures, making visual comparison misleading. Consider using a shared scale or annotating the scale difference explicitly in the caption.","section":"Figures 6 and 7"},{"comment":"The sampling from 259 books with outline metadata to 49 annotated files is not fully described. Please explain the random sampling procedure, the intended number of files per metadata depth, and why the final distribution has very few files at depths 1 and 2.","section":"Section 3"},{"comment":"The sentence 'The TOC-based PageParser cannot capture segments which are not part of a TOC' is important and could be made more prominent; it partially explains the method's lower boundary results at levels 4 and 5. It would help to quantify this limitation relative to the ground-truth coverage of the TOC metadata.","section":"Section 5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution whose main weakness is the ground-truth circularity identified in the stress-test note. I agree with the skeptic's assessment: the gold standard was seeded by the PDF outline metadata and verified with the TOC parser's own matching logic, which biases the central comparison. This is fixable in a revision by independent re-annotation of a sample and sensitivity analyses, so I do not recommend rejection. The paper would fit well in a document-analysis or digital-libraries venue. If the authors can address the circularity concern convincingly, the released benchmark would be a useful community resource."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is the resource: 49 open-access law books with 9,812 manually curated headings across up to seven hierarchy levels. That fills a genuine gap, since existing segmentation benchmarks are shallow and page-local. The paper also ships a sensible comparative setup, including a TOC-based parser and an LLM-refined pipeline that combines OCR whitespace cues, XML typography, and semantic filtering. Credit where due: the authors are candid that the TOC parser is not novel, they disclose the single-annotator ground truth, and they plan public code and data. That is more than most benchmark papers do.\n\nThe soft spot is real and it is in the middle of the evaluation. In Section 3, the ground truth is seeded from the PDF's own outline metadata, which is fine; but then the authors use the core matching logic of the TOC-Based PageParser (Algorithm 1) to 'ensure that all ground truth headings were detectable in the fulltext, i.e., there were no unmatched headings logged.' That filtering step prunes any heading that the TOC parser's normalized exact/substring/fuzzy matcher cannot find. Legitimate headings that are visually obvious but fail that matcher are excluded from the gold standard. So a TOC-free method that finds such headings gets penalized as a false positive, while the TOC parser cannot be penalized for missing them. This directly biases the precision/recall comparison in Section 5.1, and it also affects the edit-tree and boundary results through the constructed tree. The single-annotator issue is secondary; the circularity is systematic.\n\nThat said, I do not think the main qualitative conclusion collapses. The paper's claim is not 'TOC parser is best everywhere' but 'TOC parser is strongest when metadata is complete, LLM refinement helps when it is not.' That conditional claim is plausible and consistent with the figures. But the specific numbers, especially for the TOC parser's precision and recall, should not be taken at face value until the ground truth is re-annotated independently of Algorithm 1's matching logic, or at least until an ablation shows the results hold on the subset of headings that are not filtered by that logic.\n\nWho should read this: anyone building or evaluating hierarchical PDF segmentation tools, and people interested in the pitfalls of benchmarking against metadata-derived ground truth. It deserves a serious referee, but the referee should ask for a re-analysis of the ground truth construction before the numbers are trusted. I would accept it for peer review with major revision expected.","headline":"A genuinely useful benchmark for deep textbook segmentation, but the TOC parser's headline numbers are partly circular because the ground truth was filtered using the parser's own matching logic.","tokens_in":14607,"tokens_out":1822,"would_cite":true,"duration_ms":25040,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A benchmark of 49 open-access law books shows that a TOC-based parser wins when outline metadata is complete, while an LLM-refined pipeline improves heading precision and boundary quality when metadata is missing or noisy.","keywords":["PDF segmentation","hierarchical document structure","legal textbooks","table of contents extraction","LLM-based document parsing","OCR layout analysis","section boundary detection","benchmark dataset"],"falsifier":"Re-annotate a random subset of the 49 books with multiple independent annotators who do not see the PDF outline metadata; if the corrected hierarchy differs materially from the single-annotator outline-seeded ground truth, the ranking of the TOC-based versus LLM-refined pipelines could invert on those books.","tokens_in":13605,"feed_emoji":"📚","tokens_out":7031,"duration_ms":81169,"temperature":0.7,"pith_summary":"HiPS targets a problem most PDF parsers do not: recovering the full, multi-level section tree of an entire book, not just page-local headings. The paper's central claim is that two complementary pipelines together handle deep legal textbooks: a TOC-Based PageParser that exploits outline metadata when it is reliable, and an LLM-Refined PageParser that uses OCR whitespace cues and XML typography to find heading candidates and then asks a language model to filter and level them. To support this, the authors contribute a gold-standard dataset of 49 open-access law books with 9,812 manually curated headings, levels, and page anchors. A careful reader would care because deep textbook structure is what downstream applications like knowledge graphs need, and current tools are tuned to three-level research papers. The experiments suggest that no single approach dominates: metadata quality, not model size, decides which pipeline is superior.","feed_headline":"TOC matching plus LLM filtering recovers deep law-textbook headings","feed_subtitle":"When outline metadata is complete the TOC parser wins; when it is noisy, LLM refinement takes over.","key_machinery":"The pipeline pair is carried by two mechanisms. The TOC-Based PageParser uses a shared normalization step (squeezed spaces, stripped punctuation, lowercasing) and then three complementary matchers—exact, substring, and fuzzy (partial ratio at a threshold of 80)—to align outline headings with fulltext XML nodes and reading-order lines, thereby assigning section boundaries. The LLM-Refined PageParser selects candidates by detecting lines in 300-DPI OCR that are surrounded by whitespace, enriches them with XML typographic and positional features plus a short passage of trailing text, and lets an LLM filter false positives and assign consistent hierarchy levels across the whole book.","core_discovery":"The central discovery claimed in the paper is a division of labour rather than a new neural architecture. When a PDF's embedded table of contents is complete, a rule-based parser that normalizes heading text and matches it in the fulltext via exact, substring, and fuzzy criteria produces the most accurate blueprint, with edit-tree distances on hierarchy reconstruction lower than every tested alternative, especially at the deepest levels. When the outline is missing or noisy, that same parser fails; here a pipeline that selects heading candidates from 300-DPI OCR by looking for whitespace-surrounded lines, adds XML font/page features and a short trailing context, and then asks an LLM to filte","pith_inferences":["A direct extension the authors leave implicit is that the same dual-pipeline design could apply to non-legal textbooks or born-digital documents where outline metadata is systematically absent; the benchmark itself only covers law books.","Because publisher style appears to influence which method wins, a testable next step would be a publisher-aware calibration of fuzzy matching thresholds or LLM prompting, rather than a single global configuration.","If the single-annotator ground truth were re-annotated by multiple independent raters, the measured gap between the TOC-based and LLM-based pipelines could narrow or shift, making that re-annotation a worthwhile robustness check before the benchmark is used as a leaderboard.","The strongest LLM results came from the variant that combined OCR whitespace cues with XML features, suggesting that pure text-based models, no matter how large, will keep underperforming on layout-driven heading detection in visually formatted books."],"forward_implications":["Existing open-source document parsers are not reliable for deep legal textbook hierarchies, so the released 49-book benchmark can serve as a much-needed evaluation ground for document-global structure recovery.","When outline metadata is complete, a deterministic TOC matcher should be preferred over LLM-based heading detection, because it reconstructs deep hierarchy trees more accurately.","When outline metadata is missing or noisy, LLM refinement over structure-aware candidates is the stronger route, improving heading precision and boundary assignment.","Deep-level boundaries remain the hardest part for all methods, so the next practical step is to combine the TOC matcher's strong top-level results with LLM refinement focused on levels 4 and 5.","The benchmark's heading and level annotations provide a reusable target for knowledge-graph construction, entity extraction, and other tasks that depend on book-level structure."],"supporting_citations":[{"why":"Supplies the Zhang–Shasha edit-distance algorithm used to compare predicted hierarchy trees against the gold-standard outlines.","marker":"[24]"},{"why":"Defines Pk, one of the two metrics used to score section boundary assignment.","marker":"[4]"},{"why":"Defines WindowDiff, the second metric used to score boundary assignment and boundary oversegmentation.","marker":"[19]"},{"why":"Rule-based clustering of headings by font and prefix, the antecedent that motivates and contrasts with the TOC-based matching strategy.","marker":"[13]"},{"why":"Rule-based TOC extraction with an RBF-SVM ranker, informing the design and limits of the TOC-Based PageParser.","marker":"[17]"},{"why":"Sequence-labeling heading detection on XML-converted documents whose layout and text features the LLM-refined pipeline adapts.","marker":"[5]"},{"why":"Multimodal transition parser for legal contracts whose feature extraction the authors tested in preliminary experiments and found inadequate for deep hierarchies.","marker":"[14]"}],"fun_headline_variants":["TOC parser wins for clean law-book PDFs; LLM refines noisy ones","Best law-book PDF segmentation: TOC if complete, LLM if not","HiPS: match TOC when present; LLM context when missing","For legal books, TOC-based parsing beats LLM—unless TOC breaks","New benchmark: 49 law books, 9,812 headings for PDF parsing"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The ground truth headings and levels were built by hand-correcting the PDFs' own outline metadata and verified by a single annotator, so the evaluation assumes that this corrected outline is an unbiased target not unduly shaped by the TOC-based parser's matching logic.","fun_headline_variants_meta":{"raw":{"variants":["TOC parser wins for clean law-book PDFs; LLM refines noisy ones","Best law-book PDF segmentation: TOC if complete, LLM if not","HiPS: match TOC when present; LLM context when missing","For legal books, TOC-based parsing beats LLM—unless TOC breaks","New benchmark: 49 law books, 9,812 headings for PDF parsing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000356,"raw_usage":{"total_tokens":1763,"prompt_tokens":730,"completion_tokens":1033,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":929}},"tokens_in":474,"tokens_out":1033,"duration_ms":11771,"temperature":1.0,"reasoning_tokens":929,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:04:26.915481+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random subset of the 49 books with multiple independent annotators who do not see the PDF outline metadata; if the corrected hierarchy differs materially from the single-annotator outline-seeded ground truth, the ranking of the TOC-based versus LLM-refined pipelines could invert on those books.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Zhang–Shasha edit-distance algorithm used to compare predicted hierarchy trees against the gold-standard outlines."},{"cited_title":"Berger, and John D","cited_arxiv_id":null,"evidence_quote":"Defines Pk, one of the two metrics used to score section boundary assignment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines WindowDiff, the second metric used to score boundary assignment and boundary oversegmentation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Rule-based clustering of headings by font and prefix, the antecedent that motivates and contrasts with the TOC-based matching strategy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Rule-based TOC extraction with an RBF-SVM ranker, informing the design and limits of the TOC-Based PageParser."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Sequence-labeling heading detection on XML-converted documents whose layout and text features the LLM-refined pipeline adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Multimodal transition parser for legal contracts whose feature extraction the authors tested in preliminary experiments and found inadequate for deep hierarchies."}],"review_version":1}