{"id":"04a821e7-8a9d-4b4b-9d28-ce6669292b5e","arxiv_id":"2501.05082","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A comparison of NLP, computer vision, and multimodal methods for PDF metadata extraction, including a new TextMap approach, evaluated on two newly built datasets.","lead":"This paper compares machine learning and computer vision methods for pulling titles, authors, DOIs and other metadata out of the first page of scholarly PDFs, and introduces a new \"TextMap\" model that joins text meaning with page layout. The study also presents two datasets of papers with high layout variety, though the datasets and code are not yet released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported F1 scores are internally inconsistent and the 'highest F1' claim is contradicted by Table III (Vision-Language 0.92 vs TextMap-Word2Vec 0.913).","rationale":"The reader's verdict is REJECT, and the evidence supports that verdict. The most load-bearing problem is not the auto-labeling assumption, although that is also serious; it is the internal inconsistency of the reported results. The central claim requires TextMap-Word2Vec to have the highest F1-score, but Table III gives Vision-Language a higher score. Moreover, many F1 entries exceed the maximum possible value given the corresponding precision and recall, so the metric table cannot be correct as presented. This is a decisive correctness failure because it does not depend on external ground-truth quality: even with perfect labels, the numbers cannot all be true. The reader identified the contradiction in the strongest_claim and mentioned impossible scores in the rationale, but the weakest_assumption field focused on labeling quality. I partially agree: the labeling pipeline is a valid secondary concern, but the internal arithmetic is the sharper, load-bearing issue. Since the central empirical comparison is unsupported and no code or data is currently available to resolve it, the REJECT verdict should remain unchanged.","tokens_in":18311,"tokens_out":5160,"duration_ms":48738,"concrete_test":"Perform an arithmetic audit of every F1 entry in Tables III, V, VI, VII, VIII, IX, X, XI, and XII: recompute F1 = 2*P*R/(P+R) from the corresponding precision and recall in each row and flag every row where F1 > min(P,R). If the impossible entries reproduce, the F1 column is not derived from the reported precision/recall and the comparison collapses. If the authors intended a different aggregation (e.g., macro-averaged per-class F1 rather than F1 of macro P/R), they must provide the per-class confusion matrices or raw counts to verify all reported values, and they must explain how the S-PMRD and SSOAR-MVD numbers were computed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that TextMap-Word2Vec achieves the highest F1-score (0.913) on SSOAR-MVD. This is directly contradicted by Table III, which reports Vision-Language at 0.92. No qualification is given that these F1 values are computed under different protocols, so the claimed state-of-the-art result is unsupported by the paper's own data. The problem is deeper than a single misstatement: the F1 column is not arithmetically derivable from the reported precision and recall values. For any precision P and recall R, F1 = 2PR/(P+R) cannot exceed min(P,R). Yet Table VIII lists GROBID with macro precision 0.854, macro recall 0.794, and F1-score 0.821, which is impossible. Table V lists CRF Title with P=0.568, R=0.350, and F1=0.433, again impossible. Similar violations appear across Tables V-XII. This means the metric table cannot be trusted as a basis for comparing methods, independent of any concerns about the automatic labeling pipeline. The paper's footnote 4 states code 'will be released upon publication,' so no external artifact is currently available to resolve whether these are typographical errors or systemic miscalculations. The central empirical contribution is therefore not verifiable and the headline comparison is invalid.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript compares nine feature learning and prediction methods for extracting metadata from the first page of scholarly PDFs: CRF, BiLSTM, BiLSTM-CRF, GROBID, a Mask/Fast R-CNN vision model, a vision-language multimodal model, and a new TextMap approach instantiated with BERT, Word2Vec, and Char2Vec embeddings. The comparison is carried out on two datasets: SSOAR-MVD, a 50,000-document set synthesized from SSOAR metadata and templates, and S-PMRD, a curated subset of S2ORC. The paper reports token-level precision, recall, and F1 for each method, and claims in Section V.A that TextMap-Word2Vec achieves the highest F1-score of 0.913 on SSOAR-MVD. It also reports training and inference times and closes with a discussion of limitations.","tokens_in":18669,"tokens_out":6705,"duration_ms":59619,"significance":"If the reported results were reliable, the paper would offer a practical comparison of established and new extraction methods under high template variance, and the two datasets could serve as useful benchmarks. The proposed TextMap architecture, which fuses a grayscale page image with spatially interpolated text embeddings, is a plausible contribution. However, the central empirical claim is contradicted by the paper's own Tables III and IV, and the F1 scores in several tables are not arithmetically consistent with the stated precision and recall values. In addition, the SSOAR-MVD ground truth is generated by an unvalidated automatic labeling procedure. These issues undermine the headline comparison and the reliability of the datasets as presented, so the current contribution cannot be assessed as a trustworthy empirical study.","major_comments":[{"comment":"The claim that 'the proposed TextMap-Word2Vec method achieves the highest F1-score of 0.913' is directly contradicted by Table III, which reports Vision-Language with an F1-score of 0.920 and TextMap-Word2Vec with 0.913. The same ordering appears in Table IV for S-PMRD, where Vision-Language (0.903) edges out TextMap-Word2Vec (0.901). No qualification is given that these F1 values are computed under different protocols, so the paper's headline result is unsupported by its own data.","section":"Section V.A, Table III"},{"comment":"Many reported F1 values are not derivable from the listed precision and recall. For example, Table VIII (GROBID) gives Title precision 0.764 and recall 0.667, from which F1 = 2*0.764*0.667/(0.764+0.667) = 0.712, yet the table reports 0.951; the Abstract row (P=0.84, R=0.79) yields F1=0.814, not 0.935; the Email row (P=0.91, R=0.812) yields F1=0.858, not 0.893. These are not isolated typos but a pattern of systematic inconsistency in the metric tables, so the comparisons cannot be trusted without corrected computations or access to the underlying predictions. Footnote 4 states that the code 'will be released upon publication,' but no artifact is available to resolve the discrepancy.","section":"Tables V-XII"},{"comment":"The SSOAR-MVD ground truth is created by an automatic labeling procedure that assigns a text block to a metadata class when it has 'near-perfect similarity' to the corresponding SSOAR metadata value. The similarity measure and threshold are not specified, and no manual validation, inter-annotator agreement, or error analysis is reported. Every precision, recall, and F1 value in the paper depends on this unverified labeling; if the labels are noisy or biased, all cross-method comparisons are affected. The authors should define the similarity criterion, report its threshold, and provide evidence of labeling quality.","section":"Section IV.A.1"}],"minor_comments":[{"comment":"The subsection is titled 'Fast-RCNN' and the result tables refer to 'Fast-RCNN', but the described architecture is Mask R-CNN with a mask prediction branch and RoIAlign; this naming inconsistency confuses the method description.","section":"Section III.E"},{"comment":"The header 'Macro Avrerage' contains a typo; it should read 'Macro Average'.","section":"Table XI"},{"comment":"The footnote text 'Willbereleaseduponpublicaiton' is garbled; it should read 'Will be released upon publication.'","section":"Footnote 4"},{"comment":"Several references are duplicated: [3] and [24] are the same MexPub paper, [25] and [33] are the same Mask R-CNN paper, [26] and [34] are the same ResNeXt paper, and [9] and [27] are the same lecture-video metadata paper; these duplicates should be consolidated.","section":"References"},{"comment":"Some entries report standard deviations while others do not (e.g., '0.4 seconds' for TextMap-Word2Vec), and units are sometimes omitted after numeric values (e.g., '92 ± 6.2'); this makes the computational cost comparison difficult to interpret.","section":"Table XIII"},{"comment":"The introduction states that 'Section IV presents the dataset and experimental results,' but the results appear in Section V; the section cross-references should be corrected.","section":"Section I"}],"recommendation":"reject","confidential_remarks":"The paper's central claims are not supportable in its current form. The F1-score inconsistencies and the unvalidated automatic labeling of SSOAR-MVD are load-bearing flaws that would require recomputation of all metrics, validation or re-annotation of the dataset, and release of code before the comparison could be taken seriously. If the authors are able to provide corrected tables, a clear specification of the labeling procedure, and evidence of label quality, a resubmission could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the TextMap idea is worth a look, but the numbers as printed cannot be trusted. The reader's stress-test is right on both counts: Table III contradicts the headline claim, and the detailed tables list F1 scores that are not derivable from the reported precision and recall. That alone makes the empirical section unusable in its current form.\n\nWhat is genuinely new: TextMap, which interpolates token or block embeddings into a spatial feature map and fuses that with a grayscale page rendering via attention before Fast R-CNN, is a sensible extension of multimodal document analysis. The two datasets—SSOAR-MVD (synthetic, 50k) and S-PMRD (from S2ORC with PDFs)—are potentially useful resources. The paper also runs a broader set of baselines than most in this niche, which is helpful for practitioners.\n\nThe soft spots are serious. The F1 column in several tables violates F1 = 2PR/(P+R); for example, GROBID Title has P=0.764, R=0.667, and F1=0.951, which is mathematically impossible. The auto-labeling pipeline for SSOAR-MVD uses an unspecified “near-perfect similarity” threshold with no manual validation, so the ground truth itself is a question mark. S-PMRD uses fuzzy matching with more detail, but still no quality numbers. No code or data is released yet, despite the footnote promising it “upon publication.” And the central claim—that TextMap-Word2Vec achieves the highest F1 (0.913)—is contradicted by Table III, which reports Vision-Language at 0.92. These are not minor typos; they undermine the main comparative conclusion.\n\nWhere does that leave the paper? The method and datasets have merit, but the evaluation needs to be redone or at least re-reported with corrected tables and full artifact release. I would not desk-reject this. A serious referee could push for the code, the labeling thresholds, and a corrected metric table. If those come back and the numbers still favor TextMap, this could be a solid contribution. As it stands, you should treat any performance comparison in this paper as unreliable.\n\nRecommendation: send to peer review with a request for major revision, focusing on artifact release, labeling transparency, and a full re-reporting of metrics. I'd almost bring it to reading group just to discuss the architecture.","headline":"TextMap is a real architectural idea, but the paper's empirical claims are undercut by internally inconsistent F1 numbers and a headline result contradicted by its own Table III.","tokens_in":19125,"tokens_out":3126,"would_cite":false,"duration_ms":28005,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that TextMap-Word2Vec, a layout-aware approach that fuses word embeddings with page geometry, achieves the best metadata extraction score on a new 50,000-document benchmark—though its own results table reports a…","keywords":["metadata extraction","document processing","neural networks","natural language processing","computer vision","multimodal approaches","scientific documents","TextMap"],"falsifier":"Re-annotate a random sample of several hundred SSOAR-MVD pages with human labelers marking title, authors, abstract, DOI, and the other fields, then recompute F1 for TextMap-Word2Vec and Vision-Language on the same sample; if TextMap-Word2Vec no longer beats Vision-Language or drops below 0.913, the paper's central claim fails. A simpler check is to read Table III directly: it already lists Vision-Language at 0.92, above the claimed 0.913, so the \"highest\" assertion is internally falsified unless the authors meant \"best among newly proposed methods.\"","tokens_in":18143,"feed_emoji":"📄","tokens_out":5954,"duration_ms":52085,"temperature":0.7,"pith_summary":"This paper evaluates ways to pull metadata—title, authors, abstract, DOI, and similar fields—out of the first page of a scholarly PDF, focusing on documents with highly varied templates, such as those from small and mid-sized German social-science publishers. Its central proposal is TextMap, which maps text embeddings onto the spatial coordinates of the page, combines them with a grayscale layout image through attention, and runs a Fast R-CNN detector to label regions. The paper reports that TextMap with Word2Vec embeddings reaches the highest F1-score of 0.913 on its new SSOAR-MVD dataset. However, Table III in the same paper lists the Vision-Language method at 0.92, which is larger than 0.913, so the \"highest\" claim is contradicted by the paper's own numbers. If the approach holds up, it offers a cheaper, layout-aware alternative to full multimodal pipelines for making under-indexed documents findable.","feed_headline":"TextMap hits F1 of 0.913 for PDF metadata extraction","feed_subtitle":"The paper calls it best, but its own table lists a 0.92 multimodal model.","key_machinery":"The central mechanism is the TextMap interpolation–fusion pipeline: (1) a spatial stream converts page $P$ into grayscale $G = \\phi(P) \\in \\mathbb{R}^{H \\times W}$; (2) a semantic stream embeds tokens with Word2Vec or blocks with BERT and maps each embedding into the bounding-box coordinates of the region where that text appears, forming $T_m \\in \\mathbb{R}^{H \\times W \\times d}$; (3) convolutional layers bring both streams to a common channel dimension, and multi-head attention $M = \\mathrm{Attention}(F_{spatial}, F_{semantic})V$ fuses them while preserving spatial correspondence; (4) a Fast R-CNN head refines boxes and assigns metadata labels. A joint loss combines a semantic term, a spatial smoothness term that encourages neighboring same-label regions to have similar features, and a cross-modal term. The load-bearing innovation is that the text map keeps semantic information tied to page coordinates, so the detector can exploit layout even when templates vary.","core_discovery":"On its own terms, the paper's discovery is that jointly optimizing spatial and semantic information improves metadata extraction from template-varied PDFs. The TextMap method represents the first page as a grayscale image in one stream and as token or block embeddings in another, then interpolates those embeddings into the regions where the text appears, producing a \"text map\" with the same spatial dimensions as the page; convolutions on both streams are fused by multi-head attention and fed to a Fast R-CNN for label and bounding-box prediction. The authors state that the Word2Vec variant achieves the best overall F1-score, 0.913, on SSOAR-MVD and 0.901 on S-PMRD. The paper's own Table III, however, shows Vision-Language at 0.92 on SSOAR-MVD, so the asserted \"highest\" result does not match the table; a sympathetic reading is that the authors intended to claim TextMap-Word2Vec is the best among the newly introduced methods, not among all compared methods. The paper also introduces two datasets, SSOAR-MVD and S-PMRD, and reports per-category scores plus training and inference times for nine methods.","pith_inferences":["Editorial inference: if the 0.92 Vision-Language number is the true best, then the paper's conclusion shifts from \"TextMap is best\" to \"multimodal fusion wins but costs about twice as much to train and nearly nine times as long to run,\" which is still a useful engineering trade-off.","Editorial inference: because the SSOAR-MVD labels were generated automatically by similarity matching with no reported manual validation, the absolute F1 values in Tables III through XII may be optimistic; a human-annotated sample would be needed to confirm the ranking.","Editorial inference: the TextMap idea of interpolating embeddings into dense spatial maps is not limited to metadata; it could be transferred to table-structure recognition, form understanding, or any document task where layout and semantics both matter.","Editorial inference: a direct, testable follow-up is to run TextMap-Word2Vec and Vision-Language on the same 50,000 documents with human-labeled boxes and compare F1; whichever wins would resolve the table-vs-text discrepancy."],"forward_implications":["If TextMap's reported scores are taken at face value, layout-aware fusion of embeddings with page geometry reaches about 0.91 F1 on both benchmarks, competitive with a much more expensive multimodal model.","The Word2Vec variant's cost profile (about 92 hours of training, 0.4 seconds per page at inference) makes it a practical choice when GPU time or annotated data is limited.","The two new datasets, SSOAR-MVD and S-PMRD, give the community a template-diverse German social-science benchmark and an authentic S2ORC-based PDF benchmark for future comparisons.","The per-category scores suggest structured fields such as DOI and Email are easiest, while Affiliation and Address remain the weak spots across nearly all methods.","The internal contradiction between the claimed highest F1 of 0.913 and Table III's 0.92 for Vision-Language would need to be corrected in a revision before the headline result can be accepted."],"supporting_citations":[{"why":"Supplies the MexPub CV approach that the Fast-RCNN and Vision-Language baselines build on.","marker":"[3]"},{"why":"Defines the Vision-Language multimodal baseline whose 0.92 F1 in Table III is the actual top score.","marker":"[4]"},{"why":"Provides the two-layer CRF method used as the classical NLP baseline.","marker":"[29]"},{"why":"Supplies GROBID, the cascade sequence-labeling baseline.","marker":"[30]"},{"why":"Presents Mask R-CNN, the detection architecture the Fast-RCNN and TextMap pipelines are built on.","marker":"[33]"},{"why":"Provides PubLayNet, the layout dataset used to pre-train the object-detection backbone.","marker":"[38]"},{"why":"Supplies S2ORC, the source corpus from which S-PMRD is curated.","marker":"[41]"},{"why":"Provides the CrossRef API used to fetch PDF links and metadata for S-PMRD.","marker":"[42]"}],"fun_headline_variants":["TextMap vision-text fusion hits F1 0.913 on PDF metadata","Paper claims F1 0.913 best, but own table shows 0.92","New multimodal TextMap for varied PDF metadata extraction","TextMap best among new methods, not all: F1 0.913","Datasets SSOAR-MVD and S-PMRD aid PDF metadata extraction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the automatic similarity-based labeling of SSOAR-MVD is correct ground truth: blocks are assigned to metadata classes only when they have near-perfect similarity to the repository's textual metadata, with the threshold unspecified and no reported manual validation, so every precision, recall, and F1 number inherits whatever noise or bias those labels contain.","fun_headline_variants_meta":{"raw":{"variants":["TextMap vision-text fusion hits F1 0.913 on PDF metadata","Paper claims F1 0.913 best, but own table shows 0.92","New multimodal TextMap for varied PDF metadata extraction","TextMap best among new methods, not all: F1 0.913","Datasets SSOAR-MVD and S-PMRD aid PDF metadata extraction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001205,"raw_usage":{"total_tokens":4971,"prompt_tokens":958,"completion_tokens":4013,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":3913}},"tokens_in":574,"tokens_out":4013,"duration_ms":27384,"temperature":1.0,"reasoning_tokens":3913,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:21:05.490784+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of several hundred SSOAR-MVD pages with human labelers marking title, authors, abstract, DOI, and the other fields, then recompute F1 for TextMap-Word2Vec and Vision-Language on the same sample; if TextMap-Word2Vec no longer beats Vision-Language or drops below 0.913, the paper's central claim fails. A simpler check is to read Table III directly: it already lists Vision-Language at 0.92, above the claimed 0.913, so the \"highest\" assertion is internally falsified unless the authors meant \"best among newly proposed methods.\"","supporting_citations":[{"cited_title":"Mexpub: Deep transfer learning for metadata extraction from german publications,","cited_arxiv_id":null,"evidence_quote":"Supplies the MexPub CV approach that the Fast-RCNN and Vision-Language baselines build on."},{"cited_title":"Vision and natural language for meta- data extraction from scientific pdf documents: a multimodal approach,","cited_arxiv_id":null,"evidence_quote":"Defines the Vision-Language multimodal baseline whose 0.92 F1 in Table III is the actual top score."},{"cited_title":"Arctic: metadata extraction from scientific papers in pdf using two-layer crf,","cited_arxiv_id":null,"evidence_quote":"Provides the two-layer CRF method used as the classical NLP baseline."},{"cited_title":"Publaynet: largest dataset ever for document layout analysis,","cited_arxiv_id":null,"evidence_quote":"Provides PubLayNet, the layout dataset used to pre-train the object-detection backbone."},{"cited_title":"Crossref: The sus- tainable source of community-owned scholarly metadata,","cited_arxiv_id":null,"evidence_quote":"Provides the CrossRef API used to fetch PDF links and metadata for S-PMRD."}],"review_version":1}