Pith. sign in

REVIEW 3 major objections 5 minor 28 references

CRAWLDoc: A Dataset for Robust Ranking of Bibliographic Documents

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One DOI landing page can rank linked metadata sources almost perfectly

desk verdict The dataset is a genuine contribution; the near-perfect ranking scores are not yet convincing without a baseline that isolates URL/anchor overlap. read the letter →

arxiv 2506.03822 v1 pith:UOJQ4WG3 submitted 2025-06-04 cs.CL cs.IR

Fabian Karl , Ansgar Scherp This is my paper
classification cs.CLcs.IR
keywords bibliographicmetadatadocumentrankingneuralinformationretrievalDOIlandingpageswebscrapinglayoutindependencesmalllanguagemodelsdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces CRAWLDoc, a retrieval method that starts from a publication's DOI, scrapes the landing page and every resource linked from it, and ranks those resources by how relevant they are to the same publication. The authors claim that this document-as-query ranking is reliable and layout-independent: on 600 manually labeled publications from six major computer science publishers, the top-ranked document is relevant almost always, with MRR 0.967, MAP 0.987, and nDCG 0.961. A leave-one-out test that excludes a publisher from training gives nearly identical scores, which the authors offer as evidence that the method generalizes to unseen web layouts. If this holds, bibliographic services could automatically shortlist metadata sources across heterogeneous publishers instead of maintaining handwritten wrappers.

What carries the argument

The load-bearing mechanism is document-as-query ranking with a shared embedding space. A small language model (jina-embeddings-v2, a BERT-style encoder) is fine-tuned with contrastive InfoNCE loss; each linked document is turned into one string of anchor text, URL, and page text separated by [SEP], and the landing page is embedded as the query. Retrieval is then a maximum inner product search between the query embedding and document embeddings. Layout information is folded in by extracting text with bounding boxes and serializing it into a uniform JSON text format, which is why the method can handle both HTML and PDF sources.

What would settle it

Re-run the Table 1 evaluation with the URL field and anchor texts removed from both query and document inputs, keeping only rendered page text: if MRR, MAP, and nDCG fall well below 0.9, the ranking is largely surface overlap rather than layout-independent semantic understanding.

Watch

Extended reading notes

Core claim

The central discovery claim is that a single seed page—the DOI landing page—can serve as a query that separates linked documents about the same publication from unrelated ones, without relying on publisher-specific templates. CRAWLDoc encodes each linked resource by concatenating its anchor text, URL, and rendered content with a [SEP] separator, embeds the landing page with a query encoder and the resources with a document encoder, and ranks by maximum inner product search. On the new dataset of 600 publications with 72,483 labeled outgoing links, it reaches MRR 0.967, MAP 0.987, and nDCG 0.961; removing layout information lowers these to 0.950, 0.976, and 0.952, and leaving out the training publisher yields 0.959, 0.968, and 0.961. The authors interpret the small gap between in-distribution and out-of-distribution results as evidence that the model learned generalizable features of document relevance rather than publisher-specific layouts.

Load-bearing premise

The central premise is that relevance to the publication is captured by embedding the concatenation of anchor text, URL, and document content, and that all needed sources sit within one link of the DOI landing page; if URL or anchor overlap alone drives the ranking, the claimed layout-independent semantic generalization is not established.

Editorial extensions

If this is right

  • CRAWLDoc gives bibliographic databases such as DBLP a way to shortlist candidate metadata sources without per-publisher wrappers.
  • The leave-one-out results imply that a model trained on five publishers transfers to a sixth with almost no loss in ranking quality.
  • Removing layout information costs only about one to two metric points, so the method remains usable when bounding boxes are unavailable.
  • At a cut-off of k=5, the system balances precision and recall well, with recall reaching 0.870 and F1 peaking at k=4 and k=5, matching the average number of relevant documents per publication.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer from the absence of ablations that surface overlap of URLs, titles, or anchor texts may be doing much of the ranking work; deleting URL strings from the input in a follow-up experiment would settle this.
  • The one-hop crawl assumption likely caps recall for publications whose relevant material lives two or more clicks away (e.g., via a reference-list page), so multi-hop crawling is a natural stress test.
  • The same document-as-query recipe could be applied outside bibliographic data, for instance to discover related resources for any seed URI whose canonical page is known.
  • Because the robustness test uses publishers that share conventional academic page designs, the claim of layout independence is best read as applying within that design space.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces CRAWLDoc, a method and dataset for ranking web documents linked from a publication's DOI-resolved landing page. The landing page is treated as a query, and linked documents (PDFs, supplementary materials, ORCID profiles, etc.) are embedded along with their anchor texts and URLs using jina-embeddings-v2; the ranking is produced by maximum inner product search. The authors contribute a manually labeled dataset of 600 publications from six CS publishers, comprising 72,483 linked documents with binary relevance labels and author-affiliation metadata. Experiments report MRR 0.967, MAP 0.987, and nDCG 0.961 on the main test set, with a leave-one-publisher-out robustness check giving average MRR 0.959, MAP 0.968, nDCG 0.961.

Significance. If the near-perfect ranking scores reflect genuine semantic, layout-independent relevance identification, the system would be a practical tool for bibliographic metadata harvesting and the dataset would be a valuable resource for studying web-document relevance in scholarly contexts. The manual annotation of 72,483 linked documents across six publishers is a concrete contribution, as is the public release of code and labels. The leave-one-out evaluation is a sensible design for probing publisher generalization. However, the significance of the ranking results hinges on whether the model is learning content relevance rather than exploiting superficial URL or anchor-text overlap; the paper currently does not provide the baselines or ablations needed to establish that distinction.

major comments (3)
  1. [Section 3, Neural Document Ranking; Section 5, Results] The input representation is constructed by concatenating anchor text, URL, and document content with [SEP] tokens. The paper never isolates the contribution of URL and anchor-text fields. On publisher landing pages, the URLs of relevant linked documents (PDF, supplementary material, ORCID) frequently contain the paper's DOI or title, and the query landing-page URL contains the same DOI; therefore the reported MRR 0.967 and MAP 0.987 could be largely driven by token overlap rather than semantic content. The layout ablation in Section 5 ("results without layout information", MRR 0.950, MAP 0.976, nDCG 0.952) does not address this because URL and anchor text remain in the input. A content-only condition (document text without URL/anchor), a URL/anchor-only condition, and a non-neural baseline such as BM25 or exact substring matching on URL/title are needed to support the claim of semantic, layout-independent ranking.
  2. [Section 4, Dataset] The test split is 10 publications per publisher (80/10/10 split of 100 publications per publisher), giving 60 query-level observations in total. No confidence intervals or significance tests are reported for the metrics in Tables 1 and 3. For example, Springer's MRR drops from 0.800 to 0.757 between the in-distribution and leave-one-out settings, but with 10 queries this difference may be within noise. The authors should report per-query variance, bootstrap confidence intervals, and paired significance tests (e.g., permutation tests) to substantiate the robustness claims.
  3. [Section 4, Dataset; self-link removal] The self-link removal step excludes exact self-similarity (links to the landing page itself) but does not account for URL substring overlap, where a linked URL contains the query URL's DOI or title as a substring. Since URLs are part of the model input, such overlap can trivially inflate ranking scores. The paper should quantify how many test candidates have partial URL/title overlap with the query, and report performance separately on the subset of candidates without such overlap, to assess whether the high scores survive when lexical shortcuts are unavailable.
minor comments (5)
  1. [Section 5, Table 1] For Springer, MRR 0.800 and nDCG 0.800 with MAP 0.998 appear to warrant a brief explanation; the combination suggests the first relevant document is sometimes not at rank 1 while almost all relevant documents are retrieved at high ranks, but the paper does not discuss this pattern.
  2. [Section 5, Results] The phrase "less than one point" and "less than two points" should be clarified as percentage points (e.g., MRR 0.967 vs. 0.959 is a 0.8 percentage-point difference), since "one point" is ambiguous.
  3. [Section 3, Figure 2] The ablation described as "without layout information" is not fully specified; it should state which fields are removed from the JSON representation (e.g., bounding boxes only, or also anchor/URL fields) so readers can understand what the ablation actually tests.
  4. [Section 4, Dataset] There is a missing space in "an average of only5.45" (should be "only 5.45").
  5. [References] Several reference URLs contain line-break artifacts (e.g., "V ol-2080", "V ol-3178") that should be corrected for the camera-ready version.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CRAWLDoc is an empirical supervised retrieval system evaluated on its own manually labeled dataset with a proper train/test split; no derivation or prediction reduces to its inputs by construction.

full rationale

The paper makes no formal derivation that could be circular. Its central claim is empirical: that a fine-tuned neural retriever ranks relevant linked documents highly, evaluated against a manually labeled dataset of 72,483 linked documents from 600 publications. The input representation concatenates anchor text, URL, and document content (Section 3), and the query is the DOI landing page. Relevance labels were created manually and independently of the model (Section 4), and the test set excludes exact self-links to prevent trivial self-similarity inflation. The laterablations remove layout information only, and the leave-one-out publisher experiment tests held-out publishers; while these settings do not isolate URL/anchor overlap as a shortcut, the absence of such an ablation is a methodological weakness, not a circular reduction. No fitted parameter is renamed as a prediction, no result is imported from the authors' prior work, and no uniqueness or ansatz is justified by self-citation. The dataset and code are released externally, and the evaluation is self-contained against real manually labeled data. Therefore the paper does not exhibit the specific circularity patterns this analysis targets, and the score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The ledger contains no invented entities and no fitted scientific constants. The central claim rests on domain assumptions about the one-hop web graph, the accuracy of manual labels, the embedding representation, and the lossless normalization of document formats, plus hyperparameters chosen in Section 4.

free parameters (4)
  • learning_rate = 3e-05
    Chosen by hyperparameter optimization in Section 4; all reported ranking metrics depend on this setting.
  • gradient_accumulation_steps = 32
    Optimized in Section 4 and used for training; affects the training dynamics and final model.
  • early_stopping_patience = 5
    Set in Section 4; determines when training stops and therefore the model that is evaluated.
  • max_sequence_tokens = 2048
    Documents are truncated to the first 2,048 tokens due to memory restrictions (Section 3); this hand-chosen cutoff could affect ranking of long documents.
assumptions (4)
  • domain assumption All necessary metadata sources are within one hop of the DOI landing page.
    Stated in Section 3 as the primary assumption; CRAWLDoc never follows links beyond the landing page, so this bounds the problem.
  • domain assumption The manual binary relevance labels are accurate.
    Section 4 says labels were created manually; all ranking metrics treat these labels as ground truth.
  • domain assumption Embedding the concatenation of anchor text, URL, and document content captures relevance beyond trivial lexical or URL overlap.
    Section 3 constructs the input this way; no ablation or baseline isolates the contribution of anchor/URL vs. content.
  • domain assumption Converting HTML and PDF into a uniform text-plus-bounding-box JSON preserves the information needed to judge relevance.
    Section 3 converts HTML and PDF into this uniform representation; the ranking assumes the conversion is lossless for the relevance signal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CRAWLDoc: A Dataset for Robust Ranking of Bibliographic Documents." pith.science (2026). https://pith.science/paper/UOJQ4WG3

@misc{pith2026250603822,
  author       = {Pith},
  title        = {Pith review of: CRAWLDoc: A Dataset for Robust Ranking of Bibliographic Documents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UOJQ4WG3}},
  note         = {Machine review of arXiv:2506.03822}
}
read the original abstract

Publication databases rely on accurate metadata extraction from diverse web sources, yet variations in web layouts and data formats present challenges for metadata providers. This paper introduces CRAWLDoc, a new method for contextual ranking of linked web documents. Starting with a publication's URL, such as a digital object identifier, CRAWLDoc retrieves the landing page and all linked web resources, including PDFs, ORCID profiles, and supplementary materials. It embeds these resources, along with anchor texts and the URLs, into a unified representation. For evaluating CRAWLDoc, we have created a new, manually labeled dataset of 600 publications from six top publishers in computer science. Our method CRAWLDoc demonstrates a robust and layout-independent ranking of relevant documents across publishers and data formats. It lays the foundation for improved metadata extraction from web documents with various layouts and formats. Our source code and dataset can be accessed at https://github.com/FKarl/CRAWLDoc.

Figures

Figures reproduced from arXiv: 2506.03822 by the authors.

Figure 1
Figure 1. Illustration of document retrieval from heterogeneous web sources. The process starts with a DOI, which is resolved to access the landing page. Linked documents are ranked by treating the landing page as a query. A maximum inner product search (MIPS) ranks the top-𝑘 documents based on embeddings from a small language model. documents along with their associated anchor texts and URLs into a shared vector space and tr… view at source ↗
Figure 2
Figure 2. This figure illustrates our document representation methodology. The process begins with identifying all hyperlinks on the landing page, followed by integrating layout information via bounding boxes. The document is then converted into a uniform textual format and encoded into a vector representation. objective. Another approach is DocLLM [19], which does not rely on expensive image encoders but relies solely on bou… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

28 extracted references · 12 canonical work pages

  1. [1]

    Grobid, https://github.com/kermitt2/grobid, 2008–2023

    work page 2008

  2. [2]

    Tkaczyk, P

    D. Tkaczyk, P. Szostek, M. Fedoryszak, P. J. Dendek, L. Bolikowski, CERMINE: automatic extrac- tion of structured metadata from scientific literature, Int. J. Document Anal. Recognit. 18 (2015) 317–335. URL: https://doi.org/10.1007/s10032-015-0249-8. doi: 10.1007/S10032-015-0249-8

    work page doi:10.1007/s10032-015-0249-8 2015

  3. [3]

    Schenkel, Integrating and exploiting public metadata sources in a bibliographic information system, in: P

    R. Schenkel, Integrating and exploiting public metadata sources in a bibliographic information system, in: P. Mayr, I. Frommholz, G. Cabanac (Eds.), Proceedings of the 7th International Workshop on Bibliometric-enhanced Information Retrieval (BIR 2018) co-located with the 40th European Conference on Information Retrieval (ECIR 2018), Grenoble, France, Mar...

    work page 2018

  4. [4]

    Ley, DBLP - some lessons learned, Proc

    M. Ley, DBLP - some lessons learned, Proc. VLDB Endow. 2 (2009) 1493–1500. URL: http: //www.vldb.org/pvldb/vol2/vldb09-98.pdf. doi:10.14778/1687553.1687577

    arXiv 2009

  5. [5]

    Y . Zhu, H. Yuan, S. Wang, J. Liu, W. Liu, C. Deng, Z. Dou, J. Wen, Large language models for information retrieval: A survey, CoRR abs/2308.07107 (2023). URL: https://doi.org/10.48550/ arXiv.2308.07107. doi:10.48550/ARXIV.2308.07107. arXiv:2308.07107

    doi:10.48550/arxiv.2308.07107 2023

  6. [7]

    J. Guo, Y . Fan, L. Pang, L. Yang, Q. Ai, H. Zamani, C. Wu, W. B. Croft, X. Cheng, A deep look into neural ranking models for information retrieval, Inf. Process. Manag. 57 (2020) 102067. URL: https://doi.org/10.1016/j.ipm.2019.102067. doi:10.1016/J.IPM.2019.102067

    work page arXiv 2020

  7. [8]

    Zhang, M

    Y . Zhang, M. M. Rahman, A. Braylan, B. Dang, H. Chang, H. Kim, Q. McNamara, A. Angert, E. Banner, V . Khetan, T. McDonnell, A. T. Nguyen, D. Xu, B. C. Wallace, M. Lease, Neural information retrieval: A literature review, CoRR abs/1611.06792 (2016). URL: http://arxiv.org/abs/ 1611.06792. arXiv:1611.06792

    arXiv 2016

  8. [9]

    Mitra, N

    B. Mitra, N. Craswell, An introduction to neural information retrieval, Found. Trends Inf. Retr. 13 (2018) 1–126. URL: https://doi.org/10.1561/1500000061. doi:10.1561/1500000061

    doi:10.1561/1500000061 2018

Show all 28 references
  1. [10]

    Abbasiantaeb, S

    Z. Abbasiantaeb, S. Momtazi, Text-based question answering from information retrieval and deep neural network perspectives: A survey, WIREs Data Mining Knowl. Discov. 11 (2021). URL: https://doi.org/10.1002/widm.1412. doi:10.1002/WIDM.1412

    2021 doi

  2. [11]

    Devlin, M

    J. Devlin, M. Chang, K. Lee, K. Toutanova, BERT: pre-training of deep bidirectional transformers for language understanding, in: J. Burstein, C. Doran, T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Ling...

    2019

  3. [12]

    X. Tian, J. Wang, Retrieval of scientific documents based on HFS and BERT, IEEE Access 9 (2021) 8708–8717. URL: https://doi.org/10.1109/ACCESS.2021.3049391. doi:10.1109/ACCESS. 2021.3049391

    2021

  4. [13]

    J. Wang, J. X. Huang, X. Tu, J. Wang, A. J. Huang, M. T. R. Laskar, A. Bhuiyan, Utilizing BERT for information retrieval: Survey, applications, resources, and challenges, ACM Comput. Surv. 56 (2024) 185:1–185:33. URL: https://doi.org/10.1145/3648471. doi:10.1145/3648471

    2024 doi

  5. [14]

    M. Li, É. Gaussier, Intra-document block pre-ranking for bert-based long document information retrieval - abstract, in: L. Tamine, E. Amigó, J. Mothe (Eds.), Proceedings of the 2nd Joint Conference of the Information Retrieval Communities in Europe (CIRCLE 2022), Samatan, Gers...

    2022

  6. [15]

    MacAvaney, A

    S. MacAvaney, A. Yates, A. Cohan, N. Goharian, CEDR: contextualized embeddings for document ranking, in: B. Piwowarski, M. Chevalier, É. Gaussier, Y . Maarek, J. Nie, F. Scholer (Eds.), Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in I...

    2019

  7. [16]

    Khattab, M

    O. Khattab, M. Zaharia, Colbert: Efficient and effective passage search via contextualized late interaction over BERT, in: J. X. Huang, Y . Chang, X. Cheng, J. Kamps, V . Murdock, J. Wen, Y . Liu (Eds.), Proceedings of the 43rd International ACM SIGIR conference on research an...

    2020

  8. [17]

    Santhanam, O

    K. Santhanam, O. Khattab, J. Saad-Falcon, C. Potts, M. Zaharia, Colbertv2: Effective and ef- ficient retrieval via lightweight late interaction, in: M. Carpuat, M. de Marneffe, I. V . M. Ruíz (Eds.), Proceedings of the 2022 Conference of the North American Chapter of the Assoc...

    2022 doi

  9. [18]

    Huang, T

    Y . Huang, T. Lv, L. Cui, Y . Lu, F. Wei, Layoutlmv3: Pre-training for document AI with unified text and image masking, in: J. Magalhães, A. D. Bimbo, S. Satoh, N. Sebe, X. Alameda-Pineda, Q. Jin, V . Oria, L. Toni (Eds.), MM ’22: The 30th ACM International Conference on Multi...

    2022

  10. [19]

    D. Wang, N. Raman, M. Sibue, Z. Ma, P. Babkin, S. Kaur, Y . Pei, A. Nourbakhsh, X. Liu, Docllm: A layout-aware generative language model for multimodal document understanding, CoRR abs/2401.00908 (2024). URL: https://doi.org/10.48550/arXiv.2401.00908. doi:10.48550/ARXIV. 2401....

    2024 doi arXiv

  11. [20]

    Perot, K

    V . Perot, K. Kang, F. Luisier, G. Su, X. Sun, R. S. Boppana, Z. Wang, J. Mu, H. Zhang, N. Hua, LMDX: language model-based document information extraction and localization, CoRR abs/2309.10952 (2023). URL: https://doi.org/10.48550/arXiv.2309.10952. doi:10.48550/ARXIV. 2309.109...

    2023 doi arXiv

  12. [21]

    C. Chen, Z. Shen, D. Klein, G. Stanovsky, D. Downey, K. Lo, Are layout-infused language models robust to layout distribution shifts? A case study with scientific documents, in: A. Rogers, J. L. Boyd-Graber, N. Okazaki (Eds.), Findings of the Association for Computational Lingu...

    2023 doi

  13. [22]

    Z. Lu, X. Li, D. Cai, R. Yi, F. Liu, X. Zhang, N. D. Lane, M. Xu, Small language models: Survey, measurements, and insights, CoRR abs/2409.15790 (2024). URL: https://doi.org/10.48550/arXiv. 2409.15790. doi:10.48550/ARXIV.2409.15790. arXiv:2409.15790

    2024 doi arXiv

  14. [23]

    Günther, J

    M. Günther, J. Ong, I. Mohr, A. Abdessalem, T. Abel, M. K. Akram, S. Guzman, G. Mastrapas, S. Sturua, B. Wang, M. Werk, N. Wang, H. Xiao, Jina embeddings 2: 8192-token general-purpose text embeddings for long documents, CoRR abs/2310.19923 (2023). URL: https://doi.org/10.48550...

    2023 doi arXiv

  15. [24]

    Press, N

    O. Press, N. A. Smith, M. Lewis, Train short, test long: Attention with linear biases enables input length extrapolation, in: The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, OpenReview.net, 2022. URL: https://openrev...

    2022

  16. [25]

    van den Oord, Y

    A. van den Oord, Y . Li, O. Vinyals, Representation learning with contrastive predictive coding, CoRR abs/1807.03748 (2018). URL: http://arxiv.org/abs/1807.03748. arXiv:1807.03748

    2018 arXiv

  17. [26]

    dblp Team, dblp computer science bibliography – Monthly Snapshot XML Release of April 2024,

    2024

  18. [27]

    Järvelin, J

    K. Järvelin, J. Kekäläinen, Cumulated gain-based evaluation of IR techniques, ACM Trans. Inf. Syst. 20 (2002) 422–446. URL: http://doi.acm.org/10.1145/582415.582418. doi:10.1145/582415. 582418

    2002

  19. [2024]

    doi:10.4230/dblp.xml.2024-04-01

    URL: https://doi.org/10.4230/dblp.xml.2024-04-01. doi:10.4230/dblp.xml.2024-04-01

    2024 doi

  20. [4186]

    doi:10.18653/v1/n19-1423

    URL: https://doi.org/10.18653/v1/n19-1423. doi:10.18653/v1/n19-1423

    doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.