{"id":"9041aa84-c54b-415d-9e27-1966e827255f","arxiv_id":"2506.03822","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CRAWLDoc ranks linked web documents by embedding similarity to a paper's landing page, evaluated on a new manually labeled dataset of 600 publications from six publishers.","lead":"Publication databases like DBLP need reliable metadata from publisher web pages that vary widely in layout and format. This paper introduces CRAWLDoc, which follows a paper's links and ranks which linked documents describe that same paper, and releases a manually labeled dataset of 600 computer science publications with over 72,000 linked documents.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No baseline or ablation isolates URL and anchor overlap, so the near-perfect MRR/MAP may reflect token overlap rather than layout-independent semantic ranking.","rationale":"Reader's conditional verdict already targets the same weak point, and my read supports it. The central empirical claim is that a landing-page query can rank linked documents by relevance in a layout-independent way. That claim requires showing the neural retriever is not exploiting URL or anchor overlap. The paper provides one ablation (removing layout info) but none that removes URL/anchor, and no baseline. The manual labeling of 72,483 links across 600 publications is a credible dataset contribution, and the leave-one-out publisher split is a reasonable robustness design; those parts deserve credit. But with scores near ceiling, the absence of a surface-overlap control is decisive for the interpretation. A single ablation/baseline experiment can settle it; hence no change beyond the existing conditional verdict.","tokens_in":10122,"tokens_out":4653,"duration_ms":46175,"concrete_test":"Fine-tune and evaluate two ablated variants on the released data: (i) content-only input, removing the URL and anchor fields from both query and candidate [SEP] concatenations, and (ii) URL+anchor-only input. Also run a BM25 baseline over the full input. If content-only MRR stays near 0.96, the semantic claim survives; if it falls materially or the URL+anchor-only condition reproduces MRR above 0.9, the near-perfect results are explained by surface overlap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the document representation in Section 3 — 'anchor text, URL, and document content' joined with [SEP] — scores relevance semantically rather than by surface overlap. On publisher landing pages, the URLs of relevant links (PDF, supplementary material, ORCID) frequently contain the paper's DOI or title, and the landing-page query contains the same DOI. The only reported ablation, in Section 5, removes layout information and yields MRR 0.950, MAP 0.976, nDCG 0.952; this does not test URL/anchor contribution, because those fields remain in the input. No non-neural baseline (BM25, URL/anchor exact match) and no content-only condition is reported, and the test-set self-link removal in Section 4 only excludes exact self-similarity, not URL substring overlap. Consequently, Table 1's MRR 0.967 / MAP 0.987 / nDCG 0.961 and the leave-one-out Table 3 results are consistent with a trivial lexical shortcut; the 'layout-independent' generalization claim is not yet established. The manual 72,483-label dataset remains a real contribution regardless of this outcome.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CRAWLDoc, a method and dataset for ranking web documents linked from a publication's DOI-resolved landing page. The landing page is treated as a query, and linked documents (PDFs, supplementary materials, ORCID profiles, etc.) are embedded along with their anchor texts and URLs using jina-embeddings-v2; the ranking is produced by maximum inner product search. The authors contribute a manually labeled dataset of 600 publications from six CS publishers, comprising 72,483 linked documents with binary relevance labels and author-affiliation metadata. Experiments report MRR 0.967, MAP 0.987, and nDCG 0.961 on the main test set, with a leave-one-publisher-out robustness check giving average MRR 0.959, MAP 0.968, nDCG 0.961.","tokens_in":10293,"tokens_out":2981,"duration_ms":28128,"significance":"If the near-perfect ranking scores reflect genuine semantic, layout-independent relevance identification, the system would be a practical tool for bibliographic metadata harvesting and the dataset would be a valuable resource for studying web-document relevance in scholarly contexts. The manual annotation of 72,483 linked documents across six publishers is a concrete contribution, as is the public release of code and labels. The leave-one-out evaluation is a sensible design for probing publisher generalization. However, the significance of the ranking results hinges on whether the model is learning content relevance rather than exploiting superficial URL or anchor-text overlap; the paper currently does not provide the baselines or ablations needed to establish that distinction.","major_comments":[{"comment":"The input representation is constructed by concatenating anchor text, URL, and document content with [SEP] tokens. The paper never isolates the contribution of URL and anchor-text fields. On publisher landing pages, the URLs of relevant linked documents (PDF, supplementary material, ORCID) frequently contain the paper's DOI or title, and the query landing-page URL contains the same DOI; therefore the reported MRR 0.967 and MAP 0.987 could be largely driven by token overlap rather than semantic content. The layout ablation in Section 5 (\"results without layout information\", MRR 0.950, MAP 0.976, nDCG 0.952) does not address this because URL and anchor text remain in the input. A content-only condition (document text without URL/anchor), a URL/anchor-only condition, and a non-neural baseline such as BM25 or exact substring matching on URL/title are needed to support the claim of semantic, layout-independent ranking.","section":"Section 3, Neural Document Ranking; Section 5, Results"},{"comment":"The test split is 10 publications per publisher (80/10/10 split of 100 publications per publisher), giving 60 query-level observations in total. No confidence intervals or significance tests are reported for the metrics in Tables 1 and 3. For example, Springer's MRR drops from 0.800 to 0.757 between the in-distribution and leave-one-out settings, but with 10 queries this difference may be within noise. The authors should report per-query variance, bootstrap confidence intervals, and paired significance tests (e.g., permutation tests) to substantiate the robustness claims.","section":"Section 4, Dataset"},{"comment":"The self-link removal step excludes exact self-similarity (links to the landing page itself) but does not account for URL substring overlap, where a linked URL contains the query URL's DOI or title as a substring. Since URLs are part of the model input, such overlap can trivially inflate ranking scores. The paper should quantify how many test candidates have partial URL/title overlap with the query, and report performance separately on the subset of candidates without such overlap, to assess whether the high scores survive when lexical shortcuts are unavailable.","section":"Section 4, Dataset; self-link removal"}],"minor_comments":[{"comment":"For Springer, MRR 0.800 and nDCG 0.800 with MAP 0.998 appear to warrant a brief explanation; the combination suggests the first relevant document is sometimes not at rank 1 while almost all relevant documents are retrieved at high ranks, but the paper does not discuss this pattern.","section":"Section 5, Table 1"},{"comment":"The phrase \"less than one point\" and \"less than two points\" should be clarified as percentage points (e.g., MRR 0.967 vs. 0.959 is a 0.8 percentage-point difference), since \"one point\" is ambiguous.","section":"Section 5, Results"},{"comment":"The ablation described as \"without layout information\" is not fully specified; it should state which fields are removed from the JSON representation (e.g., bounding boxes only, or also anchor/URL fields) so readers can understand what the ablation actually tests.","section":"Section 3, Figure 2"},{"comment":"There is a missing space in \"an average of only5.45\" (should be \"only 5.45\").","section":"Section 4, Dataset"},{"comment":"Several reference URLs contain line-break artifacts (e.g., \"V ol-2080\", \"V ol-3178\") that should be corrected for the camera-ready version.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The dataset contribution is real and likely useful to the community, but the central claim of robust, layout-independent semantic ranking is not yet established because no baselines or ablations isolate the effect of URL and anchor-text overlap. The test set is also small (60 query-level observations) and without uncertainty estimates. These issues are fixable within the manuscript's scope by adding the missing experiments and analyses; I therefore recommend major revision rather than rejection. The paper also appears to be a workshop-style manuscript; the journal version should tighten the evaluation framing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe dataset here is the real contribution; the ranking claim is not yet established. CRAWLDoc ships a manually labeled resource of 600 publications and 72,483 linked documents with binary relevance labels plus author affiliations, and that is genuinely new. The leave-one-out publisher evaluation is a sensible robustness check, and the reported numbers are internally consistent. The method itself is a routine combination of jina-embeddings-v2, [SEP] concatenation of anchor, URL, and content, contrastive fine-tuning, and MIPS, but applying it to scholarly link ranking is a fair new task.\n\nThe soft spot is load-bearing. The near-perfect MRR/MAP/nDCG (0.967/0.987/0.961) may reflect trivial token overlap between the query landing page and the linked documents' URLs and anchors, since both often contain the DOI or title. The only ablation removes layout information, not URL or anchor. There is no BM25, exact-match, or content-only baseline. The test split is 10 publications per publisher, and no confidence intervals or significance tests are reported. Removing self-links from the test set does not address URL substring overlap. So the 'layout-independent semantic ranking' claim is under-supported. The dataset survives this; the evaluation does not.\n\nThe leave-one-out result across publishers is actually some evidence against pure layout memorization, but it does not rule out the lexical shortcut, because the overlap signal is present across publishers. The paper's threat-to-validity section is honest about publisher scope and recency bias, but it misses this more immediate issue.\n\nWho gets value: people building bibliographic metadata workflows; the dataset is a reusable benchmark, and the promised code/data availability is an asset. This deserves serious peer review, but with a firm request for baselines and ablations. I would send it to review with major-revision expectations, not desk reject.","headline":"The dataset is a genuine contribution; the near-perfect ranking scores are not yet convincing without a baseline that isolates URL/anchor overlap.","tokens_in":10853,"tokens_out":2427,"would_cite":true,"duration_ms":24304,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One DOI landing page can rank linked metadata sources almost perfectly","keywords":["bibliographic metadata","document ranking","neural information retrieval","DOI landing pages","web scraping","layout independence","small language models","retrieval dataset"],"falsifier":"Re-run the Table 1 evaluation with the URL field and anchor texts removed from both query and document inputs, keeping only rendered page text: if MRR, MAP, and nDCG fall well below 0.9, the ranking is largely surface overlap rather than layout-independent semantic understanding.","tokens_in":9869,"feed_emoji":"🔗","tokens_out":4824,"duration_ms":38746,"temperature":0.7,"pith_summary":"This paper introduces CRAWLDoc, a retrieval method that starts from a publication's DOI, scrapes the landing page and every resource linked from it, and ranks those resources by how relevant they are to the same publication. The authors claim that this document-as-query ranking is reliable and layout-independent: on 600 manually labeled publications from six major computer science publishers, the top-ranked document is relevant almost always, with MRR 0.967, MAP 0.987, and nDCG 0.961. A leave-one-out test that excludes a publisher from training gives nearly identical scores, which the authors offer as evidence that the method generalizes to unseen web layouts. If this holds, bibliographic services could automatically shortlist metadata sources across heterogeneous publishers instead of maintaining handwritten wrappers.","feed_headline":"One DOI landing page ranks linked metadata sources almost perfectly","feed_subtitle":"On 600 labeled publications from six publishers, the top-ranked linked document is relevant in nearly every case.","key_machinery":"The load-bearing mechanism is document-as-query ranking with a shared embedding space. A small language model (jina-embeddings-v2, a BERT-style encoder) is fine-tuned with contrastive InfoNCE loss; each linked document is turned into one string of anchor text, URL, and page text separated by [SEP], and the landing page is embedded as the query. Retrieval is then a maximum inner product search between the query embedding and document embeddings. Layout information is folded in by extracting text with bounding boxes and serializing it into a uniform JSON text format, which is why the method can handle both HTML and PDF sources.","core_discovery":"The central discovery claim is that a single seed page—the DOI landing page—can serve as a query that separates linked documents about the same publication from unrelated ones, without relying on publisher-specific templates. CRAWLDoc encodes each linked resource by concatenating its anchor text, URL, and rendered content with a [SEP] separator, embeds the landing page with a query encoder and the resources with a document encoder, and ranks by maximum inner product search. On the new dataset of 600 publications with 72,483 labeled outgoing links, it reaches MRR 0.967, MAP 0.987, and nDCG 0.961; removing layout information lowers these to 0.950, 0.976, and 0.952, and leaving out the training publisher yields 0.959, 0.968, and 0.961. The authors interpret the small gap between in-distribution and out-of-distribution results as evidence that the model learned generalizable features of document relevance rather than publisher-specific layouts.","pith_inferences":["I infer from the absence of ablations that surface overlap of URLs, titles, or anchor texts may be doing much of the ranking work; deleting URL strings from the input in a follow-up experiment would settle this.","The one-hop crawl assumption likely caps recall for publications whose relevant material lives two or more clicks away (e.g., via a reference-list page), so multi-hop crawling is a natural stress test.","The same document-as-query recipe could be applied outside bibliographic data, for instance to discover related resources for any seed URI whose canonical page is known.","Because the robustness test uses publishers that share conventional academic page designs, the claim of layout independence is best read as applying within that design space."],"forward_implications":["CRAWLDoc gives bibliographic databases such as DBLP a way to shortlist candidate metadata sources without per-publisher wrappers.","The leave-one-out results imply that a model trained on five publishers transfers to a sixth with almost no loss in ranking quality.","Removing layout information costs only about one to two metric points, so the method remains usable when bounding boxes are unavailable.","At a cut-off of k=5, the system balances precision and recall well, with recall reaching 0.870 and F1 peaking at k=4 and k=5, matching the average number of relevant documents per publication."],"supporting_citations":[{"why":"Supplies the jina-embeddings-v2 neural retriever used to embed queries and documents.","marker":"[23]"},{"why":"Provides the BERT architecture that the retriever is based on.","marker":"[11]"},{"why":"The DBLP dataset is the source of the 600 publications and the six-publisher selection.","marker":"[26]"},{"why":"Defines the InfoNCE contrastive loss used to fine-tune the retriever.","marker":"[25]"},{"why":"Defines nDCG, one of the central ranking metrics reported in the results.","marker":"[27]"},{"why":"Documents layout distribution shifts against which the paper positions its robustness claim.","marker":"[21]"}],"fun_headline_variants":["CRAWLDoc ranks linked docs from a single DOI landing page","Single seed URL achieves 0.967 MRR on bibliographic link ranking","Layout-free ranking of web bibliographic resources hits 0.99 MAP","One DOI link ranks 72K documents with near-perfect accuracy","CRAWLDoc: query by landing page, rank links across publishers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central premise is that relevance to the publication is captured by embedding the concatenation of anchor text, URL, and document content, and that all needed sources sit within one link of the DOI landing page; if URL or anchor overlap alone drives the ranking, the claimed layout-independent semantic generalization is not established.","fun_headline_variants_meta":{"raw":{"variants":["CRAWLDoc ranks linked docs from a single DOI landing page","Single seed URL achieves 0.967 MRR on bibliographic link ranking","Layout-free ranking of web bibliographic resources hits 0.99 MAP","One DOI link ranks 72K documents with near-perfect accuracy","CRAWLDoc: query by landing page, rank links across publishers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000399,"raw_usage":{"total_tokens":2069,"prompt_tokens":914,"completion_tokens":1155,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":1060}},"tokens_in":530,"tokens_out":1155,"duration_ms":8294,"temperature":1.0,"reasoning_tokens":1060,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:54:48.584583+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Table 1 evaluation with the URL field and anchor texts removed from both query and document inputs, keeping only rendered page text: if MRR, MAP, and nDCG fall well below 0.9, the ranking is largely surface overlap rather than layout-independent semantic understanding.","supporting_citations":[{"cited_title":"Devlin, M","cited_arxiv_id":null,"evidence_quote":"Provides the BERT architecture that the retriever is based on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The DBLP dataset is the source of the 600 publications and the six-publisher selection."}],"review_version":1}