{"id":"f9c66134-240c-4225-bc10-bbedec53e69e","arxiv_id":"2502.02167","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A multilingual DOM-LM, pre-trained and fine-tuned on a new six-language news dataset, extracts article attributes better than English-only MarkupLM and open-source tools.","lead":"The authors built a public dataset of 3,172 news web pages in six languages and tested two transformer models for extracting titles, dates, text, authors, and tags. A multilingual DOM-LM model beat the English-only MarkupLM and open-source tools on most attributes, offering a reusable benchmark and recipe for multilingual news aggregation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DOM-LM is pre-trained on the same multilingual pages later used in the 5-fold evaluation, so its reported superiority over MarkupLM and open-source tools may reflect unsupervised exposure to test pages rather than a genuinely better multilingual representation.","rationale":"The reader's weakest_assumption identifies precisely the load-bearing flaw: DOM-LM was pre-trained on the same dataset later used for evaluation, and the 5-fold site-level split does not remove that contamination. My read of Sections VI.C and VII.A confirms this is an internal-validity problem, not merely a deviation from common practice. The comparison is inequitable because DOM-LM receives additional unsupervised exposure to the evaluation pages while MarkupLM, XLM-RoBERTa, and heuristic tools do not. This could materially inflate the reported F1 scores and the conclusion that multilingual DOM-LM 'outperforms other solutions for most attributes.' The conditional verdict is appropriate: the claim should be accepted only after an ablation that excludes test-fold sites from pre-training, or after a baseline with equivalent target-domain pre-training is included. I therefore recommend no change to the reader's CONDITIONAL verdict.","tokens_in":9954,"tokens_out":2822,"duration_ms":32692,"concrete_test":"Pre-train a second DOM-LM from the same XLM-RoBERTa initialization using only the CommonCrawl-News sample plus the four training folds of the labeled dataset, excluding all pages from websites in the held-out fold, for each of the five folds. Then fine-tune and evaluate with the same site-level splits used in Table VII. If the mixed-language F1 margins over MarkupLM-EN shrink by more than a few points or reverse on any attribute, the reported superiority is attributable at least in part to unsupervised pre-training on evaluation pages and the claim should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that multilingual DOM-LM outperforms other solutions on mixed-language data—is undermined by a data-contamination confound. Section VI.C states that DOM-LM pre-training was performed on \"our multilingual news dataset\" plus a CommonCrawl-News sample. Section VII.A then describes 5-fold cross-validation in which the set of websites is split into 5 parts, 4 for training and 1 for testing, and the results are averaged. Critically, the pre-training step uses the full dataset, including pages whose websites later appear in the test fold. Although the fine-tuning splits are site-disjoint, the unsupervised pre-training has already seen every evaluation-page's raw HTML and DOM structure. This is not controlled for in the comparison: MarkupLM was pre-trained only on English CommonCrawl pages, and XLM-RoBERTa was pre-trained only on general text, so neither baseline received comparable target-domain pre-training. The masked-language-model objective can learn site-specific layout, vocabulary, and DOM patterns that transfer directly to token classification on the same pages, potentially inflating DOM-LM's F1 for title, date, text, author, and tag. The paper does not report an ablation excluding evaluation sites from pre-training, so the headline superiority claim is not yet supported as a statement about the model architecture or pre-training strategy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new multilingual dataset of 3,172 labeled news web pages from 161 websites in English, German, Russian, Chinese, Korean, and Arabic, with annotations for title, publication date, main text, authors, and tags. The authors fine-tune MarkupLM (pre-trained on English) and DOM-LM (pre-trained on their multilingual dataset plus a CommonCrawl-News sample) for node-level attribute extraction. They compare these models against XLM-RoBERTa and open-source heuristic tools (Trafilatura, Newspaper, News-please) across one-language, cross-lingual, mixed-language, and Zyte benchmark settings. The central claim is that multilingual DOM-LM outperforms other solutions for most attributes when fine-tuned and evaluated on mixed-language data.","tokens_in":10267,"tokens_out":4802,"duration_ms":46217,"significance":"If the claims hold, the paper provides a useful public multilingual benchmark for news-page attribute extraction and a practical result: a DOM-LM model pre-trained on multilingual news pages can match or exceed English-only MarkupLM with a translation pipeline, at lower inference cost. The dataset is a concrete and reusable contribution, and the evaluation protocol, including attribute-specific matching rules and 5-fold site-disjoint splits, is described in sufficient detail to be reproducible. However, the central model-comparison claim is currently undermined by a data-leakage confound, and the absence of variance reporting makes it difficult to assess whether the reported differences are meaningful.","major_comments":[{"comment":"DOM-LM pre-training is performed on the full multilingual dataset (Section VI.C: \"pre-training was performed on our multilingual news dataset\") before the 5-fold cross-validation in Section VII.A divides websites into training and test folds. Because the pre-training phase is unsupervised but sees the raw HTML and DOM structure of every evaluation page, the test pages are not truly unseen to the model at the token-classification stage. The comparison baselines (MarkupLM pre-trained on English CommonCrawl, XLM-RoBERTa pre-trained on general text) did not receive comparable target-domain pre-training, so the reported advantage of DOM-LM in Table VII conflates architecture and pre-training strategy with exposure to evaluation pages. The paper does not report an ablation excluding evaluation sites from pre-training, nor any other evidence that this overlap does not inflate F1. This issue is load-bearing for the Section IX conclusion that multilingual DOM-LM outperforms other solutions on mixed-language data.","section":"§VI.C and §VII.A"},{"comment":"All results are reported as single point estimates of F1, with no standard deviations, per-fold results, confidence intervals, or significance tests. Several differences that underlie the \"outperforms\" claim are small: in Table VII, DOM-LM versus MarkupLM-EN is 0.93 versus 0.95 for Title, 0.87 versus 0.88 for Date, and 0.41 versus 0.40 for Author. Given the small number of labeled pages for some attributes (e.g., English author and tag have 147 and 98 labeled pages, respectively, in Table III), these differences may well be within run-to-run or fold-to-fold noise. Reporting the five fold-wise results and an estimate of variance is necessary to support the headline comparison.","section":"§VII.A and Tables V, VII, VIII"}],"minor_comments":[{"comment":"In the Structor description, \"The authors retrive a node\" should read \"retrieve\".","section":"§II.B"},{"comment":"The two-line-per-language layout in Table III is difficult to parse; the counts of sites/pages with each attribute would be clearer in a standard tabular format with separate rows for sites, pages, and nodes.","section":"Table III"},{"comment":"The new-language experiment reports results only for German and Chinese, although the setup (holding out one language and training on all others) could be applied to all six languages. Reporting all held-out languages would strengthen the cross-lingual transfer claim.","section":"§VIII.B"},{"comment":"The interpretation that Chinese works better than German because the model focuses more on markup than on text is not directly supported by any analysis; consider controlling for markup similarity or reporting additional diagnostic evidence.","section":"§VIII.B"},{"comment":"The sentence \"Sites used for model training and evaluation did not intersected\" contains a grammatical error and should read \"did not intersect.\"","section":"§VII.A"}],"recommendation":"major_revision","confidential_remarks":"The data-leakage issue is the main obstacle to publication. If the authors can re-run DOM-LM pre-training using only the training folds of the site-disjoint split (or otherwise demonstrate that including evaluation pages in pre-training does not inflate scores), the paper's central comparison would become credible. The absence of error bars is also important given the small margins between methods. The dataset contribution itself is solid and could justify revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper ships a genuinely useful resource and a reasonable set of experiments, but the main superiority claim is weaker than the paper lets on because the multilingual DOM-LM was pre-trained on the same pages it is later evaluated on. That is a fixable flaw, not a fatal one.\n\nWhat is new: a public dataset of 3,172 labeled news pages in six languages from 161 sites, with title, date, text, author, and tag annotations. The construction process is described carefully (sitemaps, manual validation, proxy handling, captcha bypass). Extending the earlier Russian-only set with five languages at this scale is a real contribution. The empirical findings on translating pages to English for MarkupLM and on pre-training DOM-LM from XLM-R are also new, and the comparison against open-source tools is useful.\n\nThe main problem: Section VI.C pre-trains DOM-LM on \"our multilingual news dataset\" and Section VII.A evaluates with 5-fold CV on that same dataset. The folds are site-disjoint for fine-tuning, but the unsupervised pre-training has already seen every page's raw HTML, including pages in the test fold. Masked LM on node representations can memorize site-specific layout and vocabulary, which transfers to token classification. The paper does not ablate this, so the claim that multilingual DOM-LM \"outperforms other solutions for most attributes\" on mixed data is not yet established as an architectural win. That said, the effect size on some attributes is large (e.g., tag F1 0.73 vs 0.67), and it is plausible that a clean ablation would still show an advantage.\n\nOther, smaller issues: no error bars or significance tests anywhere; the Arabic one-language result for DOM-LM is poor (title 0.60 vs MarkupLM-EN 0.94), which complicates the \"multilingual\" story; on the Zyte benchmark DOM-LM (0.94) is slightly below MarkupLM-EN (0.95), so the headline superiority is specific to the mixed-language setting; and the translation matching for MarkupLM-EN is not described in enough detail to reproduce.\n\nBottom line: the dataset alone justifies a serious referee. The comparison needs an ablation that excludes evaluation sites from pre-training, plus confidence intervals. This is exactly the kind of paper where a round of revision can fix the main issue.\n\nRecommendation: send it out. It deserves peer review.","headline":"A useful new multilingual news extraction dataset, but DOM-LM's reported edge over baselines is weakened by unsupervised pre-training on the same pages used for evaluation.","tokens_in":10754,"tokens_out":2249,"would_cite":true,"duration_ms":21518,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multilingual DOM-LM extracts news attributes across six languages better than English-only MarkupLM and open-source tools, with no translation step.","keywords":["web data extraction","information extraction","news web pages","multilingual dataset","DOM-LM","MarkupLM","XLM-RoBERTa","transformer fine-tuning"],"falsifier":"Re-run the mixed-language 5-fold evaluation with DOM-LM pre-trained only on a corpus that is disjoint from all evaluation pages (for instance, only the sample of raw multilingual news pages or only the training folds). If its F1 on text, author, and tag falls to or below MarkupLM's with translation, the paper's main claim fails; if the score holds, the result is genuine.","tokens_in":9759,"feed_emoji":"📰","tokens_out":8431,"duration_ms":78701,"temperature":0.7,"pith_summary":"The paper builds a public multilingual benchmark for news-attribute extraction—3,172 labeled article pages in English, German, Russian, Chinese, Korean, and Arabic from 161 sites—and asks whether a single model can extract title, publication date, main text, authors, and tags across languages without translation. It finds that MarkupLM, an English-only pre-trained model, handles non-English pages poorly unless the pages are machine-translated to English first. The authors then re-initialize DOM-LM, a DOM-structure-aware transformer, from a multilingual language model and pre-train it on multilingual news pages. After fine-tuning, this multilingual DOM-LM matches or beats MarkupLM within most languages and, on mixed-language data, beats MarkupLM, a text-only multilingual transformer, and open-source heuristic tools on most attributes, with no translation overhead.","feed_headline":"Multilingual DOM-LM wins news attribute extraction in six languages","feed_subtitle":"A single model extracts title, date, text, author, and tags from news pages in English, German, Russian, Chinese, Korean, and Arabic.","key_machinery":"The load-bearing mechanism is the DOM-LM representation: a cleaned DOM tree is split into subtrees by a DOM Tree Processor, each node is encoded as the concatenation of its HTML tag, attributes, and text plus tree-position features such as depth, index, and parent index, and the whole sequence is passed through XLM-RoBERTa. The model is trained as token classification, with a node's label placed on a BOS token and prediction read from that token. For the multilingual version, the authors initialize from a multilingual language model and pre-train on raw news HTML from the labeled dataset plus a one-day sample of tens of thousands of multilingual news pages. This lets the model use layout and structure cues together with multilingual semantic cues, which the paper argues is what makes cross-lingual extraction work without translating pages.","core_discovery":"The paper's central claim is that a DOM-aware transformer can be made multilingual by swapping its English backbone for a multilingual encoder and pre-training on a small multilingual news corpus, and that the resulting model is the best tested open-source approach for extracting news attributes when the page language is unknown. In 5-fold site-level cross-validation on mixed-language data, multilingual DOM-LM reaches F1 scores of 0.93 on title, 0.87 on date, 0.93 on text, 0.41 on author, and 0.73 on tag, the highest among tested neural models and libraries on most attributes. MarkupLM needs English translation to compete, and even then leads only on title and date. Within single-language experiments the paper does not claim a clear leader; the advantage is most pronounced in the mixed-language setting. The dataset itself—3,172 pages across six languages and 161 websites—is contributed as a reusable benchmark.","pith_inferences":["If the reported advantage holds up under a clean pre-training split, the practical lesson is that a small, balanced multilingual corpus can substitute for much larger English pre-training data, making DOM-style pre-training affordable for other document extraction domains.","The paper's translation experiment implies a trade-off the authors do not state explicitly: when title and date are the priority and translation is cheap, MarkupLM-plus-translation may still be preferable; when author, tag, and text dominate, DOM-LM wins without the translation step.","The same recipe—a multilingual encoder backbone plus DOM-tree-aware pre-training on a modest domain corpus—could transfer to product pages, job listings, or other semi-structured page types where labeled data are scarce."],"forward_implications":["A single multilingual model can replace per-language models in news aggregation and media analysis, since DOM-LM reaches F1 0.93 on title and text, 0.87 on date, 0.41 on author, and 0.73 on tag in mixed-language 5-fold evaluation without translation.","Machine translation at inference remains a viable path: MarkupLM with English translation attains 0.95 on title and 0.88 on date, slightly above DOM-LM's 0.93 and 0.87, respectively, at the cost of translation time and compute.","Open-source heuristic tools are not sufficient for metadata: the three tested libraries score below the neural models on dates, authors, and tags, though they remain competitive on title and body text.","The released 3,172-page, six-language dataset gives researchers a common benchmark for news attribute extraction, with site-level splits that test generalization to unseen websites.","On the Zyte article benchmark, fine-tuned DOM-LM reaches F1 0.94, close to reported commercial services and above most open-source libraries."],"supporting_citations":[{"why":"Supplies the English-pretrained markup-aware transformer whose cross-lingual behavior, with and without translation, is the paper's main comparison point.","marker":"[1]"},{"why":"Supplies the architecture and pre-training recipe (DOM Tree Processor, node representation, BOS token prediction) that the paper adapts to multilingual data.","marker":"[2]"},{"why":"Supports the choice of DOM-LM by showing its SWDE performance is close to MarkupLM while needing far less pre-training data.","marker":"[3]"},{"why":"Supplies the seed dataset of Russian news pages and the F1 evaluation methodology (4-gram bags, sets, normalized dates, site-level 5-fold splits) reused for all experiments.","marker":"[6]"},{"why":"Supplies the multilingual backbone and tokenizer used to initialize the multilingual DOM-LM and the text-only XLM-RoBERTa baseline.","marker":"[11]"},{"why":"Supplies one of the open-source heuristic extraction baselines compared across all experiments.","marker":"[13]"},{"why":"Supplies the other open-source news-crawler baseline used in the comparisons.","marker":"[14]"}],"fun_headline_variants":["Multilingual DOM-LM tops news attribute extraction in six languages","DOM-LM beats English-first models on multilingual news extraction","Six-language news page extraction: DOM-LM leads open-source tools","Multilingual transformer DOM-LM sets new bar for news attribute extraction","DOM-LM excels at extracting news attributes across six languages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that pre-training DOM-LM on the full dataset—including pages that later appear in the 5-fold evaluation—does not inflate its measured quality; if that pre-training leaks information, the reported lead over MarkupLM and the heuristic tools is overstated.","fun_headline_variants_meta":{"raw":{"variants":["Multilingual DOM-LM tops news attribute extraction in six languages","DOM-LM beats English-first models on multilingual news extraction","Six-language news page extraction: DOM-LM leads open-source tools","Multilingual transformer DOM-LM sets new bar for news attribute extraction","DOM-LM excels at extracting news attributes across six languages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000522,"raw_usage":{"total_tokens":2504,"prompt_tokens":905,"completion_tokens":1599,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":1513}},"tokens_in":521,"tokens_out":1599,"duration_ms":11754,"temperature":1.0,"reasoning_tokens":1513,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T13:05:08.402549+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the mixed-language 5-fold evaluation with DOM-LM pre-trained only on a corpus that is disjoint from all evaluation pages (for instance, only the sample of raw multilingual news pages or only the training folds). If its F1 on text, author, and tag falls to or below MarkupLM's with translation, the paper's main claim fails; if the score holds, the result is genuine.","supporting_citations":[{"cited_title":"Markuplm: Pre-training of text and markup language for visually rich document understanding,","cited_arxiv_id":null,"evidence_quote":"Supplies the English-pretrained markup-aware transformer whose cross-lingual behavior, with and without translation, is the paper's main comparison point."},{"cited_title":"From one tree to a forest: a unified solution for structured web data extraction,","cited_arxiv_id":null,"evidence_quote":"Supports the choice of DOM-LM by showing its SWDE performance is close to MarkupLM while needing far less pre-training data."},{"cited_title":"A dataset for information extraction from news web pages,","cited_arxiv_id":null,"evidence_quote":"Supplies the seed dataset of Russian news pages and the F1 evaluation methodology (4-gram bags, sets, normalized dates, site-level 5-fold splits) reused for all experiments."},{"cited_title":"Un- supervised cross-lingual representation learning at scale,","cited_arxiv_id":null,"evidence_quote":"Supplies the multilingual backbone and tokenizer used to initialize the multilingual DOM-LM and the text-only XLM-RoBERTa baseline."},{"cited_title":"Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction,","cited_arxiv_id":null,"evidence_quote":"Supplies one of the open-source heuristic extraction baselines compared across all experiments."},{"cited_title":"news-please: A generic news crawler and extractor,","cited_arxiv_id":null,"evidence_quote":"Supplies the other open-source news-crawler baseline used in the comparisons."}],"review_version":1}