{"id":"7d278c31-aa28-45da-b226-1a936a65aec5","arxiv_id":"2508.20778","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A contrastive fine-tuning recipe that adds HTML structure and element-masking signals improves long structured document retrieval, with reported MRR@10 gains of about four points on BGE-M3.","lead":"SEAL is a fine-tuning method that teaches retrieval models to use HTML headings and section structure, by contrasting query embeddings with original, tag-stripped, and element-masked versions of the same documents. It reports consistent gains on an industrial dataset and a new 10k-word benchmark, StructDocRetrieval, making it useful for RAG and enterprise search.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Most load-bearing concern: the reported SEAL gains, including the abstract's 73.96→77.84, come from the same industrial split used to select the mask ratio and training order (Tables 6–7 vs 3), so the headline improvement is a test-set selection rather than an unbiased estimate.","rationale":"The reader's stated weakest assumption is the unverified web benchmark labels, which is a genuine concern. However, the test-set selection issue is more load-bearing because it directly affects the exact headline numbers the abstract cites. The paper's limitations section acknowledges a Chinese-language focus and open questions about generalization, but it does not acknowledge the absence of a validation split or the selection of hyperparameters on the evaluation set. The industrial click data and the online A/B test give the method real-world support, so a rejection is not warranted; the correct response is to require a clean validation protocol before the reported magnitude is trusted. Because the reader's verdict is already CONDITIONAL, my analysis does not change the verdict, hence UNCHANGED.","tokens_in":12334,"tokens_out":9615,"duration_ms":99538,"concrete_test":"Rerun the full SEAL pipeline with a held-out validation split: on the industrial training set, train all mask-ratio and training-order variants, select the best configuration by MRR@10 on a fixed validation fold, then evaluate that single configuration on the untouched test split. If the selected-on-validation MRR@10/NDCG@10 is more than 1 point below Table 3's 77.84/82.59, the reported improvement is in part a test-set selection artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim rests on the industrial evaluation table, but the protocol for choosing SEAL's two key hyperparameters is not separated from the evaluation split. In Section 4.3, Table 6 chooses the EAL mask ratio and Table 7 chooses the SAL/EAL training order by comparing metrics on the industrial structured-document retrieval data. The best rows in those tables—mask ratio 10% and EAL1−SAL2—reproduce exactly the BGE-M3 w/ SEAL row in Table 3 (HitRate@5 93.77, MRR@10 77.84, NDCG@10 82.59). Since the paper describes only a Train/Evaluation split for the industrial data (Table 2) and does not mention a separate validation set, the final reported numbers appear to be selected on the same split used for the headline result. This is a textbook test-set-selection problem: the model is chosen to maximize the very metric that is then reported as the improvement. No standard deviations or multiple-seed results are given, so the selection inflation cannot be quantified from the paper. The unverified LLM-synthesized labels for StructDocRetrieval (Section 3.1) are a secondary concern, but the industrial click data and online A/B test provide some independent support; the selection issue directly contaminates the headline number.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SEAL, a contrastive learning framework for long structured document retrieval that combines two auxiliary objectives: Structure-Aware Learning (SAL), which contrasts tagged and untagged versions of relevant documents, and Element-Aware Alignment (EAL), which randomly removes structural tags from a fraction of elements and pulls the query toward the masked relevant document. The authors also release StructDocRetrieval, a benchmark of long HTML-formatted documents with LLM-generated queries. Experiments compare SEAL against raw-text, Chunk, MCLS, and SANTA baselines on an internal industrial dataset and the new benchmark across three encoder-based PLMs (mE5-large, bge-large-zh, BGE-M3), plus an online A/B test. The paper reports consistent gains, e.g., BGE-M3 MRR@10 improves from 73.96 to 77.84 on the industrial dataset and NDCG@10 from 79.41 to 82.59.","tokens_in":12750,"tokens_out":5240,"duration_ms":44503,"significance":"If the empirical claims hold, SEAL offers a simple, potentially effective recipe for injecting HTML structure into dense retrieval fine-tuning, and the released dataset could be a useful resource for an underexplored task. The paper has clear strengths: multiple backbone models, an online A/B deployment, a public dataset release, and a transparent description of the two losses. However, the current evidence is weakened by the headline metric mislabeling, selection of hyperparameters on the evaluation split, and reliance on unverified LLM-generated relevance labels for the released benchmark. These issues must be addressed before the claimed gains can be taken at face value.","major_comments":[{"comment":"The abstract (and Section 1) states that SEAL achieves 'boosting NDCG@10 from 73.96% to 77.84% on BGE-M3.' These exact numbers do not appear under NDCG@10 in Table 3: for BGE-M3, MRR@10 is 73.96 for the base model and 77.84 with SEAL, while NDCG@10 is 79.41 → 82.59. The headline improvement is therefore labeled with the wrong metric. Please correct the abstract and all associated text, or report the true NDCG@10 gain. This is a central quantitative claim and must be accurate.","section":"Abstract; Table 3"},{"comment":"The mask ratio (Table 6) and training order (Table 7) are selected by comparing HitRate@5, MRR@10, and NDCG@10 on the industrial structured-document retrieval data. The paper describes only Train and Evaluation splits for the industrial dataset (Table 2) and does not mention a held-out validation split in Section 4.1. The best configuration from these tables (10% mask ratio, EAL1−SAL2) reproduces exactly the BGE-M3 w/ SEAL row in Table 3 (MRR@10 77.84, NDCG@10 82.59). Thus the main reported gain appears to be selected on the same split that is used for the headline evaluation, i.e., test-set selection. Please provide a separate validation split, nested evaluation, or otherwise demonstrate that hyperparameter selection does not inflate Table 3. Report multiple seeds/standard deviations.","section":"Section 4.3, Tables 6–7 vs Table 3"},{"comment":"StructDocRetrieval relevance labels are produced by 'LLM-powered query synthesis' from documents with no human relevance judgments described. Table 2 shows a single auto-generated query per document, and no details are given for negative document sampling or for filtering queries that simply echo document surface terms. Consequently, the gains on StructDocRetrieval (Table 4) could partly reflect lexical overlap rather than structural understanding. Please document the annotation protocol, include human verification or at least measure lexical overlap between queries and documents, and report how negatives were constructed.","section":"Section 3.1, Table 2"},{"comment":"Table 8 is titled 'The retrieval performance of GTE-Qwen2-1.5B on StructDocRetrieval,' but the rows are labeled mE5-large, +Chunk, +MCLS, +SEAL. The text says GTE-Qwen2-1.5B is used as an extended-context model. This mismatch makes the extended-context robustness claim unverifiable. Please correct the table/caption or the model name, and ensure all results in that subsection correspond to GTE-Qwen2-1.5B.","section":"Table 8, Section 4.3"}],"minor_comments":[{"comment":"The sentence 'The experimental validity is strengthened through robustness testing 5 10 Days' appears broken; also the x-axis labels in Figure 4 are not clearly described.","section":"Section 4.3 / Figure 4"},{"comment":"Karpukhin et al. 2020a and 2020b are duplicated entries for the same paper (Dense Passage Retrieval). Please merge them.","section":"References"},{"comment":"Table 6 says 'The Experimental results' and the table heading 'ratios (%)' is placed awkwardly; please proofread.","section":"Section 4.3"},{"comment":"No standard deviations or significance tests are reported. Given that several gains are small (e.g., some rows in Table 4), this would help assess stability.","section":"All experimental tables"},{"comment":"The T-SNE visualizations are anecdotal; consider adding a quantitative separation metric or noting this as illustrative only.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The industrial dataset is proprietary and the web benchmark is authored by the same group, so external validation is limited. The online A/B test is encouraging but does not replace a clean offline evaluation. The abstract metric mislabeling and the absence of a separate validation split are substantive issues that need to be fixed in revision; the duplicate reference and table caption inconsistency also contribute to a sense of rushed preparation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is simple and worth taking seriously: train a dense retriever with two contrastive auxiliary objectives—one that pushes queries toward both tagged and untagged versions of relevant documents, and one that masks structural tags on a random subset of elements—then fine-tune existing embedding models on long HTML documents. That recipe is architecture-agnostic and reproducible, and the reported gains are consistent across three base models on both the industrial and web datasets. Releasing StructDocRetrieval, with 10k+ word HTML documents, is a real contribution; the field needs more long structured retrieval benchmarks. The online A/B test is a nice extra signal, though it is only compared against raw-text contrastive learning, not against SANTA or other structure-aware baselines.\n\nThe soft spots are proportionate but real. First, the abstract says NDCG@10 improved from 73.96 to 77.84 on BGE-M3, but those numbers are MRR@10 in Table 3. That is a careless mislabel on the central claimed result. Second, the hyperparameter choices—10% mask ratio and the EAL-then-SAL training order—are selected using the same industrial evaluation split that produces the headline numbers. Tables 5–7 and Table 3 all report on that split, and no separate validation split is mentioned. So the reported 77.84 MRR@10 is not an unbiased estimate; it is the best row picked from a small hyperparameter grid on the test split. The paper does not report standard deviations or multiple runs, so the inflation cannot be quantified. Third, the web benchmark uses one LLM-generated query per document with no human relevance judgment, and there is no documentation of hard negatives. That makes the StructDocRetrieval results suggestive rather than definitive. Fourth, the baseline set is thin: only Chunk, MCLS, and a SANTA adaptation, with no significance testing.\n\nNone of this destroys the paper. The selection problem is fixable by rerunning with a held-out validation split and reporting test-set numbers once, or at least by being transparent that the reported numbers are selected hyperparameters on the same split. The benchmark is still useful as a community resource even if its labels are noisy. But as written, the magnitude of the improvement is not established, and the abstract overclaims.\n\nThis deserves a serious referee, mostly to force the authors to fix the split and the metric labels. If those are corrected, it is a reasonable incremental contribution to structured document retrieval. I would not rely on the numeric gains for my own work until the evaluation is cleaned up, but I would read a revised version and might use the benchmark. Bring it to a reading group if you want to discuss benchmark construction and test-set-selection pitfalls; otherwise, it is a solid 'revise and resubmit' candidate.","headline":"Useful recipe and a new benchmark, but the headline gain is overstated by a mislabeled metric and by tuning on the evaluation split.","tokens_in":13148,"tokens_out":2127,"would_cite":false,"duration_ms":24312,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Structure-aware contrastive learning lifts long-document retrieval","keywords":["long document retrieval","structured documents","contrastive learning","HTML representation","dense retrieval","retrieval benchmark","element masking","structure-aware learning"],"falsifier":"Take the released StructDocRetrieval test set and create a subset of queries that have no lexical overlap with their target document. If SEAL's advantage over raw-text contrastive learning disappears on this subset, the reported gains would be explained by term matching, not structural semantics. Conversely, removing all HTML tags at test time and seeing the gain vanish would show the model relies on markup artifacts.","tokens_in":12314,"feed_emoji":"🔍","tokens_out":5547,"duration_ms":46653,"temperature":0.7,"pith_summary":"SEAL is a contrastive fine-tuning method for long structured documents (HTML pages, technical specs, legal texts). The paper argues that existing retrieval models trained on raw text are blind to document hierarchy—headings, sections, list items—and cannot align query meaning to fine-grained elements. To fix this, SEAL adds two losses: Structure-Aware Learning pulls tagged and untagged versions of the same relevant document toward the query, while Element-Aware Alignment randomly masks structural tags so the model must use the remaining element content to judge relevance. The authors also release StructDocRetrieval, a ~10k-word-per-document benchmark with HTML annotations. Across three embedding models, two datasets, and online A/B tests, SEAL consistently outperforms chunking, multi-CLS, SANTA, and raw-text contrastive learning; the abstract's headline gain (73.96% to 77.84% on BGE-M3) appears in Table 3 under MRR@10, not NDCG@10.","feed_headline":"Structure-aware training lifts long-document retrieval","feed_subtitle":"Two added training objectives make retrievers respect HTML headings and sections, with gains on web and industrial data.","key_machinery":"The two training objectives are the central mechanism. Structure-Aware Learning (SAL) uses contrastive pairs in which positive documents appear both with and without HTML tags, teaching the encoder to map structural markup and plain text into the same embedding region. Element-Aware Alignment (EAL) randomly masks the tags of a proportion of elements and contrasts the query with the masked relevant document, forcing the model to rely on unmasked elements and textual content to infer relevance. The framework continually fine-tunes PLMs such as BGE-M3, mE5-large, and bge-large-zh from a starting point of standard contrastive learning, and uses a dataset construction pipeline that converts docum","core_discovery":"The central claim is that explicitly teaching a retriever about document structure improves long structured document retrieval without changing the model architecture or inference cost. SAL makes the model treat a query as equally similar to a relevant document with HTML tags and to its tag-stripped plain-text version, forcing the representation to be invariant to markup while preserving content semantics. EAL randomly deletes structural tags on a subset of elements (10% works best) and asks the model to keep the masked document close to the query, so it learns element-level discriminative features. The released dataset, StructDocRetrieval, provides long, HTML-annotated documents and LLM-gen","pith_inferences":["Because relevance labels in StructDocRetrieval come from LLM-generated queries on crawled documents without human verification or graded relevance, the measured gains may partly reflect lexical overlap between query and source text rather than structural understanding; a human-judged subset would let readers separate these effects.","The abstract's headline number is mislabeled: 73.96→77.84 is reported as MRR@10 in Table 3, not NDCG@10. If the intended claim is NDCG@10, the correct table entry is 79.41→82.59, still a gain but a smaller one.","The method's reliance on paired structured/unstructured variants suggests it could be extended to other structured formats such as PDFs, JSON, or markdown, or combined with structure-aware pre-training rather than only fine-tuning.","The paper's own limitations note that evaluations are mostly Chinese-language; whether the structural signal transfers to English long documents remains untested."],"forward_implications":["Retrieval systems can gain structural awareness by fine-tuning existing PLMs on HTML-tagged documents, with no new architecture or extra inference latency.","A benchmark with long documents (average >10,000 words) and explicit HTML structure is now available for apples-to-apples evaluation of structure-aware retrievers.","The training recipe—EAL before SAL, 10% element mask ratio—offers a concrete default that generalizes across encoder-only embedding models.","The gains hold in deployment: an online A/B test on ~30% of search traffic over 14 days shows higher PV CTR and UV CTR for SEAL than raw-text contrastive learning."],"supporting_citations":[{"why":"Supplies the BGE-M3 backbone and the MCLS baseline; the main model for the headline results.","marker":"Chen et al., 2024"},{"why":"SANTA is the structure-aware pre-training baseline that SEAL compares against and extends.","marker":"Li et al., 2023a"},{"why":"Justifies using HTML over plain text for representing retrieved knowledge, motivating the structural representation.","marker":"Tan et al., 2025"},{"why":"MS MARCO represents the typical short, plain-text retrieval dataset lacking structural metadata, setting up the gap.","marker":"Nguyen et al., 2016"},{"why":"Introduces dense passage retrieval, the foundational contrastive learning paradigm the method builds on.","marker":"Karpukhin et al., 2020b"},{"why":"Provides bge-large-zh, one of the three base models fine-tuned by SEAL.","marker":"Xiao et al., 2023"},{"why":"Provides multilingual E5, another base model used in the evaluation.","marker":"Wang et al., 2024"},{"why":"Supplies GTE-Qwen2, the extended-context model used to test robustness.","marker":"Li et al., 2023b"}],"fun_headline_variants":["Structure-aware training lifts long-document retrieval","New objectives make retrievers honor HTML structure","Dataset release plus structure tricks boost retrieval","SEAL framework teaches retrievers document hierarchy","Element-aware alignment improves long-doc retrieval"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The web-benchmark labels assume that an LLM-generated query for a crawled document is a valid relevance signal, with no human verification; if those queries share surface terms with the document, the measured gains may come from lexical matching rather than structural understanding.","fun_headline_variants_meta":{"raw":{"variants":["Structure-aware training lifts long-document retrieval","New objectives make retrievers honor HTML structure","Dataset release plus structure tricks boost retrieval","SEAL framework teaches retrievers document hierarchy","Element-aware alignment improves long-doc retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1070,"prompt_tokens":675,"completion_tokens":395,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":419,"completion_tokens_details":{"reasoning_tokens":330}},"tokens_in":419,"tokens_out":395,"duration_ms":4752,"temperature":1.0,"reasoning_tokens":330,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:49:08.066171+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the released StructDocRetrieval test set and create a subset of queries that have no lexical overlap with their target document. If SEAL's advantage over raw-text contrastive learning disappears on this subset, the reported gains would be explained by term matching, not structural semantics. Conversely, removing all HTML tags at test time and seeing the gain vanish would show the model relies on markup artifacts.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MS MARCO represents the typical short, plain-text retrieval dataset lacking structural metadata, setting up the gap."}],"review_version":1}