{"id":"f525ec08-4630-4168-974c-c29598cb5592","arxiv_id":"2506.16644","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"SORE uses multilingual embeddings plus approximate nearest-neighbor search to identify and remove non-core text segments from HTML documents, claiming LLM-comparable quality at much lower cost.","lead":"This paper introduces SORE, a system that removes boilerplate from web pages by embedding sentences and comparing them with document metadata and hand-picked outlier phrases, instead of using expensive LLMs. It claims near-LLM extraction quality at 25 times lower cost, with production use at millions of documents per day.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SORE's near-LLM F-score is obtained by selecting parameters, embedders, and outlier groups on the same SORE-LARGE test set where it is reported; an independent holdout evaluation is required before the claim is credible.","rationale":"The reader's verdict of REJECT is appropriate, and the reader's rationale already mentions that hyperparameters and outlier phrase sets are tuned on SORE-LARGE. However, the reader's stated weakest assumption is the ReadabilityJS-derived ground truth, which I consider secondary: SORE actually outperforms ReadabilityJS on SORE-LARGE even though the ground truth was partly generated by ReadabilityJS, so any ground-truth bias toward ReadabilityJS would make SORE's advantage conservative rather than inflated. The more load-bearing problem is explicit in-sample model selection: Figure 3 is a precision-recall sweep over parameters on the same dataset used for Table 3, and Appendix A describes outlier groups chosen through precision-recall balancing. Because the method is deterministic and has many configurable components, the reported F-score is a test-set maximum, not an estimate of expected performance. The abstract's phrase 'near-LLM extraction precision' also overstates the evidence: Table 3 reports F-score, and the underlying precision of the SORE Titan configuration appears substantially lower than the LLM baseline precision. Therefore the claim requires an independent holdout evaluation with released data, code, and pre-registered settings; without that, the reported numbers do not establish the central claim. The paper does have credible independent elements: a deployable production system, a concrete cost analysis, and a deterministic algorithm with transparent removal reasons. These do not compensate for the missing unbiased evaluation, so the reader's REJECT verdict should stand unchanged.","tokens_in":9190,"tokens_out":5311,"duration_ms":55784,"concrete_test":"Ask the authors to release the SORE-LARGE dataset and implementation, split it into disjoint tuning and test folds, and re-run the full pipeline with all parameters and outlier groups tuned only on the tuning fold. Then report a single F-score and precision/recall pair for SORE, ReadabilityJS, and the LLM tag-depth baseline on the held-out fold, ideally with bootstrap confidence intervals. If the held-out SORE F-score drops materially below 0.776 or falls below the held-out LLM F-score, Table 3's near-LLM claim is an artifact of in-sample selection rather than a reliable property of the method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim rests on Table 3: SORE with AWS Titan reaches F=0.776 versus F=0.793 for the best LLM prompt on SORE-LARGE. The experiments do not support this number as an unbiased estimate. Section 4.2.2 states that the 'best distance cutoff parameters per model family' were selected from Figure 3, which plots precision-recall curves on SORE-LARGE itself, and the reported Titan result uses the 'optimal parameters' found there. Appendix A additionally says outlier groups were 'iteratively refined based on performance metrics' and that the study presents 'the performance analysis that SORE enables in choosing these keywords.' No validation split, cross-validation, or error bars are reported. The reported F-score is therefore a selected maximum over a range of core percentages, cutoffs, embeddings, and outlier configurations, and selection on the test set is expected to inflate it. With N=20,000 but no variance estimates, the 0.017 F-score gap to the LLM baseline is well within the range that selection effects could explain. A further mismatch is that the abstract claims 'near-LLM extraction precision,' but the supporting table reports F-score; on SORE-LARGE the LLM baseline precision is 0.885 while the Titan SORE configuration shown in Figure 3 has precision around 0.65-0.70. Thus the headline claim is not established as stated, and the method's actual precision advantage over structural baselines is unclear until evaluated on a truly held-out dataset with fixed, pre-registered parameters.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SORE (Semantic Outlier Removal), a deterministic pipeline that removes extraneous text from HTML documents using multilingual sentence embeddings and approximate nearest-neighbor search. The method segments a document, embeds each segment, identifies a core set of segments by similarity to document metadata, and flags non-core segments that are either close to predefined outlier groups or far from the core. The authors evaluate SORE on two proprietary datasets (SORE-SMALL with roughly 200 hand-labeled documents and SORE-LARGE with roughly 20,000 automatically labeled documents), comparing against ReadabilityJS and three LLM prompting strategies. The central reported result is that SORE with AWS Titan embeddings reaches F=0.776 on SORE-LARGE versus F=0.793 for the best LLM prompt, at 25x lower cost and 12.5x lower latency. The paper also describes a production deployment and promises public release of implementation and datasets.","tokens_in":9440,"tokens_out":5511,"duration_ms":56003,"significance":"If the empirical claims were properly supported, SORE would be a practically valuable contribution: a transparent, deterministic, multilingual content-extraction method that approaches LLM quality at a fraction of the cost, with concrete operational advantages for large-scale web processing. The cost and latency figures, the architectural transparency, and the production deployment experience are notable strengths. However, the current evaluation has load-bearing methodological problems: hyperparameters and outlier groups appear to be selected on the same test set used for final reporting, and the large-scale ground-truth labels are partly generated by one of the baselines. These issues make the headline performance claims unverified as presented. The central idea is defensible and testable, but the evidence in this version does not establish it.","major_comments":[{"comment":"The central quantitative claim is built from parameters selected on the evaluation set. The text states that the best distance cutoff d was chosen per model family from Figure 3, which plots precision-recall curves on SORE-LARGE, and that the AWS Titan result in Table 3 uses the \"optimal parameters\" found there. Appendix A adds that the outlier groups were \"iteratively refined based on performance metrics,\" and Table 4 reports keyword accuracy on the same corpus. No validation split, cross-validation, or confidence intervals are reported. The F=0.776 in Table 3 is therefore a maximum over configurations selected on the test set, and the 0.017 gap to the LLM baseline is within the range that selection effects could explain. The authors must report results on a held-out split with hyperparameters and outlier groups fixed before the main comparison is credible.","section":"Section 4.2.2 and Table 3"},{"comment":"The SORE-LARGE ground truth is automatically generated \"using a combination of ReadabilityJS and n-gram–based content cleanup,\" and ReadabilityJS is also a baseline in Table 3 and Figure 3. Evaluating ReadabilityJS against a ground truth partly derived from ReadabilityJS makes the comparison circular: the reported recall of 0.988 and the relative F-scores are not an independent measure of SORE's capability. The manuscript should provide a bias analysis, manual validation of a random sample, or an independently labeled test set.","section":"Section 4.1 and Table 3"},{"comment":"The abstract's claim of \"near-LLM extraction precision\" is not what Table 3 reports; Table 3 reports F-scores. On SORE-LARGE the LLM tag-depth baseline has precision 0.885, while the AWS Titan SORE configuration shown in Figure 3 has precision in the range of roughly 0.65–0.70. The headline claim should either report precision for SORE explicitly or be revised to \"near-LLM F-score.\" As written, the claim is not supported by the displayed precision numbers.","section":"Abstract and Figure 3/Table 3"},{"comment":"No variance or confidence-interval estimates are provided for any reported metric. On SORE-SMALL, N is only about 200, yet the comparison in Table 1 reports point F-scores with no uncertainty. On SORE-LARGE, N is 20,000, but the parameter-selection issue in Section 4.2.2 makes point estimates even less interpretable. The paper should provide bootstrap confidence intervals or standard errors for the key comparisons.","section":"Tables 1 and 3"}],"minor_comments":[{"comment":"The name \"V oyager\" appears with an extra space; it should be \"Voyager\" throughout.","section":"Section 3.2"},{"comment":"The SORE configuration is written as \"c=0.5, k=10%\" but the parameter c is not defined in the main text; it appears to be the distance cutoff d, and this should be made consistent.","section":"Table 1"},{"comment":"The caption says each point represents different parameter configurations and that the best distance cutoff d is used per model family, but the axes and point labels do not show d; the figure would be clearer if the cutoff values were annotated or described.","section":"Figure 3 caption"},{"comment":"The cost for ReadabilityJS is listed as $7 per million documents, but this is an open-source library; please state what cost model is assumed for this row.","section":"Table 3"},{"comment":"Several outlier keywords are duplicated or overlap across categories, e.g., \"Subscribe\" appears in both \"Calls to Action\" and \"Commercial Content,\" and \"Policy\" is listed twice in \"Legal Content.\" This is a minor presentation issue but should be cleaned for reproducibility.","section":"Appendix A.2"},{"comment":"The column \"Accuracy\" is not defined in Appendix A; please specify how removal accuracy was computed and on which dataset.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The evaluation problems are serious and load-bearing: parameter and outlier-group selection on the same test set, combined with a ReadabilityJS-derived ground truth used to evaluate ReadabilityJS, invalidate the headline numbers as they stand. The contribution could still be valuable, so I would not reject outright if the authors can supply a genuinely held-out evaluation with independent labels and fixed hyperparameters. If they cannot, the paper should not be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read of arXiv:2506.16644. The central claim—that SORE gives near-LLM extraction quality at a fraction of the cost—is plausible but not supported by the experiments as run. The biggest problem is that parameters and outlier groups were selected on the same SORE-LARGE dataset where the final numbers are reported (Section 4.2.2, Figure 3, and Appendix A). That means the F=0.776 vs 0.793 is a selected maximum, not an unbiased estimate. The ground truth for SORE-LARGE is also partly generated from ReadabilityJS, one of the baselines, which biases the comparison against structural methods and in favor of SORE's recall. The abstract says \"near-LLM extraction precision,\" but the tables report F-score; the precision gap to the LLM baseline is larger than the F-score gap. So the headline claim as stated is not established.\n\nWhat is genuinely new? The actual SORE algorithm—metadata-anchored core selection with ANN outlier-group matching, plus explicit removal reasons—is a sensible, novel combination I haven't seen in the cited literature. The system is deployed in production processing millions of documents, which is real evidence of engineering viability. The cost story (25x cheaper, 12.5x lower latency) is credible as an engineering claim, even if it doesn't beat a tuned baseline. The paper is also honest about its tuning process; it doesn't hide that the outlier groups were iteratively refined. That's good practice, but it doesn't rescue the evaluation.\n\nMinor soft spots: no code or data released despite the abstract promising it, no error bars or variance estimates across the 20k samples, and the small dataset (200 docs) is too small to distinguish methods. The multilingual examples are illustrative rather than measured.\n\nWho benefits? Anyone building a high-volume content extraction pipeline should read this for the system design and cost model. It's also a useful case study in how test-set tuning can quietly undermine a comparison.\n\nFor review: I'd send it out rather than desk-reject, because the algorithmic contribution is real and the evaluation can be fixed. But it needs a substantial revision: a truly held-out evaluation with pre-registered parameters, a ground-truth process not derived from a compared baseline, and precision/recall numbers reported separately. As it stands, the performance claim is unproven, not disproven.","headline":"SORE is a sensible, deployable system, but its central performance claim is undermined by test-set tuning and a ground truth derived from a compared baseline.","tokens_in":10052,"tokens_out":2245,"would_cite":false,"duration_ms":21386,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SORE, a two-step embedding-based system, removes web-page boilerplate with near-LLM precision at one twenty-fifth the cost.","keywords":["semantic outlier removal","content extraction","boilerplate removal","sentence embeddings","multilingual embedding models","approximate nearest neighbor","HTML cleaning","LLM cost reduction"],"falsifier":"Take a random sample of about 500 documents from the SORE-LARGE corpus, have human annotators mark the true main content, and recompute F-scores for SORE, ReadabilityJS, and the LLM baselines on that subset; if SORE's advantage over ReadabilityJS shrinks or reverses against the human labels, the auto-generated ground truth was masking the true ranking.","tokens_in":8924,"feed_emoji":"🧹","tokens_out":6152,"duration_ms":57241,"temperature":0.7,"pith_summary":"The paper claims that a deterministic, language-agnostic pipeline of multilingual sentence embeddings and approximate nearest-neighbor search can strip web-page boilerplate almost as well as a prompted LLM, at a fraction of the cost and latency. The method, SORE, anchors a document's core to its metadata embedding, then removes segments that either match predefined outlier groups such as ads, legal text, and navigation, or sit far from that core. On two in-house datasets it reports F-scores within two to four points of the best LLM prompt while running 12.5 times faster and costing 25 times less, and it beats ReadabilityJS by four to eight F-score points. If these numbers hold, SORE gives production pipelines a cheap, transparent alternative to LLM-based content cleaning that processes millions of documents daily.","feed_headline":"Embedding-based SORE rivals LLM extraction at 1/25 the cost","feed_subtitle":"Two distance checks flag boilerplate in 200 ms, versus 2.5 s for LLMs, at $600 per million docs.","key_machinery":"The mechanism is a cosine-distance decision rule in embedding space. A document's metadata embedding $w_m$ serves as the semantic anchor; segments nearest to $w_m$ form the core, and non-core segments are flagged when their distance from the core exceeds a threshold $d$ or they land near one of about a dozen predefined outlier groups (dates, authorship, comments, source attribution, related links, calls to action, navigation, contact, social media, legal text, page infrastructure, commercial content, and miscellaneous boilerplate). Scalability comes from precomputed HNSW indices over the outlier-group embeddings, inner-product distance on normalized vectors, batched embedding calls, and the Voyager approximate-nearest-neighbor library, which together bring per-document latency to roughly 200 milliseconds.","core_discovery":"SORE's central claim is that a two-signal decision rule—distance to a metadata-derived core plus distance to curated outlier clusters—captures most of what an LLM does when extracting main content from HTML. The document is segmented, each segment embedded, and the top-k% of segments closest to the document's title-and-description vector become the core; every other segment is removed if it lands too close to an outlier group or too far from the core. With AWS Titan embeddings the paper reports 0.776 F-score on its large dataset versus 0.793 for the best LLM prompt, at $600 versus $15,000 per million documents and 200 ms versus 2500 ms latency, with similar margins on a smaller hand-labeled set.","pith_inferences":["A natural stress test the paper does not run is to apply the same two-signal rule to paragraph-level filtering in genres with weaker metadata anchors, such as research papers or legal filings, where the metadata vector may need a fallback to the first sentences or an LLM-generated summary.","The near-parity with LLMs suggests that boilerplate removal depends more on semantic structure than on generative reasoning; testing SORE with an LLM-written one-sentence summary as the anchor would reveal whether a stronger anchor raises the ceiling further.","Because the outlier catalog is fixed, an adaptive variant that learns per-site outlier patterns from template repetition across a site's pages could push precision higher while reusing the same HNSW machinery.","The cost-per-million-document comparison would likely shift if LLM prices fall or embedding prices rise; the paper's qualitative conclusion—that a cheap, deterministic semantic filter can rival LLM extraction on boilerplate—would still stand, but the exact multiple would change."],"forward_implications":["At $600 per million documents, semantic boilerplate removal becomes affordable for high-volume web crawls where LLM pricing was previously prohibitive, though the paper's cost figures assume their specific production setup and cloud pricing.","Because SORE's removal decisions are deterministic and logged with a reason string, it enables audit trails and fine-grained tuning of outlier groups, a property stochastic LLM outputs do not provide.","The method is language-agnostic by construction: any language covered by the embedding model is handled without additional rules, as the Chinese, French, and Spanish examples in Table 2 illustrate.","SORE can run with open-source embedders such as e5-large, which the paper tests, so the core algorithm need not depend on commercial embedding APIs.","The logged removal reasons support continuous refinement of outlier groups and could be repurposed for content moderation or compliance explanations in production systems."],"supporting_citations":[{"why":"Supplies the Boilerpipe structural baseline and text-density features that SORE must beat.","marker":"Kohlschütter et al. (2010)"},{"why":"Industry-standard structural extractor used both as a comparison baseline and as part of the SORE-LARGE ground-truth generation.","marker":"Readability.js (rea)"},{"why":"Establishes sentence-level embeddings for similarity search, the core representation SORE relies on.","marker":"Reimers and Gurevych (2019)"},{"why":"Provides massively multilingual sentence embeddings, the basis of SORE's language-agnostic operation.","marker":"Artetxe and Schwenk (2019)"},{"why":"The multilingual e5 embedder, the open-source model tested in SORE's embedding comparison.","marker":"Wang et al. (2024)"},{"why":"Motivates the LLM extraction baselines and the few-shot behavior that SORE aims to approximate cheaply.","marker":"Brown et al. (2020)"}],"fun_headline_variants":["SORE: near-LLM extraction at 1/25 the cost","Semantic outlier removal matches LLMs at 1/25 price","25x cheaper extraction with near-LLM precision","Embedding-based SORE: LLM-level precision, 25x cheaper","Boilerplate removal: SORE rivals LLMs at 1/25 cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The large-scale evaluation measures SORE and ReadabilityJS against a ground truth that was itself generated with ReadabilityJS plus n-gram cleanup, so the comparison may be biased toward whatever ReadabilityJS keeps.","fun_headline_variants_meta":{"raw":{"variants":["SORE: near-LLM extraction at 1/25 the cost","Semantic outlier removal matches LLMs at 1/25 price","25x cheaper extraction with near-LLM precision","Embedding-based SORE: LLM-level precision, 25x cheaper","Boilerplate removal: SORE rivals LLMs at 1/25 cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00084,"raw_usage":{"total_tokens":3625,"prompt_tokens":873,"completion_tokens":2752,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":2657}},"tokens_in":489,"tokens_out":2752,"duration_ms":19630,"temperature":1.0,"reasoning_tokens":2657,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:21:15.121960+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of about 500 documents from the SORE-LARGE corpus, have human annotators mark the true main content, and recompute F-scores for SORE, ReadabilityJS, and the LLM baselines on that subset; if SORE's advantage over ReadabilityJS shrinks or reverses against the human labels, the auto-generated ground truth was masking the true ranking.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes sentence-level embeddings for similarity search, the core representation SORE relies on."}],"review_version":2}