{"id":"941f0297-57f0-428c-a81c-cf301e4bb6b6","arxiv_id":"2505.17125","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new evaluation framework with XPath-based labels shows Flat JSON input gives LLMs near-perfect record extraction F1 (0.9567) with low hallucination, far outperforming Slimmed HTML and Hierarchical JSON.","lead":"This paper introduces a benchmark framework for web data record extraction, comparing traditional heuristics and LLMs on different HTML and JSON input formats. Its main finding is that Flat JSON, a map from XPath addresses to text, gives Gemini-2.5-pro the highest extraction F1 score (0.9567) and the lowest hallucination rate.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Flat JSON's reported F1 advantage may be an artifact of XPath string matching: Flat JSON supplies the exact canonical label strings, while other formats force the model to regenerate them.","rationale":"Reader identified label quality as weakest assumption; I agree labels are unaudited, but the more load-bearing and more directly testable problem is scoring fairness across input formats. Eq. (1) treats XPath expressions as atomic strings, but XPaths are not canonical: many strings can denote the same node. Because Flat JSON input keys are exactly (or near-exactly) the ground-truth label strings, that format receives a free pass on path generation that Slimmed HTML and Hierarchical JSON do not. The paper's own 'positional accuracy' validation checks existence, not equivalence. This is an internal scoring inconsistency, not a disagreement with consensus, and it can be tested with code. The hallucination metric is also too weak to rescue the claim. The paper's framework is still useful; requiring canonicalization and label audits is an addressable revision, so the reader's CONDITIONAL verdict remains appropriate.","tokens_in":13301,"tokens_out":8572,"duration_ms":76631,"concrete_test":"Recompute Table 2 after canonicalizing every predicted and ground-truth XPath to a DOM node identity (e.g., a deterministic post-order node ID assigned from the cleaned DOM) before computing the intersection in Eq. (1); equivalently, map each path to the text node it resolves to and match on node identity. If the Flat JSON F1 drops materially or the margin over Hierarchical JSON narrows, the current numbers are partly a string-matching artifact. As a check on the label-quality confound, run the same recomputation on a random sample of 20 pages re-annotated independently and compare per-page F1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result (abstract; §5.3, Table 2) depends on Eq. (1), which computes Jaccard overlap over sets of XPath strings. A DOM node has many XPath spellings (positional vs. attribute-predicate, indexed vs. unindexed siblings), and the pipeline validates only that a predicted path's text exists in the cleaned DOM (§4.3); it never canonicalizes predicted and ground-truth paths to a common node identity before intersecting them. Flat JSON (Fig. 1d) gives the model the exact absolute-positional XPath strings used as ground-truth labels, so the model can copy keys verbatim. In Slimmed HTML and Hierarchical JSON, the model must re-serialize the same paths from raw markup or nested fragments; any equivalent but differently spelled path is scored as a miss. The F1 gap (0.9567 vs. 0.4048) therefore conflates record grouping quality with XPath serialization difficulty, and the claim that Flat JSON produces 'superior extraction accuracy' is not yet established as a claim about extraction quality. The hallucination rate does not fix this: it only counts empty records, so predicted records containing valid but wrong XPaths are not penalized.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces NEXT-EVAL, a framework for evaluating web data record extraction methods on MHTML snapshots. It formalizes data records as sets of XPath expressions, constructs ground-truth labels through LLM-proposed and human-refined XPath annotations, and scores predictions with a Jaccard-based partial-credit metric. The framework compares traditional methods such as MDR with LLM-based extraction using three input representations: Slimmed HTML, Hierarchical JSON, and Flat JSON. The headline empirical claim, stated in the abstract and in Section 5.3, is that Flat JSON input enables Gemini-2.5-pro-preview to achieve an F1 score of 0.9567 with a hallucination rate of 0.0305, substantially outperforming Slimmed HTML (F1 0.1014) and Hierarchical JSON (F1 0.4048). The paper also contributes a dataset of 164 crawled web pages with 12,278 annotated records and a synthetic dataset.","tokens_in":1882,"tokens_out":1743,"duration_ms":62734,"significance":"If the central claim is established, the framework would be a useful step toward standardized benchmarking of web record extraction, addressing a real gap in the literature. The paper contributes a clean formal definition of records as XPath sets, a reproducible dataset-construction pipeline, a public synthetic dataset, and a structure-aware scoring procedure with partial credit. These are genuine strengths. However, the current evaluation does not yet support the claim that Flat JSON yields superior extraction quality: the scoring metric compares raw XPath strings, and Flat JSON supplies exactly those strings as input keys. The paper also defines hallucination in a way that ignores fabricated but non-empty XPath predictions. The framework is promising, but the headline comparison needs to be re-run under a scorer that is invariant to XPath serialization and that penalizes invalid paths.","major_comments":[{"comment":"The central F1 comparison is computed over raw XPath strings without canonicalizing predicted and ground-truth paths to a common DOM node identity. Eq. (1) defines overlap as the Jaccard similarity of XPath string sets. A DOM node admits many equivalent XPath spellings, and different input formats give the model very different access to those spellings. Flat JSON (Fig. 1d) presents the exact absolute XPath strings used as ground-truth labels as input keys, so the model can copy them verbatim; Slimmed HTML and Hierarchical JSON force the model to regenerate the same strings from markup or nested fragments, so any equivalent but differently spelled path is scored as a miss. The F1 gap of 0.9567 vs. 0.4048 therefore conflates record grouping quality with XPath-serialization difficulty. The claim that Flat JSON produces 'superior extraction accuracy' is not yet established as a claim about extraction quality. Please re-score all predictions by resolving predicted and ground-truth XPaths to DOM node IDs before computing the Jaccard overlap, and report the resulting F1 values.","section":"Section 4.4, Eq. (1); Fig. 1(d); Table 2"},{"comment":"The hallucination metric as defined only flags pages where the model predicts at least one empty record (a record containing no XPaths). It does not penalize non-empty records whose XPaths do not correspond to real DOM nodes. The paper asserts in Section 4.3 that 'every model output paths are drawn from the cleaned DOM' and that 'the model cannot hallucinate elements that do not exist,' but no procedure is described for validating each predicted path against the DOM or for counting invalid paths as hallucinations. Depending on the implementation, such fabricated paths are either silently discarded (which can inflate precision) or counted as negatively matching (which would reduce precision), but in neither case are they reflected in the reported Hallucination Rate. Please define a hallucination metric that counts predicted XPaths that do not resolve to a node in the cleaned DOM, and report that rate separately for each input format.","section":"Section 4.4, Hallucination Rate; Section 4.3"},{"comment":"Table 2 reports single precision, recall, F1, and hallucination-rate numbers for each method with no error bars, confidence intervals, or significance tests. This contradicts the checklist claim that 'we use multiple seeds for the zero-shot LLM and average the results to ensure statistical reliability.' Since Gemini-2.5-pro-preview is evaluated at temperature 1.0, run-to-run variance is expected, and the lack of any variance information makes it impossible to assess whether the large gaps in Table 2 are stable. Please report the number of runs, the mean and standard deviation (or equivalent) for each metric, and, where appropriate, a paired significance test across input formats.","section":"Section 5.3, Table 2; NeurIPS Checklist item 7"},{"comment":"Ground-truth records are constructed by having LLMs propose candidate repetitive blocks and human annotators 'review and refine' them, but the paper reports no inter-annotator reliability, no label audit, and no details about the number of annotators or the refinement protocol. Every precision, recall, and F1 value in Table 2 is computed against these labels, so label errors could shift the reported ranking, including the Flat JSON advantage. Please report the annotation protocol in detail, provide agreement statistics on a held-out subset of pages (e.g., pairwise overlap between independent annotators), and describe how ambiguous or nested record boundaries were resolved.","section":"Section 4.2"}],"minor_comments":[{"comment":"The abstract says the scoring 'specifically prevents text hallucination and allowing only for the assessment of positional hallucination,' but the Hallucination Rate defined in Section 4.4 only counts empty records and says nothing about invalid XPaths; this wording should be aligned with the actual metric.","section":"Abstract and Section 4.4"},{"comment":"The row for MDR lists 'Full / Slimmed HTML' as the input type; it is unclear whether the reported numbers come from Full HTML, Slimmed HTML, both, or a union, and this should be specified explicitly.","section":"Table 2"},{"comment":"The paper and checklist state that crawling code and dataset construction instructions are open access, but no URL or repository is given in the main text; please provide concrete links in the final version.","section":"Section 5, Checklist items 4 and 5"},{"comment":"The paper should state the exact XPath serialization convention used for ground-truth labels (e.g., positional indexes as in Fig. 1d versus attribute predicates), since the scoring and the Flat JSON input both depend on this convention.","section":"Section 4.2 and Fig. 1"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is valid and load-bearing: the headline Flat JSON result appears to be partly an artifact of the scoring metric rewarding verbatim copying of the label strings. I would ask the authors to re-run the evaluation with node-identity-based matching and to report a hallucination metric that penalizes invalid XPaths before the paper can be accepted. The framework and dataset are potentially useful, so major revision, not rejection, seems appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a useful evaluation artefact with a compromised headline result. The framework — generating datasets from MHTML snapshots, annotating with XPaths, scoring with structure-aware overlap — is genuinely reusable, and the comparison of Slimmed HTML, Hierarchical JSON, and Flat JSON as LLM inputs is new. The synthetic dataset (164 pages, 12k+ records) is a real contribution, assuming they actually release it.\n\nWhat the paper does well: it formalizes data records as sets of XPath expressions, prevents text hallucination by requiring predicted paths to have text present in the cleaned DOM, and uses Hungarian matching to give partial credit. That is a sensible baseline for the community.\n\nThe soft spots are serious, though. The F1 gap between Flat JSON (0.9567) and Hierarchical JSON (0.4048) is largely a scoring artifact. The metric compares sets of XPath strings, not DOM nodes. A DOM node has many valid XPath spellings (positional vs. attribute-based, indexed vs. unindexed siblings). Flat JSON supplies the exact absolute-positional XPath strings used as ground truth as its keys, so the model copies them verbatim. Slimmed HTML and Hierarchical JSON force the model to re-serialize those paths from markup or nested objects; any equivalent but differently spelled path is scored as a miss. So the reported gap conflates record grouping quality with XPath serialization difficulty. The circularity concern is real.\n\nThe hallucination metric also understates hallucination: it only flags empty records, so a predicted record with a valid-looking but wrong XPath (whose text exists somewhere on the page) is not penalized. The paper says it allows only assessment of positional hallucination, but the metric doesn't actually measure that — it measures whether a record is empty.\n\nThe paper claims multiple seeds and averaging in the checklist, but no error bars or per-seed numbers appear anywhere. Without them, a 0.9567 vs 0.4048 gap is hard to trust, especially with a temperature-1.0 API. Also, ground-truth labels were human-refined LLM suggestions, but no inter-annotator reliability is reported; if labels are incomplete or wrong, every number shifts.\n\nWho gets value: anyone building or comparing web extraction systems will want the framework and dataset, once released. But the headline \"Flat JSON is best\" should not be cited without re-running the evaluation with canonicalized node identity and a proper hallucination metric.\n\nRecommendation: send it to peer review. The framework deserves referee time. The evaluation protocol needs real revision before acceptance: canonicalize predicted and ground-truth XPaths to DOM node identity, fix the hallucination metric, and report variance. With those changes, the claim about Flat JSON might survive — or might not. Right now, it is not established.","headline":"Useful evaluation framework, but the Flat JSON superiority claim is largely a scoring artifact — a reworked metric is needed before the numbers can be trusted.","tokens_in":14037,"tokens_out":3191,"would_cite":false,"duration_ms":23454,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Feeding an LLM a flat map of XPath addresses to text extracts web data records with an F1 score of 0.9567, far above HTML-based input formats.","keywords":["web data record extraction","LLM evaluation","XPath annotations","Flat JSON","input representation","hallucination rate","benchmark framework","DOM structure"],"falsifier":"Take a random sample of the 164 pages and have independent annotators label data records from scratch in XPath form, then measure agreement with the paper's ground truth; if agreement is low, the reported precision, recall, and F1 numbers are not stable. Separately, count hallucinated fields inside non-empty records, not just empty records; if that count is clearly above zero for Flat JSON, the claim of minimal hallucination needs qualification.","tokens_in":13067,"feed_emoji":"🗂️","tokens_out":4032,"duration_ms":35576,"temperature":0.7,"pith_summary":"This paper tries to establish a reproducible way to compare traditional and LLM-based web data record extraction, and to show that input representation decides how well an LLM can segment a page into repeated records. The central empirical claim is that a Flat JSON format, where every absolute XPath is a key and the text content is the value, lets an off-the-shelf LLM reach an F1 score of 0.9567 with a hallucination rate of only 0.0305, while the same model on Slimmed HTML reaches only 0.1014 F1. The paper also contributes an evaluation framework that builds datasets from arbitrary MHTML snapshots, uses human-refined XPath annotations as ground truth, and scores predictions with structure-aware metrics that give partial credit. If correct, this means that simple, unambiguous localization of text fields is more important for LLM extraction than preserving the full DOM hierarchy.","feed_headline":"Flat JSON lifts LLM web extraction to F1 0.9567","feed_subtitle":"New benchmark shows input format decides success: simple XPath-to-text maps cut hallucination to about 3 percent.","key_machinery":"The load-bearing mechanism is the Flat JSON text map: a serialization that pairs every non-empty text node's absolute XPath with its textual content, for example \"/html/body/ul/li[1]/span\": \"Sample Product\". This representation removes the DOM hierarchy while preserving exact positional identity, so the LLM is effectively asked to select existing XPath keys rather than reconstruct structure from markup. The scoring framework then compares predicted record sets to ground-truth XPath sets using Jaccard overlap $Overlap(P_i, G_j) = |P_i \\cap G_j| / |P_i \\cup G_j|$, an optimal one-to-one matching via the Hungarian algorithm, and the resulting precision, recall, and F1. The hallucination rate is a per-URL binary flag for whether any predicted record is empty, which the paper uses to distinguish text hallucination (prevented by the XPath constraint) from positional hallucination.","core_discovery":"The paper claims that the input format is the dominant factor in LLM-based web data record extraction, and that a flat key-value serialization of the DOM outperforms both a hierarchy-preserving JSON and slimmed HTML. Concretely, the LLM with Flat JSON input achieves precision 0.9939, recall 0.9392, and F1 0.9567, whereas Hierarchical JSON gives F1 0.4048 and Slimmed HTML gives F1 0.1014; the traditional MDR baseline sits at F1 0.0830. The authors attribute the Flat JSON advantage to unambiguous field localization: each key is an absolute XPath, so the model must output paths that exist in the cleaned DOM, which suppresses hallucinated elements. The paper further claims that a hallucination rate defined as the fraction of URLs where the model outputs at least one empty record drops to 0.0305 with Flat JSON, versus 0.9146 with Slimmed HTML and 0.5976 with Hierarchical JSON.","pith_inferences":["The reported advantages could shift if the ground-truth labels are noisy: the annotations are LLM-proposed and human-refined, but no inter-annotator reliability is reported, so an independent re-labeling of a sample of pages would test whether the Flat JSON edge is real or an artifact of label noise.","The hallucination rate only flags records that are entirely empty, so a model that invents extra fields inside an otherwise valid record would not be penalized; the claimed 3 percent may understate positional hallucination.","The Flat JSON effect may partly reflect the model's pretraining on JSON key-value structures rather than genuine DOM reasoning; testing open-weight LLMs that have seen less structured serialization would separate these explanations.","The paper does not break down results by record layout type, so it is unclear whether Flat JSON's advantage persists for nested or non-contiguous records, where the lost hierarchy might be needed for correct grouping."],"forward_implications":["Flat JSON should become the default input serialization for LLM-based web record extraction when accuracy is the priority, since it outperforms hierarchy-preserving and HTML formats by a wide margin.","The framework lets researchers build evaluation datasets from any MHTML snapshot without redistributing crawled pages, which sidesteps licensing and robots.txt restrictions that limit previous benchmarks.","XPath-based supervision makes evaluation deterministic and DOM-grounded, so model outputs can be verified as real elements rather than plausible-sounding text.","There is a token-cost tradeoff: Flat JSON has the lowest error but the highest average token count (116,698 tokens) versus Hierarchical JSON (34,107) and Slimmed HTML (86,084), so cost-sensitive deployments may prefer a cheaper format despite worse accuracy.","The synthetic dataset created by transforming DOM structures and modifying content enables public benchmarking even for pages whose original text cannot be redistributed."],"supporting_citations":[{"why":"Supplies the MDR algorithm, the traditional DOM-structure baseline that the paper compares against.","marker":"[Liu et al., 2003]"},{"why":"Identifies the Gemini-2.5-pro-preview model used for all LLM-based extraction experiments.","marker":"[Team et al., 2023]"},{"why":"Provides PLAtE, the prior large-scale list-page extraction dataset that this paper contrasts with its more diverse benchmark.","marker":"[San et al., 2023]"},{"why":"Supplies AMBER, a rule-based extraction system with annotations that lacks a public dataset, motivating the new framework.","marker":"[Furche et al., 2012]"},{"why":"Provides the Klarna product page dataset, an e-commerce-only benchmark with redistribution restrictions that this work aims to move beyond.","marker":"[Hotti et al., 2021]"},{"why":"Represents a recent multi-record extraction benchmark focused on news pages, which the paper cites as domain-specific and not reusable.","marker":"[Kustenkov et al., 2025]"},{"why":"Defines DEPTA, a partial-tree-alignment method that the paper excludes from evaluation because it requires visual layout cues.","marker":"[Zhai and Liu, 2005]"}],"fun_headline_variants":["Flat JSON input gives LLMs best web extraction F1: 0.9567","New benchmark: Flat JSON tops LLM web extraction at F1 0.9567","LLM web extraction: Flat JSON input hits F1 0.9567","Flat JSON outperforms nested JSON: LLM F1 0.9567","Flat JSON cuts LLM hallucination to 3% in web extraction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ground-truth XPath annotations are complete and correct for every page, even though they were produced by an LLM and refined by humans with no reported check for inter-annotator agreement or label quality.","fun_headline_variants_meta":{"raw":{"variants":["Flat JSON input gives LLMs best web extraction F1: 0.9567","New benchmark: Flat JSON tops LLM web extraction at F1 0.9567","LLM web extraction: Flat JSON input hits F1 0.9567","Flat JSON outperforms nested JSON: LLM F1 0.9567","Flat JSON cuts LLM hallucination to 3% in web extraction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001506,"raw_usage":{"total_tokens":6057,"prompt_tokens":983,"completion_tokens":5074,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":4967}},"tokens_in":599,"tokens_out":5074,"duration_ms":29464,"temperature":1.0,"reasoning_tokens":4967,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:07:31.221410+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 164 pages and have independent annotators label data records from scratch in XPath form, then measure agreement with the paper's ground truth; if agreement is low, the reported precision, recall, and F1 numbers are not stable. Separately, count hallucinated fields inside non-empty records, not just empty records; if that count is clearly above zero for Flat JSON, the claim of minimal hallucination needs qualification.","supporting_citations":[],"review_version":1}