{"id":"5ac3df1d-9241-47c6-bed0-7d21796acb27","arxiv_id":"2507.21340","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A synthetic table-to-text pipeline that generates and validates key-value extraction benchmarks, revealing that LLM-generated reports keep numerical facts intact but are poorly machine-extractable.","lead":"StructText automatically creates key-value extraction benchmarks by having an LLM turn rows of existing tables into natural-language reports, then scoring the reports on factuality, coherence, and numeric/temporal fidelity. The released 71k-example benchmark shows that LLMs preserve numbers and dates well but produce narratives that automated extractors find hard to parse, so the work will interest anyone building or evaluating text-to-table systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Low extraction F1 may be an artifact of the baseline/evaluation protocol, not evidence that the generated narratives resist extraction.","rationale":"The reader's verdict is CONDITIONAL, and I agree with that disposition, but the load-bearing concern is not, in my judgment, the unvalidated LLM-as-judge. The paper's headline finding—accurate numbers and dates coexist with poor extractability—is supported by the objective numeric/temporal metrics and the baseline extraction scores, not by the 4-5 rubric scores. Even if the Llama judge were perfectly calibrated, the central claim would collapse if the baseline's low F1 is an artifact of its string-based column matching or its schema-prediction design. Conversely, if the numeric/temporal metrics are accepted and extraction is evaluated with a competent extractor, the LLM-judge scores are secondary. I therefore focus on the extraction baseline. The paper never demonstrates that a strong extractor would struggle; it only demonstrates that one particular pipeline, with Levenshtein-based column matching and an unstated policy for nulls, does. A control experiment with explicit templated reports is a minimal, cheap check that would separate text difficulty from evaluator brittleness. I also note the paper deserves credit for releasing code and data and for reporting objective numeric/temporal metrics; the issue is interpretability, not data fabrication. The recommended disposition remains CONDITIONAL: the framework is useful and scalable, but the central empirical claim needs the control and stronger-extractor evidence before it can be fully accepted.","tokens_in":14455,"tokens_out":5720,"duration_ms":69112,"concrete_test":"Run the released StructText baseline and evaluator on a control corpus generated from the same tables with templated reports that state each column value explicitly, e.g., 'revenue is 265595000000; period is 2018-09-29; ...'. If column F1 or value P/R stay near the reported 0.45/0.26 levels, the low scores are an evaluation artifact; if they rise toward 1.0, the original text genuinely impedes extraction. As a complementary check, rerun extraction on the original reports while supplying the true column schema (skipping the schema-prediction step) and using BERTScore-based column matching; near-ceiling value F1 would undermine the claim that values are embedded in a way that resists automated extraction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that StructText produces text that faithfully preserves the source row yet resists LLM extraction—rests on two empirical pillars: numeric/temporal P/R > 0.9 and baseline column F1 ≈ 0.41–0.46 with value P/R < 0.26/0.14. The first pillar does not establish that values are attached to the correct columns: Section 3.2 explicitly validates only whether normalized numbers and dates 'occur' in the text, not whether they are semantically assigned to the right fields. The second pillar is produced by a single in-house pipeline whose column-identification step predicts a schema from sampled reports and then maps predictions to ground truth using normalized Levenshtein distance (Sections 4.2, 6.3). Natural paraphrases such as 'cash and cash equivalents' versus 'cash_equivalents', or 'fiscal year 2024' versus a 2024-12-31 cell, are heavily penalized even when the extraction is semantically correct. Null-valued columns are acknowledged in Section 4.1, but no handling is described, so omitted nulls may be counted as false negatives. No stronger extractor, no human extraction study, and no control condition (e.g., templated text with explicit column-value statements) is reported. The observed gap could therefore measure brittleness of the baseline and the Levenshtein matcher rather than 'information accessibility' of the generated narratives; this directly threatens the paper's main finding.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes StructText, a two-stage pipeline that converts existing tabular data into natural-language reports (a 'plan-then-execute' generation using Qwen2.5-72B-Instruct), evaluates the generated text with LLM-based judges (Llama-3.3-70B-Instruct) on factuality, hallucination, and coherence, validates numeric and temporal information with parsers, optionally filters low-quality reports, and provides a baseline text-to-table extractor. Experiments on SEC filings and a WikiDB subset report high numeric/temporal fidelity (precision/recall mostly above 0.9) but low baseline extraction performance (column F1 around 0.41-0.46, value precision/recall below 0.26/0.14), which the authors interpret as a gap between generation fidelity and information accessibility. The framework and artifacts are released for community use.","tokens_in":14723,"tokens_out":4458,"duration_ms":50899,"significance":"If the reported pattern is robust, StructText would be a useful contribution: it addresses a real need for scalable, multi-domain benchmarks for key-value/text-to-table extraction, and the plan-then-execute design plus the release of code and data are concrete strengths. It is also commendable that the generation model and the evaluation judge are distinct (Qwen2.5-72B for generation, Llama-3.3-70B for evaluation), avoiding direct self-scoring. However, the paper's central empirical claim is not yet fully established: the two pillars supporting it—occurrence-based numeric/temporal validation and a single Levenshtein-matched extraction baseline—do not separately measure information preservation and information accessibility as cleanly as the text suggests. The missing validation of the LLM judge and the absence of error bars or significance tests further weaken the quantitative conclusions.","major_comments":[{"comment":"The numeric/temporal validation explicitly checks only whether normalized values occur in the generated text, not whether they are attributed to the correct columns ('we do not specifically check whether the numeric values are expressing the semantics in the text accurately, but only check whether the numeric values occur'). Consequently, precision/recall above 0.9 does not establish that values are attached to the right fields: a report that moves revenue figures to the net-income position could pass the occurrence check. Since the headline claim is that the generated text preserves source information with high fidelity, the evaluation must also verify value-to-attribute binding, for example by judging extracted key-value pairs against the ground-truth row or by a human annotation sample. Without this, Table 3 overstates what the validation actually measures.","section":"Section 3.2, Table 3"},{"comment":"The low extraction F1 that motivates the 'information accessibility' finding is produced by a single in-house baseline whose schema identification and column mapping depend on normalized Levenshtein distance. Natural paraphrases such as 'cash and cash equivalents' versus 'cash_equivalents', or 'fiscal year 2024' versus a 2024-12-31 cell, are penalized even when the extraction is semantically correct. Section 4.1 acknowledges null-valued columns but no handling for omitted nulls is described, so nulls may be counted as false negatives. The reported gap could therefore measure brittleness of the baseline and the matcher rather than the difficulty of extracting information from the narratives. Please add at least one stronger extractor, a templated-text control condition, or a human extraction study on a sample; without such a control, the main finding is not separable from evaluation artifacts.","section":"Sections 4.2 and 6.3, Table 4"},{"comment":"The LLM-as-judge scores for factuality, hallucination, and coherence are used as evidence for the qualitative findings (strong factual accuracy, near-zero hallucination, weaker coherence), but no human agreement study, no correlation with existing judge benchmarks, and no error analysis are reported. The rubrics and self-consistency checks are reasonable safeguards, yet they do not establish that the judge's scores correspond to valid human judgments. Please report inter-annotator agreement (human versus LLM) on a sample, or at least provide judge rationales with failure cases, so readers can assess whether the near-ceiling factuality/hallucination numbers and the coherence gap are properties of the generated text or artifacts of the judge.","section":"Sections 3.2 and 5.2"},{"comment":"The quality-filtering step and the numeric/temporal conclusions are presented without any variance or significance information. The paper reports macro averages over datasets, but no error bars, confidence intervals, or significance tests accompany the key comparisons (e.g., SEC versus WikiDB differences in Table 4). Several conclusions, such as 'temporal precision emerged as the weakest component', depend on small numeric differences (0.818 vs 0.849 precision in Table 3), and it is unclear whether these differences are stable. Please report per-dataset variability or otherwise quantify the uncertainty of the headline metrics.","section":"Sections 3.3, 5.3, 6.4"}],"minor_comments":[{"comment":"The abstract reports 71,539 examples across 49 datasets, while Section 6 says the evaluation was on 50 datasets; Section 7.2 then refers to 'a representative subset of 50 tables' from WikiDBs, although Section 5.1 states 1,000 tables. Please reconcile these numbers.","section":"Abstract, Section 6, Section 7.2"},{"comment":"The value-extraction F1 cells are left as '–' even though the text says the table shows precision, recall, and F1. Please either report the F1 values or explicitly explain why they are omitted.","section":"Table 4"},{"comment":"The sentence 'Temporal precision emerged as the weakest component (see Table 4)' appears to reference the wrong table; temporal precision is reported in Table 3, not Table 4.","section":"Section 6.4"},{"comment":"References [35] and [36] both cite the same MT-Bench paper with overlapping author lists; please deduplicate or clearly differentiate the two entries.","section":"References [35] and [36]"},{"comment":"The example in Figure 5 is presented as a parsing challenge, but it also illustrates a limitation of the validation metric: 'fiscal year 2024' is valid temporal information that is absent from the ground-truth columns, so the false positive is an annotation mismatch rather than an error in the generated text. This should be acknowledged when interpreting temporal precision scores.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable fit for a tabular-data-analysis workshop, and the released artifacts are a genuine asset. The main risk is overclaiming: the headline 'faithful but hard to extract' finding depends on two evaluation components that each have a plausible artifact (occurrence-based validation and Levenshtein-matched extraction). I would not reject the paper, but the authors should be asked to strengthen the evaluation with attribution-aware numeric/temporal validation, a second extractor or control condition, and at least a small human study or judge-validation. The dataset-count and missing-F1 inconsistencies should also be fixed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look: this is a solid, well-scoped benchmark-generation paper that ships a real stack—code, data, evaluation tools—and reports an interesting asymmetry. The new thing is the end-to-end workflow: an LLM plans column groupings from sample rows, then a grounded generator writes reports; a multi-dimensional evaluation combines LLM-as-judge rubrics with objective numeric/temporal checks and a baseline text-to-table extractor. On ~71k examples, numeric/temporal precision and recall sit above 0.9 while the baseline column F1 is 0.41–0.46 and value-level precision/recall are below 0.26/0.14. That asymmetry, if it holds up, is a genuinely useful observation for anyone building enterprise extraction pipelines.\n\nGive credit: the method is described in enough detail to reproduce, the artifacts are public, and the paper is honest about several limitations (e.g., it notes that the numeric/temporal check only verifies whether values occur, not whether they are semantically attributed to the right columns). That admission is important.\n\nThe soft spots are real but mostly fixable. The LLM judge is unvalidated against humans—no agreement study or correlation with existing judge benchmarks—so the near-ceiling factuality/hallucination numbers and the coherence gap could partly reflect rubric ambiguity. More importantly, the extraction baseline uses normalized Levenshtein and a predicted schema; natural paraphrases like \"cash and cash equivalents\" vs. cash_equivalents, or \"fiscal year 2024\" vs. a 2024-12-31 cell, get penalized even when the extraction is semantically right. Nulls are mentioned but no handling is described. There is no stronger extractor, no human extraction study, and no control condition such as templated text. So the headline claim—faithful text that resists extraction—is supported mainly by a single brittle baseline and an occurrence-only fidelity metric. I don't think the claim is false; I do think the evidence is thinner than the abstract suggests.\n\nMinor: abstract says 49 datasets, Section 6 says 50; Table 4 leaves value-extraction F1 blank; no error bars anywhere.\n\nWho this is for: anyone working on text-to-table extraction, synthetic benchmark generation, or LLM-as-judge methodology. It deserves a serious referee; the pipeline and dataset are community assets even if the central claim needs stronger validation. My recommendation: send to peer review, with a request for judge validation, embedding-based column matching, and a stronger extractor as conditions for acceptance.","headline":"A useful, reproducible benchmark-generation stack with a plausible but not fully proven main claim about faithful text resisting extraction.","tokens_in":15272,"tokens_out":3193,"would_cite":true,"duration_ms":33826,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"StructText shows LLMs can generate accurate reports that automated extractors struggle to read back into tables.","keywords":["synthetic benchmark generation","table-to-text","text-to-table extraction","key-value extraction","LLM-as-judge","factuality evaluation","numeric accuracy","plan-then-execute"],"falsifier":"Swap the baseline extractor for a stronger open-weights LLM on the same released reports and recompute value and column F1; if F1 rises above about 0.8 while numeric fidelity remains above 0.9, the paper's central 'accurate but not extractable' gap is an artifact of the baseline rather than a property of the text.","tokens_in":14250,"feed_emoji":"📊","tokens_out":6145,"duration_ms":66096,"temperature":0.7,"pith_summary":"StructText is an automated pipeline for turning existing tabular data into natural-language reports and then benchmarking systems that extract the table back out. The paper's central claim is that this pipeline produces reports that preserve the source row's numeric and temporal facts with precision and recall above 0.9, yet are hard for an LLM-based extractor to parse: column-identification F1 lands at 0.455 on SEC filings and 0.413 on WikiDB tables, with value-extraction precision/recall below 0.26/0.14. If true, the framework makes benchmark construction scalable to any domain and exposes a gap between generation fidelity and information accessibility. The authors use the generated corpus of 71,539 examples across 49 datasets to argue that current LLMs write factually strong, hallucination-light reports whose narrative organization hides the semantic relations needed for automated extraction.","feed_headline":"LLMs write accurate reports that resist automated extraction","feed_subtitle":"A table-to-text benchmark generator shows high numeric fidelity yet extraction F1 under 0.46.","key_machinery":"The load-bearing mechanism is the plan-then-execute generation loop plus a multi-dimensional validation stack. In planning, the model inspects ten sample rows and proposes one to five report types, each naming a coherent group of columns; in execution, it generates one report per row from exactly those columns, with the row's cell values as the only grounding. The evaluation stack combines LLM-as-judge 5-point rubrics for factuality, hallucination, and coherence, with a sentence-level claims self-consistency check, and objective parser-based checks for numeric and temporal fidelity using normalized values with a 0.1% relative error tolerance. A baseline extractor then recovers schema and values from the reports, and predictions are matched to ground-truth columns with a bipartite assignment maximizing similarity before precision, recall, and F1 are computed. These pieces together make the claimed fidelity-versus-extractability gap measurable.","core_discovery":"The core discovery, stated on the paper's own terms, is that the two-stage plan-then-execute generator, where an LLM first chooses column groupings and then writes grounded prose from only those columns, yields text that scores 4.58/4.56 on factuality and 4.90/4.55 on hallucination avoidance on a 5-point scale, with numeric and temporal precision and recall above 0.9, while the same text defeats the paper's baseline text-to-table extractor. The gap is quantified by extraction results: column F1 of 0.455 on SEC and 0.413 on WikiDB, and value-extraction precision/recall of 0.257/0.110 and 0.179/0.137. The authors read this as evidence that LLMs can generate accurate text but embed numeric and temporal values in narratives whose attributions to source columns are not recoverable by current extraction approaches. They position StructText as a reusable benchmark generator and release the datasets, judge prompts, filtering tools, and baseline extractor.","pith_inferences":["If the benchmark is used as a training or selection signal, optimizing for extractability may pull generators toward more templated, explicitly attributed sentences, which could in turn lower the very coherence scores the paper reports as weak.","The 0.1% numeric tolerance and the parser's set-based comparison ignore semantic roles; a value that appears in the text but attached to the wrong column would still count as correct, so the near-ceiling numeric precision may be an upper bound on true fidelity.","The filtering sweep on temporal precision removes the hardest reports; the retained corpus may be easier for future extractors, so downstream results on the released filtered data must be compared against the same filtering settings.","Plan-then-execute generation suggests a natural control experiment: if the planning step is bypassed and columns are passed in a fixed order, extraction F1 should change measurably if narrative structure is what hurts accessibility."],"forward_implications":["Any tabular dataset can be turned into a domain-specific text-to-table benchmark without manual annotation, so benchmark size and coverage are no longer limited by human effort.","Generation quality and extraction difficulty are separable axes: numeric and temporal fidelity approaching 0.9 does not imply that a reader can recover the structured record.","The published baseline extraction numbers give later systems a concrete reference point; beating column F1 of 0.455/0.413 and value precision below 0.26 means closing the accessibility gap.","Organizations evaluating LLM pipelines for report generation should measure downstream extractability in addition to factual accuracy, because the two can diverge sharply."],"supporting_citations":[{"why":"Supplies the asymmetry premise that generating from structured data is easier than extracting, which StructText exploits.","marker":"[13]"},{"why":"Provides the WikiDB dataset used as one of the two evaluation domains.","marker":"[30]"},{"why":"A prior single-domain table-to-text benchmark that StructText positions itself against.","marker":"[24]"},{"why":"Establishes the LLM-as-judge paradigm that the evaluation rubrics build on.","marker":"[36]"},{"why":"Supports the rubric-based LLM scoring approach and its alignment with human judgments.","marker":"[20]"},{"why":"Provides the open-vocabulary key-value extraction task and the similarity-based matching idea.","marker":"[22]"},{"why":"The Llama-3.3-70B model used for all LLM-as-judge scoring.","marker":"[7]"},{"why":"The Qwen2.5-72B model used for report planning and generation.","marker":"[29]"},{"why":"Stanford CoreNLP and SUTime power the numeric and temporal validation parsing.","marker":"[21]"},{"why":"Used to solve the bipartite matching that maps predicted columns to ground truth before scoring.","marker":"[26]"}],"fun_headline_variants":["LLMs write accurate text that defeats extractors","StructText: accurate LLM prose resists extraction","High-fidelity LLM narratives, low extraction scores","Benchmark: LLMs generate text that hides its numbers","When LLM accuracy undermines automated extraction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that Llama-3.3-70B-Instruct, when given the authors' 5-point rubrics, produces valid measures of factuality, hallucination, and coherence; if those judge scores do not track human judgment, the headline finding that LLMs generate accurate but poorly organized reports loses its evidentiary base.","fun_headline_variants_meta":{"raw":{"variants":["LLMs write accurate text that defeats extractors","StructText: accurate LLM prose resists extraction","High-fidelity LLM narratives, low extraction scores","Benchmark: LLMs generate text that hides its numbers","When LLM accuracy undermines automated extraction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000156,"raw_usage":{"total_tokens":1256,"prompt_tokens":1024,"completion_tokens":232,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":158}},"tokens_in":640,"tokens_out":232,"duration_ms":3469,"temperature":1.0,"reasoning_tokens":158,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:52:36.011705+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Swap the baseline extractor for a stronger open-weights LLM on the same released reports and recompute value and column F1; if F1 rises above about 0.8 while numeric fidelity remains above 0.9, the paper's central 'accurate but not extractable' gap is an artifact of the baseline rather than a property of the text.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the WikiDB dataset used as one of the two evaluation domains."},{"cited_title":"Xing, Hao Zhang, Joseph E","cited_arxiv_id":null,"evidence_quote":"Establishes the LLM-as-judge paradigm that the evaluation rubrics build on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the rubric-based LLM scoring approach and its alignment with human judgments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the open-vocabulary key-value extraction task and the similarity-based matching idea."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Stanford CoreNLP and SUTime power the numeric and temporal validation parsing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Used to solve the bipartite matching that maps predicted columns to ground truth before scoring."}],"review_version":1}