{"id":"33e22a21-5ec7-4908-bf56-473bd115a512","arxiv_id":"2411.19689","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LLMs rank similarly on human and synthetic data for extracting insights, but synthetic data does not predict how well models map insights back to source documents.","lead":"This paper introduces a new evaluation task called MIMDE, where language models extract and trace insights across documents, and tests 20 models on human and synthetic survey data. It finds that model rankings correlate strongly on insight extraction but not on mapping insights back to documents, so synthetic data only partially proxies human data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unverified synthetic ground-truth labels could explain the null document-level correlation; a label-verification pass is needed before the claim about document-level mapping is accepted.","rationale":"The reader's weakest assumption exactly matches my own: synthetic ground truth is derived from prompt injection without verification, and the document-level null result is computed against those unverified labels. This is the most load-bearing concern because the paper's headline contribution is the asymmetry between insight-level and document-level correlations; if the document-level null is an artifact of label noise, the paper's practical guidance about when synthetic data can be trusted loses its empirical basis. The insight-level correlation could also be affected, but the document-level null is the more surprising and more consequential claim. The authors deserve credit for explicitly acknowledging this limitation and for using a rigorous human annotation pipeline with majority voting on the human dataset; that makes the synthetic label problem the clear weak point rather than a general sloppiness issue. I considered whether the self-referential GPT-4 judge is a stronger concern, but the similarity-metric study in Section 4.1 does evaluate GPT-4 against human judgments and achieves F1=0.809, so the judge is at least calibrated on human data, whereas the synthetic labels have no such validation. A concrete verification pass—human labeling of a sample of synthetic responses—would settle whether the null result survives contact with accurate labels. If it does, the paper's conclusion is robust; if not, the central claim needs substantial revision. The reader's CONDITIONAL verdict is appropriate; I do not see grounds to shift it to accept or reject without that check.","tokens_in":13708,"tokens_out":2303,"duration_ms":21845,"concrete_test":"Take a random sample of the synthetic dataset (e.g., 100 responses per question) and have human annotators, blind to the generation prompts, label each response with the same insight list used during generation. Compute agreement between the injected labels and the human-verified labels. Then recompute the document-level Pearson correlation between human and synthetic performance using only the subset of responses where injected and human labels fully agree, or re-label the entire sample with human-corrected ground truth and recompute. If the correlation remains near zero, the null result stands; if it becomes substantially positive, the central negative finding is an artifact of unverified synthetic labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central negative result—no correlation between synthetic and human data at the document level (Table 4: r=0.13, p>0.1)—rests on the assumption that the synthetic responses are correctly labeled with the insights injected into the generation prompt. Section 3.2.2 explains that labels are assigned by prompting LLMs with 1–3 insights, but Section 5 concedes that the models may not have incorporated all prompted insights or may have expressed them in ways that are difficult to detect. If many injected insights are absent or paraphrased beyond recognition, the synthetic ground truth is noisy or wrong. This would simultaneously depress synthetic document-mapping scores (as observed in Table 3, where all models score lower on synthetic than on human data) and attenuate the cross-dataset correlation, mechanically producing the null result the paper interprets as a genuine limitation of synthetic data. Because the same unverified labels feed the insight-level correlation (0.71) and the document-level null, the headline conclusion is not secure until label fidelity is measured. The authors openly acknowledge this limitation, which is honest, but they do not resolve it; the claim that synthetic data 'fails to capture the complexity of document-level analysis' is exactly the claim most exposed to this artifact.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Multi-Insight Multi-Document Extraction (MIMDE), a two-step task of extracting an optimal set of insights from a document corpus and mapping them back to source documents. The authors construct paired human and synthetic datasets for five survey questions, develop an evaluation framework with three similarity-based metrics, benchmark 20 LLMs on both datasets, and analyze cross-dataset correlations of model performance. The main findings are a strong correlation (0.71) between LLM performance on synthetic and human data at the insight extraction level, but no significant correlation (0.13) at the document-level mapping level. The paper explicitly acknowledges that synthetic ground-truth labels are derived from generation prompts and may not be reliably present in the generated responses, and calls for a future verification phase.","tokens_in":14044,"tokens_out":5190,"duration_ms":44358,"significance":"The paper addresses a timely question—whether synthetic data can serve as a valid proxy for human-annotated data in LLM evaluation—and provides a new task formulation, two new datasets, and a systematic comparison of 17 similarity metrics, finding that LLM-based similarity (GPT-4) aligns best with human judgment. If the findings hold, the strong insight-level correlation would support the use of synthetic data for cheaper evaluation of extraction performance, while the document-level null would caution against its use for mapping evaluation. The authors are transparent about the main limitation (unverified synthetic labels), which is commendable. However, the central null result at the document level is directly exposed to label noise, and the evaluator choice and lack of uncertainty quantification further weaken the conclusiveness of the claims. The paper's strengths are its clear task definition, the breadth of models benchmarked, and the honest limitation statement.","major_comments":[{"comment":"The synthetic ground-truth labels are assigned by prompting LLMs to incorporate 1–3 injected insights, but the authors concede in Section 5 that 'the models may not have incorporated all prompted insights into their responses, or may have expressed them in ways that made them difficult to detect.' This is a load-bearing issue because the document-level null correlation (Table 4, r=0.13) is computed against these unverified labels. If a substantial fraction of injected insights are absent or paraphrased beyond recognition, the synthetic labels are noisy, which would depress the observed document-level scores and attenuate the cross-dataset correlation, mechanically producing the null result interpreted as a limitation of synthetic data. Please add a verification phase—e.g., manual inspection of a representative sample of synthetic responses to quantify label fidelity—and re-analyze the correlations using only responses whose labels are verified.","section":"§3.2.2, §5"},{"comment":"GPT-4 is used as the automatic evaluator for insight matching throughout the benchmark, while GPT-4 is also one of the models that generated the synthetic data and one of the 20 benchmarked models. This creates a potential circularity: GPT-4's similarity judgments may be systematically better at recognizing its own generated content, which could inflate or deflate performance on the synthetic dataset and distort the correlation analysis. As a concrete robustness test, please repeat the evaluation with an alternative similarity evaluator (e.g., another LLM such as Llama-3-70B, or a semantic embedding metric) or exclude the synthetic-data generator models from the correlation analysis. The current design does not allow the reader to assess whether the document-level null is an artifact of the evaluator.","section":"§4.1, §4.2"},{"comment":"The correlational analysis is based on 20 models, each run a single time without multiple seeds, temperature variations, or error bars. The document-level null (r=0.13, p>0.1) therefore has very low statistical power; a true moderate correlation could easily be missed with n=20. Please report bootstrap confidence intervals for all correlation coefficients, and ideally run each model multiple times (e.g., with different sampling temperatures) to quantify the variability of the performance scores. Without such uncertainty quantification, the claim of 'little to no relationship' at the document level is not statistically secure.","section":"§4.3, Table 4"},{"comment":"The thresholds for each similarity metric are tuned on the insight-mapping dataset, which is constructed from the same human and synthetic datasets used later for evaluation. This introduces a potential optimism bias: the chosen metric (GPT-4) and its threshold are selected to maximize agreement on data drawn from the evaluation corpus, and the same threshold is then applied to that corpus. Please use a held-out validation set or cross-validation to select thresholds, and confirm that the performance ranking of metrics and the subsequent correlation results are stable under this procedure.","section":"§3.3.1"}],"minor_comments":[{"comment":"The sentence 'Performance on the synthetic dataset consistently exceeded that of the human dataset by 0.1–0.2' is only true at the insight level; at the document level (Table 3) synthetic scores are consistently lower than human scores. Please qualify this statement to refer to the insight level only.","section":"Section 4.2"},{"comment":"The model names are inconsistent (e.g., 'llama-3-1.405B' vs. 'llama-3.1-405B', 'llama-3-1.8b-instruct' vs. 'llama-3-1-8b-instruct'), and 'llama-2-70b-chat' appears twice in Table 2. Please standardize model names and remove duplicates.","section":"Tables 2 and 3"},{"comment":"The formal definition of the relevancy rate, TP/MP, is unclear: MP is described as 'the number of matched predictions between true and predicted insights,' but it is not specified whether this counts matched predicted insights, matched true insights, or matched pairs. Please provide an unambiguous definition with a small example.","section":"Section 3.3.2"},{"comment":"The embedding model is referred to as 'text-3-large,' which is likely a typo for 'text-embedding-3-large.' Please correct.","section":"Section 3.3.1"},{"comment":"The datasets and code are described as 'available upon request' and 'will be published publicly at a later date.' For reproducibility and for the community to use the benchmark, please make the datasets and evaluation code publicly available with the manuscript.","section":"Data Availability"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and tackles an important evaluation question. The main technical risk is that the central document-level null result rests on synthetic ground-truth labels that are not verified. The authors acknowledge this, but the manuscript as it stands does not resolve it. I would be willing to consider a revised version that includes a label-fidelity verification study, an alternative evaluator analysis, and uncertainty quantification for the correlations. I also note that the cost comparison (human £7,771.83 vs synthetic £442.87) is an interesting contribution that would benefit from more careful interpretation, as the synthetic dataset was generated using human-derived insights and thus does not represent a fully autonomous pipeline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful, honest benchmark paper. It defines the MIMDE task family, builds paired human and synthetic datasets from five survey questions, benchmarks 20 LLMs, and finds a strong correlation between synthetic and human performance at the insight-extraction level (r=0.71) but none at the document-mapping level (r=0.13). The split finding is interesting and likely to be cited.\n\nWhat's genuinely new: the task definition is a sensible fusion of unsupervised extraction and classification; the human annotation process is careful (three annotators, majority voting, 81% full agreement); the comparison of 17 similarity metrics against human judgments is a useful reference; and the cost transparency (human £7,772 vs synthetic £443) is a nice touch. The authors are candid about their assumptions, particularly in Section 5 where they admit the synthetic ground truth may be wrong because models may not have incorporated all prompted insights. That honesty is to their credit.\n\nSoft spots, in proportion: the main one is indeed the unverified synthetic labels. The document-level null rests on those labels, and if many injected insights are absent or paraphrased, the null could be an artifact. The authors flag it but don't resolve it. Second, GPT-4 is both the similarity judge and one of the evaluated models, so there's a mild self-preference risk, though the similarity-metric test showed GPT-4's judgments align best with humans, so this is defensible. Third, each model runs once, and the correlation is over only 20 data points with no error bars, so the non-significant document-level r should be read with extra caution. Fourth, no code or data is released yet, which limits reproducibility. None of these overturn the insight-level correlation, which is robust across multiple metrics, but they do mean the document-level claim is provisional.\n\nThe paper is worth a serious referee. The right referee will push for a verification pass on synthetic labels, multiple runs, and public data/code. For a workshop or conference, I'd probably accept with those revisions required. For a journal, it's a solid empirical contribution but not a paradigm shift. The reader's take is about right: conditional, moderate confidence, genuinely new task and finding.","headline":"A solid, honest benchmark paper with a genuinely useful split finding; the main caveat is the one the authors already admit — unverified synthetic ground truth for the document-level null.","tokens_in":611,"tokens_out":681,"would_cite":false,"duration_ms":24529,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces the MIMDE task and shows that synthetic data reliably rank LLMs for extracting insights from documents, but not for mapping those insights back to their source documents.","keywords":["multi-insight multi-document extraction","synthetic data evaluation","LLM benchmarking","survey response analysis","insight-to-document mapping","semantic similarity metrics","correlation analysis","ground truth annotation"],"falsifier":"Take the synthetic responses and have independent annotators verify, for each response, whether each prompted insight is present and detectable; then recompute the document-level correlation between synthetic and human performance using only responses whose labels are verified. If the correlation rises substantially, the reported null result is an artifact of label noise; if it stays near zero, the claim that synthetic data fails to capture document-level complexity holds.","tokens_in":13506,"feed_emoji":"📊","tokens_out":7191,"duration_ms":56284,"temperature":0.7,"pith_summary":"Large language models are increasingly asked to read a pile of documents and pull out the few useful insights, then connect each insight back to the documents that support it. This paper names that task MIMDE, builds a human-annotated dataset and a paired synthetic dataset from the same five survey questions, and benchmarks 20 LLMs on both. Its central question is whether cheap synthetic data can stand in for expensive human data when evaluating such systems. The answer is split: for the extraction half, model rankings on synthetic data strongly track rankings on human data (recall correlation 0.71), but for the mapping half there is essentially no relationship (correlation 0.13). So synthetic data can rank models on the 'finding insights' part of the work, but not on the 'knowing where each insight came from' part.","feed_headline":"Synthetic data ranks LLMs on insight mining, not mapping","feed_subtitle":"Across 20 models, synthetic scores track human data for extracting insights but not for tracing them to source documents.","key_machinery":"The central object is the insight: a piece of information that is useful to a human analyst, specific and actionable, and potentially shared across documents. The load-bearing machinery is the paired-dataset design. Human responses were annotated by three annotators with majority-vote labels; synthetic responses were produced by prompting LLMs with a stance plus 1--3 pre-selected insights, so each synthetic response carries a label by construction. Evaluation runs at two levels: insight-level extraction asks whether a model produces each true insight at all, and document-level mapping asks whether it attaches each insight to the right response. A frontier LLM judge, validated against human judgments on 9,378 insight pairs (peak F1 0.809), decides whether a predicted insight matches a true one.","core_discovery":"The paper defines MIMDE as a two-step task: extract an optimal set of insights from a corpus and map each insight back to the documents it comes from. To test whether synthetic data can substitute for expensive human annotations in evaluating LLMs on this task, it builds paired human and synthetic datasets from the same five survey questions and benchmarks 20 LLMs. The central finding is split: at the insight-extraction level, model performance on synthetic data tracks human-data performance strongly ($r = 0.71$, $p < 0.001$ for recall, and similarly for F1, precision, and redundancy), but at the document-mapping level there is no significant relationship ($r = 0.13$, $p > 0.1$). The paper concludes that synthetic data can rank-order LLMs for extracting insights but not for tracing insights to source documents.","pith_inferences":["If the generating models sometimes omit or rephrase prompted insights, then adding a verification pass that filters unlabelled synthetic responses would probably raise synthetic document-level scores; whether the human--synthetic correlation rises would indicate whether the null result is an artifact of label noise or a real difference in document-level complexity.","The insight-level correlation may be inflated by self-similarity, since the same families of LLMs that generated the synthetic text also performed the extraction; generating synthetic data with a different model family than the evaluated models would test this.","For safety-relevant applications such as patient-feedback analysis or public consultations, the document-mapping half is the component that determines whether an insight is traceable to evidence, so relying on synthetic-only evaluation in those settings is risky until the mapping correlation is improved.","Because all three annotators agreed on the full insight set for only 23% of human responses, the human ground truth itself is noisy; using soft labels from multiple annotators could change both the benchmark scores and the correlation structure."],"forward_implications":["For the insight-extraction half of MIMDE, synthetic data can replace human data when the goal is to rank or select among LLMs, at roughly one twentieth of the annotation cost.","For the document-mapping half, synthetic data cannot be trusted to predict which model will do well on human responses; benchmark rankings built on synthetic mapping scores are likely to mislead.","Because document-level scores on synthetic data are lower across all models, any combined MIMDE benchmark that mixes extraction and mapping will be dominated by the mapping difficulty and will not transfer to human-data performance.","Even the best automatic similarity judge agrees with humans only moderately, so evaluation pipelines that need high-confidence insight matching should keep a manual verification step."],"supporting_citations":[{"why":"Shows LLMs can outperform crowd workers on text annotation, supporting the use of LLMs both to generate synthetic responses and to judge insight similarity.","marker":"[Gilardi et al., 2023]"},{"why":"Surveys synthetic data in human analysis, grounding the paper's claim that synthetic datasets are a standard substitute when evaluation data are scarce or sensitive.","marker":"[Joshi et al., 2024]"},{"why":"Argues benchmark improvements should hold across datasets, the comparative logic behind correlating model performance on synthetic and human data.","marker":"[Liu et al., 2021]"},{"why":"Demonstrates LLM-assisted thematic coding of qualitative responses, the application domain MIMDE formalizes.","marker":"[Gamieldien et al., 2023]"},{"why":"Provides methods and challenges for generative synthetic data, informing the prompt-injection generation procedure.","marker":"[Guo and Chen, 2024]"}],"fun_headline_variants":["Synthetic data ranks LLMs for extraction, not mapping","Synthetic eval misses mapping insights to documents","LLM benchmarks: synthetic data only half works","Synthetic data can't link insights to source docs","Extraction yes, mapping no: synthetic data limits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The synthetic ground truth is assumed to be present in the responses because it was written into the generation prompt, and the paper admits the generating models may have omitted or distorted those insights, so the document-level correlation is computed against labels that may not match what the models actually wrote.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic data ranks LLMs for extraction, not mapping","Synthetic eval misses mapping insights to documents","LLM benchmarks: synthetic data only half works","Synthetic data can't link insights to source docs","Extraction yes, mapping no: synthetic data limits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000252,"raw_usage":{"total_tokens":1545,"prompt_tokens":915,"completion_tokens":630,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":556}},"tokens_in":531,"tokens_out":630,"duration_ms":5264,"temperature":1.0,"reasoning_tokens":556,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:55:08.658684+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the synthetic responses and have independent annotators verify, for each response, whether each prompted insight is present and detectable; then recompute the document-level correlation between synthetic and human performance using only responses whose labels are verified. If the correlation rises substantially, the reported null result is an artifact of label noise; if it stays near zero, the claim that synthetic data fails to capture document-level complexity holds.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows LLMs can outperform crowd workers on text annotation, supporting the use of LLMs both to generate synthetic responses and to judge insight similarity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Surveys synthetic data in human analysis, grounding the paper's claim that synthetic datasets are a standard substitute when evaluation data are scarce or sensitive."},{"cited_title":"Do Question Answering Modeling Improvements Hold Across Benchmarks?","cited_arxiv_id":"2102.01065","evidence_quote":"Argues benchmark improvements should hold across datasets, the comparative logic behind correlating model performance on synthetic and human data."},{"cited_title":"M., and Katz, A","cited_arxiv_id":null,"evidence_quote":"Demonstrates LLM-assisted thematic coding of qualitative responses, the application domain MIMDE formalizes."}],"review_version":1}