Pith. sign in

REVIEW 4 major objections 5 minor 53 references

MIMDE: Exploring the Use of Synthetic vs Human Data for Evaluating Multi-Insight Multi-Document Extraction Tasks

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces the MIMDE task and shows that synthetic data reliably rank LLMs for extracting insights from documents, but not for mapping those insights back to their source documents.

desk verdict A solid, honest benchmark paper with a genuinely useful split finding; the main caveat is the one the authors already admit — unverified synthetic ground truth for the document-level null. read the letter →

arxiv 2411.19689 v1 pith:ERKMLBY6 submitted 2024-11-29 cs.CL

John Francis , Saba Esnaashari , Anton Poletaev , Sukankana Chakraborty , Youmna Hashem , Jonathan Bright This is my paper · ORCID
classification cs.CL
keywords multi-insightmulti-documentextractionsyntheticdataevaluationLLMbenchmarkingsurveyresponseanalysisinsight-to-documentmappingsemanticsimilaritymetricscorrelationgroundtruthannotation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models are increasingly asked to read a pile of documents and pull out the few useful insights, then connect each insight back to the documents that support it. This paper names that task MIMDE, builds a human-annotated dataset and a paired synthetic dataset from the same five survey questions, and benchmarks 20 LLMs on both. Its central question is whether cheap synthetic data can stand in for expensive human data when evaluating such systems. The answer is split: for the extraction half, model rankings on synthetic data strongly track rankings on human data (recall correlation 0.71), but for the mapping half there is essentially no relationship (correlation 0.13). So synthetic data can rank models on the 'finding insights' part of the work, but not on the 'knowing where each insight came from' part.

What carries the argument

The central object is the insight: a piece of information that is useful to a human analyst, specific and actionable, and potentially shared across documents. The load-bearing machinery is the paired-dataset design. Human responses were annotated by three annotators with majority-vote labels; synthetic responses were produced by prompting LLMs with a stance plus 1--3 pre-selected insights, so each synthetic response carries a label by construction. Evaluation runs at two levels: insight-level extraction asks whether a model produces each true insight at all, and document-level mapping asks whether it attaches each insight to the right response. A frontier LLM judge, validated against human judgments on 9,378 insight pairs (peak F1 0.809), decides whether a predicted insight matches a true one.

What would settle it

Take the synthetic responses and have independent annotators verify, for each response, whether each prompted insight is present and detectable; then recompute the document-level correlation between synthetic and human performance using only responses whose labels are verified. If the correlation rises substantially, the reported null result is an artifact of label noise; if it stays near zero, the claim that synthetic data fails to capture document-level complexity holds.

Watch

Extended reading notes

Core claim

The paper defines MIMDE as a two-step task: extract an optimal set of insights from a corpus and map each insight back to the documents it comes from. To test whether synthetic data can substitute for expensive human annotations in evaluating LLMs on this task, it builds paired human and synthetic datasets from the same five survey questions and benchmarks 20 LLMs. The central finding is split: at the insight-extraction level, model performance on synthetic data tracks human-data performance strongly ($r = 0.71$, $p < 0.001$ for recall, and similarly for F1, precision, and redundancy), but at the document-mapping level there is no significant relationship ($r = 0.13$, $p > 0.1$). The paper concludes that synthetic data can rank-order LLMs for extracting insights but not for tracing insights to source documents.

Load-bearing premise

The synthetic ground truth is assumed to be present in the responses because it was written into the generation prompt, and the paper admits the generating models may have omitted or distorted those insights, so the document-level correlation is computed against labels that may not match what the models actually wrote.

Editorial extensions

If this is right

  • For the insight-extraction half of MIMDE, synthetic data can replace human data when the goal is to rank or select among LLMs, at roughly one twentieth of the annotation cost.
  • For the document-mapping half, synthetic data cannot be trusted to predict which model will do well on human responses; benchmark rankings built on synthetic mapping scores are likely to mislead.
  • Because document-level scores on synthetic data are lower across all models, any combined MIMDE benchmark that mixes extraction and mapping will be dominated by the mapping difficulty and will not transfer to human-data performance.
  • Even the best automatic similarity judge agrees with humans only moderately, so evaluation pipelines that need high-confidence insight matching should keep a manual verification step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the generating models sometimes omit or rephrase prompted insights, then adding a verification pass that filters unlabelled synthetic responses would probably raise synthetic document-level scores; whether the human--synthetic correlation rises would indicate whether the null result is an artifact of label noise or a real difference in document-level complexity.
  • The insight-level correlation may be inflated by self-similarity, since the same families of LLMs that generated the synthetic text also performed the extraction; generating synthetic data with a different model family than the evaluated models would test this.
  • For safety-relevant applications such as patient-feedback analysis or public consultations, the document-mapping half is the component that determines whether an insight is traceable to evidence, so relying on synthetic-only evaluation in those settings is risky until the mapping correlation is improved.
  • Because all three annotators agreed on the full insight set for only 23% of human responses, the human ground truth itself is noisy; using soft labels from multiple annotators could change both the benchmark scores and the correlation structure.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Multi-Insight Multi-Document Extraction (MIMDE), a two-step task of extracting an optimal set of insights from a document corpus and mapping them back to source documents. The authors construct paired human and synthetic datasets for five survey questions, develop an evaluation framework with three similarity-based metrics, benchmark 20 LLMs on both datasets, and analyze cross-dataset correlations of model performance. The main findings are a strong correlation (0.71) between LLM performance on synthetic and human data at the insight extraction level, but no significant correlation (0.13) at the document-level mapping level. The paper explicitly acknowledges that synthetic ground-truth labels are derived from generation prompts and may not be reliably present in the generated responses, and calls for a future verification phase.

Significance. The paper addresses a timely question—whether synthetic data can serve as a valid proxy for human-annotated data in LLM evaluation—and provides a new task formulation, two new datasets, and a systematic comparison of 17 similarity metrics, finding that LLM-based similarity (GPT-4) aligns best with human judgment. If the findings hold, the strong insight-level correlation would support the use of synthetic data for cheaper evaluation of extraction performance, while the document-level null would caution against its use for mapping evaluation. The authors are transparent about the main limitation (unverified synthetic labels), which is commendable. However, the central null result at the document level is directly exposed to label noise, and the evaluator choice and lack of uncertainty quantification further weaken the conclusiveness of the claims. The paper's strengths are its clear task definition, the breadth of models benchmarked, and the honest limitation statement.

major comments (4)
  1. [§3.2.2, §5] The synthetic ground-truth labels are assigned by prompting LLMs to incorporate 1–3 injected insights, but the authors concede in Section 5 that 'the models may not have incorporated all prompted insights into their responses, or may have expressed them in ways that made them difficult to detect.' This is a load-bearing issue because the document-level null correlation (Table 4, r=0.13) is computed against these unverified labels. If a substantial fraction of injected insights are absent or paraphrased beyond recognition, the synthetic labels are noisy, which would depress the observed document-level scores and attenuate the cross-dataset correlation, mechanically producing the null result interpreted as a limitation of synthetic data. Please add a verification phase—e.g., manual inspection of a representative sample of synthetic responses to quantify label fidelity—and re-analyze the correlations using only responses whose labels are verified.
  2. [§4.1, §4.2] GPT-4 is used as the automatic evaluator for insight matching throughout the benchmark, while GPT-4 is also one of the models that generated the synthetic data and one of the 20 benchmarked models. This creates a potential circularity: GPT-4's similarity judgments may be systematically better at recognizing its own generated content, which could inflate or deflate performance on the synthetic dataset and distort the correlation analysis. As a concrete robustness test, please repeat the evaluation with an alternative similarity evaluator (e.g., another LLM such as Llama-3-70B, or a semantic embedding metric) or exclude the synthetic-data generator models from the correlation analysis. The current design does not allow the reader to assess whether the document-level null is an artifact of the evaluator.
  3. [§4.3, Table 4] The correlational analysis is based on 20 models, each run a single time without multiple seeds, temperature variations, or error bars. The document-level null (r=0.13, p>0.1) therefore has very low statistical power; a true moderate correlation could easily be missed with n=20. Please report bootstrap confidence intervals for all correlation coefficients, and ideally run each model multiple times (e.g., with different sampling temperatures) to quantify the variability of the performance scores. Without such uncertainty quantification, the claim of 'little to no relationship' at the document level is not statistically secure.
  4. [§3.3.1] The thresholds for each similarity metric are tuned on the insight-mapping dataset, which is constructed from the same human and synthetic datasets used later for evaluation. This introduces a potential optimism bias: the chosen metric (GPT-4) and its threshold are selected to maximize agreement on data drawn from the evaluation corpus, and the same threshold is then applied to that corpus. Please use a held-out validation set or cross-validation to select thresholds, and confirm that the performance ranking of metrics and the subsequent correlation results are stable under this procedure.
minor comments (5)
  1. [Section 4.2] The sentence 'Performance on the synthetic dataset consistently exceeded that of the human dataset by 0.1–0.2' is only true at the insight level; at the document level (Table 3) synthetic scores are consistently lower than human scores. Please qualify this statement to refer to the insight level only.
  2. [Tables 2 and 3] The model names are inconsistent (e.g., 'llama-3-1.405B' vs. 'llama-3.1-405B', 'llama-3-1.8b-instruct' vs. 'llama-3-1-8b-instruct'), and 'llama-2-70b-chat' appears twice in Table 2. Please standardize model names and remove duplicates.
  3. [Section 3.3.2] The formal definition of the relevancy rate, TP/MP, is unclear: MP is described as 'the number of matched predictions between true and predicted insights,' but it is not specified whether this counts matched predicted insights, matched true insights, or matched pairs. Please provide an unambiguous definition with a small example.
  4. [Section 3.3.1] The embedding model is referred to as 'text-3-large,' which is likely a typo for 'text-embedding-3-large.' Please correct.
  5. [Data Availability] The datasets and code are described as 'available upon request' and 'will be published publicly at a later date.' For reproducibility and for the community to use the benchmark, please make the datasets and evaluation code publicly available with the manuscript.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the central correlation analysis is an empirical comparison with independent human labels, and the acknowledged synthetic-label verification gap is a validity threat, not a circular reduction.

full rationale

The paper's derivation chain is not circular. The synthetic ground-truth labels are defined by prompt injection (Section 3.2.2) and the human ground-truth labels by majority-vote annotation (Section 3.2.1); these are independent measurements of the same task. The choice of GPT-4 as the automatic matcher is justified by a separate human-labeled similarity benchmark (Section 3.3.1, Figure 1), not by the target correlation result. The correlations in Table 4 are empirical statistics over 20 models, not identities or fitted outputs. The acknowledged limitation in Section 5 — that generators may omit or paraphrase prompted insights — is a genuine measurement-error concern: unverified synthetic labels could attenuate or distort the document-level correlation, and a verification pass would strengthen the conclusion. However, this is a threat to validity, not a case where a 'prediction' reduces by construction to an input or where a fitted parameter is renamed as a prediction. No self-citation chain or imported uniqueness theorem is load-bearing. Therefore no circular step is present.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on two modeling choices: the similarity threshold for matching insights, tuned on a derived human-labeled set, and the majority-vote definition of human ground truth. The synthetic ground truth assumes prompt compliance. No new physical or conceptual entities are postulated; MIMDE is a task abstraction, not an invented entity requiring independent evidence.

free parameters (2)
  • GPT-4 similarity threshold = Implicit, chosen to maximize F1 on the 9,378-pair insight-mapping dataset
    The threshold for deciding whether a predicted insight matches a true insight is tuned on human-labeled pairs from the same datasets, and this same evaluator is then used to score all benchmark models.
  • Annotator agreement threshold = 2 of 3 annotators
    The human ground truth is defined by majority voting of three annotators. This choice affects both datasets and the reliability of the gold labels, and the paper notes all three agreed on only 23% of responses.
assumptions (3)
  • domain assumption Insights have an objective ground truth per document
    The paper acknowledges this assumption in Section 5 and notes that all three annotators agreed on the exact same set of insights for only 23% of responses.
  • domain assumption LLM-generated responses correctly realize the prompted insights
    The synthetic labels are taken from the prompts. The paper admits in Section 5 that models may not have incorporated all prompted insights or may have expressed them in hard-to-detect ways.
  • standard math Pearson correlation across 20 model scores is a meaningful measure of proxy validity
    The analysis uses Pearson's r with n=20 models. No confidence intervals are reported, and the small sample size makes the correlation estimate sensitive to outliers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MIMDE: Exploring the Use of Synthetic vs Human Data for Evaluating Multi-Insight Multi-Document Extraction Tasks." pith.science (2026). https://pith.science/paper/ERKMLBY6

@misc{pith2026241119689,
  author       = {Pith},
  title        = {Pith review of: MIMDE: Exploring the Use of Synthetic vs Human Data for Evaluating Multi-Insight Multi-Document Extraction Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ERKMLBY6}},
  note         = {Machine review of arXiv:2411.19689}
}
read the original abstract

Large language models (LLMs) have demonstrated remarkable capabilities in text analysis tasks, yet their evaluation on complex, real-world applications remains challenging. We define a set of tasks, Multi-Insight Multi-Document Extraction (MIMDE) tasks, which involves extracting an optimal set of insights from a document corpus and mapping these insights back to their source documents. This task is fundamental to many practical applications, from analyzing survey responses to processing medical records, where identifying and tracing key insights across documents is crucial. We develop an evaluation framework for MIMDE and introduce a novel set of complementary human and synthetic datasets to examine the potential of synthetic data for LLM evaluation. After establishing optimal metrics for comparing extracted insights, we benchmark 20 state-of-the-art LLMs on both datasets. Our analysis reveals a strong correlation (0.71) between the ability of LLMs to extracts insights on our two datasets but synthetic data fails to capture the complexity of document-level analysis. These findings offer crucial guidance for the use of synthetic data in evaluating text analysis systems, highlighting both its potential and limitations.

Figures

Figures reproduced from arXiv: 2411.19689 by the authors.

Figure 1
Figure 1. Evaluating Potential MIMDE Metrics (Precision) Measures how often true matches to human mapped insights are correctly identified. (Recall) Measures how well the model identifies true positives from all the true positives in the dataset. (F1 ) A weighted mean between Precision and Recall. 4.1 Evaluating Similarity Metrics [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Relationship between Synthetic and Human data performance (Recall) Measures how often true insights have been correctly identified. At the insight level, shown on the left side of [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 34 canonical work pages

  1. [1]

    Abdullin, Y., Molla-Aliod, D., Ofoghi, B., Yearwood, J., and Li, Q. (2024). Synthetic dialogue dataset generation using llm agents. arXiv preprint arXiv:2401.17461

    arXiv 2024

  2. [2]

    Alhelbawy, A., Lattimer, M., Kruschwitz, U., Fox, C., and Poesio, M. (2020). An nlp-powered human rights monitoring platform. Expert Systems with Applications , 153:113365

    work page 2020

  3. [3]

    and Clarke, V

    Braun, V. and Clarke, V. (2012). Thematic analysis. American Psychological Association

    work page 2012

  4. [4]

    Byrne, B., Krishnamoorthi, K., Sankar, C., Neelakantan, A., Duckworth, D., Yavuz, S., Goodrich, B., Dubey, A., Cedilnik, A., and Kim, K.-Y. (2019). Taskmaster-1: Toward a realistic and diverse dialog dataset. arXiv preprint arXiv:1909.05358

    arXiv 2019

  5. [5]

    and Davidson, T

    Chae, Y. and Davidson, T. (2023). Large language models for text classification: From zero-shot learning to fine-tuning. Open Science Foundation

    work page 2023

  6. [6]

    Chan, X., Wang, X., Yu, D., Mi, H., and Yu, D. (2024). Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint arXiv:2406.20094

    arXiv 2024

  7. [7]

    and Mago, V

    Chandrasekaran, D. and Mago, V. (2021). Evolution of semantic similarity—a survey. ACM Computing Surveys (CSUR) , 54(2):1--37

    work page 2021

  8. [8]

    S., Ceder, G., Persson, K

    Dagdelen, J., Dunn, A., Lee, S., Walker, N., Rosen, A. S., Ceder, G., Persson, K. A., and Jain, A. (2024). Structured information extraction from scientific text with large language models. Nature Communications , 15(1):1418

    work page 2024

Show all 53 references
  1. [9]

    Dai, S.-C., Xiong, A., and Ku, L.-W. (2023). Llm-in-the-loop: Leveraging large language model for thematic analysis. arXiv preprint arXiv:2310.15100

    2023 arXiv

  2. [10]

    S., Ceder, G., Persson, K., and Jain, A

    Dunn, A., Dagdelen, J., Walker, N., Lee, S., Rosen, A. S., Ceder, G., Persson, K., and Jain, A. (2022). Structured information extraction from complex scientific text with fine-tuned large language models. arXiv preprint arXiv:2212.05238

    2022 arXiv

  3. [11]

    and Caragea, C

    Florescu, C. and Caragea, C. (2017). Positionrank: An unsupervised approach to keyphrase extraction from scholarly documents. In Proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers) , pages 1105--1115

    2017

  4. [12]

    M., and Katz, A

    Gamieldien, Y., Case, J. M., and Katz, A. (2023). Advancing qualitative analysis: An exploration of the potential of generative ai and nlp in thematic coding. Available at SSRN 4487768

    2023

  5. [13]

    A., Montiel, R., Ledeneva, Y., Rend \'o n, E., Gelbukh, A., and Cruz, R

    Garc \' a-Hern \'a ndez, R. A., Montiel, R., Ledeneva, Y., Rend \'o n, E., Gelbukh, A., and Cruz, R. (2008). Text summarization by sentence extraction using unsupervised learning. In Mexican International Conference on Artificial Intelligence , pages 133--143. Springer

    2008

  6. [14]

    Gatto, J., Sharif, O., Seegmiller, P., Bohlman, P., and Preum, S. M. (2023). Text encoders lack knowledge: Leveraging generative llms for domain-specific semantic textual similarity. arXiv preprint arXiv:2309.06541

    2023 arXiv

  7. [15]

    Gilardi, F., Alizadeh, M., and Kubli, M. (2023). Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences , 120(30):e2305016120

    2023

  8. [16]

    H., Hao, X., Jaber, B., Reddy, S., Kartha, R., et al

    Goel, A., Gueta, A., Gilon, O., Liu, C., Erell, S., Nguyen, L. H., Hao, X., Jaber, B., Reddy, S., Kartha, R., et al. (2023). Llms accelerate annotation for medical information extraction. In Machine Learning for Health (ML4H) , pages 82--100. PMLR

    2023

  9. [17]

    and Chen, Y

    Guo, X. and Chen, Y. (2024). Generative ai for synthetic data generation: Methods, challenges and the future. arXiv preprint arXiv:2403.04190

    2024 arXiv

  10. [18]

    Hristova, G., Bogdanova, B., and Netov, N. (2022). Design of ml-based ai system for mining public opinion on e-government services in bulgaria. In AIP Conference Proceedings , number 1. AIP Publishing

    2022

  11. [19]

    and He, G

    Huang, C. and He, G. (2024). Text clustering as classification with llms. arXiv preprint arXiv:2410.00927

    2024

  12. [20]

    Jagannathan, M., Roy, D., and Delhi, V. S. K. (2022). Application of nlp-based topic modeling to analyse unstructured text data in annual reports of construction contracting companies. CSI Transactions on ICT , 10(2):97--106

    2022

  13. [21]

    N., and Weller, A

    Jordon, J., Szpruch, L., Houssiau, F., Bottarelli, M., Cherubin, G., Maple, C., Cohen, S. N., and Weller, A. (2022). Synthetic data--what, why and how? arXiv preprint arXiv:2205.03257

    2022 arXiv

  14. [22]

    Joshi, I., Grimmer, M., Rathgeb, C., Busch, C., Bremond, F., and Dantcheva, A. (2024). Synthetic data in human analysis: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence

    2024

  15. [23]

    Kowalski, R., Esteve, M., and Jankin Mikhaylov, S. (2020). Improving public services by mining citizen feedback: An application of natural language processing. Public administration , 98(4):1011--1026

    2020

  16. [24]

    and Nithya, M

    Leelavathy, S. and Nithya, M. (2021). Public opinion mining using natural language processing technique for improvisation towards smart city. International journal of speech technology , 24(3):561--569

    2021

  17. [25]

    F., Lee, T., Jia, R., and Liang, P

    Liu, N. F., Lee, T., Jia, R., and Liang, P. (2021). Do question answering modeling improvements hold across benchmarks? arXiv preprint arXiv:2102.01065

    2021 arXiv

  18. [26]

    Section 4: Consulting residents

    Local Government Association (2024). Section 4: Consulting residents. https://www.local.gov.uk/our-support/communications-and-community-engagement/resident-communications/understanding-views-2. Accessed: Sep 18, 2024

    2024

  19. [27]

    Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021). Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664

    2021 arXiv

  20. [28]

    Meng, H., Yang, Y., Li, Y., Lee, J., and Lee, Y.-C. (2024). Exploring the potential of human-llm synergy in advancing qualitative analysis: A case study on mental-illness stigma. arXiv preprint arXiv:2405.05758

    2024

  21. [29]

    and Tarau, P

    Mihalcea, R. and Tarau, P. (2004). Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing , pages 404--411

    2004

  22. [30]

    W., Abdi, A., and Amrit, C

    Prakoso, D. W., Abdi, A., and Amrit, C. (2021). Short text similarity measurement methods: a review. Soft Computing , 25:4699--4723

    2021

  23. [31]

    D., Wibawa, A

    Prasetya, D. D., Wibawa, A. P., and Hirashima, T. (2018). The performance of text similarity algorithms. International Journal of Advances in Intelligent Informatics , 4(1):63--69

    2018

  24. [32]

    Pu, X., Gao, M., and Wan, X. (2023). Summarization is (almost) dead. arXiv preprint arXiv:2309.09558

    2023 arXiv

  25. [33]

    Qin, L., Chen, Q., Feng, X., Wu, Y., Zhang, Y., Li, Y., Li, M., Che, W., and Yu, P. S. (2024). Large language models meet nlp: A survey. arXiv preprint arXiv:2405.12819

    2024 arXiv

  26. [34]

    Quan, C., Wang, M., and Ren, F. (2014). An unsupervised text mining method for relation extraction from biomedical literature. PloS one , 9(7):e102039

    2014

  27. [35]

    Rajpurkar, P. (2016). Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250

    2016 arXiv

  28. [36]

    Ray, P. P. (2023). Chatgpt: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems , 3:121--154

    2023

  29. [37]

    Reddy, S., Chen, D., and Manning, C. D. (2019). Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics , 7:249--266

    2019

  30. [38]

    Rietz, T., Toreini, P., and Maedche, A. (2020). Cody: An interactive machine learning system for qualitative coding. In Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology , pages 90--92

    2020

  31. [39]

    and Goharian, N

    Soldaini, L. and Goharian, N. (2016). Quickumls: a fast, unsupervised approach for medical concept extraction. In MedIR workshop, sigir , pages 1--4

    2016

  32. [40]

    G., and Parai, G

    Sripada, S., Kasturi, V. G., and Parai, G. K. (2005). Multi-document extraction based summarization. CS 224N, Final Project

    2005

  33. [41]

    Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615

    2022 arXiv

  34. [42]

    Sun, L., Huang, Y., Wang, H., Wu, S., Zhang, Q., Gao, C., Huang, Y., Lyu, W., Zhang, Y., Li, X., et al. (2024). Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561

    2024 arXiv

  35. [43]

    Tang, R., Han, X., Jiang, X., and Hu, X. (2023). Does synthetic data generation of llms help clinical text mining? arXiv preprint arXiv:2303.04360

    2023 arXiv

  36. [44]

    Y., and Wang, H

    Tao, X., Li, Y., Lau, R. Y., and Wang, H. (2012). Unsupervised multi-label text classification using a world knowledge ontology. In Advances in Knowledge Discovery and Data Mining: 16th Pacific-Asia Conference, PAKDD 2012, Kuala Lumpur, Malaysia, May 29-June 1, 2012, Proceedin...

    2012

  37. [45]

    Tulkens, S., S uster, S., and Daelemans, W. (2019). Unsupervised concept extraction from clinical text through semantic composition. Journal of biomedical informatics , 91:103120

    2019

  38. [46]

    Verma, J. P. and Patel, A. (2017). Evaluation of unsupervised learning based extractive text summarization technique for large scale review and feedback data. Indian Journal of Science and Technology , 10(17):1--6

    2017

  39. [47]

    Viswanathan, V., Gashteovski, K., Lawrence, C., Wu, T., and Neubig, G. (2023). Large language models enable few-shot clustering. arXiv preprint arXiv:2307.00524

    2023 arXiv

  40. [48]

    Wang, W., Jiang, X., Tian, S., Liu, P., Dang, D., Su, Y., Lookman, T., and Xie, J. (2022). Automated pipeline for superalloy data by text mining. NPJ Computational Materials , 8(1):9

    2022

  41. [49]

    A., Ceder, G., and Jain, A

    Weston, L., Tshitoyan, V., Dagdelen, J., Kononova, O., Trewartha, A., Persson, K. A., Ceder, G., and Jain, A. (2019). Named entity recognition and normalization applied to large-scale information extraction from the materials science literature. Journal of chemical information...

    2019

  42. [50]

    Yih, W.-t., Goodman, J., Vanderwende, L., and Suzuki, H. (2007). Multi-document summarization by maximizing informative content-words. In IJCAI , volume 7, pages 1776--1782. Citeseer

    2007

  43. [51]

    and Bo, L

    Yujian, L. and Bo, L. (2007). A normalized levenshtein distance metric. IEEE transactions on pattern analysis and machine intelligence , 29(6):1091--1095

    2007

  44. [52]

    J., and Bing, L

    Zhang, W., Deng, Y., Liu, B., Pan, S. J., and Bing, L. (2023). Sentiment analysis in the era of large language models: A reality check. arXiv preprint arXiv:2305.15005

    2023 arXiv

  45. [53]

    and Eskenazi, M

    Zhao, T. and Eskenazi, M. (2018). Zero-shot dialog generation with cross-domain latent actions. arXiv preprint arXiv:1805.04803

    2018 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.