Pith. sign in

REVIEW 5 major objections 6 minor 20 references

Do LLMs Understand Why We Write Diaries? A Method for Purpose Extraction and Clustering

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that large language models can extract and cluster the purposes behind diary writing accurately enough to make large personal-narrative corpora analyzable, with GPT-4o and o1-mini performing best and a template baseline…

desk verdict A genuinely new LLM pipeline for diary-purpose extraction, but the gold standard is model-defined so the headline recall claims are upper bounds; the demographic analysis is the weakest part, yet the paper is honest and deserves refereeing. read the letter →

arxiv 2506.00985 v1 pith:HOSKEYP4 submitted 2025-06-01 cs.CL

Valeriya Goloviznina , Alexander Sergeev , Mikhail Melnichenko , Evgeny Kotelnikov This is my paper
classification cs.CL
keywords LLMdiaryanalysispurposeextractionclusteringarchivaldocumentsdigitalhumanitiesSoviet-eradiariesrelativerecall
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models can identify and cluster the purposes behind diary writing at a quality that makes quantitative analysis of large personal-narrative collections practical. The paper demonstrates this on 38,332 Soviet-era diary entries from 1922–1929 drawn from a digital archive of personal diaries, prompting three models to extract stated intentions such as documenting life, self-reflection, or practicing language skills. GPT-4o and o1-mini achieve the best precision, DeepSeek gives the widest relative recall, and the union of GPT-4o with o1-mini reaches the best relative F1-score, while a template-based baseline performs poorly. Manual reading of the whole corpus would take roughly 300 hours per person, whereas the models processed it in about an hour, which is the practical payoff if the claim holds. The same pipeline also clusters the extracted purposes and reveals demographic and temporal patterns in why people wrote.

What carries the argument

The load-bearing mechanism is the extraction-and-clustering pipeline together with its relative-recall evaluation design. In the extraction step, each model receives up to 10 diary entries per prompt and returns any purposes it finds; the union of all models' outputs becomes the annotation pool, and the set of annotator-approved extractions defines the gold standard. Because the full corpus cannot be manually labeled, Recall is computed relative to that gold set rather than to a true corpus-wide set, so the reported Recall values are 'relative Recall'. Clustering is done by an iterative LLM algorithm: the model first proposes cluster names for the complete purpose list, then assigns purposes to those clusters, and the assignment step is repeated until no purposes remain unassigned; partitions are scored with the Rand index against manual partitions. This machinery gives the paper's claims about feasibility and demographic patterns measurable, repeatable steps.

What would settle it

Manually label a random sample of 500 diary entries for the presence of purposes without consulting model outputs, then measure each model's and the union's Recall against this complete gold set; if the union misses a substantial share of true purpose-bearing entries—for instance if true Recall falls below 0.6—the reported relative-Recall estimates are optimistic and the demographic findings may be biased.

Watch

Extended reading notes

Core claim

The paper's central claim is that a three-step pipeline—prompt-based purpose extraction, human annotation of model outputs, and an iterative LLM clustering algorithm—can recover the intentions behind diary entries accurately enough for use in digital humanities research. On the entry-identification task, GPT-4o reaches 0.7685 precision and 0.4882 relative recall, o1-mini reaches 0.6761 and 0.2824, and DeepSeek reaches 0.4828 and 0.8235; the union of all three models reaches 0.9471 relative recall with 0.4834 precision, and the union of GPT-4o and o1-mini balances at 0.6993 precision and 0.6625 relative F1. For purpose extraction on entries that genuinely contain purposes, GPT-4o and o1-mini each exceed 0.89 precision. Clustering by the iterative assignment algorithm produces Rand indices up to 0.8693 for GPT-4o on its own extracted purposes. From those clusters the paper derives qualitative findings, such as men favoring memory retention and personal growth, women favoring emotional release and self-analysis, and authors over 60 writing only to preserve memories.

Load-bearing premise

The evaluation assumes that the set of all correct diary purposes in the corpus is well approximated by the union of what the four models extract, because the annotators only saw entries the models selected; the authors state in the Limitations section that they cannot label the full corpus, so any purpose-bearing entry missed by all four models is absent from the gold set.

Editorial extensions

If this is right

  • A researcher can replace roughly 300 hours of manual reading with about an hour of LLM processing, making large-corpus diary studies feasible without sacrificing acceptable quality.
  • For high precision, GPT-4o is the best choice; for broad coverage at low cost followed by manual review, DeepSeek is preferable; for a balanced trade-off, the union of GPT-4o and o1-mini works best.
  • The 13 purpose clusters produced by GPT-4o, such as Preservation of Memories, Emotional Relief, and Writing Practice, form a reusable taxonomy for studying personal narratives.
  • The documented error types—confusing other people's diaries, other writing types, and plans or style with purposes—give concrete guidance for designing prompts and annotation schemas in future ego-document studies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the gold set is limited to annotator-approved extracts from the four models, the reported Recall figures are upper bounds for absolute recall; a complete manual sample would likely lower them, and the observed gender, age, and temporal patterns could shift if missed purposes correlate with those demographics.
  • The iterative clustering recipe—generate category names, then assign items in rounds—is a general-purpose approach to open-ended category induction and could transfer to letters, memoirs, or open-ended survey responses.
  • The paper's historical-context caveat implies that prompt accuracy depends on domain knowledge; supplying period background or few-shot examples could reduce hallucinations and improve extraction precision.
  • A natural next test is applying the same pipeline to later-period diaries or modern personal writing to see whether the purpose taxonomy and demographic differences persist across eras.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes a three-step LLM pipeline for extracting and clustering the purposes behind diary writing, applied to Soviet-era diaries (1922-1929) from the Prozhito archive. Three LLMs (GPT-4o, o1-mini, DeepSeek-V3) and a template-based baseline first identify diary entries that contain a purpose and extract the purpose statements; annotators label the union of the extracted entries; performance is reported as precision, relative recall, and relative F1; and an iterative LLM clustering algorithm groups the extracted purposes, evaluated against manual partitions with the Rand index. The paper also reports demographic trends by gender, age, and writing period, and claims that GPT-4o and o1-mini achieve the best performance while the baseline is significantly less effective.

Significance. If the central claim were fully supported, the paper would offer a scalable and comparatively cheap method for quantitative analysis of large personal-narrative corpora, a useful contribution to digital humanities and computational social science. The paper is transparent about several limitations, reports concrete cost figures, and compares multiple proprietary and open-weight models. However, the evaluation's gold standard is constructed from the union of the models' own outputs, so relative recall is an upper bound and cannot substantiate the claim that the method reliably extracts purposes from unseen corpora. The demographic conclusions rest on a small, model-selected sample without statistical testing. These gaps are load-bearing for the paper's headline claims, though they are addressable with additional annotation and more cautious framing.

major comments (5)
  1. [Section 5, Table 5 and Limitations] The gold standard for relative recall is the 170 correct entries among the 460 entries extracted by the four models. Any purpose-bearing diary entry missed by all models is invisible to the annotators and absent from the denominator. Consequently, every relative recall and relative F1 value in Table 5 is an upper bound on true recall, and the Abstract/Conclusion's claim that the method can reliably and scalably extract purposes from large corpora is not supported by the presented evidence. To support absolute-recovery claims, the authors should annotate an independently sampled set of diary entries (e.g., a random sample of the 38,332 entries) and re-estimate recall on that sample; otherwise the contribution should be explicitly reframed as a ranking of models on a model-selected set.
  2. [Section 6, Fig. 2-4 and Conclusion] The demographic analyses are based on 109 purposes from GPT-4o's clusters, a subset of the model-selected 170 labeled entries, and several categories are extremely small (e.g., the over-60 age group contains 6 entries). The manuscript reports no statistical tests, confidence intervals, or corrections for multiple comparisons, so statements such as 'over 60 primarily write to preserve memories' and 'women place greater emphasis on emotional release and self-analysis' are not supported. These observations should be labeled as exploratory hypotheses or accompanied by appropriate inferential statistics (e.g., permutation tests or Bayesian interval estimates) and a discussion of selection bias from the model-based gold set.
  3. [Section 5, Step 2, Table 6] Purpose-level inter-annotator agreement is low (Krippendorff's alpha = 0.598), and the precision differences among GPT-4o (0.8936), o1-mini (0.9059), and DeepSeek (0.8462) may be within annotation noise. The authors should report per-annotator agreement, adjudicated or consensus labels, and confidence intervals for precision estimates before drawing fine-grained conclusions about which model is best at purpose extraction.
  4. [Section 3, Step 3 and Table 8] The clustering evaluation uses a single manual partition per purpose set, with no information about who constructed it, how conflicts were resolved, or whether the manual clustering is reliable. Rand indices in Table 8 therefore conflate model clustering quality with the subjectivity of the reference partition. Provide the annotation protocol, multiple independent clusterings, or an inter-clusterer agreement measure.
  5. [Abstract and Section 7] The statement that 'GPT-4o and o1-mini achieve the best performance' is ambiguous and partly contradicted by Table 5: DeepSeek has a higher relative F1 (0.6087) than GPT-4o (0.5971) and o1-mini (0.3983), while GPT-4o has the highest precision. The paper should specify which metric (purpose-extraction precision, entry-level precision, relative F1, or clustering Rand index) supports the 'best performance' claim and qualify the conclusion accordingly.
minor comments (6)
  1. [Appendices A-G] All appendices state that prompts and examples are 'available at the repository' but no repository URL or identifier is provided, so readers cannot access the materials.
  2. [Section 4.2] The exclusion of GPT-4o-mini, DeepSeek-R1, and Qwen-2.5 is justified only by 'preliminary analysis showed a large number of incorrectly selected entries,' with no quantitative criterion or reported numbers; this model-selection step should be reported transparently.
  3. [Table 4] Baseline token counts are measured in words while model token counts use subword tokenizers, so the average token numbers and cost comparison are not directly comparable across methods.
  4. [Figures 1-4] The figures would be easier to interpret with error bars or by displaying the underlying counts directly in the plot, since several categories have very small bases.
  5. [Section 1] The term 'purpose' is used to cover both reasons for starting a diary and ongoing purposes while writing; a single operational definition and consistent terminology would help the reader follow the annotation task.
  6. [Section 3] The phrase 'labor-intensive of annotating' should be reworded, for example to 'the labor-intensive nature of annotating.'

Circularity Check

1 steps flagged · score 1.0 of 10

No load-bearing circularity: the only self-referential element is the explicitly labeled relative Recall denominator, which the paper acknowledges as a limitation rather than presenting as an independent gold standard.

  1. self definitional [Section 3 Methodology, Step 2; reiterated in Limitations]
    "In our case, we consider the set of all correct answers to be the set of correct answers labeled by the annotators. ... Another issue is that we lack information about the complete set of diary entries in our corpus that contain purposes, as labeling the entire corpus is too time-consuming. Therefore, we utilize relative Recall (and consequently relative F1-score) since we only consider the set of entries extracted by all four models."

    Relative Recall's denominator is the set of correct answers found in the union of the four models' extractions, so each model's recall is measured against a gold set its own outputs helped construct. Any purpose-bearing entry missed by all four models is invisible to the annotators and absent from the denominator, making relative Recall and relative F1 upper bounds relative to an independent gold standard. However, the paper explicitly names the metric 'relative,' explains why true Recall cannot be computed, and repeats the caveat in Limitations. The main precision comparisons and the clustering Rand-index evaluation rest on independent human annotation and manual partitions, so the central claim does not reduce to this self-referential definition.

full rationale

The claimed derivation chain is an empirical evaluation, not a formal derivation from first principles. The core evidence for the headline result—GPT-4o and o1-mini perform best—comes from human-annotated precision on model outputs (Table 5, Table 6) and from Rand indices comparing model clusterings against manually constructed reference partitions (Table 8). These evaluations are not circular: annotators independently judge whether an extracted entry actually contains a purpose, and manual clustering is created without the models' labels. The one genuinely self-referential element is the computation of relative Recall and relative F1, whose denominator is the set of correct entries identified from the union of the four models' outputs. This makes the recall-related numbers upper bounds and makes any recall-based ranking partly dependent on which entries the models happened to retrieve. Critically, the paper does not disguise this: it labels the metric 'relative,' states that true recall cannot be computed because the whole corpus cannot be labeled, and explicitly acknowledges in Limitations that the study lacks information about the complete set of purpose-containing entries. There is no fitted parameter renamed as a prediction, no load-bearing self-citation chain, and no imported uniqueness theorem. The demographic and temporal observations are presented as hypotheses from a small, model-selected sample and are accompanied by explicit majority-voting and sample-size caveats. Overall, the study's central method and comparative claims retain independent content despite the acknowledged evaluation limitation, so the circularity score is minimal.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters were fitted to data, and no new physical or formal entities are introduced. The load-bearing input assumptions are annotation-as-ground-truth despite moderate agreement, and the union-of-extractions as the search space for correctness.

assumptions (4)
  • domain assumption Annotator majority labels are the ground truth for whether an entry contains a purpose and whether an extracted purpose is correct.
    All precision and relative F1 scores are computed against majority-voted labels. Purpose-level inter-annotator agreement is only 0.598, so the ground truth is not stable. See Section 5, Step 2 text and Table 6.
  • domain assumption The union of the four models' extracted entries contains all or nearly all purpose-bearing entries.
    The authors state they cannot label the whole corpus and therefore use relative recall over the annotated union. If some purpose-bearing entries are missed by all models, the performance ranking could change. See Methodology Step 2 and Limitations.
  • domain assumption LLM extractions are treated as faithful text analysis results, aside from the noted error types.
    Model outputs are the raw material for clustering and demographic analysis. The limitations section acknowledges that misunderstanding historical context could lead to hallucination and incorrect labeling.
  • domain assumption Prozhito corpus publication permissions and anonymization make the diaries ethically usable for automated analysis.
    The ethics statement says diaries were used only with author or heir permission and were anonymized. If permission metadata is incomplete, the dataset and the results inherit that issue.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do LLMs Understand Why We Write Diaries? A Method for Purpose Extraction and Clustering." pith.science (2026). https://pith.science/paper/HOSKEYP4

@misc{pith2026250600985,
  author       = {Pith},
  title        = {Pith review of: Do LLMs Understand Why We Write Diaries? A Method for Purpose Extraction and Clustering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HOSKEYP4}},
  note         = {Machine review of arXiv:2506.00985}
}
read the original abstract

Diary analysis presents challenges, particularly in extracting meaningful information from large corpora, where traditional methods often fail to deliver satisfactory results. This study introduces a novel method based on Large Language Models (LLMs) to identify and cluster the various purposes of diary writing. By "purposes," we refer to the intentions behind diary writing, such as documenting life events, self-reflection, or practicing language skills. Our approach is applied to Soviet-era diaries (1922-1929) from the Prozhito digital archive, a rich collection of personal narratives. We evaluate different proprietary and open-source LLMs, finding that GPT-4o and o1-mini achieve the best performance, while a template-based baseline is significantly less effective. Additionally, we analyze the retrieved purposes based on gender, age of the authors, and the year of writing. Furthermore, we examine the types of errors made by the models, providing a deeper understanding of their limitations and potential areas for improvement in future research.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 18 canonical work pages

  1. [1]

    Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., Yang, D.: Can large language models transform computational social science? Computational Linguistics 50(1), 237–291 (2024)

    work page 2024

  2. [2]

    and others: The Impact of Digital Analysis and Large Language Models in Digital Humanity

    Cigliano, A., Fallucchi, F., Gerardi, M. and others: The Impact of Digital Analysis and Large Language Models in Digital Humanity. In ICYRIME 2024: 9th International Conference of Yearly Reports on Informatics, Mathematics, and Engineering, pp. 1. CEUR Workshop Pro- ceedings

    work page 2024

  3. [3]

    Available at SSRN 5022988 (2024)

    Kataishi, R.: The Technological Trajectory of Semantic Analysis: A Historical -Methodo- logical Review of NLP in Social Sciences. Available at SSRN 5022988 (2024)

    work page 2024

  4. [4]

    L.: Diary Methods

    Hyers, L. L.: Diary Methods. Oxford University Press, New York, NY (2018)

    work page 2018

  5. [5]

    Encyclopedia of Quality of Life and Well-Being Research, pp

    Lischetzke, T., Könen, T.: Daily Diary Methodology. Encyclopedia of Quality of Life and Well-Being Research, pp. 1563–1570, Springer International Publishing, Cham (2023)

    work page 2023

  6. [6]

    G., Pooley, M

    Pooley, C. G., Pooley, M. E.: The Value of Diary Writing. Everyday Mobilities in Nine- teenth- and Twentieth-Century British Diaries, pp. 21-53, Springer International Publishing, Cham (2022)

    work page 2022

  7. [7]

    V., Miret, S., Koch, C

    Schilling-Wilhelmi, M., Ríos -García, M., Shabih, S., Gil, M. V., Miret, S., Koch, C. T., Márquez L. A., Jablonka K. M.: From text to insight: large language models for chemical data extraction. Chem. Soc. Rev. 54, 1125-1150 (2025)

    work page 2025

  8. [8]

    Preprint, ChemRxiv (2023)

    Patiny, L., Godin, G.: Automatic extraction of FAIR data from publications using LLM. Preprint, ChemRxiv (2023). 7 https://prozhito.org/page/corpus/ 15

    work page 2023

Show all 20 references
  1. [9]

    Preprint, arXiv:2308.03107

    Peng, R., Liu, K., Yang, P., Yuan, Z., Li, S.: Embedding -based Retrieval with LLM for Effective Agriculture Information Extracting from Unstructured Data. Preprint, arXiv:2308.03107

    arXiv

  2. [10]

    S., Wolf, F., Le βmann, M-E., van Treeck, M ., Ferber, D

    Wiest, I. S., Wolf, F., Le βmann, M-E., van Treeck, M ., Ferber, D. Zhu, J., Boehme, H., Bressem, K., Ulrich, H., Ebert, M. P., Kather, J. N.: LLM-AIx: An open source pipeline for Information Extraction from unstructured medical text based on privacy preserving Large Language ...

    2024

  3. [11]

    : Phenotypes Extraction from Text: Analysis and Perspective in the LLM Era

    Baddour, M., Paquelet, S., Rollier, P., De Tayrac, M., Dameron, O., Labbe, T. : Phenotypes Extraction from Text: Analysis and Perspective in the LLM Era. In 2024 IEEE 12th Inter- national Conference on Intelligent Systems (IS), pp. 1–8

    2024

  4. [12]

    J Med Internet Res 26, e54617 (2024)

    Shin, D., Kim, H., Lee, S., Cho, Y., Jung, W.: Using Large Language Models to Detect Depression From User-Generated Diary Text Data as a Novel Approach in Digital Mental Health Screening: Instrument Validation Study. J Med Internet Res 26, e54617 (2024)

    2024

  5. [13]

    Y., Ma, X.: DiaryHelper: Exploring the Use of an Automatic Contextual Information Recording Agent for Elicitation Diary Study

    Li, J., He, C., Hu, J., Jia, B., Halevy, A. Y., Ma, X.: DiaryHelper: Exploring the Use of an Automatic Contextual Information Recording Agent for Elicitation Diary Study. In Proceed- ings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, New York, NY, ...

    2024

  6. [14]

    M., Bragança, I

    Pezzato, L. M., Bragança, I. F. de S., Prado G. do V. T.: What can a diary do? A look at the academic literature. Revista Brasileira de Educação, 19 (2024)

    2024

  7. [15]

    In Proceedings of the 9th Workshop on Computa- tional Linguistics and Clinical Psychology (CLPsych 2024), pp

    Alhamed, F., Ive, J., Specia, L.: Using Large Language Models (LLMs) to Extract Evidence from Pre-Annotated Social Media Data. In Proceedings of the 9th Workshop on Computa- tional Linguistics and Clinical Psychology (CLPsych 2024), pp. 232–237, St. Julians, Malta. Association...

    2024

  8. [16]

    : A practical guide to evaluating sensitivity of literature search strings for systematic reviews using relative recall

    Lagisz, M., Yang, Y., Young, S., Nakagawa, S. : A practical guide to evaluating sensitivity of literature search strings for systematic reviews using relative recall. Research Synthesis Methods, 1–14 (2025)

    2025

  9. [17]

    Searching for the social determinants of health: observations from evidence synthesis publications

    Hanneke, R., Brunskill, A. Searching for the social determinants of health: observations from evidence synthesis publications. Syst Rev 13, 134 (2024)

    2024

  10. [18]

    M.: Objective Criteria for the Evaluation of Clustering Methods

    Rand, W. M.: Objective Criteria for the Evaluation of Clustering Methods. Journal of the American Statistical Association 66(336), 846-850 (1971)

    1971

  11. [19]

    J.: A conception of adult development

    Levinson, D. J.: A conception of adult development. Am. Psychol. 41, 3–13 (1986)

    1986

  12. [20]

    and others: DeepSeek-V3 Technical Re- port

    DeepSeek-AI, Liu A., Feng, B., Xue, B., Wang, B. and others: DeepSeek-V3 Technical Re- port. Computing Research Repository, arXiv:2412.19437. Appendix A. Prompt for purpose extraction Prompt for purpose extraction is available at the repository, Appendix A. Appendix B. Prompt ...

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.