Pith. sign in

REVIEW 4 major objections 4 minor 66 references

SAGE: A Context-Aware Approach for Mining Privacy Requirements Relevant Reviews from Mental Health Apps

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SAGE combines mental-health-specific NLI with a zero-shot GPT classifier to find privacy-relevant app reviews without labeled training data.

desk verdict The headline F1 is in-sample because the NLI thresholds were tuned on the same benchmark used to report SAGE's performance; the 748-review corpus and the 17 MH-specific hypotheses are the real contributions. read the letter →

arxiv 2507.09051 v2 pith:Q7JGRAHF submitted 2025-07-11 cs.SE

Aakash Sorathiya , Gouri Ginde This is my paper · ORCID
classification cs.SE
keywords privacyrequirementsappreviewsmentalhealthappsnaturallanguageinferencezero-shotclassificationlargemodelsengineeringconcerns
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes SAGE, a two-stage pipeline that identifies privacy-relevant reviews in mental health app stores without needing labeled training data. In the first stage, a natural language inference model scores each review against seventeen privacy-violation hypotheses specific to mental health apps, and a threshold rule keeps likely candidates. In the second stage, a zero-shot GPT classifier labels the candidates as privacy or not. On a 1,376-review manually labeled benchmark, the paper reports F1 = 0.85, close to a fine-tuned SVM and ahead of fine-tuned BERT and T5, and it reports discovering 748 privacy reviews in 42,271 unlabeled reviews that keyword filtering had missed. If these numbers hold, privacy requirements can be mined from user feedback at scale without the usual annotation cost.

What carries the argument

The load-bearing mechanism is the set of 17 mental-health-specific privacy hypotheses, each a sentence stating a privacy violation such as "Mental health data is shared with third parties." Each review is treated as a premise; a DeBERTa-based NLI model returns an entailment probability for each hypothesis. The heuristic $NE(i,t)$ counts how many hypotheses scored above threshold $t$ for review $i$, and a review is passed to the LLM stage if $NE(i,0.85) \geq 1$, or $NE(i,0.75) \geq 3$, or $NE(i,0.7) \geq 5$. A GPT-4o-mini classifier, prompted with role instructions and run five times with majority voting at temperature 0, makes the final privacy or not-privacy call. The NLI stage is what the paper credits with filtering irrelevant reviews and providing domain context, while the GPT stage is what removes the need for fine-tuning.

What would settle it

Re-run SAGE with the threshold rules selected by cross-validation on a training split and then evaluated on a held-out split; if the F1 drops toward the 0.76 of ChatGPT without the NLI stage, the headline result is in-sample selection rather than a property of the method. A second check would apply SAGE to a fresh batch of mental-health app reviews from apps or stores outside the 204K corpus and measure how many extracted reviews survive independent human inspection.

Watch

Extended reading notes

Core claim

The central claim is that a context-aware hybrid of domain-specific NLI and zero-shot LLM classification can find privacy-relevant app reviews with accuracy comparable to supervised classifiers while using no labeled training data. The paper argues that generic keyword or regex filters miss reviews that describe privacy concerns in different words, and that generic NLI hypotheses lack the mental-health context needed to separate true privacy complaints from incidental uses of words like "private." SAGE's design turns each mental-health privacy concept from a published taxonomy into explicit violation hypotheses, uses NLI entailment probabilities to retain candidate reviews, and then asks GPT to make the final binary decision. On the 1,376-review benchmark the paper reports F1 = 0.85 and Cohen's kappa = 0.71 against human labels, and on 42,271 unlabeled reviews it reports that human inspection confirmed 748 new privacy reviews that keyword-based filtering had missed.

Load-bearing premise

The central result assumes that choosing the NLI threshold rules on the full 1,376-review labeled set and then reporting F1 on that same set does not inflate the score; if that assumption fails, the comparison with fine-tuned baselines overstates the zero-shot advantage.

Editorial extensions

If this is right

  • Privacy requirements can be elicited from app stores without building a labeled training set, since the SAGE pipeline operates in a zero-shot setting.
  • The NLI pre-filter is central to the reported accuracy: removing it and using ChatGPT alone drops F1 from 0.85 to 0.76.
  • Domain-specific hypotheses outperform generic ones, with F1 of 0.85 versus 0.77, indicating that privacy wording in specialized domains needs domain-tailored hypotheses.
  • Keyword-based filtering can overlook a substantial fraction of privacy reviews; SAGE reports finding 748 such reviews that a keyword-based pipeline had missed.
  • Because the two-stage design depends only on the hypothesis set, the same architecture can be adapted to other ethical concerns, such as safety or transparency, by replacing the hypotheses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An evaluation that selects the NLI thresholds on one fold and scores them on another would clarify whether the reported F1 reflects the method or the threshold choice; this check is not reported in the paper.
  • Because the final classifier is a closed, API-based model, reproducibility at scale depends on model availability and cost; re-running SAGE with an open-weight model would test how much of the result is tied to GPT-4o-mini.
  • The 748 "new" reviews are defined relative to one specific keyword filter and annotation procedure; a different filter would likely shift the number, so it is best read as a demonstration of recall rather than a population estimate.
  • The same two-stage design could be pointed at other sensitive domains, such as period-tracking or financial apps, by writing a new set of domain hypotheses; the paper gestures at this direction but does not test it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes SAGE, a two-stage pipeline for mining privacy requirements-relevant app reviews from mental health apps. The first stage uses a DeBERTa-v3 NLI model with 17 manually derived, MH-specific privacy hypotheses to filter candidate reviews via entailment-score heuristics; the second stage uses zero-shot GPT-4o-mini with a role-based prompt to classify the candidates as privacy or not-privacy. The authors evaluate on 1,376 manually labeled reviews from Ebrahimi et al. and report F1 = 0.85 without fine-tuning, comparing favorably with fine-tuned BERT and T5 baselines, and report extracting 748 new privacy reviews from 42,271 unlabeled reviews. The paper also compares domain-specific hypotheses against generic hypotheses, evaluates agreement with human labels via Cohen's kappa, and releases a replication package.

Significance. If the quantitative claims were valid, SAGE would be a practically useful, low-labeling-cost approach to a privacy-requirements mining task, and the MH-specific hypothesis set would be a reusable contribution. The paper has concrete strengths: it uses a publicly available dataset, ships an open-source replication package with the hypotheses and extracted reviews, reports inter-annotator agreement (kappa = 0.74) for the qualitative study, and addresses a real limitation of keyword-based sampling. However, the headline F1 is not a trustworthy estimate of zero-shot performance because the NLI heuristics were selected on the same benchmark used for the final evaluation, and the comparison against supervised baselines uses mismatched evaluation protocols. These issues are load-bearing for the central claim, so the current evidence does not support the paper's conclusions as stated.

major comments (4)
  1. [III-A, Table II; IV-B; Table IV] The NLI heuristic set (Set 2) was selected by computing Precision, Recall, and F1 against the full 1,376-review ground-truth benchmark, and SAGE's reported F1 = 0.85 in Table IV is then computed on that same 1,376-review benchmark. This makes the headline F1 an in-sample estimate, not a zero-shot generalization result. The heuristics are part of the SAGE pipeline, so the system is effectively tuned on the test set. The internal-threats paragraph in Section VII acknowledges that altering the heuristics could affect results but does not address this selection bias. Please re-select the heuristics on a validation split or via nested cross-validation and report the F1 on a held-out test set.
  2. [IV-B, IV-C; Table IV] For RQ1, the supervised baselines (SVM, BERT, T5) are trained on 80% of the labeled data and evaluated on a 20% stratified holdout, whereas SAGE is evaluated on all 1,376 labeled reviews. These are not comparable protocols, so the differences in Table IV (SAGE 0.85 vs. BERT 0.84, T5 0.79) could be explained by the evaluation mismatch alone. No confidence intervals or significance tests are provided, so even the gap with SVM (0.91) is unquantified. Please evaluate every system on the same held-out test split (or use identical cross-validation) and report confidence intervals or significance tests.
  3. [V, RQ5] The qualitative evaluation validates only the reviews that SAGE flagged as privacy (1,155 GPT 'yes' reviews), of which manual inspection confirmed 748. This measures precision on a selected positive sample, but it does not measure recall on the 42,271 unlabeled reviews, so the claim that SAGE 'extracted 748 privacy reviews previously overlooked by keyword-based methods' is not an effectiveness comparison against the keyword baseline. Please report recall or an estimate based on a sampled set of negatives, and frame RQ5 as a discovery case study rather than a comparative effectiveness result.
  4. [Abstract; V (RQ1); VI (Discussion)] The abstract and conclusion state that SAGE 'outperforms' the fine-tuned classifiers BERT and T5, but Table IV shows that SVM achieves the highest F1 (0.91) and BERT is within 0.01 of SAGE. Given the protocol mismatch and lack of significance testing, the superiority claim over BERT and T5 is not established. The authors should either temper this claim or support it with a statistically valid comparison.
minor comments (4)
  1. [V (RQ1) vs. VI] The text in Section V reports T5's F1 as 0.79, but Section VI states 0.78; please reconcile the numbers.
  2. [III-A] The text says Set 2 achieved the highest F1 score of 0.54, but Table II reports 0.55; please correct the inconsistency.
  3. [IV-C] The paper says it uses macro-averaged P, R, and F1, but it does not clarify whether the reported F1 is the macro-average across classes or the F1 for the privacy class; please specify, since this is a binary classification task.
  4. [VI] The subsection heading 'Abstract tool for app developers' appears to be a typo; consider renaming it, e.g., 'An abstraction for app developers' or 'A generalizable tool for app developers'.

Circularity Check

1 steps flagged · score 6.0 of 10

SAGE's reported zero-shot F1=0.85 is in part an in-sample result: the NLI threshold set was selected on the same 1,376-review labeled benchmark used to compute that F1.

  1. fitted input called prediction [Section III-A (Defining heuristics, Table II); Section IV-B (Evaluation dataset); Section V (Table IV)]
    "Thus, using this intuition method, we performed experiments with 4 different sets of heuristics (shown in Table II) ranging from high to low values of entailment score and selected the best set of heuristics that yielded the minimum number of false positives (FP) (0-labeled reviews annotated as 'maybe-privacy') and false negatives (FN) (1-labeled reviews annotated as 'maybe-not-privacy'). We used the F1 score to perform this analysis. ... For the quantitative evaluation (RQ1-RQ4) of SAGE, we used the labeled dataset of 1,376 reviews."

    Set 2 is a fixed stage of SAGE: reviews pass to GPT only if they satisfy NE(i,0.85)>=1 or NE(i,0.75)>=3 or NE(i,0.7)>=5. These thresholds were chosen by comparing four candidate threshold sets against the full 1,376-review ground truth and picking the highest F1 (Set 2, F1=0.55). The same 1,376 labeled reviews are then used to produce the advertised SAGE F1=0.85 in Table IV, so the reported zero-shot F1 is an in-sample estimate after test-set model selection, not an independent prediction. The RQ1 comparison is also asymmetric: the supervised baselines train on 80% and test on 20% of the data, while SAGE's number uses all 1,376 reviews, so the comparison with BERT and T5 is not a valid measure of zero-shot generalization.

full rationale

SAGE's only label-based fitting step is the choice of the NLI heuristic thresholds. Section III-A states that the four candidate threshold sets in Table II were scored with F1 against the ground-truth annotations, and Set 2 was selected; Section IV-B then uses the same 1,376 labeled reviews as the benchmark for RQ1-RQ4, and Section V reports SAGE's F1=0.85 on it. That is test-set model selection: the headline result is partly a product of having chosen the filtering thresholds to maximize performance on the benchmark, so the 'zero-shot without fine-tuning' claim is overstated. The GPT classification and DeBERTa NLI models themselves are pretrained externally and are not fine-tuned on the labels, which is why this is partial circularity rather than the entire derivation being a fit. The RQ5 qualitative study on 42,271 unlabeled reviews with manual verification (kappa=0.74) is a genuinely independent application and supports the claim that SAGE can surface non-keyword privacy reviews, though it measures precision rather than recall. No load-bearing self-citation chain is present; the citation to the authors' own prior work [12] is only a pointer for later requirement refinement. Overall, the quantitative zero-shot claim is not self-contained because its key parameter was fitted to the evaluation benchmark, so it receives a 6.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; the 17 hypotheses are textual templates derived from an existing taxonomy, so the invented-entities ledger is empty. The main burden is the hand-fitted NLI threshold set and the assumed reliability of the external benchmark and models.

free parameters (2)
  • NLI heuristic thresholds and counts (Set 2) = NE(i,0.85)>=1 or NE(i,0.75)>=3 or NE(i,0.7)>=5
    Selected as the best of four hand-designed threshold sets by F1 on the full 1,376-review evaluation benchmark (Section III-A, Table II), then reused for the reported SAGE F1.
  • LLM inference configuration = temperature=0; majority over 5 repeated prompts
    Hand-set to reduce output variability and improve determinism (Section III-B); not fit to the evaluation labels but part of the pipeline definition.
assumptions (4)
  • domain assumption The 1,376 manually annotated reviews from Ebrahimi et al. [17] are a correct and representative gold standard for privacy-relevant MH app reviews.
    Used as ground truth for threshold selection, RQ1-RQ4 evaluation, and kappa agreement; any label noise propagates to all results (Section IV-B).
  • domain assumption The privacy taxonomy of Iwaya et al. [26], converted into 17 MH-specific hypotheses, adequately covers privacy violations voiced by app users.
    The NLI stage only detects reviews that entail one of these 17 hypotheses; privacy concerns outside this taxonomy are invisible to SAGE (Section III-A, Table I).
  • domain assumption DeBERTa-v3 fine-tuned on MNLI, ANLI, and FeverNLI provides trustworthy entailment probabilities for 23,392 premise-hypothesis pairs.
    The threshold heuristics and the downstream F1 depend on the accuracy and calibration of the NLI entailment scores (Section III-A).
  • domain assumption GPT-4o-mini zero-shot with the designed role prompt returns reliable yes/no privacy labels for candidate reviews.
    LLM inference is the final arbiter of the label; the prompt, temperature, and majority-vote protocol are assumed to make the output faithful (Section III-B).

how reviews work

0 comments
Cite this review

Pith. "Pith review of SAGE: A Context-Aware Approach for Mining Privacy Requirements Relevant Reviews from Mental Health Apps." pith.science (2026). https://pith.science/paper/Q7JGRAHF

@misc{pith2026250709051,
  author       = {Pith},
  title        = {Pith review of: SAGE: A Context-Aware Approach for Mining Privacy Requirements Relevant Reviews from Mental Health Apps},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q7JGRAHF}},
  note         = {Machine review of arXiv:2507.09051}
}
read the original abstract

Mental health (MH) apps often require sensitive user data to customize services for mental wellness needs. However, such data collection practices in some MH apps raise significant privacy concerns for users. These concerns are often mentioned in app reviews, but other feedback categories, such as reliability and usability, tend to take precedence. This poses a significant challenge in automatically identifying privacy requirements-relevant reviews (privacy reviews) that can be utilized to extract privacy requirements and address users' privacy concerns. Thus, this study introduces SAGE, a context-aware approach to automatically mining privacy reviews from MH apps using Natural Language Inference (NLI) with MH domain-specific privacy hypotheses (provides domain-specific context awareness) and a GPT model (eliminates the need for fine-tuning). The quantitative evaluation of SAGE on a dataset of 204K app reviews achieved an F1 score of 0.85 without any fine-tuning, outperforming the fine-tuned baseline classifiers BERT and T5. Furthermore, SAGE extracted 748 privacy reviews previously overlooked by keyword-based methods, demonstrating its effectiveness through qualitative evaluation. These reviews can later be refined into actionable privacy requirement artifacts.

Figures

Figures reproduced from arXiv: 2507.09051 by the authors.

Figure 1
Figure 1. SAGE design process. At the core, we have domain-specific NLI and a GPT model. NLI with MH domain-specific hypotheses is used to extract [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Role-based prompt designed by following the guidelines from [36]. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. SAGE evaluation process. RQ1-RQ4 represent the quantitative [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 51 canonical work pages

  1. [1]

    The state of mental health in america 2024,

    M. Reinert et al., “The state of mental health in america 2024,” 2024

    work page 2024

  2. [2]

    Smartphones in mental health: a critical review of background issues, current status and future concerns,

    M. Bauer et al. , “Smartphones in mental health: a critical review of background issues, current status and future concerns,” International journal of bipolar disorders , vol. 8, pp. 1–19, 2020

    work page 2020

  3. [3]

    ” money doesn’t buy you happiness

    T. Eagle et al. , “” money doesn’t buy you happiness”: negative con- sequences of using the freemium model for mental health apps,” Proc. of the ACM on Human-Computer Interaction , vol. 6, no. CSCW2, pp. 1–38, 2022

    work page 2022

  4. [4]

    Dichotomies in the development and implementation of digital mental health tools,

    J. Torous et al. , “Dichotomies in the development and implementation of digital mental health tools,” Psychiatric Services, vol. 69, no. 12, pp. 1204–1206, 2018

    work page 2018

  5. [5]

    Current regulation of mobile mental health applications,

    D. E. McNiel et al. , “Current regulation of mobile mental health applications,” J Am Acad Psychiatry Law , vol. 46, pp. 204–11, 2018

    work page 2018

  6. [6]

    Needed innovation in digital health and smartphone applications for mental health: transparency and trust,

    J. Torous et al. , “Needed innovation in digital health and smartphone applications for mental health: transparency and trust,” JAMA psychiatry, vol. 74, no. 5, pp. 437–438, 2017

    work page 2017

  7. [7]

    What do mobile app users complain about?

    H. Khalid et al. , “What do mobile app users complain about?” IEEE software, vol. 32, no. 3, pp. 70–77, 2014

    work page 2014

  8. [8]

    Government mobile apps: Analysing citizen feedback via app reviews,

    T. Aamir, M. B. Chhetri, M. Chamikara, and M. Grobler, “Government mobile apps: Analysing citizen feedback via app reviews,” in 2023 38th IEEE/ACM International Conference on Automated Software Engineer- ing (ASE). IEEE, 2023, pp. 1858–1863

    work page 2023

Show all 66 references
  1. [9]

    Mobile app privacy in software engineering research: A systematic mapping study,

    F. Ebrahimi et al., “Mobile app privacy in software engineering research: A systematic mapping study,”Information and Software Technology, vol. 133, p. 106466, 2021

    2021

  2. [10]

    Mining and searching app reviews for requirements engineering: Evaluation and replication studies,

    J. Dabrowski, E. Letier, A. Perini, and A. Susi, “Mining and searching app reviews for requirements engineering: Evaluation and replication studies,” Information Systems, vol. 114, p. 102181, 2023

    2023

  3. [11]

    Analysing app reviews for software engineering: a systematic literature review,

    J. Dabrowski et al., “Analysing app reviews for software engineering: a systematic literature review,” Empirical Software Engineering , vol. 27, no. 2, p. 43, 2022

    2022

  4. [12]

    Towards extracting ethical concerns- related software requirements from app reviews,

    A. Sorathiya and G. Ginde, “Towards extracting ethical concerns- related software requirements from app reviews,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, 2024, pp. 2251–2255

    2024

  5. [13]

    Analyzing and automatically labelling the types of user issues that are raised in mobile app reviews,

    S. McIlroy, N. Ali, H. Khalid, and A. E. Hassan, “Analyzing and automatically labelling the types of user issues that are raised in mobile app reviews,” Empirical Software Engineering, vol. 21, pp. 1067–1106, 2016

    2016

  6. [14]

    Investigating user per- ceptions of mobile app privacy: An analysis of user-submitted app reviews,

    A. R. Besmer, J. Watson, and M. S. Banks, “Investigating user per- ceptions of mobile app privacy: An analysis of user-submitted app reviews,” International Journal of Information Security and Privacy (IJISP), vol. 14, no. 4, pp. 74–91, 2020

    2020

  7. [15]

    Analyzing user perspectives on mobile app privacy at scale,

    P. Nema, P. Anthonysamy, N. Taft, and S. T. Peddinti, “Analyzing user perspectives on mobile app privacy at scale,” in Proc. of the 44th International Conference on Software Engineering , 2022, pp. 112–124

    2022

  8. [16]

    Hark: A deep learning system for navigating privacy feedback at scale,

    H. Harkous, S. T. Peddinti, R. Khandelwal, A. Srivastava, and N. Taft, “Hark: A deep learning system for navigating privacy feedback at scale,” in 2022 IEEE Symposium on Security and Privacy (SP) . IEEE, 2022, pp. 2469–2486

    2022

  9. [17]

    Unsupervised summarization of privacy concerns in mobile application reviews,

    F. Ebrahimi and A. Mahmoud, “Unsupervised summarization of privacy concerns in mobile application reviews,” in Proc. of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022, pp. 1–12

    2022

  10. [18]

    Short text, large effect: Measuring the impact of user reviews on android app security & privacy,

    D. C. Nguyen, E. Derr, M. Backes, and S. Bugiel, “Short text, large effect: Measuring the impact of user reviews on android app security & privacy,” in 2019 IEEE symposium on Security and Privacy (SP) . IEEE, 2019, pp. 555–569

    2019

  11. [19]

    An empirical study on user reviews targeting mobile apps’ security & privacy,

    D. Mukherjee, A. Ahmadi, M. V . Pour, and J. Reardon, “An empirical study on user reviews targeting mobile apps’ security & privacy,” arXiv preprint arXiv:2010.06371, 2020

    2010 arXiv

  12. [20]

    Privacy analysis of period tracking mobile apps in the post-roe v. wade era,

    Z. Dong, L. Wang, H. Xie, G. Xu, and H. Wang, “Privacy analysis of period tracking mobile apps in the post-roe v. wade era,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022, pp. 1–6

    2022

  13. [21]

    Lessons from the use of natural language inference (nli) in requirements engineering tasks,

    M. Fazelnia, V . Koscinski, S. Herzog, and M. Mirakhorli, “Lessons from the use of natural language inference (nli) in requirements engineering tasks,” in 2024 IEEE 32nd International Requirements Engineering Conference (RE). IEEE, 2024, pp. 103–115

    2024

  14. [22]

    Transforming question answering datasets into natural language inference datasets,

    D. Demszky et al. , “Transforming question answering datasets into natural language inference datasets,” arXiv preprint arXiv:1809.02922 , 2018

    2018 arXiv

  15. [23]

    Semantic diversity in dialogue with natural language inference,

    K. Stasaski et al., “Semantic diversity in dialogue with natural language inference,” in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, J...

    2022

  16. [24]

    Large language models for text classification: From zero-shot learning to fine-tuning,

    Y . Chae et al. , “Large language models for text classification: From zero-shot learning to fine-tuning,” Open Science Foundation , vol. 10, 2023

    2023

  17. [25]

    An extended model of natural logic,

    B. MacCartney et al. , “An extended model of natural logic,” in Proc. of the eight international conference on computational semantics , 2009, pp. 140–156

    2009

  18. [26]

    On the privacy of mental health apps: An empirical investigation and its implications for app development,

    L. H. Iwaya et al., “On the privacy of mental health apps: An empirical investigation and its implications for app development,” Empirical Software Engineering, vol. 28, no. 1, p. 2, 2023

    2023

  19. [27]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Pro- ceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolo...

    2019

  20. [28]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu et al., “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692 , 2019

    1907 arXiv

  21. [29]

    Deberta: Decoding-enhanced bert with disentangled attention,

    P. He et al. , “Deberta: Decoding-enhanced bert with disentangled attention,” in 2021 International Conference on Learning Representations , May 2021, under review. [On- line]. Available: https://www.microsoft.com/en-us/research/publication/ deberta-decoding-enhanced-bert-with...

    2021

  22. [30]

    Debertav3: Improving deberta using electra- style pre-training with gradient-disentangled embedding sharing,

    He, Pengcheng et al. , “Debertav3: Improving deberta using electra- style pre-training with gradient-disentangled embedding sharing,” arXiv preprint arXiv:2111.09543, 2021

    2021 arXiv

  23. [31]

    Transformers: State-of-the-art natural language process- ing,

    T. Wolf et al., “Transformers: State-of-the-art natural language process- ing,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations . Online: Association for Computational Linguistics, Oct. 2020, pp. 38–45

    2020

  24. [32]

    A broad-coverage challenge corpus for sentence un- derstanding through inference,

    A. Williams et al., “A broad-coverage challenge corpus for sentence un- derstanding through inference,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) . A...

    2018

  25. [33]

    Adversarial NLI: A new benchmark for natural language understanding,

    Y . Nie et al., “Adversarial NLI: A new benchmark for natural language understanding,” in Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics . Association for Computational Linguistics, Jul. 2020, pp. 4885–4901

    2020

  26. [34]

    The fever2. 0 shared task,

    J. Thorne et al. , “The fever2. 0 shared task,” in Proc. of the second workshop on Fact Extraction and VERification , 2019, pp. 1–6

    2019

  27. [35]

    Evaluating semantic accuracy of data-to-text generation with natural language inference,

    O. Du ˇsek et al., “Evaluating semantic accuracy of data-to-text generation with natural language inference,” in Proceedings of the 13th Interna- tional Conference on Natural Language Generation . Association for Computational Linguistics, Dec. 2020, pp. 131–137

    2020

  28. [36]

    Unleashing the potential of prompt engineering for large language models,

    B. Chen et al., “Unleashing the potential of prompt engineering for large language models,” Patterns, 2025

    2025

  29. [37]

    Llama: Open and efficient foundation language models,

    H. Touvron et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971 , 2023

    2023 arXiv

  30. [38]

    Revisiting sentiment analysis for software engineering in the era of large language models,

    T. Zhang et al., “Revisiting sentiment analysis for software engineering in the era of large language models,” ACM Trans. Softw. Eng. Methodol., vol. 34, no. 3, Feb. 2025

    2025

  31. [39]

    An empirical study of the non-determinism of chatgpt in code generation,

    S. Ouyang et al., “An empirical study of the non-determinism of chatgpt in code generation,” ACM Trans. Softw. Eng. Methodol. , vol. 34, no. 2, Jan. 2025

    2025

  32. [40]

    A comprehensive capability analysis of gpt-3 and gpt-3.5 series models,

    J. Ye et al., “A comprehensive capability analysis of gpt-3 and gpt-3.5 series models,” arXiv preprint arXiv:2303.10420 , 2023

    2023 arXiv

  33. [41]

    Gpt-4o mini: Advancing cost-efficient intelli- gence,

    OpenAI, “Gpt-4o mini: Advancing cost-efficient intelli- gence,” Jul 2024. [Online]. Available: https://openai.com/index/ gpt-4o-mini-advancing-cost-efficient-intelligence

    2024

  34. [42]

    Want to reduce labeling cost? GPT-3 can help,

    S. Wang et al. , “Want to reduce labeling cost? GPT-3 can help,” in Findings of the Association for Computational Linguistics: EMNLP

  35. [43]

    Machine learning in automated text categorization,

    F. Sebastiani, “Machine learning in automated text categorization,” ACM computing surveys (CSUR) , vol. 34, no. 1, pp. 1–47, 2002

    2002

  36. [44]

    Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit

    J. Cohen, “Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.” Psychological bulletin , vol. 70, no. 4, p. 213, 1968

    1968

  37. [45]

    Text categorization with support vector machines: Learn- ing with many relevant features,

    T. Joachims, “Text categorization with support vector machines: Learn- ing with many relevant features,” in European conference on machine learning. Springer, 1998, pp. 137–142

    1998

  38. [46]

    Support vector machines for text categorization,

    A. Basu et al. , “Support vector machines for text categorization,” in 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the . IEEE, 2003, pp. 7–pp

    2003

  39. [47]

    A taxonomy of privacy,

    D. J. Solove, “A taxonomy of privacy,” U. Pa. l. Rev., vol. 154, p. 477, 2005

    2005

  40. [48]

    Privacy-enhancing technologies,

    Y . Wang, “Privacy-enhancing technologies,” in Handbook of research on social and organizational liabilities in information security . IGI Global, 2009, pp. 203–227

    2009

  41. [49]

    Understanding interobserver agreement: the kappa statistic,

    A. J. Viera et al. , “Understanding interobserver agreement: the kappa statistic,” Fam med, vol. 37, no. 5, pp. 360–363, 2005

    2005

  42. [50]

    Exploring the capability of chatgpt to reproduce human labels for social computing tasks,

    Y . Zhu et al., “Exploring the capability of chatgpt to reproduce human labels for social computing tasks,” in International Conference on Advances in Social Networks Analysis and Mining . Springer, 2024, pp. 13–22

    2024

  43. [51]

    The parrot dilemma: Human-labeled vs. LLM- augmented data in classification tasks,

    A. G. Møller et al. , “The parrot dilemma: Human-labeled vs. LLM- augmented data in classification tasks,” in Proceedings of the 18th Con- ference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers) . Association for Computational ...

    2024

  44. [52]

    Narratives: the unforeseen influencer of privacy concerns,

    Z. S. Li, M. Sihag, N. N. Arony, J. B. Junior, T. Phan, N. Ernst, and D. Damian, “Narratives: the unforeseen influencer of privacy concerns,” in 2022 IEEE 30th International Requirements Engineering Conference (RE). IEEE, 2022, pp. 127–139

    2022

  45. [53]

    The llama 3 herd of models,

    A. Dubey et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  46. [54]

    Introducing the technology innovation institute’s falcon 3 making advanced ai accessible and available to everyone, everywhere,

    E. Almazrouei et al. , “Introducing the technology innovation institute’s falcon 3 making advanced ai accessible and available to everyone, everywhere,” Nov 2023. [Online]. Available: https: //falconllm.tii.ae/falcon-models.html

    2023

  47. [55]

    Mistral 7b,

    A. Q. Jiang et al. , “Mistral 7b,” Sep 2023. [Online]. Available: https://mistral.ai/news/announcing-mistral-7b

    2023

  48. [56]

    Finding the needle in a haystack: On the automatic identification of accessibility user reviews,

    E. A. AlOmar et al., “Finding the needle in a haystack: On the automatic identification of accessibility user reviews,” in Proc. of the 2021 CHI conference on human factors in computing systems , 2021, pp. 1–15

    2021

  49. [57]

    Large language models for software engineering: A systematic literature review,

    X. Hou et al. , “Large language models for software engineering: A systematic literature review,”ACM Trans. Softw. Eng. Methodol., vol. 33, no. 8, Dec. 2024

    2024

  50. [58]

    Leveraging gpt-like llms to automate issue labeling,

    G. Colavito, F. Lanubile, N. Novielli, and L. Quaranta, “Leveraging gpt-like llms to automate issue labeling,” in 2024 IEEE/ACM 21st In- ternational Conference on Mining Software Repositories (MSR) . IEEE, 2024, pp. 469–480

    2024

  51. [59]

    Advancing requirements engineering through generative ai: Assessing the role of llms,

    C. Arora, J. Grundy, and M. Abdelrazek, “Advancing requirements engineering through generative ai: Assessing the role of llms,” in Generative AI for Effective Software Development . Springer, 2024, pp. 129–148

    2024

  52. [60]

    Exploring the efficacy of chatgpt in generating requirements: An experimental study,

    L. Bencheikh and N. H ¨oglund, “Exploring the efficacy of chatgpt in generating requirements: An experimental study,” 2023

    2023

  53. [61]

    Investigating chatgpt’s potential to assist in requirements elicitation processes,

    K. Ronanki, C. Berger, and J. Horkoff, “Investigating chatgpt’s potential to assist in requirements elicitation processes,” in 2023 49th Euromi- cro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, 2023, pp. 354–361

    2023

  54. [62]

    Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design,

    J. White, S. Hays, Q. Fu, J. Spencer-Smith, and D. C. Schmidt, “Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design,” in Generative ai for effective software development. Springer, 2024, pp. 71–108

    2024

  55. [63]

    Autonomous agents in software development: A vi- sion paper,

    Z. Rasheed et al., “Autonomous agents in software development: A vi- sion paper,” in International Conference on Agile Software Development. Springer Nature Switzerland Cham, 2024, pp. 15–23

    2024

  56. [64]

    Chatgpt: A study on its utility for ubiquitous software engineering tasks,

    G. Sridhara et al., “Chatgpt: A study on its utility for ubiquitous software engineering tasks,” arXiv preprint arXiv:2305.16837 , 2023

    2023 arXiv

  57. [65]

    Using chatgpt in software requirements engineering: A comprehensive review,

    N. Marques, R. R. Silva, and J. Bernardino, “Using chatgpt in software requirements engineering: A comprehensive review,” vol. 16, no. 6, p. 180, 2024

    2024

  58. [2021]

    2021, pp

    Association for Computational Linguistics, Nov. 2021, pp. 4195– 4205

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.