Pith. sign in

REVIEW 4 major objections 6 minor 32 references

Assessing Reliability of BERT-Based Models on Question Answering Tasks

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Monte Carlo dropout gives BERT QA models a usable reliability signal: enabling it at 10% during inference leaves scores statistically unchanged while exposing per-sample variability, and paraphrasing shows reliability rankings depend on…

desk verdict A reproducible empirical benchmark whose headline rankings don't match its own tables, and whose key correlation evidence is inflated by averaging over MCD samples. read the letter →

arxiv 2608.10806 v1 pith:3OW5NLRG submitted 2026-08-11 cs.CL

Pooja Yadav , Priyanka Harjule , Basant Agarwal , Marko Robnik Šikonja This is my paper · ORCID
classification cs.CL
keywords reliabilityestimationMonteCarlodropoutquestionansweringBERTvariantspredictionstabilityparaphrasingperturbationSQuADQuAC
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the reliability of BERT-family question-answering models can be measured by perturbing the model and the input rather than by accuracy alone. Reliability is defined as the stability of answers when Monte Carlo dropout is enabled at inference and when questions are paraphrased, with cosine similarity and $\mathrm{F_1}$ against the gold answer as the scoring functions. On SQuAD and QuAC, the authors report that enabling Monte Carlo dropout leaves accuracy and prediction rankings statistically intact, which they read as validating dropout as a reliability metric. They also report that reliability and accuracy come apart: RoBERTa is generally the most consistent, DistilBERT is stable under internal variation, ALBERT is competitive under paraphrasing but highly variable on answerable questions, and BERT-Base is the weakest overall. If the argument holds, model selection for real QA systems should weigh stability under perturbation as heavily as benchmark accuracy.

What carries the argument

The load-bearing mechanism is repeated stochastic inference: each question is passed through the model 50 times with a 10% dropout rate enabled, and the mean and standard deviation of cosine similarity and $\mathrm{F_1}$ scores against the gold answer summarize the answer distribution. For input perturbation, a pre-trained paraphrasing model rewrites each question and a cosine filter keeps paraphrases within a 0.75–0.98 similarity band before the paraphrased answer is scored against the gold answer. The operating point was chosen after a sweep: below 5% dropout yields no stochastic spread, while at or above 15% models start emitting blank responses on answerable questions. The paper then uses the correlation between baseline and dropout-era scores as evidence that dropout does not disrupt inference dynamics, and uses Welch's t-test and Bonferroni-corrected Wilcoxon signed-rank tests to decide which model differences are statistically meaningful.

What would settle it

A calibration study would settle it: take a stratified sample of SQuAD and QuAC predictions, have annotators label each answer as correct, partially correct, or incorrect, and compare those labels with the cosine-similarity-to-gold scores and the 0.95 threshold. If many predictions above the threshold are judged wrong by humans, or many below it are judged right, the reliability rankings do not measure what they claim. A second check is to rerun the Monte Carlo dropout protocol at several dropout rates and random seeds and see whether the model-level ranking by variance is stable across runs.

Watch

Extended reading notes

Core claim

The central claim is that a single hyperparameter change—keeping dropout active during prediction—turns a QA model into a distribution over answers whose spread is a usable reliability signal. With a 10% dropout rate and 50 stochastic samples per question, the paper finds high per-question correlation between baseline and Monte-Carlo-dropout scores (coefficients from about 0.857 to 0.946 across models and datasets), while the standard deviation of cosine similarity across samples exposes which models are stable. On this measure RoBERTa and DistilBERT come out most consistent, ALBERT shows large variance on answerable questions, and BERT-Base is middling; paraphrasing the input changes the ranking, with ALBERT leading on QuAC and RoBERTa leading on SQuAD. The paper takes these results to show both that accuracy and reliability are partly independent and that Monte Carlo dropout is a valid, non-disruptive way to estimate reliability for QA.

Load-bearing premise

The load-bearing premise is that cosine similarity between the vector representation of the predicted answer and the vector representation of the gold answer measures whether an answer is correct and stable; the 0.95 threshold used to define accuracy is never calibrated against human judgments, and the human evaluation only re-examines automatically incorrect responses.

Editorial extensions

If this is right

  • Enabling Monte Carlo dropout at 10% during QA inference can serve as a reliability check without retraining or materially changing test-time accuracy.
  • RoBERTa and DistilBERT are the safer choices among the four when answer consistency matters on standard QA benchmarks.
  • Accuracy and reliability are partly decoupled: a high-accuracy model such as RoBERTa still shows detectable variance, while ALBERT's low answerable accuracy on QuAC coexists with competitive paraphrase stability.
  • Reliability rankings are dataset-specific; on QuAC, where answerable accuracy is much lower, model differences under paraphrasing reverse the SQuAD ordering.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not in the paper: the same protocol could be used as a retraining-free stress test during model development, inserting a stability screen before deployment to reject models whose answer distributions widen under dropout.
  • Not in the paper: if cosine similarity to the gold answer is accepted as the correctness signal, the framework is a cheap proxy for uncertainty estimation, and a natural extension is to test whether the 10% dropout protocol transfers to generative QA models, where the paper's appendix suggests accuracy may be too low for the metric to be informative.
  • Not in the paper: the 0.75–0.98 paraphrase filter is a tunable knob, and reliability rankings should be checked across several similarity bands before treating them as fixed model properties.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This manuscript proposes a framework for assessing the reliability of four BERT-based extractive QA models (BERT-Base, RoBERTa, DistilBERT, ALBERT) on SQuAD 2.0 and QuAC. Reliability is operationalized in two ways: stability under Monte Carlo dropout (MCD), quantified by the mean and standard deviation of per-instance cosine-similarity and F1 scores over N=50 stochastic forward passes, and stability under input paraphrasing filtered to an S-BERT cosine interval of 0.75–0.98. The paper defines a semantic exact-match threshold of 0.95, computes correlations between unperturbed scores and MCD-averaged scores, runs Welch t-tests and Bonferroni-corrected Wilcoxon tests, reports a human evaluation with Fleiss' kappa, and provides qualitative error tables. The central claims are that enabling MCD during prediction does not disrupt inference dynamics, thereby validating MCD as a reliability metric, and that RoBERTa maintains higher reliability while ALBERT and DistilBERT exhibit significant inconsistencies.

Significance. The study addresses a genuine gap: accuracy metrics alone do not capture the consistency of QA models under stochastic inference and input reformulation, and the paper ships reproducible code, a clear algorithmic protocol (Algorithms 1 and 2), a systematic dropout-rate sweep, Bonferroni-corrected pairwise tests, and inter-annotator agreement analysis. These are substantive strengths. Nevertheless, the key statistical evidence for the non-disruptive claim (Table 5) is an averaging artifact, the headline reliability ranking contradicts the paper's own Tables 3–5 and its Welch test, and the 0.95 accuracy threshold is uncalibrated. The limitations acknowledged in Section 6 (encoder-only models, QA-only scope, two datasets, one perturbation approach per type) are reasonable and do not by themselves block the claims; the blocking issues are statistical and internal-consistency problems that can in principle be fixed within the manuscript's scope.

major comments (4)
  1. [Section 4.2, Table 5] The correlations used to validate MCD are computed between the unperturbed per-instance score and the per-instance score averaged over N=50 stochastic samples, as the text states: 'for the perturbed model, evaluation metrics are averaged across stochastic samples to obtain a single score per input.' If the per-sample metric for input i satisfies y_ij = x_i + e_ij with zero-mean fluctuations, the average converges to x_i as N grows, so the reported coefficients (0.857–0.946) approach 1 regardless of the magnitude of the individual fluctuations e_ij. The per-instance standard deviations in Tables 3–4 (e.g., 0.0906–0.2449 on SQuAD) show that the individual samples vary substantially, and Section 4.1 itself states that the variance bands widen at 10% dropout. The abstract's claim that 'enabling MCD during prediction does not disrupt inference dynamics' therefore does not follow from Table 5. Please report correlations between unperturbed scores and individual stochastic samples (not averages), or use the per-instance standard deviation directly as the stability measure, or provide a span-level agreement analysis across the 50 samples.
  2. [Abstract and Section 4.2] The headline claim that 'RoBERTa maintains higher reliability, whereas AlBERT and DistilBERT exhibit significant inconsistencies' is contradicted by the manuscript's own results. Table 3 (Total rows) shows DistilBERT with lower standard deviations than RoBERTa on SQuAD (cosine 0.1131 vs 0.1543; F1 0.1285 vs 0.1772), and Table 4 shows the same pattern on QuAC (cosine 0.1470 vs 0.1761; F1 0.1360 vs 0.1761). Table 5 shows RoBERTa with the lowest unperturbed/perturbed correlations of the four models on both datasets and both metrics, and the Welch test on QuAC yields p = 0.4547, i.e., no significant difference between RoBERTa and DistilBERT. Section 7 even concludes that 'DistilBERT is more stable in handling internal configuration variations.' The abstract and the Section 4.2 narrative must be rewritten so that the reliability ranking is consistent with the reported standard deviations, correlations, and significance tests.
  3. [Section 4.2, MCD Accuracy; Section 4.3] Accuracy is defined by a semantic similarity threshold of 0.95 between predicted and ground-truth answer embeddings, but no evidence links this threshold to human judgments of answer correctness. The human evaluation described in Section 4.3 explicitly re-labels only responses 'automatically classified as incorrect' and covers just two of the four models, so it cannot calibrate the 0.95 cutoff or validate the positive classifications that underlie the Acc. and MCD Acc. columns of Tables 3 and 4. Because the near-equality of Acc. and MCD Acc. is offered as additional evidence that stochastic inference does not degrade performance, this unvalidated cutoff is load-bearing. The fix is either a calibration analysis (e.g., a threshold sweep against human judgments, or against lexical exact match) or removal of the semantic-exact-match accuracy framing in favor of the score distributions.
  4. [Sections 3.2 and 4, Tables 3–6] The manuscript does not state which fine-tuned checkpoints or fine-tuning datasets were used for each model, nor which evaluation subsets produced Tables 3–6. This matters because the QuAC results (e.g., RoBERTa total accuracy of about 21% in Table 4) are consistent with evaluating a SQuAD-fine-tuned model on an out-of-domain dataset, which would change the interpretation of the 'low accuracy' dataset comparison that motivates Objective 3 in Section 1. Please specify the exact model identifiers used (e.g., Hugging Face checkpoint names), the fine-tuning data for each model, and the number of evaluation instances per dataset, so that the accuracy–reliability comparison is reproducible and interpretable.
minor comments (6)
  1. [Section 1] The sentence 'This paper is organized into five sections' is immediately followed by a list that runs from Section 2 through Section 7; the count should be corrected.
  2. [Abstract and throughout] The model name is spelled inconsistently ('AlBERT' in the abstract, 'ALBERT' elsewhere, and 'alBERT' in some passages); use a single spelling throughout.
  3. [Section 3.5] The rationale for choosing N=50 ('experiments... demonstrated that there is no variation in the model's performance on increasing the number of samples') is asserted without supporting evidence; please include the saturation curve or a table showing mean and variance as a function of N.
  4. [Section 4.2] The Welch test p-value is reported as '0.0'; report it as p < 0.0001 (or an inequality with a specific bound) rather than an exact zero, which is an artifact of floating-point precision.
  5. [Section 5, Tables 7 and 8] The layout of the error-analysis tables makes it difficult to map each question to the answer columns and to see which output is being compared to which ground truth; please add column headings per model, bold the divergent answers, and state the criterion used to label an output as a hallucination.
  6. [Section 3.1] The description of QuAC mentions context-dependent conversational questions but does not state how the dialogue history was encoded (or whether it was discarded) when the model was applied; a sentence on this is needed for reproducibility, since it can materially affect QuAC accuracy.

Circularity Check

1 steps flagged · score 6.0 of 10

Table 5's MCD validation is circular: perturbed scores are averaged over N=50 samples, so high correlations with unperturbed scores are guaranteed by the law of large numbers rather than by prediction stability.

  1. self definitional [Section 4.2, Table 5 (and Section 3.5 for the choice of N=50)]
    "For the perturbed model, evaluation metrics are averaged across stochastic samples to obtain a single score per input. This is done for the entire dataset, and the correlation between the resulting scores is analyzed to evaluate the effect of perturbations on model predictions. ... The empirical results consistently show high correlation coefficients across all models and datasets, indicating that dropout has a minimal impact on prediction stability."

    The paper's reliability definition is output variance ('Reliability is quantified by analyzing the variance in outputs'), yet Section 4.2 defines the 'perturbed' score as the mean of N=50 stochastic samples. If s_ij = x_i + e_ij with zero-mean noise, the averaged score has noise variance σ_e²/N and Corr(x_i, \bar s_i) = 1/sqrt(1+σ_e²/(N σ_x²)). With N=50 this approaches 1 regardless of per-sample noise; reported correlations of 0.86–0.95 can coexist with σ_e several times σ_x (0.90 implies σ_e ≈ 3.4 σ_x). N=50 was itself selected because larger N 'produced semantically similar responses' (Sec. 3.5). Hence Table 5's high correlations are an artifact of averaging, not evidence that 'enabling MCD during prediction does not disrupt inference dynamics.'

full rationale

The paper is not globally circular: it benchmarks four BERT variants on public SQuAD/QuAC data, and most of its per-model reliability claims (Tables 3, 4) rest on directly reported standard deviations over stochastic samples, which are independent evidence and can be checked from the released code. The self-citations to Miok et al. [16, 17] are used as motivation and as a weak consistency check for the 10% dropout rate, not as the sole justification, so no load-bearing self-citation chain is present. The circularity is localized to the central validation of MCD as non-disruptive: Section 4.2 measures the effect of dropout by correlating unperturbed scores with scores that are themselves averaged over 50 stochastic samples. Because averaging reduces noise variance by a factor of 50, the correlation is high by construction even when individual stochastic predictions are unstable, and N=50 was selected using the same stability observation. The paper's own definition of reliability as output variance makes this a case where the validation statistic suppresses the quantity being validated. This does not invalidate the empirical std-based findings, but it does mean the abstract's claim that 'enabling MCD during prediction does not disrupt inference dynamics' is not established by Table 5. Score 6 reflects one central reduction-by-construction while the rest of the framework retains independent content.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities or forces. Its free parameters are operational thresholds chosen on the evaluation data. The axioms are standard assumptions for embedding-based semantic evaluation, with the paraphrase interval being the most paper-specific.

free parameters (4)
  • Dropout rate for MC dropout = 0.10
    Selected after sweeping 0-35% because 0-5% gave too little variance and >=15% produced blank answers (Section 4.1).
  • Number of stochastic samples N = 50
    Chosen as a mid-range value after observing no variation across 10-100 samples (Section 3.5).
  • Paraphrase similarity interval = [0.75, 0.98]
    Chosen via a 10% sample of SQuAD answerable questions to balance semantic preservation and lexical variation (Section 3.5, Table 2).
  • Semantic exact-match threshold = 0.95
    Used to convert cosine similarity into accuracy; no theoretical or empirical calibration is provided (Section 4.2).
assumptions (4)
  • domain assumption Pretrained transformer models used as-is provide meaningful answer representations for QA.
    The evaluation treats BERT, RoBERTa, DistilBERT, and ALBERT as ready-made QA systems without questioning whether their output distributions are well-calibrated (Section 3.2).
  • domain assumption Cosine similarity between answer embeddings is a valid measure of semantic correctness.
    The paper uses cosine similarity between predicted and ground-truth answers as the primary reliability metric (Section 3.3.1).
  • domain assumption Monte Carlo dropout samples represent meaningful model uncertainty.
    The method assumes that stochastic forward passes with dropout reveal reliability, rather than merely adding noise (Section 3.4).
  • ad hoc to paper Paraphrases with S-BERT cosine similarity between 0.75 and 0.98 preserve meaning while introducing useful variation.
    This interval is justified only by a small internal validation on a 10% sample of SQuAD (Section 3.5, Table 2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Assessing Reliability of BERT-Based Models on Question Answering Tasks." pith.science (2026). https://pith.science/paper/3OW5NLRG

@misc{pith2026260810806,
  author       = {Pith},
  title        = {Pith review of: Assessing Reliability of BERT-Based Models on Question Answering Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3OW5NLRG}},
  note         = {Machine review of arXiv:2608.10806}
}
read the original abstract

Reliability estimation of large language models is in many cases as crucial as their accuracy, as reliable models are more trustworthy, robust, and suitable for practical applications. Recent advancements in natural language processing (NLP), particularly those based on transformer architectures, have significantly accelerated progress across various NLP tasks. This study focuses on the reliability of transformer-based question answering (QA) models, specifically BERT models and its variants (RoBERTa, ALBERT, DistilBERT). These encoder-only pretrained transformers have demonstrated remarkable accuracy in QA tasks that can be treated as classification tasks. However, their reliability remains underexplored. This study evaluates the reliability of four BERT-based models by assessing response stability under two conditions: (1) internal model variations induced via Monte Carlo Dropout (MCD) and (2) input perturbations through paraphrasing. Using the SQuAD and QuAC datasets, we investigate how dropout rates affect prediction consistency and whether lexical changes impact answer stability. Our findings reveal that RoBERTa maintains higher reliability, whereas AlBERT and DistilBERT exhibit significant inconsistencies. Statistical analyses confirm that enabling MCD during prediction does not disrupt inference dynamics, validating its effectiveness as a reliability metric. These findings underscore the importance of evaluating both accuracy and stability in QA models to ensure stability in real-world applications.

Figures

Figures reproduced from arXiv: 2608.10806 by the authors.

Figure 1
Figure 1. Visual Representation of the SQuAD Dataset. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Visual Representation of the QuAC Dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Analyzing Output Stability of SQuAD Under Varying Dropout Per [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Analyzing Output Stability of QuAC Under Varying Dropout Per [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Comparison of Cosine Similarity and F1 Scores for SQuAD Across BERT Models [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 7
Figure 7. Figure 7: Pairwise Wilcoxon signed-rank test heatmaps with [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 23 canonical work pages

  1. [1]

    A deep network model for paraphrase detection in short text 29 messages.Information Processing & Management, 54(6):922–937, 2018

    Basant Agarwal, Heri Ramampiaro, Helge Langseth, and Massimiliano Ruocco. A deep network model for paraphrase detection in short text 29 messages.Information Processing & Management, 54(6):922–937, 2018. doi: https://doi.org/10.1016/j.ipm.2018.06.005

    work page doi:10.1016/j.ipm.2018.06.005 2018

  2. [2]

    Evalu- ating ChatGPT as a question answering system: A comprehensive analysis and comparison with existing models.arXiv preprint arXiv:2312.07592, 2023

    Hossein Bahak, Farzaneh Taheri, Zahra Zojaji, and Arefeh Kazemi. Evalu- ating ChatGPT as a question answering system: A comprehensive analysis and comparison with existing models.arXiv preprint arXiv:2312.07592, 2023

    arXiv 2023

  3. [3]

    Understanding dropout.Advances in neural information processing systems, 26, 2013

    Pierre Baldi and Peter J Sadowski. Understanding dropout.Advances in neural information processing systems, 26, 2013

    work page 2013

  4. [4]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Ka- plan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  5. [5]

    QuAC: Question answering in context

    Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. QuAC: Question answering in context. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, edi- tors,Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184

    work page 2018

  6. [6]

    BERT: Pre-training of deep bidirectional transformers for language un- derstanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language un- derstanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186, 2019

    work page 2019

  7. [7]

    A review on different methods of paraphras- ing

    Ashwini Gadag and BM Sagar. A review on different methods of paraphras- ing. In2016 International conference on electrical, electronics, communica- tion, computer and optimization techniques (ICEECCOT), pages 188–191, 2016

    work page 2016

  8. [8]

    Natural language based analysis of SQuAD: An analytical approach for BERT.Expert Systems with Applications, 195:116592, 2022

    Zekeriya Anil Guven and Murat Osman Unalir. Natural language based analysis of SQuAD: An analytical approach for BERT.Expert Systems with Applications, 195:116592, 2022

    work page 2022

Show all 32 references
  1. [9]

    A survey on hallucination in large language models: Principles, tax- onomy, challenges, and open questions.ACM Transactions on Information Systems, 43(2):1–55, 2025

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Hao- tian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, tax- onomy, challenges, and open questions.ACM Transactions on Informa...

    2025

  2. [10]

    AlBERT: A lite BERT for self-supervised learning of language representations.International Conference on Learning Representations., 2020

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. AlBERT: A lite BERT for self-supervised learning of language representations.International Conference on Learning Representations., 2020

    2020

  3. [11]

    The measurement of observer agree- ment for categorical data.biometrics, pages 159–174, 1977

    J Richard Landis and Gary G Koch. The measurement of observer agree- ment for categorical data.biometrics, pages 159–174, 1977. 30

    1977

  4. [12]

    Ensemble AlBERT on SQuAD 2.0

    Shilun Li, Renee Li, and Veronica Peng. Ensemble AlBERT on SQuAD 2.0. CoRR, abs/2110.09665, 2021. URLhttps://arxiv.org/abs/2110.09665

    2021 arXiv

  5. [13]

    How can rec- ommender systems benefit from large language models: A survey.ACM Transactions on Information Systems, 43(2):1–47, 2025

    Jianghao Lin, Xinyi Dai, Yunjia Xi, Weiwen Liu, Bo Chen, Hao Zhang, Yong Liu, Chuhan Wu, Xiangyang Li, Chenxu Zhu, et al. How can rec- ommender systems benefit from large language models: A survey.ACM Transactions on Information Systems, 43(2):1–47, 2025

    2025

  6. [14]

    RoBERTa: A robustly optimized BERT pretraining approach.arXiv preprint arXiv:1907.11692, 2019

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach.arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  7. [15]

    Le- dlcm: Decoupled learner and course modeling with large language mod- els for enhanced course recommendation.Knowledge-Based Systems, page 115135, 2025

    Jinjin Ma, Zhuo Zhao, Zhiwen Xie, Yi Zhang, and Guangyou Zhou. Le- dlcm: Decoupled learner and course modeling with large language mod- els for enhanced course recommendation.Knowledge-Based Systems, page 115135, 2025

    2025

  8. [16]

    Prediction uncertainty estimation for hate speech classi- fication

    Kristian Miok, Dong Nguyen-Doan, Blaˇ zˇSkrlj, Daniela Zaharie, and Marko Robnik-ˇSikonja. Prediction uncertainty estimation for hate speech classi- fication. InStatistical Language and Speech Processing: 7th International Conference, SLSP 2019, Proceedings 7, pages 286–298, 2019

    2019

  9. [17]

    To BAN or not to BAN: Bayesian attention networks for reliable hate speech detection.Cognitive Computation, 14(1):353–371, 2022

    Kristian Miok, Blaˇ zˇSkrlj, Daniela Zaharie, and Marko Robnik- ˇSikonja. To BAN or not to BAN: Bayesian attention networks for reliable hate speech detection.Cognitive Computation, 14(1):353–371, 2022

    2022

  10. [18]

    Extractive text summarization.International Journal of Current Engineering and Technology, 4(2), 2014

    Namita Mittal, Basant Agarwal, Himanshu Mantri, Rahul Kumar Goyal, and Manoj Kumar Jain. Extractive text summarization.International Journal of Current Engineering and Technology, 4(2), 2014

    2014

  11. [19]

    Transformer models used for text- based question answering systems.Applied Intelligence, 53(9):10602–10635, 2023

    Khalid Nassiri and Moulay Akhloufi. Transformer models used for text- based question answering systems.Applied Intelligence, 53(9):10602–10635, 2023

    2023

  12. [20]

    Comparative analysis of state-of-the-art Q&A models: BERT, RoBERTa, DistilBERT, and ALBERT on SQuAD v2 dataset

    Cem ¨Ozkurt. Comparative analysis of state-of-the-art Q&A models: BERT, RoBERTa, DistilBERT, and ALBERT on SQuAD v2 dataset. ADBA Chaos and Fractals, 1(1):19–30, 2024. doi: https://doi.org/10. 69882/adba.chf.2024073

    2024

  13. [21]

    A com- parative study of transformer-based language models on extractive question answering.CoRR, abs/2110.03142, 2021

    Kate Pearce, Tiffany Zhan, Aneesh Komanduri, and Justin Zhan. A com- parative study of transformer-based language models on extractive question answering.CoRR, abs/2110.03142, 2021. URLhttps://arxiv.org/abs/ 2110.03142

    2021 arXiv

  14. [22]

    Measuring relia- bility of large language models through semantic consistency.arXiv preprint arXiv:2211.05853, 2022

    Harsh Raj, Domenic Rosati, and Subhabrata Majumdar. Measuring relia- bility of large language models through semantic consistency.arXiv preprint arXiv:2211.05853, 2022. 31

    2022 arXiv

  15. [23]

    Semantic consistency for assuring reliability of large language models.arXiv preprint arXiv:2308.09138, 2023

    Harsh Raj, Vipul Gupta, Domenic Rosati, and Subhabrata Majumdar. Semantic consistency for assuring reliability of large language models.arXiv preprint arXiv:2308.09138, 2023

    2023 arXiv

  16. [24]

    SQuAD: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. InPro- ceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392. doi: 10.18653/v1/D16-1264

    2016 doi

  17. [25]

    Comparative analysis of transformer based models for question answering

    Anchal Rawat and Surender Singh Samant. Comparative analysis of transformer based models for question answering. In2022 2nd Interna- tional Conference on Innovative Sustainable Computational Technologies (CISCT), pages 1–6, 2022

    2022

  18. [26]

    Llm4rec: a comprehensive sur- vey on the integration of large language models in recommender sys- tems—approaches, applications and challenges.Future Internet, 17(6):252, 2025

    Sarama Shehmir and Rasha Kashef. Llm4rec: a comprehensive sur- vey on the integration of large language models in recommender sys- tems—approaches, applications and challenges.Future Internet, 17(6):252, 2025

    2025

  19. [27]

    How does BERT answer questions? A layer-wise analysis of transformer representations

    Betty Van Aken, Benjamin Winter, Alexander L¨ oser, and Felix A Gers. How does BERT answer questions? A layer-wise analysis of transformer representations. InProceedings of the 28th ACM international conference on information and knowledge management, pages 1823–1832, 2019

    2019

  20. [28]

    Attention is all you need.Advances in Neural Information Processing Systems, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in Neural Information Processing Systems, 2017

    2017

  21. [29]

    Assessing factual reliability of large language model knowledge

    Weixuan Wang, Barry Haddow, Alexandra Birch, and Wei Peng. Assessing factual reliability of large language model knowledge. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume ...

    2024

  22. [30]

    A survey on large language models for recommendation.World Wide Web, 27(5):60, 2024

    Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, et al. A survey on large language models for recommendation.World Wide Web, 27(5):60, 2024

    2024

  23. [31]

    Paraphrasing for style

    Wei Xu, Alan Ritter, William B Dolan, Ralph Grishman, and Colin Cherry. Paraphrasing for style. InProceedings of COLING 2012, pages 2899–2914, 2012

    2012

  24. [32]

    Pegasus: Pre- training with extracted gap-sentences for abstractive summarization

    Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. Pegasus: Pre- training with extracted gap-sentences for abstractive summarization. In International conference on machine learning, pages 11328–11339. PMLR, 2020. 32

    2020

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.