Pith. sign in

REVIEW 4 major objections 6 minor 34 references

Question: How do Large Language Models perform on the Question Answering tasks? Answer:

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper shows that fine-tuned small models outperform prompt-only LLMs on in-distribution question answering, while a 70B LLaMA wins on most out-of-distribution sets.

desk verdict Useful SQuAD2 comparison with a genuinely new single-inference prompt and a promising-but-under-evidenced OOD claim; deserves review with revisions. read the letter →

arxiv 2412.12893 v1 pith:IRDXWI6E submitted 2024-12-17 cs.CL

Kevin Fischer , Darren Fürst , Sebastian Steindl , Jakob Lindner , Ulrich Schäfer This is my paper · ORCID
classification cs.CL
keywords questionansweringSQuAD2largelanguagemodelsfine-tunedout-of-distributiongeneralizationpromptingstrategyLevenshteindistancechain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether out-of-the-box instruction-following large language models can replace smaller, task-specialized models in question answering. It compares six LLMs with three fine-tuned models on SQuAD2 and finds that the fine-tuned models still win in-distribution: RoBERTa reaches 82.43 F1 versus 73.68 for the best LLM, LLaMA-3.1-70B. On five additional QA datasets used without fine-tuning, however, the same LLaMA model outperforms all fine-tuned models on three of them, suggesting that large prompted models generalize better to new distributions at higher inference cost. The paper also contributes a single-inference prompt that handles answerable and unanswerable questions in one forward pass, and a Levenshtein-distance analysis of near-miss answers.

What carries the argument

The carrying mechanism is the single-inference prompting strategy: a persona prompt that tells the LLM it is a reading-comprehension assistant, asks it to 'think step-by-step' (which elicits chain-of-thought reasoning), and instructs it to reply with the exact quote from the context or the literal string 'unanswerable'. This replaces the previous double-inference approach, where the model first decides answerability and then answers in a second pass, halving the forward passes. The comparison is completed by a standard SQuAD2 evaluation pipeline plus a Levenshtein-distance extension that measures how many character edits separate a prediction from the best ground truth, and a breakdown by interrogative pronoun. For the fine-tuned models, answerability is decided by a no-answer probability threshold chosen as the mean of the best EM and best F1 thresholds.

What would settle it

Rerun the out-of-distribution evaluation on the five named datasets using the full validation sets (or a documented stratified sample) and the same prompts; if LLaMA-3.1-70B no longer beats RoBERTa on three of the five datasets in EM and F1, the paper's central generalization claim fails.

Watch

Extended reading notes

Core claim

The central empirical discovery is the two-part ranking. On the SQuAD2 validation set, the fine-tuned extractive models dominate: RoBERTa scores 79.97 EM and 82.43 F1, DistilBERT 67.89 EM and 70.17 F1, and Flan-T5 64.55 EM and 66.02 F1, while every tested LLM scores lower in EM, with LLaMA-3.1-70B the strongest LLM at 57.13 EM and 73.68 F1. When the same models are evaluated on 1000-question subsamples of five other QA datasets without any fine-tuning, LLaMA-3.1-70B beats all fine-tuned models on three of the five datasets in both EM and F1, while RoBERTa remains best on the other two. The paper interprets this as evidence that sufficiently large prompted LLMs can rival or exceed specialized models when the test distribution differs from the training distribution, at the price of much higher inference cost and a tendency toward longer, less exact answers.

Load-bearing premise

The claim that LLaMA-3.1-70B outperforms fine-tuned models on 3 of 5 out-of-distribution datasets rests on the assumption that those five datasets are genuine QA benchmarks comparable in format to SQuAD2 and that the 1000-question subsample drawn from each is representative; the paper names neither the datasets nor the sampling method.

Editorial extensions

If this is right

  • A single-inference prompt can halve the compute cost of handling unanswerable questions in QA, matching the capability that previously required two separate forward passes.
  • For in-distribution question answering, a small fine-tuned model such as RoBERTa remains the better choice in accuracy per parameter, since it outperforms all tested LLMs despite being roughly 500 times smaller than LLaMA-3.1-70B.
  • For out-of-distribution or rapidly changing domains where fine-tuning data is unavailable, a sufficiently large instruction-following LLM can be deployed zero-shot; LLaMA-3.1-70B beats every fine-tuned model on three of the five test datasets.
  • The Levenshtein-distance results show that allowing even a few character edits barely changes EM for all models, but a 20-character allowance lifts several LLMs substantially, so string-based metrics underestimate the semantic correctness of LLM answers.
  • All models perform worst on 'why' questions, indicating a common bottleneck in causal reading comprehension that is not specific to LLMs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the five out-of-distribution datasets are never named and the 1000-question sampling procedure is not described, the 3-of-5 result should be treated as provisional until it is reproduced on identifiable public benchmarks; the paper itself provides no identifiers to check representativeness.
  • The Levenshtein-distance analysis suggests a cheap testable extension: apply more aggressive model-specific post-processing to LLM outputs (e.g., stripping courtesy phrases) and measure whether the sharp EM jump at 8-10 character changes for LLaMA-3.1-70B disappears, which would indicate recoverable score.
  • A hybrid pipeline combining a large prompted LLM for answerability judgment and a small extractive model for answer spans could combine the OOD robustness of the former with the exactness of the latter; the paper does not test this.
  • The single-inference prompt might be evaluated on long-context or multi-hop QA, where the 'unanswerable' instruction could behave differently because the context itself contains partial answers that invite hallucination.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This manuscript reports an empirical comparison of three small fine-tuned models (Flan-T5, DistilBERT, RoBERTa) and six out-of-the-box instruction-tuned LLMs (GPT-4 Turbo and several LLaMA variants) on the SQuAD2 validation set using a newly proposed single-inference prompt designed to handle both answerable and unanswerable questions. The authors report standard EM/F1 scores, a no-answer EM score, a breakdown by interrogative pronoun, and a Levenshtein-distance-based analysis. They also evaluate the best-performing LLM (LLaMA-3.1-70B) and the FT-LMs on 1000 questions from each of five additional QA datasets to test out-of-distribution generalization, claiming that the LLM outperforms the FT-LMs on 3 of 5 datasets. The main SQuAD2 result is that RoBERTa achieves the best F1 (82.43) and EM (79.97), while LLaMA-3.1-70B is the best LLM (F1 73.68).

Significance. If supported, the SQuAD2 comparison provides a useful data point that fine-tuned small extractive models still lead on in-distribution QA, while a large prompted LLM can close much of the gap and may generalize better to new distributions, at higher inference cost. The paper is transparent about its SQuAD2 evaluation setup (prompt in the appendix, standard metrics, per-question-type breakdowns) and explicitly states its limitations. However, the OOD generalization claim, which is the main novel contribution, currently rests on unnamed datasets, unspecified subsampling, unquantified LLM post-processing, and no variance estimates; the claim cannot be accepted as stated.

major comments (4)
  1. [5] The five OOD datasets are never named and the procedure for selecting 1000 questions per dataset is not described, so the central claim that LLaMA-3.1-70B outperforms the FT-LMs on 3 of 5 datasets is not independently verifiable or reproducible. Please specify the dataset names and versions, the answer-format characteristics of each, and the exact sampling procedure (e.g., random seed, stratification), or provide the sampled question IDs.
  2. [5] The FT-LMs are extractive QA systems that can only return spans from the context, while the LLM is free-form. If any of the five OOD datasets contains abstractive, yes/no, or otherwise non-span answers, the FT-LMs are structurally handicapped and the 3-of-5 result may reflect format incompatibility rather than generalization. Please report the answer type of each OOD dataset and confirm that all ground-truth answers are extractable spans; if not, the comparison needs to be adjusted or the limitation acknowledged.
  3. [4.3] The hand-written regex post-processing applied to LLM predictions is described only qualitatively ('greatly improved all calculated scores') and no before/after numbers are given. Since this post-processing is applied only to LLM outputs and is model-specific, it is a potential source of bias in the comparison, including the OOD results. Please provide the exact post-processing rules, the scores before and after application for every LLM and dataset, and a sensitivity check showing that the reported rankings are stable under reasonable variations of the rules.
  4. [5] The 1000-question subsample estimates in Figs. 3-4 are reported without variance or confidence intervals, despite the sampling step and the stochastic LLM decoding (temperature 0.1). The 3-of-5 superiority claim is a point estimate; please report per-dataset bootstrap intervals or standard errors and the number of independent runs, so the robustness of the ranking can be assessed.
minor comments (6)
  1. [1] There are typographical errors throughout (e.g., 'Additonally' in Section 1, 'perfomance' in Section 2, 'Skript' in the Table 3 caption, 'T able' in several captions) that should be corrected.
  2. [3.3] The NAP threshold is chosen as the mean of the best-EM and best-F1 thresholds on the same validation split; this is a tuned parameter, and the sensitivity of the reported FT-LM scores to this choice should be stated, at least for the no-answer rows.
  3. [3.2] The claim that the proposed single-inference prompt elicits the same unanswerable-question handling as the double-inference method is not empirically tested in this paper; consider adding a direct comparison or rephrasing this as a design goal rather than an equivalence.
  4. [4.1] The statement 'The evaluation in [4] utilizes previous questions and answers on a document as additional context' cites the LLaMA 3 model card; please cite the original work that introduced this prompting practice, as [4] may not be the primary source.
  5. [4.2] Figure 2 would be clearer if the y-axis label and the normalization used for the Levenshtein allowance were defined in the caption.
  6. [2] Some references lack arXiv identifiers or URLs (e.g., references 3, 6, 8, 20, 23, 24, 25, 29); please complete the reference list for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical benchmark comparison with independent external evaluation metrics, not a derivation that reduces to its inputs.

full rationale

This paper reports a direct empirical comparison between fine-tuned models and instruction-following LLMs on SQuAD2 and on five additional QA datasets. There is no derivation chain in which an output quantity is defined in terms of the claimed result. The SQuAD2 scores are computed using the official SQuAD2 evaluation script and standard EM/F1 metrics, which are external to the models' training objectives. The OOD generalization claim ('LLaMA-3.1-70B Instruct was able to outperform the FT-LMs ... on three of the five datasets') is an empirical statement about measured scores, not a quantity fitted into the evaluation. The paper's prompt construction is an input design choice and does not encode the outcome; the Levenshtein-distance extension is an additional descriptive metric, not a result derived from itself. The threshold used for FT-LM unanswerable detection is chosen on the validation split ('the overall mean between the best-achieved F1 threshold and the best-achieved EM threshold'), which is post-hoc selection rather than circular reasoning, and it does not make the headline comparison true by construction. The main weakness is that the five OOD datasets and the 1000-question subsampling procedure are never named or described, making the 3-of-5 claim difficult to reproduce; however, that is an external-validity and reporting limitation, not circularity. No load-bearing self-citation or imported uniqueness theorem appears. Accordingly, no specific circular step can be quoted, and the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced. The empirical claims depend on benchmark assumptions and hand-chosen evaluation settings rather than derived quantities.

free parameters (2)
  • NAP threshold per FT-LM = Flan-T5 ~0.515, DistilBERT ~0.716, RoBERTa ~0.849
    Sec. 3.3 chooses the mean of the best EM and best F1 thresholds computed on the same validation split used for reporting, so FT-LM scores are partly optimized on the test set.
  • LLM decoding temperature and repetition penalty = 0.1, 1.1
    Hand-chosen in Sec. 3.1 without ablation; stochastic outputs get no error bars and the effect on answer conciseness is not measured.
assumptions (4)
  • domain assumption The five OOD QA datasets are valid out-of-distribution targets and the 1000-question subsamples are representative.
    Sec. 5 says 'five further QA datasets' but never names them or describes sampling, so the OOD comparison cannot be audited.
  • domain assumption LLMs have not memorized the test questions during pretraining.
    No contamination check is reported; GPT-4 and LLaMA training corpora may include SQuAD2 or similar QA data and this is not discussed.
  • ad hoc to paper The single-inference CoT prompt is functionally equivalent to the double-inference approach.
    Abstract and Sec. 3.2 claim the prompt elicits the same unanswerable-question behavior, but the double-inference method is never run as a baseline.
  • ad hoc to paper The hand-written LLM post-processing is fair and does not favor LLMs.
    Sec. 4.3 says post-processing 'greatly improved' scores but raw before/after numbers are not reported and rules are only qualitative.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Question: How do Large Language Models perform on the Question Answering tasks? Answer:." pith.science (2026). https://pith.science/paper/IRDXWI6E

@misc{pith2026241212893,
  author       = {Pith},
  title        = {Pith review of: Question: How do Large Language Models perform on the Question Answering tasks? Answer:},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IRDXWI6E}},
  note         = {Machine review of arXiv:2412.12893}
}
read the original abstract

Large Language Models (LLMs) have been showing promising results for various NLP-tasks without the explicit need to be trained for these tasks by using few-shot or zero-shot prompting techniques. A common NLP-task is question-answering (QA). In this study, we propose a comprehensive performance comparison between smaller fine-tuned models and out-of-the-box instruction-following LLMs on the Stanford Question Answering Dataset 2.0 (SQuAD2), specifically when using a single-inference prompting technique. Since the dataset contains unanswerable questions, previous work used a double inference method. We propose a prompting style which aims to elicit the same ability without the need for double inference, saving compute time and resources. Furthermore, we investigate their generalization capabilities by comparing their performance on similar but different QA datasets, without fine-tuning neither model, emulating real-world uses where the context and questions asked may differ from the original training distribution, for example swapping Wikipedia for news articles. Our results show that smaller, fine-tuned models outperform current State-Of-The-Art (SOTA) LLMs on the fine-tuned task, but recent SOTA models are able to close this gap on the out-of-distribution test and even outperform the fine-tuned models on 3 of the 5 tested QA datasets.

Figures

Figures reproduced from arXiv: 2412.12893 by the authors.

Figure 1
Figure 1. The mean F1 score of the models discerned by interrogative pronouns [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Mean EM score if n character changes (cf. Levenshtein Distance) to the answer were allowed [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. EM-Scores of each model on the OOD-Datasets [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: F1-Scores of each model on the OOD-Datasets [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: The prompt used for GPT4-Turbo [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: The prompt used for LLaMA-2-7B [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 27 canonical work pages

  1. [1]

    Evaluating chatgpt as a question answering system: A comprehensive analysis and comparison with existing models, 2023

    Bahak, H., Taheri, F., Zojaji, Z., and Kazemi, A. Evaluating chatgpt as a question answering system: A comprehensive analysis and comparison with existing models, 2023

    work page 2023

  2. [2]

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, ...

    2020

  3. [3]

    W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., W ang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S

    Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., W ang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suz- gun, M., Chen, X., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., V alter, D., Narang, S., Mishra, G., Yu, A., Zhao, V., Huang, Y., Dai, A., Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J., Robe...

    work page 2022

  4. [4]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., F an, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783(2024)

    arXiv 2024

  5. [5]

    F., Wold, A

    Green, B. F., Wold, A. K., Chomsky, C., and Laughery, K. Baseball: an automatic question-answerer. IRE-AIEE’61 (Western) (1961), 219–224

    work page 1961

  6. [6]

    Edg-based question decomposition for complex question answering over knowledge bases

    Hu, X., Shu, Y., Huang, X., and Qu, Y. Edg-based question decomposition for complex question answering over knowledge bases. In The Semantic Web - ISWC 2021 - 20th International Semantic Web Conference, ISWC 2021, Virtual Event, October 24-28, 2021, Proceedings(2021), A. Hotho, E. Blomqvist, S. Dietze, A. Fokoue, Y. Ding, P. M. Barnaghi, A. Haller, M. Dra...

    work page 2021

  7. [7]

    Y., Shi, X., Chen, P.-Y., Liang, Y., Li, Y.-F., Pan, S., et al

    Jin, M., W ang, S., Ma, L., Chu, Z., Zhang, J. Y., Shi, X., Chen, P.-Y., Liang, Y., Li, Y.-F., Pan, S., et al. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728(2023)

    arXiv 2023

  8. [8]

    P., and Ba, J

    Kingma, D. P., and Ba, J. Adam: A method for stochastic optimization, 2017

    work page 2017

Show all 34 references
  1. [9]

    ChatGPT: Jack of all trades, master of none

    Koco´n, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szyd lo, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., Koco´n, A., Koptyra, B., Mieleszczenko-Kowszewicz, W., Mi lkowski, P., Oleksy, M., Piasecki, M., Radli´nski, L., Wojtasik, K., Wo´zniak, S., and Ka...

    2023

  2. [10]

    S., Reid, M., Matsuo, Y., and Iwasawa, Y.Large Language Models are Zero-Shot Reasoners, Jan

    Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y.Large Language Models are Zero-Shot Reasoners, Jan. 2023. arXiv:2205.11916 [cs]

    2023 arXiv

  3. [11]

    Roberta: A robustly optimized bert pretraining approach, 2019

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach, 2019

    2019

  4. [12]

    A universal question- answering platform for knowledge graphs, 2023

    Omar, R., Dhall, I., Kalnis, P., and Mansour, E. A universal question- answering platform for knowledge graphs, 2023

    2023

  5. [13]

    Chatgpt versus traditional question answering for knowledge graphs: Current status and future directions towards knowledge graph chatbots, 2023

    Omar, R., Mangukiya, O., Kalnis, P., and Mansour, E. Chatgpt versus traditional question answering for knowledge graphs: Current status and future directions towards knowledge graph chatbots, 2023

    2023

  6. [14]

    A comparative study of transformer-based language models on extractive question answering, 2021

    Pearce, K., Zhan, T., Komanduri, A., and Zhan, J. A comparative study of transformer-based language models on extractive question answering, 2021. 14 Kevin Fischer, Darren F¨ urst et al

    2021

  7. [15]

    Is ChatGPT a General-Purpose Natural Language Processing Task Solver?, Nov

    Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., and Yang, D. Is ChatGPT a General-Purpose Natural Language Processing Task Solver?, Nov

  8. [16]

    Know what you don’t know: Unanswer- able questions for squad, 2018

    Rajpurkar, P., Jia, R., and Liang, P. Know what you don’t know: Unanswer- able questions for squad, 2018

    2018

  9. [17]

    Know What You Don’t Know: Unan- swerable Questions for SQuAD, June 2018

    Rajpurkar, P., Jia, R., and Liang, P. Know What You Don’t Know: Unan- swerable Questions for SQuAD, June 2018. arXiv:1806.03822 [cs]

    2018 arXiv

  10. [18]

    Squad: 100,000+ questions for machine comprehension of text, 2016

    Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. Squad: 100,000+ questions for machine comprehension of text, 2016

    2016

  11. [19]

    S.Comparative analysis of transformer based models for question answering

    Rawat, A., and Samant, S. S.Comparative analysis of transformer based models for question answering. 2022 2nd International Conference on Innovative Sustain- able Computational Technologies (CISCT)(2022), 1–6

    2022

  12. [20]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020

    Sanh, V., Debut, L., Chaumond, J., and Wolf, T. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020

    2020

  13. [21]

    Can chatgpt replace traditional kbqa models? an in-depth analysis of the question answering performance of the gpt llm family, 2023

    Tan, Y., Min, D., Li, Y., Li, W., Hu, N., Chen, Y., and Qi, G. Can chatgpt replace traditional kbqa models? an in-depth analysis of the question answering performance of the gpt llm family, 2023

    2023

  14. [22]

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N....

    2023

  15. [23]

    Gpt-ner: Named entity recognition via large language models

    W ang, S., Sun, X., Li, X., Ouyang, R., Wu, F., Zhang, T., Li, J., and W ang, G. Gpt-ner: Named entity recognition via large language models. arXiv preprint arXiv:2304.10428 (2023)

    2023 arXiv

  16. [24]

    White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., El- nashar, A., Spencer-Smith, J., and Schmidt, D. C. A prompt pattern catalog to enhance prompt engineering with chatgpt, 2023

    2023

  17. [25]

    Woods, W. A. Progress in natural language understanding: an applcation to lunar geology. AFPIS ’73 (1973), 441–450

    1973

  18. [26]

    Instruction Tuning for Large Language Models: A Survey, Oct

    Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., W ang, S., Li, J., Hu, R., Zhang, T., Wu, F., and W ang, G. Instruction Tuning for Large Language Models: A Survey, Oct. 2023

    2023

  19. [27]

    Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., and Hashimoto, T. B. Benchmarking large language models for news summariza- tion, 2023

    2023

  20. [28]

    E., and Stoica, I

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J. E., and Stoica, I. Judg- ing llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Infor- mation Processing Systems(2023), A. Oh, T. ...

    2023

  21. [29]

    Comparative analysis of state-of-the-art q&a models: Bert, roberta, distilbert, and albert on squad v2 dataset, 2024

    ¨Ozkurt, C. Comparative analysis of state-of-the-art q&a models: Bert, roberta, distilbert, and albert on squad v2 dataset, 2024. How do LLMs perform on the QA task? 15 A Additional Results T able 3.Full results of the FT-LMs on SQuAD2’s official validation dataset based on Hu...

    2024

  22. [33]

    The ideal response should be a exact repetition of the answer as it appears in the context, without any additions, modifications, or interpretations

    Answer the question with a exact quote from the context. The ideal response should be a exact repetition of the answer as it appears in the context, without any additions, modifications, or interpretations. Reply with the answer from the context, as short as possible. Do not u...

  23. [34]

    unanswerable

    Decide whether the question is answerable by thoroughly evaluating the context. If you can not answer it from the text reply with "unanswerable" and do not continue with the next step

  24. [35]

    Find the concise answer in the text

  25. [36]

    The ideal response should be a exact repetition of the answer as it appears in the context, without any additions, modifications, or interpretations

    Answer the question with a exact quote from the context. The ideal response should be a exact repetition of the answer as it appears in the context, without any additions, modifications, or interpretations. Reply with the answer from the context, as short as possible. Do not u...

  26. [2023]

    arXiv:2302.06476 [cs] version: 2

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.