Pith. sign in

REVIEW 35 cited by

A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2303.10420 v2 pith:SO3NGBEN submitted 2023-03-18 cs.CL

Junjie Ye , Xuanting Chen , Nuo Xu , Can Zu , Zekai Shao , Shichun Liu , Yuhan Cui , Zeyang Zhou
show 7 more authors
Chao Gong Yang Shen Jie Zhou Siming Chen Tao Gui Qi Zhang Xuanjing Huang
This is my paper · ORCID
classification cs.CL
keywords modelsseriesgpt-3capabilitiesabilitytasksanalysisattention
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

GPT series models, such as GPT-3, CodeX, InstructGPT, ChatGPT, and so on, have gained considerable attention due to their exceptional natural language processing capabilities. However, despite the abundance of research on the difference in capabilities between GPT series models and fine-tuned models, there has been limited attention given to the evolution of GPT series models' capabilities over time. To conduct a comprehensive analysis of the capabilities of GPT series models, we select six representative models, comprising two GPT-3 series models (i.e., davinci and text-davinci-001) and four GPT-3.5 series models (i.e., code-davinci-002, text-davinci-002, text-davinci-003, and gpt-3.5-turbo). We evaluate their performance on nine natural language understanding (NLU) tasks using 21 datasets. In particular, we compare the performance and robustness of different models for each task under zero-shot and few-shot scenarios. Our extensive experiments reveal that the overall ability of GPT series models on NLU tasks does not increase gradually as the models evolve, especially with the introduction of the RLHF training strategy. While this strategy enhances the models' ability to generate human-like responses, it also compromises their ability to solve some tasks. Furthermore, our findings indicate that there is still room for improvement in areas such as model robustness.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 35 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 187 citations worldwide. Full citation record

  1. SensorQA: A Question Answering Benchmark for Daily-Life Monitoring

    cs.CL 2025-01 conditional novelty 7.0 of 10

    SensorQA is a new crowdsourced benchmark showing that current AI models answer only about 28% of daily-life sensor-data questions correctly.

  2. SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information

    cs.CL 2026-08 conditional novelty 6.0 of 10

    A new human-curated benchmark for LLM mobile assistants shows models struggle to locate scattered personal information, with the best model at 57.3% accuracy.

  3. TigerCoder: A Novel Suite of LLMs for Code Generation in Bangla

    cs.CL 2025-09 conditional novelty 6.0 of 10

    TigerCoder is a fine-tuned Bangla code-generation LLM family that posts 0.82 Pass@1 on the new MBPP-Bangla benchmark, but its gains partly come from model selection on the test set.

  4. Another Turn, Better Output? A Turn-Wise Analysis of Iterative LLM Prompting

    cs.AI 2025-09 conditional novelty 6.0 of 10

    Iterative LLM refinement helps early in ideation and code, but in math only late under elaboration prompting; vague feedback tends to plateau or degrade quality.

  5. Understood: Real-Time Communication Support for Adults with ADHD Using Mixed Reality

    cs.HC 2025-07 conditional novelty 6.0 of 10

    A HoloLens-based mixed reality assistant for adults with ADHD reduced conversation pause and off-topic recovery times in a within-subjects lab study of 10 participants.

  6. RecRankerEval: A Flexible and Extensible Framework for Top-k LLM-based Recommendation

    cs.IR 2025-07 conditional novelty 6.0 of 10

    A reimplementation of RecRanker shows its pointwise variant's high top-k scores come from ground-truth data leakage in the prompts, and the new RecRankerEval framework finds listwise tuning, DBSCAN sampling, XSimGCL, ...

  7. Un-considering Contextual Information: Assessing LLMs' Understanding of Indexical Elements

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A 1,600-question English benchmark shows LLMs interpret 'I' almost perfectly while struggling with 'you', 'here', and 'tomorrow', with quotation affecting each indexical differently.

  8. CODEMENV: Benchmarking Large Language Models on Code Migration

    cs.SE 2025-06 conditional novelty 6.0 of 10

    CODEMENV provides 922 examples and three tasks for evaluating LLMs on cross-version code migration, finding models are much better at migrating old code to new versions (up to 43.84% pass@1) than the reverse.

  9. Large Language Models in the Task of Automatic Validation of Text Classifier Predictions

    cs.CL 2025-05 conditional novelty 6.0 of 10

    LLM-based annotators using token-probability thresholds, RAG, and reasoning fine-tuning matched or exceeded human annotator quality on a proprietary 250-class intent-validation task.

  10. Is Your LLM-Based Multi-Agent a Reliable Real-World Planner? Exploring Fraud Detection in Travel Planning

    cs.MA 2025-05 conditional novelty 6.0 of 10

    Multi-agent LLM travel planners are frequently deceived by injected fake listings, coordinated fake reviews, and multi-round scam conversations, and a simple anti-fraud reviewer helps only some models.

  11. SensorChat: Answering Qualitative and Quantitative Questions during Long-Term Multimodal Sensor Interactions

    cs.AI 2025-02 conditional novelty 6.0 of 10

    A three-stage pipeline with LLM decomposition, pretrained embedding retrieval, and LLM assembly outperforms prior sensor QA systems on long-duration, high-frequency data, with caveats on evaluation leakage.

  12. Watching the AI Watchdogs: A Fairness and Robustness Analysis of AI Safety Moderation Classifiers

    cs.CL 2025-01 conditional novelty 6.0 of 10

    A fairness and robustness audit of four closed-source moderation APIs finds measurable group disparities and shows that LLM-based paraphrasing can bypass unsafe-content flags.

  13. Automatically Detecting Heterogeneous Bugs in High-Performance Computing Scientific Software

    cs.SE 2025-01 conditional novelty 6.0 of 10

    HeteroBugDetect combines LLM-generated simulation scripts, subsystem extraction, and differential CPU/GPU testing to detect 8 of 20 benchmark bugs and 2 previously unknown bugs in LAMMPS.

  14. Level-Navi Agent: A Framework and benchmark for Chinese Web Search Agents

    cs.IR 2024-12 conditional novelty 6.0 of 10

    The paper proposes Level-Navi Agent, a training-free Chinese web search agent framework, plus the Web24 benchmark and a composite evaluation metric for comparing LLMs.

  15. Efficient Code Analysis via Graph Representation Learning-Guided Large Language Models

    cs.SE 2026-01 reject novelty 5.0 of 10

    A GNN selects suspicious subgraphs of a Python project and an LLM judges only those subgraphs, beating direct whole-project LLM analysis on PyPI malware benchmarks.

  16. LLM as an Execution Estimator: Recovering Missing Dependency for Practical Time-travelling Debugging

    cs.SE 2025-08 conditional novelty 5.0 of 10

    RecovSlicing recovers dynamic data dependencies from partially recorded execution traces by using a language model to reconstruct missed variable values and aliases.

  17. SAGE: A Context-Aware Approach for Mining Privacy Requirements Relevant Reviews from Mental Health Apps

    cs.SE 2025-07 reject novelty 5.0 of 10

    SAGE combines domain-specific NLI hypotheses with a zero-shot GPT classifier to identify privacy-related app reviews, reporting F1 0.85 on 1,376 labels and extracting 748 new privacy reviews.

  18. A Modular Multitask Reasoning Framework Integrating Spatio-temporal Models and LLMs

    cs.CL 2025-06 conditional novelty 5.0 of 10

    STReason uses in-context learning to convert spatio-temporal queries into executable programs with specialized modules, outperforming plain LLMs on a new 150-query benchmark.

  19. MATE: LLM-Powered Multi-Agent Translation Environment for Accessibility Applications

    cs.MA 2025-06 conditional novelty 5.0 of 10

    MATE is an open-source multi-agent system for accessibility that uses a fine-tuned BERT model, trained on a new AI-generated dataset, to recognize and execute modality conversion tasks.

  20. StepProof: Step-by-step verification of natural language mathematical proofs

    cs.LO 2025-06 conditional novelty 5.0 of 10

    Decomposing natural-language proofs into sentence-level formal subproofs improves autoformalization success rates and efficiency compared with whole-proof formalization.

  21. Towards medical AI misalignment: a preliminary study

    cs.CY 2025-05 conditional novelty 5.0 of 10

    A custom role-playing prompt called the Goofy Game made four major LLMs produce plausible but incorrect medical recommendations.

  22. TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Use

    cs.CL 2024-12 conditional novelty 5.0 of 10

    TL-Training, a task-feature-based training framework, lets a 7B CodeLLaMA-2 model reach competitive tool-use performance using only 1,217 training trajectories.

  23. Large Language Models for Scholarly Ontology Generation: An Extensive Analysis in the Engineering Field

    cs.DL 2024-12 conditional novelty 5.0 of 10

    Zero-shot LLMs, especially Claude 3 Sonnet and a fine-tuned 7B Mistral variant, classify semantic relations between engineering research topics with high F1 on the new IEEE-Rel-1K benchmark.

  24. AIDBench: A benchmark for evaluating the authorship identification capability of large language models

    cs.CL 2024-11 conditional novelty 5.0 of 10

    A new benchmark shows GPT-4 and several other LLMs can attribute anonymous texts to their authors at rates well above random chance, though performance drops sharply in harder cross-topic settings.

  25. Evaluating Large Language Models for Diacritic Restoration in Romanian Texts: A Comparative Study

    cs.CL 2025-11 conditional novelty 4.0 of 10

    GPT-4o with a three-shot prompt is the best tested LLM for Romanian diacritic restoration (TAS 0.9639), beating an echo baseline by about 19%.

  26. Accelerating Automatic Program Repair with Dual Retrieval-Augmented Fine-Tuning and Patch Generation on Large Language Models

    cs.SE 2025-07 conditional novelty 4.0 of 10

    A dual-retrieval, threshold-gated, full-parameter fine-tuned code LLM (SelRepair) reports state-of-the-art exact-match patch generation on Tufano and VulRepair benchmarks.

  27. CMER: A Context-Aware Approach for Mining Ethical Concern-related App Reviews

    cs.SE 2025-07 conditional novelty 4.0 of 10

    A hybrid NLI-plus-LLM pipeline with finance-specific hypotheses extracted 2,178 manually validated privacy/security reviews from mobile investment app reviews.

  28. Fine-Tuning and Prompt Engineering of LLMs, for the Creation of Multi-Agent AI for Addressing Sustainable Protein Production Challenges

    cs.AI 2025-06 conditional novelty 4.0 of 10

    A proof-of-concept multi-agent GPT system for microbial protein literature extraction shows both fine-tuning and prompt engineering improve cosine-similarity scores, with fine-tuning slightly ahead but more variable.

  29. A Practical Guide for Supporting Formative Assessment and Feedback Using Generative AI

    cs.CY 2025-05 conditional novelty 4.0 of 10

    A narrative review that aligns generative AI tools with formative assessment principles, provides classroom prompt examples, and identifies missing evaluation metrics for AI feedback.

  30. Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarking

    cs.LG 2025-05 conditional novelty 4.0 of 10

    A benchmark and two low-cost tricks (weight refactorization and momentum reset) that make low-rank LLM pre-training competitive with GaLore and Fira at about 25% lower memory.

  31. ICH-Qwen: A Large Language Model Towards Chinese Intangible Cultural Heritage

    cs.CL 2025-05 reject novelty 4.0 of 10

    They fine-tuned Qwen2.5-7B on Chinese intangible cultural heritage texts to build ICH-Qwen, and report n-gram metric wins over general LLMs on 100-sample ICH QA tasks.

  32. Adapting Biomedical Abstracts into Plain language using Large Language Models

    cs.CL 2025-01 conditional novelty 4.0 of 10

    A one-shot GPT-4 prompt driven by distilled PLABA annotation guidelines ranked first on simplicity and third on accuracy in the plain-language biomedical abstract adaptation task.

  33. Kajal: Extracting Grammar of a Source Code Using Large Language Models

    cs.SE 2024-12 reject novelty 4.0 of 10

    Kajal uses GPT-3.5 with prompt engineering, few-shot examples, and parser feedback to infer grammars for DSL code snippets, reaching 60% self-parse accuracy on a 20-snippet dataset.

  34. IntellectSeeker: A Personalized Literature Management System with the Probabilistic Model and Large Language Model

    cs.IR 2024-12 reject novelty 4.0 of 10

    IntellectSeeker combines a fine-tuned GPT-3.5-turbo term translator and a probabilistic relevance filter for personalized academic search, reporting BLEU 0.93 and ROUGE-1 0.94 on a self-generated corpus.

  35. To Ensemble or Not: Assessing Majority Voting Strategies for Phishing Detection with Large Language Models

    cs.CL 2024-11 conditional novelty 3.0 of 10

    Majority-voting ensembles of LLMs for phishing URL detection improve on the best single model only when ensemble members have comparable performance.

Pith tools