Pith. sign in

REVIEW 5 major objections 8 minor 14 references

Challenges and Applications of Large Language Models: A Comparison of GPT and DeepSeek family of models

T0 review · 5 major / 8 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GPT-4o and DeepSeek are complementary trade-offs, not ranked alternatives: closed models offer safety, open models offer control.

desk verdict A useful but derivative practitioner survey whose qualitative trade-offs are right and whose quantitative assertions need sourcing. read the letter →

arxiv 2508.21377 v1 pith:OLIV5AWG submitted 2025-08-29 cs.CL cs.AIcs.LG

Shubham Sharma , Sneha Tuli , Narendra Badam This is my paper
classification cs.CLcs.AIcs.LG
keywords largelanguagemodelsGPT-4oDeepSeekclosedsourcevsopenmodelsafetyhallucinationMixture-of-Expertsselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is a structured comparison of two contemporary large language models: the closed GPT-4o and the open-weight DeepSeek-V3-0324. It argues that neither model is simply better; instead, each embodies a trade-off. The closed model offers stronger safety alignment, lower hallucination, and more predictable behavior in high-stakes and user-facing settings, while the open model offers lower training and deployment cost, full customization, and transparency. The paper works through sixteen known LLM challenges and then maps the two models' relative strengths onto application domains, concluding that model choice should be driven by whether a use case prioritizes safety and reliability or control, privacy, and cost.

What carries the argument

The paper's organizing instrument is a taxonomy of sixteen LLM challenges, grouped into design, behavioral, and evaluation categories and inherited from [5]. Each challenge is applied to GPT-4o and DeepSeek as a fixed comparison lens, yielding a per-challenge verdict; those verdicts are then mapped onto seven application domains to produce model recommendations. The recurring mechanism that carries the argument is the contrast between a closed, centrally aligned deployment, which delivers safety by design at the cost of transparency and control, and an open-weight, sparsely activated Mixture-of-Experts model, which delivers efficiency and adaptability by design at the cost of built-in safety

What would settle it

Take one fixed evaluation battery: the same hallucination test set, the same long-context documents beyond 20K tokens, the same safety and prompt-injection probes, and the same math and coding benchmarks, and run it on GPT-4o and DeepSeek-V3-0324 under identical conditions. If DeepSeek's hallucination rate is not higher than GPT-4o's, or GPT-4o does not beat DeepSeek on safety probes, the paper's central trade-off claim would fail its empirical test.

Watch

Extended reading notes

Core claim

The paper's central claim is that the practical difference between GPT-4o and DeepSeek-V3-0324 is best described as a closed-source versus open-source trade-off. GPT-4o is presented as the safer, more reliable, more polished system: reinforcement learning from human feedback, red-teaming, content filtering, a reported hallucination rate of 1.5% versus DeepSeek's 3.9%, reliable handling of long contexts, and strong refusal behavior make it the recommended default for chatbots, content creation, education, and high-stakes advice. DeepSeek is presented as the more efficient and adaptable system: a Mixture-of-Experts architecture with about 37B active parameters per token, roughly $5–6 million t

Load-bearing premise

The load-bearing premise is that the performance and safety numbers quoted for both models, such as GPT-4o's 89% accuracy and DeepSeek's 3.9% hallucination rate, are accurate, current, and directly comparable; the paper does not provide primary sources for several of them, and its informal screenshot tests are assumed to represent typical usage.

Editorial extensions

If this is right

  • A use case can be screened by four questions: Is it user-facing? Is the cost of error high? Is the data confidential? Does the team need full customization? The answers point to GPT-4o or DeepSeek respectively.
  • Open models at DeepSeek's cost level put competitive pressure on closed providers, and the paper expects MoE-style efficiency techniques to be absorbed into next-generation closed models.
  • Hybrid deployments become viable: a closed, aligned model for the front end and an open model for internal document processing or specialized coding.
  • If the safety gap is real, deploying open models in public-facing or regulated settings requires an extra layer of monitoring and filtering that the paper treats as the adopter's responsibility.
  • The paper's verdicts imply that model selection guidance should be updated frequently, because both models are versioned and the trade-off magnitudes will shift.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The comparison's empirical edge cases are only as strong as third-party numbers that the paper cites without sources; a reproducible benchmark battery would be the natural next step.
  • The closed-versus-open axis likely generalizes to other API-based aligned models and open-weight models, but the specific margin of safety versus flexibility will shift as new versions land.
  • A testable extension: run the same sixteen-challenge rubric on later model versions, or on fine-tuned open models with added alignment, to see whether the safety gap can be closed by community effort.
  • The paper implicitly assumes that informal screenshot probes represent typical usage; formalizing those probes into a fixed prompt set would make the qualitative findings falsifiable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. The manuscript is a survey-style comparison of GPT-4o and DeepSeek-V3-0324 across 16 LLM challenges, organized into design, behavioral, and evaluation issues, followed by application-level recommendations. The central thesis is that GPT-4o is preferable when safety, reliability, and minimal maintenance are priorities, while DeepSeek is preferable when control, customization, and cost are priorities. The paper claims to 'showcase' these trade-offs using a combination of literature-based descriptions, uncited benchmark figures, and informal screenshot demonstrations.

Significance. The paper's contribution is mainly organizational and practical. It provides a readable taxonomy of known LLM challenges and maps them onto concrete model choices. The qualitative direction of the trade-off (closed models being safer and more polished, open models being more efficient and controllable) agrees with broad field consensus, and the paper cites some relevant sources, including the DeepSeek-V3 technical report and a recent medical-exam evaluation. However, the manuscript does not provide machine-checked proofs, code, or reproducible experiments. Its quantitative evidence is largely uncited, and the informal tests do not meet a reproducibility standard. As such, the central claim is not yet supported at the level the paper's confident language suggests.

major comments (5)
  1. [Section II.B] Section II.B introduces four quantitative performance/safety claims (89% accuracy, 87% precision, 82% disallowed-content reduction, 2x faster inference, 50% lower cost) with no reference. These are the paper's first concrete evidence for the GPT-4o reliability advantage and they recur as implicit support in Sections III.A, III.H, and III.I. Please provide primary sources or remove the numbers; if removed, the subsequent safety verdicts must be re-grounded in citable evidence rather than assertion.
  2. [Section III.H] Section III.H states, 'According to Vectara's HHEM 2.1 benchmark, DeepSeek has a hallucination rate of 3.9%, compared to GPT-4o's 1.5%' and adds a 14.3% figure for DeepSeek-R1. No date, URL, test-set description, or configuration is given, and Sections III.O and III.P describe both models as opaque. Since the hallucination comparison is one of the two central pillars of the safety/reliability verdict, please supply the exact benchmark version, dataset, prompt settings, and access date, and indicate whether the rates are directly comparable.
  3. [Section III.I] Section III.I invokes 'independent audits' for a series of safety metrics (77% prompt-attack success, 69.2% evasion, 35.6% WMD pass, 53.3% hate speech, 48.9% self-harm, and failure on all Pliny injections) without citation. These numbers are the sole quantitative support for the conclusion that 'GPT-4o is clearly superior in safety and alignment.' Please cite the specific audits (organization, date, methodology, and model checkpoint), or label these as author-conducted stress tests with a full protocol. As written, the evidence is not verifiable.
  4. [Figures 3-7 and Sections III.A, III.B, III.I] Figures 3-7 are presented as results of 'our testing' with no protocol. There is no description of the number of prompts, selection criteria, model version/API parameters, temperature, or environment; the screenshots are therefore anecdotal and potentially unrepresentative. Since these examples are used to illustrate and partially justify the safety and tokenization verdicts, please either add a methodology appendix specifying how the prompts were generated and chosen, or explicitly downgrade the figures to illustrative, non-evidentiary examples.
  5. [Section III.C vs. Sections I and II.C] The parameter count for DeepSeek is inconsistent across the manuscript: 671B in Sections I and II.C, but '685B' in Section III.C. The DeepSeek-V3 technical report gives 671B total with 37B active. This inconsistency, combined with the uncited numbers above, suggests that the quantitative details have not been carefully checked. Please correct the value and audit the remaining quantitative statements for accuracy.
minor comments (8)
  1. [Abstract and Title] Typo: 'close source' should be 'closed source.' The title 'GPT and DeepSeek family of models' is grammatically awkward; consider 'GPT and DeepSeek Families of Models.'
  2. [Section II.B] The statement that GPT-4o has 'an estimated several hundred billion parameters' is vague and unsourced; provide a citation or mark it as an author estimate.
  3. [Section III.C] The training-cost estimates ($5-6M for DeepSeek, >$100M for GPT-4) are presented without a source. Please cite the underlying reports and clarify that these are estimates.
  4. [Section III.F] The claim that 'users report steep performance drops' after ~20K tokens and failures at ~56K tokens is not cited. Add a source or explicitly label it as anecdotal.
  5. [Section III.G] The claim that DeepSeek's temperature setting of 1.0 corresponds to an effective temperature of 0.3 is not sourced; cite the DeepSeek documentation or remove the statement.
  6. [Section IV.H] The statement that 'GPT-4o powers trusted tools like Duolingo and Khan Academy' needs a citation or should be hedged, as these integrations may change over time.
  7. [References] Several statements in Section III (e.g., inverse scaling in III.N, watermarking in III.M) could be supported by existing references [8], [13], etc.; please add citations at the point of claim rather than only in the reference list.
  8. [Figures 1-2] The architecture diagrams are labeled 'illustrative' but no source is given. Please state whether they are original or redrawn from existing sources, and cite the source if applicable.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is a comparative survey whose conclusions rest on external evidence and direct observations, not on a derivation that reduces to its own inputs.

full rationale

This is a survey/comparison paper, not a derivation or modeling paper. It makes no fitted parameters, no equations, and no predictions that are constructed from the data they claim to explain. The central trade-off (GPT-4o for safety/reliability, DeepSeek for control/cost) is stated in the introduction and reiterated in the conclusion, but the body supplies independent content: cited third-party benchmark results (e.g., Vectara hallucination rates in Section III.H, the Polish medical exam study in Section IV.H), references to technical reports (DeepSeek-V3, GPT-4), and the authors' own screenshot-based tests (Figures 3–7). None of these reduce by construction to the paper's conclusions. The uncited quantitative claims (GPT-4o's 89% accuracy in Section II.B, the 'independent audits' in Section III.I) are verification/correctness concerns, not circularity: they are not the paper's own outputs being fed back into the analysis. The paper does not import a uniqueness theorem from its own authors, does not smuggle an ansatz via self-citation, and does not rename a known result as a new derivation. Therefore the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities or fitted parameters; its load-bearing inputs are the cited framework, vendor claims, third-party benchmarks, and the authors' selected examples, none of which are independently verifiable from the manuscript.

assumptions (4)
  • domain assumption Kaddour et al.'s 16-challenge taxonomy is a valid and complete frame for LLM challenges
    The survey's entire structure (Sections III.A-P) is inherited from reference [5], which the paper states explicitly in the introduction; any gap in that taxonomy is inherited without review.
  • domain assumption Vendor-disclosed model facts are accurate (GPT-4o context window, performance gains, safety metrics; DeepSeek training cost and token counts)
    Section II.B quotes 89%/87% performance and an 82% disallowed-content reduction, and Section I quotes a $5-6M training cost and 2.788M H800 GPU hours, all without primary sources.
  • domain assumption The uncited 'independent audits' of DeepSeek safety are reliable
    Section III.I's safety verdict rests on unspecified audits (Pliny prompt injections, 77% attack success, 69.2% evasion, 35.6% WMD pass) with no named source or access date.
  • domain assumption The authors' informal prompt tests (Figures 3-7) are representative of typical usage
    These screenshots are selected to illustrate verdicts; no sampling method, prompt inventory, model version pinning, or repetition protocol is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Challenges and Applications of Large Language Models: A Comparison of GPT and DeepSeek family of models." pith.science (2026). https://pith.science/paper/OLIV5AWG

@misc{pith2026250821377,
  author       = {Pith},
  title        = {Pith review of: Challenges and Applications of Large Language Models: A Comparison of GPT and DeepSeek family of models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OLIV5AWG}},
  note         = {Machine review of arXiv:2508.21377}
}
read the original abstract

Large Language Models (LLMs) are transforming AI across industries, but their development and deployment remain complex. This survey reviews 16 key challenges in building and using LLMs and examines how these challenges are addressed by two state-of-the-art models with unique approaches: OpenAI's closed source GPT-4o (May 2024 update) and DeepSeek-V3-0324 (March 2025), a large open source Mixture-of-Experts model. Through this comparison, we showcase the trade-offs between closed source models (robust safety, fine-tuned reliability) and open source models (efficiency, adaptability). We also explore LLM applications across different domains (from chatbots and coding tools to healthcare and education), highlighting which model attributes are best suited for each use case. This article aims to guide AI researchers, developers, and decision-makers in understanding current LLM capabilities, limitations, and best practices.

Figures

Figures reproduced from arXiv: 2508.21377 by the authors.

Figure 1
Figure 1. GPT-4o Architecture Diagram experts, thereby reducing compute costs. LLMs are pre-trained on vast corpora to predict the next token, and later fine-tuned for specific tasks. Scaling model size and data drove progress: GPT-3’s 175B parameters enabled few-shot learning [3], but high costs prompted efforts toward computing-optimal training to balance size and data [4]. B. GPT-4o GPT-4o, released by OpenAI in May 2024, … view at source ↗
Figure 2
Figure 2. illustrates the detailed architecture of DeepSeek-V3-0324. Trained on 14.8T tokens using 2.788 million H800 GPU hours (approximately $5–6M), DeepSeek is cost-efficient compared to GPT-4o (approximately $100M). Fine-tuning was layered on top of earlier models: DeepSeek￾R1 for reasoning, DeepSeek-V2.5 for creative writing and used Grouped Relative Policy Optimization (GRPO) on 1.5M human rated prompts for alignment. O… view at source ↗
Figure 3
Figure 3. When asked to pick a religion that is most peaceful, GPT rejected the prompt but DeepSeek did not hesitate to share [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: During our testing of different prompts across the 2 models to evaluate tokenization, GPT had direct and concise [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Similarly, in another example GPT interpreted the word to be fictional that implies grandeur and playfulness. DeepSeek [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: We tested different prompts and GPT consistently rejected those associated with unethical, harmful or stereotypical [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: But if we update the prompt as shown in this example, to introduce slight ambiguity, GPT was successful in recognizing [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 9 canonical work pages

  1. [1]

    Attention is All You Need,

    A. Vaswani et al., “Attention is All You Need,” NeurIPS, 2017

    work page 2017

  2. [2]

    Outrageously Large Neural Networks: The Sparsely- Gated Mixture-of-Experts Layer,

    N. Shazeer et al., “Outrageously Large Neural Networks: The Sparsely- Gated Mixture-of-Experts Layer,” ICLR, 2017

    work page 2017

  3. [3]

    Language Models are Few-Shot Learners,

    T. Brown et al., “Language Models are Few-Shot Learners,” NeurIPS, 2020

    work page 2020

  4. [4]

    Training Compute-Optimal Large Language Mod- els,

    J. Hoffmann et al., “Training Compute-Optimal Large Language Mod- els,” arXiv:2203.15556, 2022

    arXiv 2022

  5. [5]

    Challenges and Applications of Large Language Models,

    J. Kaddour, J. Harris, M. Mozes, et al., “Challenges and Applications of Large Language Models,” arXiv preprint arXiv:2307.10169 , 2023

    arXiv 2023

  6. [6]

    Performance of GPT-4o and DeepSeek-R1 in the Polish Infectious Diseases Specialty Exam,

    Z. Błecha, D. Jasi ´nski, A. Jaworski, A. Latkowska, W. Jaworski, O. Sysło, N. Rubik, I. Jastrzebska, K. Harazi ´nski, W. Goliat, M. Gmur, M. Gajewski, B. Sławi ´nska, and N. Maryniak, “Performance of GPT-4o and DeepSeek-R1 in the Polish Infectious Diseases Specialty Exam,” Cureus, vol. 17, no. 4, e82870, 2025. doi:10.7759/cureus.82870

    work page doi:10.7759/cureus.82870 2025

  7. [7]

    Training Language Models to Follow Instructions with Human Feedback,

    L. Ouyang, J. Wu, X. Jiang, et al., “Training Language Models to Follow Instructions with Human Feedback,” Advances in Neural Information Processing Systems, vol. 35, 2022

    work page 2022

  8. [8]

    A Watermark for Large Language Models,

    J. Kirchenbauer, et al., “A Watermark for Large Language Models,” Proceedings of the 2023 International Conference on Machine Learning (ICML), 2023

    work page 2023

Show all 14 references
  1. [9]

    DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature,

    E. Mitchell, et al., “DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature,” Transactions of the ACL , vol. 11, pp. 2448–2461, 2023

    2023

  2. [10]

    Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K¨uttler, M. Lewis, W. Y . Wang, and S. Riedel, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” NeurIPS, vol. 33, pp. 9459–9474, 2020

    2020

  3. [11]

    DeepSeek-V3 Technical Report,

    DeepSeek-AI, A. Liu, B. Feng, B. Xue, et al., “DeepSeek-V3 Technical Report,” arXiv preprint arXiv:2412.19437 , 2025

    2025 arXiv

  4. [12]

    GPT-4 Technical Report,

    OpenAI, “GPT-4 Technical Report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  5. [13]

    Self-Refine: Iterative Re- finement with Self-Feedback,

    A. Madaan, X. Liu, M. Yazdanbakhsh, et al., “Self-Refine: Iterative Re- finement with Self-Feedback,” arXiv preprint arXiv:2303.17651 , 2023

    2023 arXiv

  6. [14]

    Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation,

    O. Press, N. A. Smith, and M. Lewis, “Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation,” International Conference on Learning Representations (ICLR) , 2022

    2022

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.