{"id":"6f9f5fdf-3495-4047-9726-e3ee9de491ef","arxiv_id":"2412.03220","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":0.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A broad but error-prone survey of LLM and MLLM architectures, training methods, benchmarks, and challenges.","lead":"This paper surveys the main types of large language models, how they are trained and fine-tuned, and how they are tested. It is a broad index of the field, but it contains many factual errors and duplicated text, so it should be read with caution.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The survey's central claim of comprehensive, accurate coverage is undermined by demonstrably incorrect model descriptions and unsourced benchmark figures; factual reliability is the load-bearing concern.","rationale":"The reader's weakest assumption correctly identifies faithful representation of cited sources as the load-bearing premise. My independent reading of the full text confirms specific factual errors (LLaMA dating, duplicated Megatron text) and adds the observation that the benchmark figures are unsourced, which directly undermines the survey's comparative analysis. These defects are objective, not matters of interpretation, and they affect the core utility of a survey. The reader's CONDITIONAL verdict is appropriate: the paper needs major revision with fact-checking and sourcing before it can be accepted as a reliable overview. My concern does not shift the verdict; it reinforces it. The proposed test would provide a quantitative error rate to guide the revision and would settle whether the problem is superficial or pervasive.","tokens_in":52493,"tokens_out":1750,"duration_ms":18036,"concrete_test":"Select a random sample of 50 factual claims from Sections II, IV, VI, and VII (e.g., release years, parameter counts, architectural components, benchmark scores). Independently verify each against the cited primary paper or official documentation. Record the error rate and note any claims that cannot be traced to any source. If the error rate exceeds 5% or any benchmark figure in Figures 5-11 lacks a verifiable data source, the central reliability claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's value rests entirely on the accuracy and representativeness of its survey content, yet multiple objective defects show this premise is not secure. Section VI-B-5 dates LLaMA to 2022 and cites the LLaMA-2 paper [10] for LLaMA-1, conflating the two releases. Section VI-B-4 contains a duplicated Megatron paragraph, indicating copy-paste errors that compromise reliability. Additionally, the benchmark figures (Figures 5-11) plot named models with no accompanying data tables, sources, or evaluation protocols, making their accuracy and comparability impossible to verify. Without a stated inclusion methodology or a fact-checking process, the survey cannot support its claim to be a 'comprehensive' and 'holistic' map of the field. The known errors are not isolated typos; they signal a systematic risk that many other descriptions of models, benchmarks, and methods are inaccurate or fabricated from secondary sources. If a reader follows the survey to learn about an unfamiliar model, they may absorb misinformation that is hard to detect without checking every primary source.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper is a survey of Large Language Model (LLM) and Multimodal LLM (MLLM) research, aiming to cover architectures (encoder-only, decoder-only, encoder-decoder), benchmark evaluations, fine-tuning and pre-training methods, applications, and challenges, with coverage of models up to mid-2024 and a reference list of 427 entries. It positions itself as a comprehensive, holistic map of the field, explicitly claiming in Section III to delve deeply into Architecture, Benchmark, and Challenges aspects.","tokens_in":52613,"tokens_out":4460,"duration_ms":41191,"significance":"If the survey were factually reliable, its breadth—spanning model evolution, PEFT taxonomies, benchmark comparisons, and challenge taxonomies—would make it a useful entry point for researchers and practitioners. The manuscript has genuine strengths: it compiles a large reference base, includes a comparative table of earlier surveys (Table 3), and offers a structured taxonomy of parameter-efficient fine-tuning methods (Table 5). However, the paper's value rests entirely on the accuracy and representativeness of its content, and the demonstrated factual errors, duplicated passages, and unverifiable benchmark figures currently undermine that value. The errors are correctable within the scope of a survey, so the appropriate path is a major revision rather than rejection, but the revision must be substantive rather than cosmetic.","major_comments":[{"comment":"The section states that 'The LLaMA model was introduced by Meta in 2022' and cites reference [10], which is the LLaMA-2 paper, for claims about the original LLaMA. LLaMA-1 was released in February 2023 and has its own technical report; conflating the two releases corrupts the historical timeline that the survey claims to provide. This needs correction, including separate references for LLaMA-1 and LLaMA-2.","section":"§VI-B-5"},{"comment":"The Megatron paragraph is duplicated nearly verbatim: the text beginning 'Nvidia Megatron [220] is a framework proposed by Nvidia...' appears twice, with the second copy mislabeled as following from 'ChatGPT Nvidia's Megatron'. The two copies also give contradictory definitions: the first says inter-layer parallel is also known as tensor parallel, while the second says intra-layer parallelism is also known as tensor parallelism. This is a load-bearing editing failure in a technical survey, and the passage must be rewritten with a single, correct definition of tensor, pipeline, and data parallelism.","section":"§VI-B-4"},{"comment":"Multiple model descriptions contain factual errors that are not isolated typos. In §VI-B-1, GPT is described as having 'employed a 12-layer transformer encoder,' but GPT is decoder-only, contradicting the paper's own classification in §II-C. In §VI-C-1, BART is attributed to 'the Google research team,' whereas BART is from Facebook AI Research. In §VI-B-7, CogView is described as 'a 4T parameter Chinese multimodal LLM' (CogView is a 4B-parameter text-to-image model), and Mistral 7B is said to 'use mix-of-expert,' which is actually a property of Mixtral, not Mistral 7B. Together with the LLaMA error, these indicate a systemic reliability problem in the model descriptions, which is the core content of a survey.","section":"§VI-B-1, §VI-C-1, §VI-B-7"},{"comment":"The benchmark performance figures plot named models (MMLU, HellaSwag, ARC, WinoGrande, NLVR2, VQA) without any accompanying data tables, source citations, or evaluation-protocol details. The reader cannot verify the plotted values, determine whether they come from the original benchmark papers or secondary leaderboards, or assess comparability across models with different prompting and few-shot settings. Since the paper explicitly claims in §III to delve deeply into benchmarking, these unsupported figures are load-bearing for that claim and must be replaced or supplemented with tables that give exact scores, sources, and protocols.","section":"§IV-A, Figures 5–11"},{"comment":"The paper does not state inclusion criteria for the models, benchmarks, or prior surveys it discusses. The claim of being 'comprehensive' and 'holistic' (abstract and §III) is therefore unsupported by any reproducible methodology. To make the survey's coverage verifiable, the authors should specify how models and benchmark results were selected (e.g., date ranges, release venues, availability of primary sources) and how the comparative table of prior surveys (Table 3) was compiled.","section":"§III, §IV"}],"minor_comments":[{"comment":"Reference [10] is cited for both LLaMA and LLaMA-2 claims; these should be separate references, with the original LLaMA report cited for LLaMA-1.","section":"§VI-B-5, References"},{"comment":"The positional-encoding formulas are printed as '100002i/dim' in the denominator; this should be typeset as 10000^(2i/dim) to be unambiguous.","section":"§II-A, Equations (1)–(2)"},{"comment":"Cross-references are incorrect: the text says 'figure 15 shows the common source of the datasets' but the relevant figure is Figure 28, and 'Table 7 shows transformer based pruning technology' but the pruning methods are in Table 9.","section":"§VIII-A-3, §VIII-B-1"},{"comment":"The phrase 'ChatGPT Nvidia's Megatron' appears in the running text as a leftover editing artifact and should be removed.","section":"§VI-B-4"},{"comment":"There are numerous typos and terminological inconsistencies, including 'Megatron-Tuiring NLG' (§VI-B-4), 'the the datasets' (§VIII-A-3), and inconsistent spelling of 'auto-regressive' vs. 'autoregressive' and 'LLaMA' vs. 'Llama'.","section":"Throughout"},{"comment":"The evolutionary tree and parameter-count figures lack explicit sources and some lack axis labels (e.g., Figure 13 uses a log scale without stating the base); these should be clarified for a reader to interpret the data.","section":"Figures 12–14"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a survey whose value lies in its accuracy and representativeness, and the number of demonstrated factual errors—GPT described as encoder-based, BART attributed to Google, LLaMA dated 2022 and cited to the LLaMA-2 paper, CogView described as 4T, a duplicated Megatron paragraph with contradictory definitions—is high enough that I cannot certify its reliability as a reference work. I would ask the editor to treat this as a substantive revision requiring a full fact-check pass and a stated methodology section, not a copy-edit. The paper's breadth is commendable, but breadth without reliability is not sufficient for publication in its current form. There is no circularity or fabrication concern beyond the cited errors; the issues are purely factual and editorial."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a broad, recent-vintage LLM survey that would be handy if it were reliable, but it isn't yet. I counted several concrete factual errors and at least one duplicated paragraph, and the benchmark charts lack sources. That's enough to keep me from trusting the rest without cross-checking.\n\nWhat it does well: the scope is genuinely wide—models up to early/mid 2024, including Gemini, Claude 3, Mistral, plus the usual GPT/LLaMA families. The PEFT taxonomy in Table 5 is a useful cheat sheet. Table 3, the chronological map of 50+ earlier surveys, is the kind of orientation aid that newcomers will appreciate. The LoRA math is correct. The organization is clear, and the prose is readable.\n\nThe soft spots are not minor. LLaMA is dated 2022 (it was February 2023). BART is credited to a \"Google research team\" (it's Facebook/Meta). CogView is called a 4T-parameter model (it's 4B, and not really a Chinese MLLM in the way described). Figure 12's evolutionary tree places GPT-1/2/3, Gopher, Chinchilla, and XLNet under \"Encoder Only,\" which is simply wrong. The Megatron section contains a full duplicated paragraph, with \"ChatGPT Nvidia's Megatron\" stranded mid-sentence. The BLIP discussion has a literal placeholder citation \"[paper, paper, paper].\" The benchmark plots (Figures 5–11) show named models and scores with no data tables or sources, so the numbers can't be checked. And the paper never states its inclusion criteria for models, benchmarks, or methods, which matters for a survey's core claim of representativeness.\n\nBecause the paper's entire value proposition is accurate synthesis, these defects are load-bearing. A few typos would be fixable; this cluster suggests the manuscript needs a systematic fact-check, not just copy-editing.\n\nThat said, I'd still send it to peer review. The authors have done a real literature sweep and assembled material that, once corrected, would be a legitimate resource. A conscientious referee could turn the error list into a concrete revision path, and the field does need up-to-date orientation surveys. I just wouldn't accept it now or cite it in its current form.\n\nFor a reading group, it's a maybe—useful as a case study in how surveys should be vetted, less useful as a source of facts.","headline":"A useful-in-principle LLM survey, but the factual errors and duplicated copy make it unreliable in its current form.","tokens_in":53148,"tokens_out":3535,"would_cite":false,"duration_ms":32465,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey claims that the evolution of large language models can be comprehensively mapped through three Transformer-based architectural families—encoder-only, decoder-only, and encoder-decoder—and provides a comparative analysis of…","keywords":["Large Language Models (LLMs)","Transformer Architecture","Generative Models","Survey","Multimodal Learning","Deep Learning","Natural Language Processing (NLP)","Benchmarks"],"falsifier":"Spot-check at least ten model descriptions from the survey against the cited technical reports; for example, verify the release year, parameter count, and architectural details claimed for LLaMA, Megatron, and KOSMOS-1. Any material mismatch, such as dating LLaMA to 2022 when its cited source describes the 2023 LLaMA-2 model, would indicate that the survey's secondary summaries cannot be trusted without independent verification.","tokens_in":52259,"feed_emoji":"🧠","tokens_out":4836,"duration_ms":40822,"temperature":0.7,"pith_summary":"This survey aims to give a single, up-to-date map of large language models: how they are built, how they are trained and fine-tuned, how they are evaluated, and where they still fail. It organizes the field into three Transformer-based families—encoder-only (auto-encoding), decoder-only (auto-regressive), and encoder-decoder (sequence-to-sequence)—and traces the lineage of major models from BERT and GPT through LLaMA, PaLM, GPT-4, and multimodal systems. The intended payoff is practical: a reader could use the survey to classify any major LLM, compare it to peers on standard benchmarks, and identify the dominant techniques for adaptation and compression. If the survey is correct, it saves researchers and practitioners the effort of assembling this picture from dozens of primary sources.","feed_headline":"New survey sorts six years of LLMs into three architecture families","feed_subtitle":"From BERT and GPT to GPT-4o and Gemini, one reference maps models, benchmarks, and fine-tuning.","key_machinery":"The organizing machinery is the three-way architectural taxonomy built on the Transformer foundation: auto-encoding (encoder-only) models such as BERT, ERNIE, and ALBERT; auto-regressive (decoder-only) models such as GPT, LLaMA, PaLM, and KOSMOS-1; and sequence-to-sequence (encoder-decoder) models such as T5, BART, Pangu, and GLM. The Transformer—a neural architecture using multi-head self-attention and positional encoding—is treated as the shared substrate, and each family is defined by which part of the Transformer it keeps and what training objective it uses. This taxonomy carries the survey's comparative analysis: benchmark tables and lineage diagrams are organized by family, and fine-tuning and compression techniques are discussed as ways of adapting models within a family.","core_discovery":"On its own terms, the paper discovers nothing new about language models; its claim is that the recent history of LLMs is now mature enough to be summarized in a coherent narrative, and that the right narrative is an architectural one. Grouping models by encoder-only, decoder-only, and encoder-decoder designs, the survey traces evolution from 2018's GPT and BERT through the open-weights LLaMA family, the Pathways-based PaLM series, and the multimodal GPT-4, KOSMOS-1, and Gemini models. It further claims that standardized benchmarks—MMLU, SuperGLUE, HellaSwag, ARC, WinoGrande for language, NLVR2 and VQA for vision-language—plus a taxonomy of fine-tuning methods (LoRA and parameter-efficient techniques) and challenges (data quality, compression, distributed training, multimodality) provide a reliable basis for comparing models and guiding practice. The contribution is the synthesis and the comparative framing, not a new model or algorithm.","pith_inferences":["Editorial inference: the survey's own comparison tables suggest that benchmark scores, not parameter counts, are becoming the field's common currency, but the paper does not argue for any particular evaluation protocol.","Editorial inference: because the paper includes a few clear factual slips (for example, dating LLaMA to 2022 and citing the LLaMA-2 paper for LLaMA-1, and duplicating the Megatron description), a careful reader should treat specific numbers and dates as leads to check against primary sources rather than as authoritative.","Editorial inference: the architectural taxonomy could be extended to organize future model families, but doing so would require explicit inclusion criteria, which the survey does not provide."],"forward_implications":["A reader can classify any new LLM release by its architectural family and predict its likely strengths and trade-offs, since the survey links family to task suitability.","Practitioners can use the benchmark figures (MMLU, HellaSwag, ARC, WinoGrande, NLVR2, VQA) to compare models released in different years without re-running evaluations.","The survey's catalog of LoRA and other parameter-efficient fine-tuning methods offers concrete, lower-cost routes for adapting large models to specialized tasks.","The explicit list of challenges—data quality and bias, model compression, distributed computation, and multimodal alignment—marks where near-term research effort is most needed."],"supporting_citations":[{"why":"Supplies the foundational Transformer architecture on which every surveyed model family is built.","marker":"[1]"},{"why":"Defines the auto-encoding (encoder-only) family and its masked-language-modeling and next-sentence-prediction training tasks.","marker":"[3]"},{"why":"Defines the auto-regressive (decoder-only) family and the generative pre-training approach that later models extend.","marker":"[2]"},{"why":"Source for the LLaMA family descriptions, including the open-weights lineage and the fine-tuning derivatives the survey catalogs.","marker":"[10]"},{"why":"Defines the sequence-to-sequence (text-to-text) family and the masking strategies used by T5 and its successors.","marker":"[11]"},{"why":"The prior general LLM survey that this paper explicitly positions itself against and extends with 2023-2024 developments.","marker":"[81]"}],"fun_headline_variants":["LLM survey sorts models into encoder, decoder, and hybrid families","From GPT to Gemini: one survey tracks LLM architectures and tests","Six years of LLM evolution, grouped by architecture and benchmarks","Multimodal included: new survey covers LLMs, benchmarks, and limits","A reference map of LLMs: architecture, fine-tuning, and challenges"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The survey's value rests on the assumption that its selection of models, benchmarks, and methods is representative and that the descriptions of cited works faithfully reflect their primary sources; if either fails, the overview can mislead despite its breadth.","fun_headline_variants_meta":{"raw":{"variants":["LLM survey sorts models into encoder, decoder, and hybrid families","From GPT to Gemini: one survey tracks LLM architectures and tests","Six years of LLM evolution, grouped by architecture and benchmarks","Multimodal included: new survey covers LLMs, benchmarks, and limits","A reference map of LLMs: architecture, fine-tuning, and challenges"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000833,"raw_usage":{"total_tokens":3646,"prompt_tokens":969,"completion_tokens":2677,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":2584}},"tokens_in":585,"tokens_out":2677,"duration_ms":17018,"temperature":1.0,"reasoning_tokens":2584,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:38:27.004944+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Spot-check at least ten model descriptions from the survey against the cited technical reports; for example, verify the release year, parameter count, and architectural details claimed for LLaMA, Megatron, and KOSMOS-1. Any material mismatch, such as dating LLaMA to 2022 when its cited source describes the 2023 LLaMA-2 model, would indicate that the survey's secondary summaries cannot be trusted without independent verification.","supporting_citations":[],"review_version":1}