{"id":"e3907277-8c01-4bb7-9996-97327f360f6f","arxiv_id":"2412.06249","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-task GPT-4 model is said to beat single-task GPT-4, GPT-3, BERT, and Bi-LSTM on classification and summarization, but the experimental evidence is not reported.","lead":"This paper claims that a multi-task learning setup based on GPT-4 outperforms single-task and older models on text classification and summarization, reporting 93.6% accuracy and 44.8 ROUGE-1. It matters as a test of whether sharing a feature extractor across tasks helps modern large language models, but the evidence is too thin to support the claim.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The summary-generation experiment is not anchored to any named dataset: GLUE, the only dataset introduced, contains no summarization task, so the reported ROUGE-1 advantage is untestable.","rationale":"The reader rejected the paper on grounds of non-reproducibility and unmatched baselines. I agree with that rejection, but I identify a more fundamental and concrete flaw: the summary-generation task has no named dataset, and the only dataset introduced in the paper, GLUE, does not contain a summarization task. This makes the ROUGE-1 column of Table 1 meaningless as reported. The central claim is an empirical superiority claim; without a defined evaluation corpus, no amount of baseline matching could salvage the generation result. I also note that the gradient-cosine regularization sign in Eq. (4) appears questionable, since minimizing cosine similarity between gradients would push gradients toward opposition rather than reducing conflict, but that is secondary because the empirical evidence fails before reaching the method's details. No code or data is provided, so independent verification is impossible. The verdict remains REJECT, and my analysis does not change the reader's decision.","tokens_in":6866,"tokens_out":5708,"duration_ms":58377,"concrete_test":"Analytical check: enumerate the official GLUE task list from the GLUE paper or website and confirm that none is a summarization or generation task; then grep the manuscript for any standard summarization corpus, such as CNN/DailyMail, XSum, or Gigaword. If no corpus is named, the reported ROUGE-1 numbers cannot be traced to a concrete dataset, and the generation-task claim is unverifiable. If a corpus is later supplied, re-run the proposed multi-task configuration on that corpus's official test split to check whether 44.8 ROUGE-1 is reproduced.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV-A introduces GLUE as the dataset, describing its nine subtasks (CoLA, MNLI, STS-B, etc.), none of which is a summarization task. Section IV-B then states that two tasks were selected: text classification and 'summary automatic generation task.' Table 1 reports ROUGE-1 values for this generation task (single-task GPT-4 at 38.6, proposed model at 44.8). The paper never names a summarization corpus, never specifies its train/validation/test split, and the abstract asserts that 'the experiment uses multiple subtasks of the GLUE dataset' to compare performance. This is an internal inconsistency: ROUGE-1 cannot be computed on any GLUE subtask because GLUE contains no text-generation or summarization benchmark. Consequently, the central claim of surpassing all comparison models on summary generation is not attached to any identifiable evaluation target, making it impossible to reproduce or even interpret. The baseline-fairness issue noted by the reader is real, but it is downstream of this more basic gap: before asking whether baselines were matched, one must know what data and metric define the task. The absence of a dataset name is a factual, checkable flaw in the experimental design, not merely a missing hyperparameter.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-task learning (MTL) framework built on GPT-4 as a shared feature extractor with task-specific heads for text classification and automatic summarization. The method section defines a weighted joint loss with a gradient-conflict penalty, and the experiments compare the proposed model against single-task GPT-4, a GPT-3 multi-task version, BERT base, and Bi-LSTM+Attention. The central claim, stated in Section IV-C, is that the proposed model surpasses all comparison models on both tasks, with 93.6% classification accuracy and 44.8 ROUGE-1, while maintaining stable loss convergence.","tokens_in":7105,"tokens_out":3009,"duration_ms":30933,"significance":"If the empirical claims were supported by a reproducible setup, the paper would demonstrate a practically useful result: sharing a GPT-4 backbone between classification and summarization improves both tasks relative to single-task GPT-4. The paper makes a concrete quantitative comparison and states its evaluation metrics, which is a strength. However, the contribution is entirely empirical, and the evidence presented is not sufficient to establish the claims: the summarization dataset is not identified, baseline evaluation protocols are not described, and the loss curves are not actually included. The central result is therefore unverifiable from the manuscript as written, so the potential significance does not translate into a publishable contribution without major experimental additions.","major_comments":[{"comment":"The only dataset introduced is GLUE, and Section IV-A describes its nine subtasks, none of which is a summarization benchmark; Section IV-B nevertheless defines a 'summary automatic generation task' and Table 1 reports ROUGE-1 for it. Since no summarization corpus, train/validation/test split, or generation dataset details are given anywhere in the manuscript, the claim that the proposed model surpasses all comparison models on summary generation is not attached to any identifiable evaluation target and is impossible to reproduce or even interpret.","section":"Section IV-A, IV-B, Table 1"},{"comment":"The central claim rests entirely on Table 1, but the manuscript never describes how the single-task GPT-4, GPT-3 multi-task, BERT base, and Bi-LSTM+Attention baselines were trained, validated, or evaluated, nor whether they were matched in data, prompt format, hyperparameters, or compute. Without matched protocols and without error bars or significance tests, the reported improvements of 2.1 accuracy points and 6.2 ROUGE-1 points cannot be distinguished from configuration noise.","section":"Section IV-C, Table 1"},{"comment":"The loss-convergence claim is supported only by a caption; the actual loss curves of Figure 2 are not present in the manuscript. The text states that train and test loss decline and remain close, but with no figure or logged values this cannot be checked, and the claim of no overfitting is unsubstantiated.","section":"Section IV-C, Figure 2"},{"comment":"The method section presents generic equations for a shared encoder, task heads, weighted loss, and gradient-conflict penalty, but it does not specify how these components are instantiated with GPT-4 (e.g., whether GPT-4 is frozen, which layers/adapters/LoRA are used, how the generation head is trained, or how the gradient penalty lambda and task weights were chosen). Since the paper's contribution is an empirical optimization recipe, the absence of these implementation details makes the experiment non-reproducible.","section":"Section III and IV-B"}],"minor_comments":[{"comment":"The abstract says 'the experiment uses multiple subtasks of the GLUE dataset,' but Section IV-B selects only two tasks; either the experiments or the abstract wording should be corrected to avoid this inconsistency.","section":"Abstract and Section IV-B"},{"comment":"Reference [22] concerns gradient nanostructured materials and does not appear to support the claim about using gradient information for task optimization; a relevant optimization reference should be cited instead.","section":"Section III, Reference [22]"},{"comment":"The manuscript contains numerous typographical and formatting errors, including 'Indianpolis' and 'The Chinese University of Hon Kong' in the author affiliations, and garbled mathematical notation in Section III.","section":"Throughout"},{"comment":"The evaluation section reports only a single ROUGE-1 number without precision/recall decomposition, decoding hyperparameters, or variance measures, making it difficult to assess the reliability of the generation result even if the dataset issue were resolved.","section":"Table 1 and Section IV-C"}],"recommendation":"reject","confidential_remarks":"The manuscript falls below the evidentiary standard expected for this venue. The central empirical claim cannot be verified because no summarization dataset is named and no baseline evaluation protocol is described; addressing these issues would require new experiments rather than a revision of the existing text. I also note that the citation list contains several references that appear tangential to the corresponding claims, which compounds the impression that the paper was not carefully prepared."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper claims a clean win for multi-task GPT-4 but gives no way to check it. The more basic problem: the only dataset named is GLUE, which contains no summarization task, yet Table 1 reports ROUGE-1 scores for a summary generation task. That is an internal inconsistency, not a missing hyperparameter.\n\nTo give credit where it is due: the paper correctly describes the standard shared-encoder MTL setup, and the idea of fine-tuning GPT-4 with a classification and a generation head is not implausible. The writing is clear, and the structure moves from method to experiment in a logical order.\n\nBut the experimental section falls apart. Table 1 gives single numbers with no error bars, no data splits, no training configuration, and no description of how baselines were evaluated. Figure 2 is referenced, but the loss curves are not actually included. The citation for gradient cosine similarity regularization [22] is a materials design paper that does not support the technique. Hyperparameters are tuned on the same tasks used for the final results, which is a mild but real form of selection bias.\n\nThe dataset inconsistency is load-bearing. The abstract and Section IV-A say the experiment uses GLUE, listing its nine subtasks, none of which is summarization. Section IV-B then says they selected text classification and summary generation, but no summarization corpus is ever named. So the claimed 44.8 ROUGE-1 is not anchored to any identifiable evaluation set, making the central claim untestable. The baseline-fairness concern the reader raised is real, but it is downstream of this more basic gap.\n\nWho is this paper for? A practitioner who wants to know whether multi-task fine-tuning helps GPT-4 would find the conclusion plausible but unsupported. The paper deserves a desk reject, not a full review. I would not send it to external referees. If the authors supply the actual data, dataset names, training details, and matched baselines, a revision might be worth another look, but this version is not a serious empirical contribution.","headline":"Multi-task GPT-4 claims rest on an inconsistent dataset description and no reproducible evidence.","tokens_in":7616,"tokens_out":2213,"would_cite":false,"duration_ms":19696,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that sharing one GPT-4 backbone between text classification and summarization outperforms single-task GPT-4, GPT-3 multi-task, BERT-base, and Bi-LSTM+Attention, with 93.6% accuracy and 44.8 ROUGE-1.","keywords":["multi-task learning","large language models","GPT-4","text classification","summary generation","GLUE","shared feature extractor","ROUGE"],"falsifier":"Reproduce Table 1 under controlled conditions: train single-task GPT-4 and the proposed multi-task GPT-4 on the same GLUE classification and summarization splits, with the same optimizer, epochs, and hyperparameter search budget, and evaluate with the same metric script. If single-task GPT-4 reaches or exceeds 93.6% accuracy and 44.8 ROUGE-1 in those controlled runs, the central claim is refuted.","tokens_in":6680,"feed_emoji":"🧠","tokens_out":6055,"duration_ms":50870,"temperature":0.7,"pith_summary":"This paper claims that one large language model can do both text classification and summary generation better when trained on the two tasks at once than when trained on either alone. The proposed model shares a GPT-4 feature extractor across two task-specific heads, with a weighted joint loss and a gradient-similarity penalty meant to keep the tasks from fighting. On the reported experiments it reaches 93.6% classification accuracy and 44.8 ROUGE-1, ahead of single-task GPT-4, multi-task GPT-3, BERT-base, and Bi-LSTM+Attention. The reason to care is that this suggests multi-task training can be a cheap way to get more capability out of an existing large language model rather than fine-tuning separate copies.","feed_headline":"Multi-task GPT-4 beats single-task on classification and summaries","feed_subtitle":"Sharing one GPT-4 backbone across two tasks lifts accuracy to 93.6% and ROUGE-1 to 44.8%.","key_machinery":"The load-bearing machinery is a two-part architecture plus a joint objective. A shared GPT-4 extractor $\\vartheta(x;\\theta)$ maps input text to a high-dimensional representation $h$; each task $t$ then has its own output head $g_t(h;\\phi_t)$, so classification and summarization share the backbone but keep separate output spaces. The joint loss is a weighted sum of task losses with a cosine-similarity regularizer on task gradients, $L_{\\cos} = \\sum_{t_1<t_2} \\cos(\\nabla_\\theta L_{t_1}, \\nabla_\\theta L_{t_2})$, intended to balance gradient directions and reduce task conflict. Training proceeds by pre-training the shared extractor, then joint fine-tuning with alternating batches from the two tasks.","core_discovery":"The paper's central discovery is reported in Table 1: a multi-task model built on a shared GPT-4 representation with task-specific heads outperforms every comparison model on both measured tasks. The gains are attributed to the combined design of a shared feature extractor and separate classification and generation heads, optimized through a joint objective $L_{\\mathrm{total}} = \\sum_t \\alpha_t L_t + \\lambda L_{\\cos}$, where the cosine term discourages conflicting task gradients. Table 1 reports 93.6% accuracy, exceeding single-task GPT-4 by 2.1 points and BERT-base by 6.3 points, and 44.8 ROUGE-1, exceeding single-task GPT-4 by 6.2 points and GPT-3 multi-task by 3.6 points. The paper interprets these numbers as evidence that collaborative learning between classification and generation improves generalization.","pith_inferences":["Inference: If the claimed gains hold under matched training budgets, the same shared-extractor-plus-heads recipe should transfer to other GPT-scale models, open or closed, without per-task fine-tuning.","Inference: The paper reports only aggregate accuracy and ROUGE-1, so an immediate extension is a per-GLUE-subtask breakdown, for instance CoLA, MNLI, and STS-B, to see which task pairs help or hurt each other.","Inference: A strong test of whether multi-task sharing rather than extra hyperparameter tuning drives the result is to run single-task GPT-4 and the multi-task model with identical LoRA or adapter settings and identical search budgets."],"forward_implications":["The multi-task framework improves text classification accuracy by 2.1 points over single-task GPT-4 on the reported GLUE-based setup.","Summary generation quality improves by 6.2 ROUGE-1 points over single-task GPT-4 when the same backbone also trains on classification.","Both tasks are handled by one shared GPT-4 backbone, so the approach adds task-specific heads rather than duplicating a full model, which the paper says avoids excessive computational cost.","Training loss and test loss decline together with a small gap throughout training, which the paper reads as stable convergence without obvious overfitting."],"supporting_citations":[{"why":"Supplies the GLUE benchmark data and its standard train-validation-test splits used for the experiments.","marker":"[23]"},{"why":"Backs the gradient-based dynamic weight adjustment and gradient-similarity regularization that the joint objective uses to reduce task conflict.","marker":"[22]"},{"why":"Supports the alternating-batch joint training strategy that the paper uses to share knowledge between classification and generation tasks.","marker":"[24]"}],"fun_headline_variants":["Multi-task GPT-4 outperforms single-task on two NLP tasks","One model, two tasks: multi-task GPT-4 wins both","Multi-task learning lifts GPT-4 accuracy and ROUGE","Sharing a GPT-4 backbone for multi-task beats dedicated models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the baseline models, single-task GPT-4, GPT-3 multi-task, BERT-base, and Bi-LSTM+Attention, were trained and evaluated on the same data with the same evaluation protocol as the proposed multi-task model, yet the paper does not describe how those baselines were run.","fun_headline_variants_meta":{"raw":{"variants":["Multi-task GPT-4 outperforms single-task on two NLP tasks","One model, two tasks: multi-task GPT-4 wins both","Multi-task learning lifts GPT-4 accuracy and ROUGE","Sharing a GPT-4 backbone for multi-task beats dedicated models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000453,"raw_usage":{"total_tokens":2292,"prompt_tokens":975,"completion_tokens":1317,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":1243}},"tokens_in":591,"tokens_out":1317,"duration_ms":9644,"temperature":1.0,"reasoning_tokens":1243,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:51:19.421683+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce Table 1 under controlled conditions: train single-task GPT-4 and the proposed multi-task GPT-4 on the same GLUE classification and summarization splits, with the same optimizer, epochs, and hyperparameter search budget, and evaluate with the same metric script. If single-task GPT-4 reaches or exceeds 93.6% accuracy and 44.8 ROUGE-1 in those controlled runs, the central claim is refuted.","supporting_citations":[{"cited_title":"GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding,","cited_arxiv_id":null,"evidence_quote":"Supplies the GLUE benchmark data and its standard train-validation-test splits used for the experiments."},{"cited_title":"Effective design space exploration of gradient nanostructured materials using active learning based surrogate models,","cited_arxiv_id":null,"evidence_quote":"Backs the gradient-based dynamic weight adjustment and gradient-similarity regularization that the joint objective uses to reduce task conflict."}],"review_version":1}