{"id":"2c0a7393-d657-4f6d-9ce3-00c6167b4663","arxiv_id":"2412.00959","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ChatGPT and Gemini generally underperform task-specific models on requirements engineering benchmarks, except ChatGPT achieves a new top F1 score on the REQuestA question answering dataset.","lead":"This paper tests ChatGPT and Gemini on four software requirements tasks: extraction, classification, named entity recognition, and question answering, using three levels of prompt detail. It finds the models trail specialized predictors on three tasks, but ChatGPT edges out the best existing model on question answering.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pretraining contamination and unmatched evaluation protocols make the QA state-of-the-art claim (F1 0.91 vs 0.90) unestablished.","rationale":"The reader's weakest_assumption already identified the two key issues: comparability of F1 scores across different protocols and pretraining contamination. The strongest claim, that ChatGPT achieves SOTA QA performance, is exactly the premise that would need those issues resolved. I agree with the CONDITIONAL verdict and do not see a reason to move it: the paper's contribution is a useful empirical case study with plausible but unverified headline numbers. The load-bearing vulnerability is the QA comparison because it is the one positive result that flips the narrative from 'GLMs underperform' to 'GLMs are SOTA on QA'. If contamination or protocol mismatch explains the 0.91, the conclusion changes to GLMs underperforming on all four tasks. I explicitly flag that the supplementary prompts and extraction scripts are referenced but absent, which is a concrete missing-artifact issue consistent with the reader's note. I also flag Figure 6 as internally ambiguous: the text says AlBERT beats all other existing predictors, but the figure legend lists MiniLM as the top baseline, and the caption narrative is inconsistent with the table values claimed in the text; this does not change the fundamental concern but reinforces that the comparison needs a careful re-derivation. The proposed test is a single check that settles the concern: reproduce the QA comparison under a matched protocol with contamination control.","tokens_in":24531,"tokens_out":1587,"duration_ms":14126,"concrete_test":"Run the REQuestA QA comparison under a matched protocol: (1) fine-tune the five baseline models (AlBERT, MiniLM, RoBERTa, etc.) on the standard REQuestA split and evaluate on the identical 387 QA pairs used for ChatGPT/Gemini, or alternatively evaluate ChatGPT/Gemini with no passage provided (closed-book) and verified non-memorized questions; (2) report exact prompts and NER parsing scripts as an artifact; (3) perform a contamination check by prompting ChatGPT for known REQuestA answer strings on held-out questions and by checking n-gram overlap between dataset passages and pretraining corpora. If ChatGPT's F1 remains above the best fine-tuned baseline under the matched protocol, the state-of-the-art claim stands.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that ChatGPT achieves state-of-the-art on REQuestA QA (F1 0.91 vs 0.90 for AlBERT) is load-bearing: it is the only task where GLMs allegedly outperform specialized predictors, and the paper's practical conclusion ('useful for QA') rests on it. This claim is not yet supported for two reasons, both in Section 4.3.4. First, the literature baselines in Figure 6 were fine-tuned end-to-end on REQuestA training pairs, whereas ChatGPT and Gemini were prompted zero-shot with optional passages (Section 4.1.1 says prompts 2 and 3 contain the text containing answers), so the comparison is not apples-to-apples. Second, REQuestA is constructed from public SRS documents and Wikipedia (Section 3.2.4); ChatGPT-3.5 pretraining almost certainly included at least the Wikipedia passages, so retrieval-style answer extraction can be inflated by memorization. The same concern applies, though less centrally, to the other three tasks. The authors neither state this limitation nor run a contamination check. Additionally, exact prompts are not in the paper (supplementary tables are referenced but not included), so the evaluation cannot be reproduced. A positive QA finding could survive these checks, but as reported the headline comparison is not a fair test.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript reports a case study comparing ChatGPT-3.5 and Gemini on four requirements engineering tasks: requirements extraction (Pure), requirements classification (PROMISE), named entity recognition (Aerospace), and question answering (REQuestA). For each task, the authors design three prompt types with increasing levels of domain knowledge, evaluate the models using accuracy/F1 and related metrics, and compare against previously published task-specific predictors. The headline findings are that both generative models underperform state-of-the-art predictors on extraction, classification, and NER, while ChatGPT reaches F1=0.91 on REQuestA QA, slightly above the existing best AlBERT result of 0.90, and that Gemini requires more careful prompt engineering than ChatGPT.","tokens_in":24714,"tokens_out":4786,"duration_ms":41160,"significance":"The study is useful as a broad empirical snapshot of two widely used closed-source generative models across standard RE benchmarks. It contributes a prompt-level comparison, error analysis via Venn diagrams, and a small reproducibility analysis, and it goes beyond anecdote by evaluating on public datasets. However, the paper's only positive headline result—state-of-the-art QA performance—rests on a comparison that is confounded by unequal access to answer-bearing context and by possible pretraining contamination. Because the practical conclusion in Sections 5 and 6 (that generative models are useful for QA) depends on this single result, the central claim is not yet established; additional experiments or clearly stated limitations are required.","major_comments":[{"comment":"The QA comparison is not apples-to-apples: the baseline predictors (MiniLM, RoBERTa, ELECTRA, DistilBERT, BERT, AlBERT) were fine-tuned on REQuestA training pairs, whereas ChatGPT and Gemini were prompted, and Section 4.1.1 states that prompts 2 and 3 contain the text that contains the answers. This gives the generative models an extractive advantage and invalidates the claim that ChatGPT 'achieves state-of-the-art performance' in the question answering task. Please rerun the comparison with baselines that have the same passage access, or evaluate the GLMs without answer-bearing passages and with fine-tuned variants, or clearly reframe the claim as 'prompted extractive QA' rather than state-of-the-art.","section":"Section 4.3.4, Table 7, Figure 6"},{"comment":"Pretraining contamination is not addressed. REQuestA is constructed from public SRS documents and Wikipedia articles, and ChatGPT-3.5's training data likely includes at least the Wikipedia content; retrieval from memorized passages can inflate QA F1. The authors neither test for text overlap between the benchmark and pretraining data nor discuss this threat. A contamination analysis (for example, n-gram overlap statistics or evaluation on newly collected held-out QA pairs) is needed before the 0.91 vs 0.90 result can be treated as evidence of general QA capability.","section":"Section 3.2.4 and Section 4.3.4"},{"comment":"The evaluation is not reproducible as reported. Section 3.1 refers to supplementary Tables 1-4 for the exact prompts, but the supplementary file is not included with the reviewed version; Section 4 mentions a rule-based content and tags extractor for NER but does not provide the script; and Section 4 does not give API versions or access dates for ChatGPT-3.5 and Gemini. Without the prompts, the NER parsing script, and model version/date information, the prompt-dependence results cannot be audited or replicated.","section":"Section 3.1, Section 4, and Section 4.1.1"},{"comment":"There is an inconsistency in the reported size of the REQuestA dataset: the abstract states 300 QA pairs, while Section 3.2.4 reports 387 QA pairs. If the experiments use 387 pairs, the abstract is incorrect; if they use 300, the comparison to Ezzini et al.'s baseline scores may be on a different subset. Please reconcile the number and state exactly which split or subset was used for both the GLMs and the baselines.","section":"Abstract and Section 3.2.4"},{"comment":"The conclusion that 'for both classification and NER tagging tasks, basic prompts outperform knowledge rich prompts' is not supported by the reported data. Gemini's extraction accuracy increases monotonically with prompt level (0.746, 0.753, 0.790) and classification accuracy is flat between prompts 2 and 3; ChatGPT's best NER F1 occurs at prompt 2, not prompt 1. Please restrict the conclusion to the tasks and models for which the data actually show the pattern, and temper the generalization accordingly.","section":"Section 5 and Tables 4-6"}],"minor_comments":[{"comment":"The text says the QA evaluation uses '8 different evaluation measures' but then lists nine: precision, recall, F1-score, ROUGE-1, ROUGE-2, ROUGE-L, ROUGE-S, METEOR, and BLEU.","section":"Section 3.3"},{"comment":"The legend contains 'MoW+SVM', which appears to be a typo for 'BoW+SVM'.","section":"Figure 5"},{"comment":"Section 4.3.1 says the comparison includes '3 existing predictors' while Section 4.3's summary says 1 predictor for requirements extraction; please clarify that the three predictors are all from Ivanov et al. [60].","section":"Section 4.3.1 vs Section 4.3"},{"comment":"The capitalization of ROUGE is inconsistent across the manuscript (for example, 'Rouge-1', 'Rouge-2', and 'Rouge-L' versus 'ROUGE-S'); please standardize.","section":"Tables 4-7"},{"comment":"The reproducibility analysis is based on only 30 samples from a single dataset; please state this limitation explicitly in the text rather than leaving the reader to infer it.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on 2412.00959. The paper is a genuinely useful empirical mapping: ChatGPT and Gemini, three prompt levels, four RE benchmark tasks. The main finding—that except for QA, both LLMs trail specialized predictors—is believable and consistent with the numbers: extraction F1 0.76-0.79 vs BERT 0.86, classification 0.71-0.79 vs FnReq-Net 0.97, NER 0.25-0.36 vs Aero-BERT 0.92. Those gaps are too large to be artifacts of minor protocol differences. The prompt sensitivity analysis is also new and worth having, especially the claim that Gemini is more sensitive to prompt wording than ChatGPT. The paper is honest about its limitations and includes a small reproducibility check.\n\nThe soft spot is exactly where the reader put it: the QA state-of-the-art claim. ChatGPT's F1 0.91 vs AlBERT 0.90 is the only place where LLMs beat specialized models, and the comparison is not controlled. The literature baselines were fine-tuned on REQuestA; the LLMs were prompted, and for prompts 2 and 3 the passage containing the answer was given in the prompt. That makes the comparison retrieval-with-given-context versus closed-book fine-tuned QA. Add to that the pretraining contamination risk: REQuestA is partly built from Wikipedia, and ChatGPT-3.5 was certainly trained on Wikipedia. The 0.91 number may survive a proper controlled evaluation, but as reported it is not a fair test. The paper neither acknowledges this nor runs a contamination check.\n\nOther issues: the supplementary prompts and NER parsing script are referenced but not included, so the experiment is not reproducible as-is. The abstract says 300 QA pairs, the body says 387. These are minor but should be fixed. The other three task comparisons have caveats too (different evaluation protocols, random splits for NER), but those don't threaten the qualitative conclusion.\n\nVerdict: this deserves peer review. It is a solid empirical contribution that needs major revision to make the QA claim credible: release the prompts and parsing code, rerun the baselines under matched conditions, and test for memorization. For the other three tasks, the results are likely to stand.","headline":"Useful four-task evaluation of ChatGPT and Gemini for requirements engineering, but the one headline claim—ChatGPT beating AlBERT on QA—is not supported as reported.","tokens_in":25279,"tokens_out":2533,"would_cite":false,"duration_ms":22550,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompted ChatGPT and Gemini reach the previous best only on requirements question answering, with ChatGPT edging past it at 0.91 versus 0.90 F1.","keywords":["requirements engineering","generative language models","ChatGPT","Gemini","prompt engineering","requirements classification","named entity recognition","question answering"],"falsifier":"Run ChatGPT and Gemini on the REQuestA question-answer pairs with the answer-bearing passages withheld, and separately on a contamination-checked version of the benchmark; if F1 drops below the prior best of 0.90, the claimed QA advantage is an artifact of in-prompt retrieval or memorization. For extraction, classification, and NER, retrain the prior predictors and the LLMs under identical train/test splits, evaluation scripts, and decoding settings; if the F1 gaps shrink to near zero, the reported underperformance is an artifact of protocol mismatch rather than a real model limitation.","tokens_in":24291,"feed_emoji":"🤖","tokens_out":7767,"duration_ms":65091,"temperature":0.7,"pith_summary":"The paper asks whether two general-purpose generative language models, ChatGPT and Gemini, can take over four requirements engineering jobs: extracting requirements from documents, classifying them as functional or non-functional, tagging named entities, and answering questions about requirements. It reports that, except for question answering, both models lag the best task-specific predictors on the same public benchmarks. ChatGPT reaches 0.91 F1 on the REQuestA question-answering benchmark, just above the reported best prior score of 0.90, while Gemini reaches 0.88. The authors also find that Gemini needs more domain-specific prompt detail than ChatGPT, and that richer prompts help on some tasks but push predictions toward one class on others. If the findings hold, the current practical value of generative LLMs in requirements engineering lies mainly in question answering, not in extraction, classification, or NER.","feed_headline":"ChatGPT beats prior systems on requirements QA, lags elsewhere","feed_subtitle":"Across four benchmarks, prompted ChatGPT and Gemini beat state-of-the-art only on REQuestA question answering.","key_machinery":"The machinery is a three-level prompt protocol. Level 1 gives the model only the task query, Level 2 adds background definitions, and Level 3 adds definitions plus worked examples; the same three prompts are fed to both models for each task. The outputs are scored with standard measures (accuracy, precision, recall, F1, and for QA also ROUGE, BLEU, and METEOR). This protocol is what lets the authors separate model capability from prompt informativeness, and it underlies both main conclusions: Gemini needs more expert knowledge in its prompts than ChatGPT, and adding domain-specific keywords can steer predictions toward one class.","core_discovery":"The central claim is that, across four public benchmarks, ChatGPT and Gemini underperform the current state-of-the-art predictors for requirements extraction, requirements classification, and named entity recognition, but match or slightly beat them for requirements question answering. On extraction (Pure dataset) the best prior F1 is 0.86, while ChatGPT and Gemini score 0.76 and 0.77; on classification (PROMISE) the best prior F1 is 0.96, while both score 0.78; on NER (Aerospace) the best prior F1 is 0.92, while ChatGPT scores 0.36 and Gemini 0.25. On REQuestA, ChatGPT's best F1 is 0.91 against 0.90 for the best prior predictor, and Gemini reaches 0.88. The paper concludes that generative models are currently useful for building requirements question-answering applications, while traditional machine/deep learning models or foundational language models remain the better choice for the other three tasks.","pith_inferences":["A testable extension is to vary how the answer-bearing text is supplied in the QA prompts; the paper's gains at prompt level 3 may come largely from giving the model the relevant passage, so retrieval quality, not generative skill, could be the deciding factor.","The observed sensitivity to prompt wording suggests that automatic hard- or soft-prompt optimization could narrow the classification and NER gaps; the paper names prompt optimization as a future direction but does not test it.","A contamination check would settle whether the QA advantage transfers to unseen requirements documents: if REQuestA text appeared in pretraining, the reported 0.91 F1 would be inflated and would not generalize.","The reproducibility results imply that any LLM benchmark report should include variance across repeated API calls; the paper reports such variance for 30 extraction samples only, not for the other tasks."],"forward_implications":["For requirements question answering, ChatGPT is a viable option: on REQuestA its best F1 of 0.91 marginally exceeds the best prior predictor's 0.90.","For extraction, classification, and NER, practitioners should expect fine-tuned task-specific predictors to beat prompted ChatGPT and Gemini by wide margins (0.10, 0.18, and 0.56 F1, respectively).","Prompt content matters: generic prompts sometimes outperform knowledge-rich prompts, and domain-specific keywords bias predictions toward a particular class, so prompt design should be treated as part of the evaluation.","Gemini needs more carefully engineered, domain-specific prompts to reach its best results, while ChatGPT is comparatively better with generic prompts.","Repeated runs on the same extraction samples produce inconsistent labels for both models, so single-shot evaluations are unreliable and averaging over multiple iterations is advisable."],"supporting_citations":[{"why":"supplies the Pure dataset and the BERT baseline whose 0.86 F1 is the extraction comparison target.","marker":"[60]"},{"why":"supplies the PROMISE requirements classification dataset used for all classification comparisons.","marker":"[118]"},{"why":"supplies the Aerospace NER dataset and the Aero-BERT predictor whose 0.92 weighted F1 is the NER comparison target.","marker":"[97]"},{"why":"supplies the REQuestA question-answering benchmark and the six prior QA predictors; its reported best prior F1 is 0.90.","marker":"[44]"},{"why":"provides the FnReq-Net classification predictor whose 0.96 F1 is the best prior result on PROMISE.","marker":"[11]"}],"fun_headline_variants":["For requirements tasks, LLMs beat SOTA only on QA, lag on rest","ChatGPT tops classic models for requirement QA, but loses elsewhere","LLMs underperform on most requirement engineering benchmarks","Gemini requires careful prompts; ChatGPT only wins at requirement QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the F1 scores previously reported for task-specific predictors can be fairly compared with the prompted LLM results, even though the earlier systems were trained or fine-tuned under different split and evaluation protocols, and that ChatGPT and Gemini had not already memorized the public benchmark texts during pretraining.","fun_headline_variants_meta":{"raw":{"variants":["For requirements tasks, LLMs beat SOTA only on QA, lag on rest","ChatGPT tops classic models for requirement QA, but loses elsewhere","LLMs underperform on most requirement engineering benchmarks","Gemini requires careful prompts; ChatGPT only wins at requirement QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000525,"raw_usage":{"total_tokens":2605,"prompt_tokens":1087,"completion_tokens":1518,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":703,"completion_tokens_details":{"reasoning_tokens":1446}},"tokens_in":703,"tokens_out":1518,"duration_ms":9769,"temperature":1.0,"reasoning_tokens":1446,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:48:49.456815+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ChatGPT and Gemini on the REQuestA question-answer pairs with the answer-bearing passages withheld, and separately on a contamination-checked version of the benchmark; if F1 drops below the prior best of 0.90, the claimed QA advantage is an artifact of in-prompt retrieval or memorization. For extraction, classification, and NER, retrain the prior predictors and the LLMs under identical train/test splits, evaluation scripts, and decoding settings; if the F1 gaps shrink to near zero, the reported underperformance is an artifact of protocol mismatch rather than a real model limitation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the PROMISE requirements classification dataset used for all classification comparisons."},{"cited_title":"In: AIAA SCITECH 2023 Forum, p","cited_arxiv_id":null,"evidence_quote":"supplies the Aerospace NER dataset and the Aero-BERT predictor whose 0.92 weighted F1 is the NER comparison target."},{"cited_title":"AI-based Question Answering Assistance for Analyzing Natural-language Requirements","cited_arxiv_id":"2302.04793","evidence_quote":"supplies the REQuestA question-answering benchmark and the six prior QA predictors; its reported best prior F1 is 0.90."}],"review_version":1}