{"id":"2be80161-c811-4bb3-9c1e-9bd7f8d8a191","arxiv_id":"2411.09269","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An ensemble of five RAG-assisted LLMs identifies the presence of deep-learning methodology details in biodiversity papers, agreeing with human annotations on 417 of 600 comparisons.","lead":"A pipeline that combines five open-source LLMs with retrieval-augmented generation and a majority-vote classifier automatically checks whether biodiversity papers report key deep-learning methodology details, matching human annotations in 69.5% of 600 checks. The approach is pitched as a way to audit methodological reporting and could extend to other scientific fields.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 69.5% headline is computed against a gold standard that saw code, figures, and supplementary material and used different criteria; it mixes extraction error with annotation-scheme mismatch rather than measuring text-only accuracy.","rationale":"The paper is transparent and the pipeline is reproducible (code is released), and the authors themselves acknowledge the dataset/framework/hyperparameter criterion differences in the Discussion. However, the central claim is conditional on the evaluation protocol being a faithful measure of text-only extraction quality, and that protocol is the weakest link: the gold standard comes from the authors' prior study, with access to non-textual artifacts and different inclusion criteria. Table 3 shows the lowest agreement exactly for the variables where those criteria differ, so the headline number conflates extraction error with annotation-scheme mismatch. This is not a fatal flaw: a text-only, same-criteria re-annotation would settle it, and the current number may even underestimate text-only agreement for datasets since the LLM counts dataset names the humans ignored. I also note that the word 'enhances' lacks a direct single-LLM baseline for the 600 comparisons, but the gold-standard mismatch is the more load-bearing issue because it affects the validity of the headline metric itself. The reader's CONDITIONAL verdict therefore stands without adjustment.","tokens_in":17768,"tokens_out":5885,"duration_ms":53008,"concrete_test":"Re-annotate the 100 evaluation publications using only the full text (no code, figures, tables, or supplementary material) and the exact binary criteria of the six CQs in Table 3 (e.g., a named dataset counts even without a persistent identifier; a framework counts only if stated in the text). Have annotators blind to the prior annotations and to the LLM outputs, then recompute the voting-classifier agreement against this text-only gold standard, reporting per-variable agreement and a 95% confidence interval. If agreement stays near 417/600, the mismatch concern fails; if it moves materially, the headline 69.5% is an artifact of the mismatched gold standard.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline accuracy of 417/600 is not yet an accuracy of text-only extraction. The gold standard is the manual annotations from the prior work (Ahmed et al., 2024b), produced by overlapping authors who had access to code, figures, tables, and supplementary material, and whose criteria differ from the LLM's competency questions. The paper itself states in the Discussion that, for the 'Dataset' variable, the prior annotations counted a dataset only when a persistent identifier was provided, whereas the LLM counts a named dataset even without an identifier; for frameworks and hyperparameters, the prior annotations could draw on accompanying code that the text-only pipeline never sees. Under this protocol, every disagreement is a mixture of genuine extraction errors and annotation-scheme or input-modality mismatch, so the 69.5% figure does not measure how well the pipeline retrieves information that is actually stated in the text. Table 3 corroborates the problem: the two lowest agreement scores, 'Dataset' (63/100) and 'Open source framework' (53/100), are exactly the variables where the criteria diverge. Without a text-only, same-criteria gold standard, the central 'enhances retrieval' claim rests on an incommensurate comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an automated information-extraction pipeline that uses five open-source LLMs (Llama-3 70B, Llama-3.1 70B, Mixtral-8x22B, Mixtral 8x7B, and Gemma 2 9B) with retrieval-augmented generation (RAG) to answer 28 competency questions about deep-learning methodology from 464 biodiversity publications. Textual answers are converted to binary yes/no responses by the same LLMs and combined via majority voting. The main evaluation compares the voting output with manual annotations from the authors' prior work (Ahmed et al., 2024b) for six reproducibility variables over 100 publications, reporting 417/600 agreements (69.5%). The paper additionally reports per-LLM agreement with human labels for the text-to-categorical conversion step on 30 publications, an analysis of publication filtering, pairwise LLM cosine similarity and inter-annotator agreement, and an environmental-footprint estimate.","tokens_in":18034,"tokens_out":6780,"duration_ms":62491,"significance":"The work is a useful and transparent case study: the pipeline is described in enough detail to be reproduced, the model parameters (temperature=0, fixed majority voting, chunking settings) are fixed rather than tuned on the evaluation set, and code and data are provided. If the 69.5% agreement were shown to reflect text-only extraction quality under a commensurate annotation scheme, the result would be practically valuable for reproducibility screening and knowledge-graph construction. However, the current evidence does not establish the 'enhances retrieval' claim, because there is no single-LLM baseline on the same task and the gold standard is incommensurate with the text-only input in exactly the areas of lowest agreement.","major_comments":[{"comment":"The claim that the multi-LLM, RAG-assisted pipeline 'enhances the retrieval' of DL methodological information is not supported by a same-task comparison against a single-LLM baseline. The 417/600 result in Table 3 is computed for the majority-vote outputs only; Table 2 evaluates the text-to-categorical conversion step on 30 publications per LLM, which is a different task. To support the 'enhances' wording, the authors should report accuracy of each individual LLM on the same six competency questions over the same 100 evaluation publications, or otherwise revise the claim to state that the voting outputs agree with prior annotations at a certain rate.","section":"Abstract; §Results (Table 3)"},{"comment":"The headline 69.5% agreement is not a clean measure of text-only extraction accuracy. As the Discussion concedes, the prior human annotations in Ahmed et al. (2024b) were produced with access to code, figures, and tables, and used criteria that differ from the LLM competency questions: datasets were recorded only when a persistent identifier was given, whereas the LLM counts a named dataset without an identifier; frameworks and hyperparameters could be inferred from accompanying code. Every disagreement therefore mixes genuine extraction errors with annotation-scheme and input-modality mismatch, and Table 3 shows the lowest agreement precisely on Dataset (63/100) and Open source framework (53/100), the two variables with divergent criteria. The authors should either construct a text-only, same-criteria gold standard for at least a subset of the 100 publications, or explicitly present 69.5% as agreement under a mismatched protocol rather than as retrieval accuracy.","section":"§Discussion; §Results (Table 3)"},{"comment":"The evidence base is narrow for the strength of the conclusions: only 6 of the 28 competency questions have human ground truth, only the 100 prior-work publications are scored, and no uncertainty quantification is provided for the 417/600 estimate or for the per-variable counts. A binomial 95% confidence interval for 417/600 is approximately 65.7% to 73.1%, and the per-variable intervals (e.g., 53/100 to 89/100) overlap substantially, so the variable-level ordering in Table 3 should not be over-interpreted. Adding confidence intervals, a majority-class or chance baseline, and a per-competency-question breakdown would make the reliability claim commensurate with the evidence.","section":"§Evaluation; §Results (Table 3)"},{"comment":"The 93% accuracy reported for the publication-filtering component is not informative without additional statistics, because the evaluation set consists of 100 publications that all use DL methods and therefore contains no true negatives. Reporting only accuracy on a one-class sample cannot distinguish sensitivity from specificity; the authors should report precision, recall, and the false-positive and false-negative counts, ideally on a sample that includes non-DL papers, before using the filtering result to support the conclusion that filtering 'improved our findings.'","section":"§Additional analysis (Publication filtering); §Discussion"}],"minor_comments":[{"comment":"In the in-context conversion prompt, Example 1 asks about 'methods are utilized for collecting raw data', but the answer discusses whether a code repository is available; the question-answer pair is mismatched and should be corrected, since in-context examples directly influence the conversion step.","section":"§Methods (Prompt for converting textual to categorical responses)"},{"comment":"The text says 'randomly selected 30 publications' twice, but it is not clear whether the same 30 publications were used for all five LLMs or a different random sample was drawn for each LLM; please clarify, as this affects the comparability of the per-model kappa scores in Table 2.","section":"§Evaluation"},{"comment":"The two annotators who produced the ground truth for the 30-publication conversion evaluation are the first and last authors, but no inter-annotator agreement between them is reported; reporting a kappa value for the human annotations would strengthen the validity of the conversion evaluation.","section":"§Evaluation"},{"comment":"Please report confidence intervals or standard errors alongside the Cohen's kappa values; with 840 comparisons the differences between models (e.g., 0.7128 vs. 0.7708) may not be statistically meaningful.","section":"§Results (Table 2)"},{"comment":"The description of the retrieval query states that keywords were divided into five sets due to the platform's limit of 8 boolean connectors per search, but it is not stated whether the five result sets were unioned after deduplication or how the 991 citation records were combined; a sentence clarifying the union and deduplication step would improve reproducibility.","section":"§Methods (Dataset from Ecological Informatics)"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's scope as an information-retrieval case study, and the pipeline is described with commendable transparency. My main concern is that the abstract overstates what the evaluation supports; the requested ablation and re-annotation would put the central claim on solid ground. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is an honest, reproducible case study, and the 417/600 agreement figure is computed correctly from Table 3, but the paper overstates what that figure measures. The stress-test note is right: the gold standard from the prior work saw code, figures, tables, and supplementary material, and used different annotation criteria in at least two of the six variables. The two lowest agreement scores—Dataset (63/100) and Open source framework (53/100)—are exactly the variables where the criteria diverge. So 69.5% is not a clean measure of text-only extraction quality; it mixes extraction error with annotation-scheme and input-modality mismatch. The authors themselves admit this in the Discussion, which is a point in their favor, but the abstract still calls it accuracy.\n\nWhat is actually new: a five-LLM hard-voting RAG pipeline applied to 464 biodiversity publications, a new corpus-level run on 364 Ecological Informatics papers, and an explicit comparison of the voting classifier against six reproducibility variables from prior annotations. No parameter fitting on the target set—fixed majority, temperature zero, chunk settings given. Code and data are on GitHub, and the environmental footprint is reported with the Green Algorithms calculator. That is real evidence and deserves credit.\n\nThe soft spots are the ones the reader identified. There is no same-task single-LLM baseline, so \"enhances\" is unsupported; the multi-LLM majority vote may beat the worst model but we cannot see that from the paper. Only 6 of 28 CQs have human ground truth in the 600-comparison evaluation. No confidence intervals or significance tests are reported. The prior annotations being produced by overlapping authors matters, though it is a validity concern rather than a fatal flaw.\n\nOverall, this is a serious, transparent piece of applied work. I would send it to peer review with a request for three things: a text-only, same-criteria gold standard on a random sample, a direct single-LLM comparison on the same 600 items, and reframing of the headline as agreement with prior human annotations rather than accuracy. Who is it for: people building LLM extraction pipelines for scientific literature and reproducibility auditors. It is a solid case study, not a breakthrough. I would bring it to the reading group, mainly because the evaluation mismatch is a good teaching example.","headline":"A transparent, reproducible incremental case study whose 69.5% headline overstates what is measured: the gold standard saw code and figures and used different criteria, so the number mixes extraction error with annotation-scheme mismatch.","tokens_in":18569,"tokens_out":2522,"would_cite":false,"duration_ms":51706,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A majority vote of five open-source LLMs extracts deep-learning methodological details from biodiversity papers, agreeing with human annotations on 417 of 600 comparisons (69.5%), even though the models see only the article text.","keywords":["multi-LLM ensemble","retrieval-augmented generation","information extraction","deep learning reproducibility","biodiversity literature","voting classifier","competency questions","categorical annotation"],"falsifier":"Run the same five-LLM voting pipeline on a fresh sample of biodiversity deep-learning papers whose six variables are annotated by a second, independent team applying the same text-only criteria, and check whether the majority vote's agreement with those annotations is significantly above the per-variable majority-class baseline; if it is not, the 69.5% figure was an artifact of the earlier annotation criteria rather than a measure of extraction quality.","tokens_in":17586,"feed_emoji":"🤖","tokens_out":10036,"duration_ms":93972,"temperature":0.7,"pith_summary":"Deep-learning methods in scientific papers are usually described in unstructured prose, which makes it hard to find out how a model was built, trained, and evaluated. This paper tries to show that a pipeline of five open-source large language models, each combined with retrieval-augmented generation, can automatically turn the text of a paper into categorical yes/no answers about reproducibility variables. Tested on 464 biodiversity publications, a majority-vote ensemble of the five models matched the manual annotations from the authors' earlier study on 417 of 600 comparisons, or 69.5%, across six variables such as model architecture, datasets, hyperparameters, and source-code links. This agreement is notable because the human annotators had access to code, figures, and tables, while the LLMs saw only the text. If the claim is right, the pipeline offers a scalable way to screen research papers for methodological transparency and to help authors and reviewers check whether a deep-learning study is reproducible.","feed_headline":"Five LLMs vote to extract deep learning methods at 69.5%","feed_subtitle":"This could let researchers and reviewers screen methodological reporting across entire fields without reading every paper.","key_machinery":"The load-bearing mechanism is a hard voting classifier built from five open-source LLMs: Llama-3 70B, Llama-3.1 70B, Mixtral-8x22B-Instruct-v0.1, Mixtral 8x7B, and Gemma 2 9B. Each LLM is paired with retrieval-augmented generation, a setup that retrieves the most relevant chunks of a paper's full text and feeds them to the model so that answers are grounded in the source. Each model answers 28 competency questions covering the whole deep-learning pipeline, and a second pass converts each textual answer into a categorical 'yes' or 'no' using the same model. The five categorical votes are then combined by simple majority rule, which is what makes the final decision an ensemble verdict rather than the output of any single model. A final RAG-assisted filtering step removes publications that only mention DL keywords without actually presenting a DL pipeline.","core_discovery":"The paper's central discovery is that the majority vote of five different open-source LLMs, each fed a paper's full text through a retrieval-augmented pipeline, is a usable partial proxy for human judgment about whether six deep-learning reproducibility variables are present in a biodiversity publication. On a curated set of 100 papers from earlier work, the voting classifier matched the prior manual annotations in 417 out of 600 question-answer pairs, or 69.5%; agreement by variable ranged from 53/100 for open-source frameworks to 89/100 for model architecture. The authors further show that filtering out papers that mention DL keywords without actually employing a DL pipeline raises the fraction of answered queries from 27.12% to 35.77%, and that the filter itself achieved 93% accuracy on the 100-paper set. They present the results as a proof of concept that a multi-LLM voting pipeline can retrieve DL methodology from text alone and can be transferred to other scientific domains.","pith_inferences":["The 69.5% figure probably understates the models' text-extraction ability on the dataset variable, because the human gold standard counted a dataset only when a persistent identifier was given, while the LLMs counted the dataset by name; a criterion-aligned evaluation would be a fairer test.","Because the evaluation covered only the six of 28 competency questions that overlap with the earlier study, the ensemble's accuracy on the remaining 22 questions, such as deployment platforms, bias handling, and randomness strategies, is still unmeasured; a full 28-question gold standard would show whether the proxy extends beyond the best-matched variables.","The pipeline could be turned into a preprint-time reproducibility checklist: authors could run their own manuscripts through the same 28 competency questions, and the unanswered ones would flag missing methodological details before review.","A sharp test of ensemble value would be to measure agreement between the five models on the six variables and re-run the evaluation with each model alone; the paper's moderate pairwise kappas suggest the majority vote should beat the best single model, but that comparison is not explicitly reported."],"forward_implications":["The majority-vote outputs of five LLMs can serve as a partial proxy for human annotation in large-scale reproducibility audits, letting researchers screen thousands of papers for DL methodological reporting without reading each one.","The filtering step that removes non-DL papers is itself a usable component: it reached 93% accuracy on the 100-paper evaluation set and raised the pipeline's positive response rate from 27.12% to 35.77%.","Because the pipeline relies only on text and not on biodiversity-specific features, the same RAG-plus-multi-LLM procedure should transfer to other scientific domains where methodological reporting matters.","Individual LLMs diverge, with pairwise Cohen's kappa between 0.53 and 0.79, so an ensemble is more trustworthy than picking one model; Llama 3.1 70B and Llama 3 70B answer most alike, while Gemma 2 9B and Mixtral 8x22B diverge most.","The pipeline can report its own environmental footprint as a routine byproduct: 177.55 kWh and 60.14 kg CO2e for RAG text responses, and 50.63 kWh and 17.15 kg CO2e for categorical conversion."],"supporting_citations":[{"why":"Provides the curated 100-publication evaluation set and the manual yes/no annotations on the six reproducibility variables that serve as ground truth for the 69.5% agreement result.","marker":"Ahmed et al., 2024b"},{"why":"Supplies the 28 competency questions and the earlier RAG-based CQ-answering pipeline that the multi-LLM pipeline extends.","marker":"Kommineni et al., 2024b"},{"why":"Introduces retrieval-augmented generation, the mechanism that retrieves relevant passages from each paper before the LLMs answer.","marker":"Lewis et al., 2020"},{"why":"Documents the manual extraction of deep-learning variables in biodiversity papers and the data used as comparison for the pipeline's categorical outputs.","marker":"Ahmed et al., 2024a"}],"fun_headline_variants":["Voting LLMs hit 69.5% extracting DL methods","Five LLM ensemble reads papers, votes on methods","Multi-LLM voting extracts DL details at 69.5%","69.5% accuracy: five LLMs vote on methods extraction","LLM voting pipeline hits 69.5% on method extraction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 69.5% accuracy number assumes the earlier manual annotations are correct ground truth for the six reproducibility variables, even though those annotations were made by partially overlapping authors who could see code, figures, and tables and who used criteria, such as requiring persistent identifiers for datasets, that differ from what a text-only LLM can observe.","fun_headline_variants_meta":{"raw":{"variants":["Voting LLMs hit 69.5% extracting DL methods","Five LLM ensemble reads papers, votes on methods","Multi-LLM voting extracts DL details at 69.5%","69.5% accuracy: five LLMs vote on methods extraction","LLM voting pipeline hits 69.5% on method extraction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000753,"raw_usage":{"total_tokens":3414,"prompt_tokens":1076,"completion_tokens":2338,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":692,"completion_tokens_details":{"reasoning_tokens":2249}},"tokens_in":692,"tokens_out":2338,"duration_ms":16912,"temperature":1.0,"reasoning_tokens":2249,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:49:14.662582+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same five-LLM voting pipeline on a fresh sample of biodiversity deep-learning papers whose six variables are annotated by a second, independent team applying the same text-only criteria, and check whether the majority vote's agreement with those annotations is significantly above the per-variable majority-class baseline; if it is not, the 69.5% figure was an artifact of the earlier annotation criteria rather than a measure of extraction quality.","supporting_citations":[],"review_version":1}