{"id":"32807b12-1f17-43bc-b7fa-97292ca30398","arxiv_id":"2507.20520","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"AQUA is presented as the first aquaculture-specific LLM, fine-tuned on roughly 3 million synthetic QA pairs filtered by an LLM judge, with claimed advantages over general models based on a four-question comparison.","lead":"AQUA is a language model fine-tuned for aquaculture advice, built from millions of AI-generated question-answer pairs that experts helped filter. The paper claims it beats general chatbots on fish-farming tasks, but the evidence is only four example questions and no independent test.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'consistent outperformance' claim rests on four handpicked questions with no random sampling, no blinding, and no significance testing; that evidence cannot support the central claim.","rationale":"The reader's weakest assumption concerns generalization of the fine-tuned GPT-4.1 judge across the full 3-million-pair dataset; that is a legitimate and load-bearing concern for the data-quality claim. My concern sits one step downstream and is more directly attached to the paper's headline: the only head-to-head evidence for 'consistently outperformed' is a four-question, handpicked comparison without statistical support. Even granting perfect judge generalization, the central performance claim would still be unsupported. The firstness claim is also unverified and secondary to the performance claim, and the BLEU/ROUGE numbers on a same-pipeline validation set do not measure expert quality. I therefore partially agree with the reader: the judge concern is real, but the comparative evaluation is the most load-bearing weak point. Since the reader has already rejected the paper, my finding does not change the verdict; if anything, the proposed large-model-blind expert evaluation would be the minimal experiment that could either substantiate or falsify the headline.","tokens_in":14440,"tokens_out":4550,"duration_ms":53115,"concrete_test":"Construct a fresh evaluation set of 100-200 questions sampled from the 11 AQUADAPT categories, using sources not included in the AQUADAPT pipeline (e.g., FAO, NACA, or extension-service aquaculture manuals). Generate answers from AQUA, Qwen-2.5-7B, and Llama-3.1-8B under identical decoding settings, then have multiple aquaculture experts rate the responses in randomized, model-blind order using the Section VIII rubric. Compute per-model mean ratings with paired bootstrap confidence intervals and pairwise significance tests. If AQUA does not significantly outperform both baselines on the full set, the claim of consistent outperformance should be withdrawn or revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that AQUA 'consistently outperformed' Qwen-2.5-7B and Llama-3.1-8B (Section IX) is supported by exactly four questions, one per domain, shown in Section VIII. These questions are hand-constructed rather than sampled; the paper reports no repeated trials, no confidence intervals, no significance test, and no description of whether the expert raters were blinded to model identity. Even if the fine-tuned judge in Section VI-D generalizes perfectly, which is the reader's concern, a four-example comparison cannot establish a distributional superiority claim. Moreover, the BLEU/ROUGE evaluation in Section VII-B uses a held-out set drawn from the same AQUADAPT pipeline, so those scores measure fidelity to synthetic references rather than expert-level reasoning. The released Hugging Face checkpoints are useful for reproducibility, but they do not close the evidentiary gap in the headline comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AQUA, a 7B-parameter instruction-tuned model for aquaculture, built by fine-tuning Mistral-7B-Instruct-v0.3 with LoRA on approximately 3 million question-answer pairs generated by the AQUADAPT agentic pipeline. The pipeline combines web-scraped literature, expert-curated seed questions, dual-path generation using fine-tuned GPT-4.1 and Gemini 2.0 Flash, BM25 filtering, and an LLM-as-a-judge filtering stage calibrated on 10,000 expert-rated examples. The authors claim that AQUA is the first aquaculture-specific LLM and that it \"consistently outperformed\" general-purpose models such as Qwen-2.5-7B and Llama-3.1-8B. The evaluation consists of BLEU/ROUGE scores on a 20,000-pair held-out set and a four-question expert-rated comparison spanning fish health, feeding, economics, and water quality.","tokens_in":14599,"tokens_out":5243,"duration_ms":54131,"significance":"If the performance claims were supported, AQUA would be a useful community resource: the authors release model checkpoints and test data, document training hyperparameters, and describe a modular data-generation framework that could be adapted to other specialized domains. The expert-in-the-loop seed curation and judge-calibration strategy is a reasonable methodological direction. However, the current evidence does not support the central claims. The headline comparison rests on four handpicked questions, the held-out evaluation is drawn from the same synthetic pipeline that produced the training data, and no comparison to the base Mistral model is reported. The claimed \"expert-level reasoning\" is therefore unsubstantiated, even though the limitations section shows appropriate awareness of domain-anchoring risks.","major_comments":[{"comment":"The claim in Section IX that AQUA \"consistently outperformed general-purpose instruction models\" is supported only by the four hand-constructed questions in Section VIII. No sampling frame, repeated trials, confidence intervals, significance test, or information about rater blinding is reported, and the average ratings (AQUA 4.75, Llama 3.25, Qwen 2.75) are computed over exactly four responses. This evidence cannot establish a distributional superiority claim.","section":"Section VIII; Fig. 4"},{"comment":"The BLEU-4 and ROUGE evaluation is conducted on a held-out set of QA pairs generated by the same AQUADAPT pipeline used to create the training data. Without comparison to the base Mistral-7B-Instruct-v0.3 checkpoint or to general-purpose models, these scores primarily measure fidelity to the synthetic reference distribution and do not demonstrate expert-level reasoning. The paper should include baseline comparisons and, ideally, an independently authored test set.","section":"Section VII-B; Table 4"},{"comment":"The evaluation loop is circular: the fine-tuned GPT-4.1 judge is selected for agreement with 10,000 expert ratings (Section VI-C), then applied to filter the entire 3-million-pair dataset (Section VI-D); AQUA is fine-tuned on the filtered set (Section VII-A) and evaluated on a held-out set drawn from the same pipeline (Section VII-B). This design bakes the judge's preferences into both training and evaluation. An external validation set with expert labels, generated independently of the AQUADAPT filter, is needed to break the loop.","section":"Section VI"},{"comment":"The paper reports that approximately 3 million QA pairs passed the judge's threshold of 4 but does not report the retention rate, the distribution of judge scores, or any estimate of judge reliability on the full corpus, as opposed to the 10,000-pair gold standard. Without this information, the assertion that the judge's agreement with experts generalizes to the whole dataset is unsupported.","section":"Section VI-D"}],"minor_comments":[{"comment":"The tool name \"Dockling\" appears to be a typo for \"Docling\" (reference [21]).","section":"Section III"},{"comment":"The BM25 threshold tau is described as tuned, but its value is not reported; please provide the value or describe the tuning procedure.","section":"Section V-B"},{"comment":"The related-work paragraph cites \"Li et al. (2024)\" for OceanGPT, but reference [19] is \"Deng et al. (2023)\"; please correct the citation.","section":"Section II"},{"comment":"The sentence \"see extracted reference on AQUA-1B with Raspberry Pi and MQTT\" does not point to any entry in the reference list; either add the reference or remove the pointer.","section":"Section X"},{"comment":"Reference [15] is listed with the same page range as reference [11] and appears to describe a different work; please verify the bibliographic details.","section":"References"},{"comment":"The two source quantities sum to 55,105, but the collection strategy suggests possible overlap between \"Web\" and \"Open-access Source\"; please clarify whether the categories are disjoint.","section":"Table 1"},{"comment":"The caption of Fig. 1 says \"Aquaculture Categories\" but the figure is not visible in the submitted text; please ensure the figure is included.","section":"Section III; Fig. 1"},{"comment":"The novelty claim \"first large language model tailored for aquaculture\" is stated without a systematic comparison to prior domain-adapted agricultural or marine models; consider softening the claim.","section":"Abstract and Introduction"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has the character of an industrial report: the system description is detailed and the released checkpoints are a useful contribution, but the evaluation is not yet at the standard required for a research claim of consistent superiority. The circularity of the judge-filtered training and evaluation, together with the four-question headline comparison, would need to be addressed with a genuinely external evaluation before acceptance. The citation inconsistencies and the unresolved \"extracted reference\" suggest the manuscript was assembled quickly; these are fixable but should be corrected carefully."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. First, this paper ships a real artifact: AQUA-7B weights and a 20k-question test set are on HuggingFace, and the AQUADAPT pipeline is described in enough detail to reproduce the broad strokes. Second, the central claim—that AQUA 'consistently outperformed' Qwen-2.5-7B and Llama-3.1-8B—rests on exactly four handpicked questions, one per domain, with no random sampling, no blinding, no repeated trials, and no significance testing. That claim does not survive contact with the evidence.\n\nWhat is actually new: this is (as far as the cited literature goes) the first aquaculture-specific LLM, and the AQUADAPT framework is a coherent example of the now-standard domain-LLM recipe: expert-authored seeds, synthetic QA generation with two large models, BM25 literature filtering, LLM-as-a-judge quality filtering, and LoRA fine-tuning of Mistral-7B. The category taxonomy is thoughtful, and the judge-selection step (comparing three candidate judges against a 10k-sample expert gold standard, reporting Spearman, Kendall, Pearson, MAE, and kappa) is more careful than many papers in this space.\n\nThe soft spots are exactly where the claims outrun the evidence. Four questions cannot establish distributional superiority, and the fact that AQUA's answers are longer and include thresholds is a plausible but anecdotal indication of domain tuning. The BLEU/ROUGE scores in Table 4 are computed against synthetic references from the same AQUADAPT pipeline, so they measure how well the model matches that pipeline's style, not expert-level reasoning. The circularity the reader flags is real: the fine-tuned GPT-4.1 judge is validated on the gold standard, then used to filter the 3M training pairs, and the model is evaluated on a held-out set from the same filtered distribution. That tells you AQUA learned the judge's preferences, not that it is genuinely expert-level. Missing details: BM25 threshold tau is never reported, and there is no comparison to the base Mistral-7B model, which would have been trivial and is arguably the most important baseline. The paper's own limitations section (XII) is honest about over-specificity and rigidity, but doesn't mention these evaluation gaps.\n\nWho is this for? Practitioners who want a starting point for aquaculture-specialized assistants, and methodologists who want a worked example of domain-LLM building. As a research claim, the evaluation is currently inadequate. That said, the system is real, the artifacts are public, and the pipeline is described transparently enough to warrant a serious referee. I would send it to peer review with a strong request that the comparative evaluation be redone on a sampled, blinded, statistically tested set, and that the base-model comparison be added. As is, it's a useful technical report, not a validated superiority claim.","headline":"Ships a real aquaculture LLM and a coherent data pipeline, but the central 'consistently outperforms' claim rests on four unblinded, handpicked questions and a circular evaluation.","tokens_in":15205,"tokens_out":3172,"would_cite":false,"duration_ms":31654,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AQUA, the first aquaculture-specific LLM, is claimed to outperform general-purpose models on disease, feeding, water, and farm-economics advice.","keywords":["Aquaculture","Large Language Models","Artificial Intelligence","AQUA","AQUADAPT","Sustainability","Automation","LLM-as-a-Judge"],"falsifier":"Have independent aquaculture experts blindly rate a random sample of pairs the judge rejected (scores below 4) and a matching sample it accepted, then compare rating distributions: substantial numbers of 4-or-above scores among rejected pairs, or below-4 scores among accepted pairs, would falsify the judge's generalization. The complementary test is a fresh expert-written question set, independent of AQUADAPT-generated data, on which AQUA and the general-purpose baselines are rated blind.","tokens_in":14246,"feed_emoji":"🐟","tokens_out":11806,"duration_ms":107159,"temperature":0.7,"pith_summary":"AQUA is built by fine-tuning a 7-billion-parameter instruction-tuned model on roughly three million synthetic question–answer pairs produced by AQUADAPT, an agentic pipeline anchored in an expert-built taxonomy of eleven aquaculture categories and over sixty subcategories. The paper's aim is to show that AQUA is the first large language model purpose-built for aquaculture and that this domain-specific training lets it consistently outperform general-purpose instruction models across fish health, feeding and nutrition, production economics, and water quality. The load-bearing design is a fine-tuned LLM judge: calibrated against a gold standard of ten thousand expert-rated pairs, it scores every generated pair and keeps only those rated 4 or above, turning scarce expert judgment into a scalable dataset filter. If the claim holds, expert-level aquaculture advisory can be delivered cheaply and at scale, including on low-power farm devices through a companion small-model IoT system.","feed_headline":"First aquaculture LLM beats general-purpose models on farm advice","feed_subtitle":"A 7B model fine-tuned on 3M expert-filtered Q&As answers disease, feeding, water, and economics like a specialist.","key_machinery":"The central machine is AQUADAPT (Data Acquisition, Processing, and Tuning), an agentic data pipeline whose output is the roughly three-million-pair instruction dataset, with the Scoring Agent as the load-bearing component. The Scoring Agent is a fine-tuned GPT-4.1 judge that, prompted with few-shot expert-rated examples, assigns every QA pair a score from 2 to 5 and filters out anything below 4; the paper argues this transfer is sound because the judge outperforms two alternatives on agreement (Spearman $\\rho = 0.85$, Kendall $\\tau = 0.79$, Pearson $r = 0.89$; pairwise consistency 88.5%; weighted Cohen's $\\kappa = 0.76$). Domain specialization is then applied with low-rank adapters on the surviving pairs.","core_discovery":"On the paper's own terms, the discovery is that expert-anchored, judge-filtered synthetic data is enough to make a compact 7-billion-parameter model answer aquaculture questions at a level a rubric-based comparison treats as expert. AQUA receives an average rating of 4.75 out of 5 across four topical areas, versus 3.25 for Llama-3.1-8B and 2.75 for Qwen-2.5-7B, and its answers reproduce expert norms such as feed conversion ratios near 1.5, daily monitoring intervals, and dissolved-oxygen thresholds above 5 mg/L. The claimed novelty is the AQUADAPT agentic framework: a Data Agent cleans 55,105 documents; an Expert Agent supplies seed question–answer pairs and the category taxonomy; a QA Agent generates pairs from prompts and from BM25-filtered literature; and a Scoring Agent applies a fine-tuned GPT-4.1 judge, chosen because it best matches expert ratings (Spearman $\\rho = 0.85$, Kendall $\\tau = 0.79$, Pearson $r = 0.89$, MAE 0.42), to admit only pairs scoring 4 or above.","pith_inferences":["A direct ablation the paper does not report — training on the unfiltered pool versus the judge-filtered subset — would test whether threshold filtering itself is what creates the expert-level answers, not just the volume of domain text.","Because the held-out validation set and the four illustrative comparisons come from the same generation pipeline, the sharper test is an independent expert-written benchmark; until that exists, part of AQUA's apparent edge may be label agreement with its own filter.","The IoT scenario suggests that once on-device advisory is reliable, the practical bottleneck in real farms shifts to sensor calibration, data latency, and alarm fatigue rather than model quality.","If the judge-generalization assumption holds across domains, the AQUADAPT pipeline implies that small, cheap models can rival much larger general models in any vertical with a willing expert community and enough open-access literature."],"forward_implications":["AQUA could serve as a low-cost advisory layer for fish farmers and hatchery operators, supplying procedural answers with concrete intervals, thresholds, and corrective actions.","The AQUADAPT recipe — expert seeds, dual-path synthetic QA generation, and a judge-model filter calibrated to a small gold standard — is portable to other specialized domains where expert annotation is scarce.","A 1-billion-parameter version of AQUA can run on edge hardware such as a Raspberry Pi, using MQTT sensor feeds to generate real-time water-quality alerts and corrective recommendations without internet access.","Domain specialization measurably improves factual precision over general models in the paper's comparisons, such as correct feed-conversion ratios and global seafood-production shares where generalist baselines give wrong values.","The expert-calibrated LLM judge establishes a template for scalable quality control of synthetic instruction data in future domain-specific fine-tuning."],"supporting_citations":[{"why":"Provides the LoRA parameter-efficient fine-tuning method used to adapt the base model on roughly three million QA pairs.","marker":"[34]"},{"why":"Supplies the open-weight 7-billion-parameter base model that AQUA is fine-tuned from.","marker":"[35]"},{"why":"Documents the Qwen model family, one of the general-purpose instruction baselines AQUA is compared against.","marker":"[36]"},{"why":"Defines the BLEU metric used to measure AQUA's phrase-level fidelity on the held-out validation set.","marker":"[37]"}],"fun_headline_variants":["AQUA: 7B model answers aquaculture like a specialist","Expert-filtered synthetic data makes compact LLM expert","First aquaculture LLM scores 4.75/5 on advisor test","Agentic pipeline crafts 7B model that rivals specialists"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the judge model scores the roughly three million question–answer pairs as faithfully as experts scored the ten thousand gold-standard pairs, so the filtered training data — and anything learned from it — is genuinely expert-level.","fun_headline_variants_meta":{"raw":{"variants":["AQUA: 7B model answers aquaculture like a specialist","Expert-filtered synthetic data makes compact LLM expert","First aquaculture LLM scores 4.75/5 on advisor test","Agentic pipeline crafts 7B model that rivals specialists"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000402,"raw_usage":{"total_tokens":2105,"prompt_tokens":965,"completion_tokens":1140,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":1069}},"tokens_in":581,"tokens_out":1140,"duration_ms":8685,"temperature":1.0,"reasoning_tokens":1069,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:41:51.393303+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent aquaculture experts blindly rate a random sample of pairs the judge rejected (scores below 4) and a matching sample it accepted, then compare rating distributions: substantial numbers of 4-or-above scores among rejected pairs, or below-4 scores among accepted pairs, would falsify the judge's generalization. The complementary test is a fresh expert-written question set, independent of AQUADAPT-generated data, on which AQUA and the general-purpose baselines are rated blind.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the BLEU metric used to measure AQUA's phrase-level fidelity on the held-out validation set."}],"review_version":2}