{"id":"2ab44ab4-f6ad-42da-bbc5-852aaa79726d","arxiv_id":"2506.21567","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A proposed Persian biomedical LLM, BioPars, is evaluated on medical QA datasets and reported to beat GPT-4 on a self-built Persian QA benchmark, but the training setup is not described.","lead":"This paper introduces a Persian medical question-answering dataset and a model called BioPars, which it claims outperforms GPT-4 on that dataset. The paper is internally inconsistent about whether BioPars is a benchmark or a model, and it does not provide training details.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No training setup, model size, data split, or leakage control is reported, and the text calls BioPars both a 'measure' and a model; the claimed pretrained Persian biomedical LLM and its BioParsQA gains are therefore unverifiable.","rationale":"The reader's weakest_assumption is exactly the point: the paper assumes a pretrained BioPars exists and was evaluated without leakage, yet provides no training setup, model size, data splits, or leakage controls. I agree. Additional specifics strengthen this concern: Section 5's architecture is not tied to any training run; the abstract's 'measure' language conflicts with the model claim; BioParsQA has no documented split or deduplication; and Table 7 contains an implausible ROUGE-L value while Table 5's caption is mislabeled. This is not a disagreement with external consensus; it is an internal verification failure. The proposed repository inspection and rerun is the concrete check because the paper names that repository as the sole source of code and weights. If the repository contains the checkpoint and the rerun reproduces the result, the central claim gains support; if not, the paper's central claim is unverifiable and the evaluation may be circular. Therefore I recommend REJECT, matching the reader's verdict.","tokens_in":18493,"tokens_out":3828,"duration_ms":41733,"concrete_test":"Clone the repository at https://github.com/amirap80/BioPars and verify three artifacts: (1) a downloadable pretrained checkpoint plus tokenizer; (2) a training configuration specifying model size, pretraining corpus (and its overlap with BioParsQA), token budget, and compute; (3) evaluation scripts reproducing Table 7. Then run the released checkpoint against BioParsQA with a held-out split excluded from any pretraining/fine-tuning and recompute ROUGE-L, BERTScore, MoverScore, and BLEURT for the stated ranking methods. If any artifact is missing, or the reproduced ROUGE-L is not approximately 29.99, the claim that a pretrained BioPars outperforms GPT-4 cannot be sustained.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim—that BioPars is a pretrained Persian biomedical LLM that outperforms GPT-4 on BioParsQA—requires two things: that such a model was actually pretrained on Persian biomedical text, and that Tables 3–7 report measurements of that model. Neither is established. Section 5 describes an architecture (Eqs. 1–12), but gives no training objective, token budget, model size, optimizer, hardware, or procedure connecting BIOPARS-BENCH (§3) to pretraining. The abstract itself calls BioPars 'a simple but accurate measure designed to assess LLMs,' and §5.3.1 lists 'Unknown Scalability' as a restriction. BioParsQA (§4) is a self-built set of 5,231 Q&A pairs with no train/test split, no deduplication against the pretraining corpus, and no leakage controls, so the headline ROUGE-L/BERTScore gains may be circular rather than evidence of domain adaptation. Table 7 also contains an unchecked value (LLaMA2-13b at 88.52 ROUGE-L), and Table 5's caption is mislabeled, suggesting the evaluation tables were not carefully audited. The only route to verification is the GitHub link, but the paper gives no checkpoint identifier, model card, or training config there. The conclusion's assertion that BioPars 'implements the entire training pipeline from pre-training to final testing' is therefore unsupported by the manuscript.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces three artifacts: BIOPARS-BENCH, a Persian biomedical corpus compiled from roughly 10,000 articles, textbooks, and websites; BioParsQA, a self-constructed set of 5,231 Persian medical question-answer pairs; and BioPars, which is described in the abstract and introduction as a benchmark or 'measure' but in Section 5 and the conclusion as a pretrained Persian biomedical large language model with a gated attention architecture based on damped exponential moving averages, C-EMA, TimestepNorm, normalized attention, and a pre-norm two-hop residual. The evaluation compares BioPars against GPT-4, LLaMA2-13b, and Galactica 30B on LiveQA, ExpertQA-Bio, ExpertQA-Med, MedicationQA, and BioParsQA using ROUGE-L, BERTScore, MoverScore, and BLEURT, reporting that BioPars achieves the best scores on several datasets, most notably ROUGE-L 29.99 on BioParsQA versus 28.99 for GPT-4.","tokens_in":18879,"tokens_out":5479,"duration_ms":64847,"significance":"If substantiated, a pretrained Persian biomedical LLM together with a Persian biomedical benchmark would fill a genuine gap in low-resource medical NLP, and BIOPARS-BENCH and BioParsQA could become useful community resources. The inclusion of external English medical QA datasets is also a positive partial check on the model's behavior. However, the manuscript does not establish that the claimed pretrained model exists as described: there is no training description, no data-split or leakage-control protocol, and the evaluation tables contain internally inconsistent values. The current strengths are therefore limited to a resource proposal; the central model and performance claims are not verifiable from the manuscript as written.","major_comments":[{"comment":"The central claim that BioPars is a pretrained biomedical LLM is not supported by any training description. Section 5.1–5.4 presents architectural equations (Eqs. 1–12) but gives no training objective, token budget, model size, optimizer, learning-rate schedule, hardware, or procedure connecting the BIOPARS-BENCH corpus of Section 3 to pretraining. Section 5.3.1 explicitly lists 'Unknown Scalability' and states that stability at billion-parameter scale has not been tested, while Section 5.4.4 refers to 'even at 7 B parameters.' The conclusion's assertion that BioPars 'implements the entire training pipeline from pre-training to final testing' is therefore unsupported, and the paper's main claim cannot be independently checked.","section":"§5, §8"},{"comment":"The headline result is measured on BioParsQA, a dataset constructed by the same authors, yet Section 4 gives no train/test split, no deduplication against the pretraining corpus, no statement about whether BioParsQA was used for training, and no leakage-control procedure. Because BioPars's best ROUGE-L score is reported on this self-built dataset, the apparent improvement over GPT-4 may reflect overlap between training and evaluation data rather than domain-adaptive ability. The manuscript must state whether BioParsQA was seen during training and provide a split or contamination analysis before the BioParsQA comparison can be interpreted.","section":"§4, Table 7"},{"comment":"Table 7 reports a ROUGE-L of 88.52 for LLaMA2-13b under the Sim method, while the same model's other ROUGE-L entries are between 27 and 28 and the competing models are in the 27–29 range. No explanation is offered for this value, and it is so far outside the range of the other rows that it cannot be a simple metric fluctuation. An unchecked or mislabeled value in the central comparison table undermines confidence in all the reported numbers.","section":"Table 7"},{"comment":"The object being evaluated is described inconsistently: the abstract calls BioPars 'a simple but accurate measure designed to assess LLMs,' Section 1 calls it an 'open but powerful testing platform,' while Section 5 and Section 8 describe it as a neural model with parameters and pretraining. These are incompatible descriptions. If BioPars is a benchmark or metric, comparing it against GPT-4 and Galactica in Section 7 is not meaningful; if it is a model, the paper needs to specify what was actually implemented and evaluated.","section":"Abstract, §1, §5"},{"comment":"The evaluation protocol for the external English datasets is not described. LiveQA, ExpertQA-Bio, ExpertQA-Med, and MedicationQA are all non-Persian datasets, but the manuscript does not state whether the questions were translated into Persian, whether BioPars was run in Persian or English, what prompting or few-shot settings were used, or what decoding parameters were applied. Without this information, the comparative results in Tables 3–6 cannot be reproduced or interpreted, and the claim that BioPars outperforms other models on these datasets is not verifiable.","section":"§7, Tables 3–6"}],"minor_comments":[{"comment":"The caption of Table 5 says 'Automatic evaluation scores on ExpertQA-Bio,' but the table body, the surrounding text, and Figure 5 all refer to ExpertQA-Med; the caption should be corrected.","section":"Table 5"},{"comment":"The text says BioPars achieved 'a maximum ERTScore=83.90'; this appears to be a typo for BERTScore, and the reported BERTScore values are expressed as percentages in the tables but as fractions (e.g., 'above 0.88') in the prose, so the unit should be stated consistently.","section":"§7, MedicationQA paragraph"},{"comment":"The Mifepristone example states that the drug 'is usually provided to treat high blood sugar,' which is clinically inaccurate; since the paper is in the biomedical domain, such an error in a dataset example raises concerns about the curation quality and should be corrected or removed.","section":"§4"},{"comment":"The phrase 'an improvement over GPT-4 1.0' is unclear; it should be phrased as an improvement of 1.00 ROUGE-L points, and the relationship between the reported BioParsQA scores and the corresponding rows of Table 7 should be stated precisely.","section":"Abstract"}],"recommendation":"reject","confidential_remarks":"The manuscript has no training section, no leakage control for its headline dataset, and at least one impossible value in its main comparison table. These are not local presentation issues; they affect the core claim that BioPars is a pretrained Persian biomedical LLM that outperforms GPT-4. The paper would need a complete reworking of the experimental methodology and a re-audit of all tables before it could be considered for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Know this paper only if you care about Persian medical NLP. It points at a real gap and ships a sizeable new dataset, but the paper's core claim—that BioPars is a pretrained Persian biomedical LLM—cannot be checked from the manuscript, and the evaluation contains a few numbers that are simply wrong.\n\nWhat's genuinely useful: BIOPARS-BENCH and BioParsQA. A Persian medical QA set of 5,231 expert-approved pairs is a real contribution if it's actually released. Section 5 is also a coherent, fairly detailed description of a gated-EMA/attention hybrid; it's derivative of RWKV/RetNet-style ideas, but it's not hand-wavy. Running GPT-4, Llama, and Galactica on external English datasets (LiveQA, ExpertQA, MedicationQA) is a sensible sanity check.\n\nThe soft spots are load-bearing. There is no description of how BioPars was trained: no corpus used, no token budget, no model size, no optimizer or hardware. The abstract calls BioPars 'a simple but accurate measure', while the title and conclusion call it a pretrained LLM—that's not a minor typo. BioParsQA has no train/test split and no leakage control, so the headline gain over GPT-4 might be circular, especially since the paper never states whether the model saw that data. Table 7 lists an impossible ROUGE-L of 88.52 for LLaMA2-13b, and Table 5's caption says ExpertQA-Bio while the text is about ExpertQA-Med. The claim of being the first Persian biomedical LLM is unverified because the paper doesn't cite prior Persian medical NLP work. And the conclusion's statement that BioPars 'implements the entire training pipeline from pre-training to final testing' is simply not supported by anything in the body.\n\nIf you're building a Persian medical QA dataset, you might look at this for the resource. If you're evaluating an LLM claim, this paper isn't a reliable source. I'd send it back with a request for training details, splits, and corrected tables, but I wouldn't spend referee time on it as it stands. Desk reject, or at most a 'revise before review' at a venue that allows it.","headline":"A real Persian medical QA dataset and a coherent architecture, but the central claim that BioPars is a trained model is unverifiable and the evaluation tables contain errors.","tokens_in":19375,"tokens_out":3381,"would_cite":false,"duration_ms":39595,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that BioPars, presented as the first Persian biomedical large language model, outperforms GPT-4, Llama, and Galactica on Persian medical question answering, reaching ROUGE-L 29.99 on its own 5,231-question…","keywords":["Persian biomedical NLP","biomedical large language model","Persian medical question answering","BioParsQA benchmark","gated attention","exponential moving average","domain-specific pretraining","low-resource languages"],"falsifier":"Ask for the released weights and the exact pretraining and evaluation configuration, then rerun the BioParsQA evaluation on a split from which every reference answer has been confirmed absent from the training data; if the reported ROUGE-L 29.99 and BERTScore 90.87 cannot be reproduced, or if a contamination check finds BioParsQA texts inside the pretraining corpus, the claimed margin over GPT-4 is not established.","tokens_in":18347,"feed_emoji":"🩺","tokens_out":10109,"duration_ms":95892,"temperature":0.7,"pith_summary":"The paper sets out to show that Persian biomedical question answering cannot be handled well by general-purpose LLMs and needs a dedicated, language-specific model — and that such a model can in fact be built and measured. To make that case it introduces three artifacts: BIOPARS-BENCH, a Persian biomedical corpus drawn from over 10,000 scientific articles, textbooks, and medical websites; BioParsQA, 5,231 Persian medical question-answer pairs vetted by at least two specialist doctors; and BioPars, a pretrained Persian biomedical model whose attention is guided by a damped exponential moving average. Its central empirical claim is that BioPars outperforms ChatGPT (GPT-4), Llama 2-13B, and Galactica 30B on BioParsQA, reaching ROUGE-L 29.99 (1.0 over GPT-4), BERTScore 90.87, MoverScore 60.43, and BLEURT 50.78, while also posting the best numbers on three of the four English medical QA sets tested. If the claim holds, the practical consequence is that a purpose-built Persian model beats far larger general systems at generating usable medical answers, and the paper positions BioPars as the first Persian biomedical LLM for long-form medical question answering.","feed_headline":"Persian medical model tops GPT-4 on new QA benchmark","feed_subtitle":"BioPars claims the first Persian biomedical LLM, scoring 29.99 ROUGE-L on its own 5,231-question test set.","key_machinery":"The object that carries the argument is a transformer block whose self-attention is guided by a damped exponential moving average, extended to the complex plane as C-EMA: each input channel is lifted to an $h$-dimensional latent space, propagated through a complex-valued decay $h^j_t = \\alpha^j(\\cos\\theta^j + i\\sin\\theta^j) u^j_t + (1-\\alpha^j\\delta^j)(\\cos\\theta^j + i\\sin\\theta^j) h^j_{t-1}$, and projected back to the token width. Around that core sit a causal TimestepNorm that stabilizes activations without revealing future tokens, a normalized attention that derives queries and keys from an $\\ell^2$-normalized shared representation, a pre-norm two-hop residual that routes the feed-forward branch back to the block input, and chunk-wise attention plus parallelism so that only the final C-EMA state crosses devices during pretraining. The claimed role of this machinery is to give long-context, autoregressive stability during pretraining on Persian biomedical corpora, so the evaluation numbers are presented as the joint payoff of the architecture and the domain corpus.","core_discovery":"The paper's central claim is that BioPars is the first Persian biomedical large language model, and that its pretraining on Persian biomedical text makes it better than larger or more general models at answering real Persian medical questions in free-form prose. On its own benchmark, BioParsQA, the authors report the best scores among the four tested systems on all four metrics — ROUGE-L 29.99 against 28.99 for GPT-4, BERTScore 90.87 under the MMR ranking method, MoverScore 60.43, and BLEURT 50.78 — with the advantage appearing where the model must generate an answer rather than pick one. The same model records the top ROUGE-L on LiveQA, MedicationQA, and the ExpertQA-Med subset of the English-language sets, while trailing GPT-4 and Galactica on ExpertQA-Bio. From the pattern that ChatGPT, Llama, and Galactica retrieve learned facts well but stumble on higher-level, real-world questions and fine-grained inferences, the paper concludes that targeted pretraining on authentic Persian biomedical data is necessary for viable applications, rather than relying on zero-shot or generic capabilities.","pith_inferences":["My inference: the margin that matters is small — 1.0 ROUGE-L over GPT-4 — so the paper's strongest possible reading survives only if the benchmark is clean; the authors leave the leakage test implicit, and that test is the first thing an independent reader should run.","My inference: if the architecture, not just the corpus, is responsible for the gains, the gated-EMA block should transfer to other low-resource medical languages; a cheap experiment would be to pretrain the same block on Arabic or Urdu medical text and check BioParsQA-style benchmarks.","My inference: the paper's framing that general models 'remember' but cannot 'infer' is testable directionally — a diagnostic split of BioParsQA into fact-retrieval versus multi-step reasoning questions would show where the 29.99 ROUGE-L actually comes from.","My inference: the authors' own strongest unmet promise is clinical safety; a natural extension they mention but do not attempt is routing generated answers through the same two-doctor approval protocol used to build the dataset."],"forward_implications":["Persian medical question answering can be served by a dedicated, smaller model rather than a closed commercial API, with the reported numbers showing the largest gains where answers must be generated sentence by sentence rather than selected.","BioParsQA supplies the Persian biomedical community with a fixed, doctor-approved evaluation set of 5,231 question-answer pairs, so future Persian medical models can be compared on identical ground.","The reported shortfall of ChatGPT, Llama, and Galactica on higher-level and fine-grained questions is taken as evidence that zero-shot prompting of general LLMs is insufficient for Persian biomedical applications.","The architecture recipe — complex-domain damped EMA, TimestepNorm, and chunked normalized attention — is claimed to be a stable base for long-context autoregressive pretraining.","Because BioPars also posts the best scores on LiveQA and MedicationQA in the paper's tables, the authors' evidence suggests the gains are not confined to Persian."],"supporting_citations":[{"why":"BioMistral, the open medical LLM collection whose domain-pretraining approach and multilingual limitations BioPars positions itself against.","marker":"34"},{"why":"Defines BERTScore, one of the four metrics the comparison is reported on.","marker":"49"},{"why":"Defines ROUGE, the metric behind the headline ROUGE-L 29.99 result.","marker":"50"},{"why":"Defines MoverScore, the semantic-similarity metric used in the evaluations.","marker":"51"},{"why":"Defines BLEURT, the learned metric used in the evaluations.","marker":"52"},{"why":"The ChatGPT reference; GPT-4 is the strongest baseline BioPars must beat on BioParsQA.","marker":"53"},{"why":"Llama 2, the open 7B/13B baseline model in the comparison.","marker":"54"},{"why":"Galactica 30B, the science-specialized baseline model in the comparison.","marker":"55"},{"why":"Source of the dataset statistics for the English QA sets in the experimental protocol.","marker":"56"}],"fun_headline_variants":["First Persian biomedical LLM tops GPT-4 in QA","Persian med model beats GPT-4 on new QA benchmark","BioPars: Persian biomed LLM edges GPT-4 in long-form QA","Persian biomedical LLM outscores GPT-4 on its own test","New Persian biomedical model beats GPT-4 on medical QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result depends on BioPars having actually been trained on Persian biomedical text and then tested on questions it had never seen, yet the paper provides no training setup, data splits, model size, or leakage controls.","fun_headline_variants_meta":{"raw":{"variants":["First Persian biomedical LLM tops GPT-4 in QA","Persian med model beats GPT-4 on new QA benchmark","BioPars: Persian biomed LLM edges GPT-4 in long-form QA","Persian biomedical LLM outscores GPT-4 on its own test","New Persian biomedical model beats GPT-4 on medical QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000243,"raw_usage":{"total_tokens":1632,"prompt_tokens":1149,"completion_tokens":483,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":765,"completion_tokens_details":{"reasoning_tokens":393}},"tokens_in":765,"tokens_out":483,"duration_ms":5501,"temperature":1.0,"reasoning_tokens":393,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:24:20.634498+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask for the released weights and the exact pretraining and evaluation configuration, then rerun the BioParsQA evaluation on a split from which every reference answer has been confirmed absent from the training data; if the reported ROUGE-L 29.99 and BERTScore 90.87 cannot be reproduced, or if a contamination check finds BioParsQA texts inside the pretraining corpus, the claimed margin over GPT-4 is not established.","supporting_citations":[{"cited_title":"Y ., Miao, B","cited_arxiv_id":null,"evidence_quote":"The ChatGPT reference; GPT-4 is the strongest baseline BioPars must beat on BioParsQA."}],"review_version":1}