{"id":"36f83828-dcda-41b1-9971-81bd9603b867","arxiv_id":"2502.00063","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"LLAMA3-based NER preprocessing plus LoRA fine-tuning of CAMeL-BERT gives 83% disease type and 69% severity accuracy on Arabic telehealth posts, though without significance testing.","lead":"This paper tests whether preprocessing Arabic medical social media posts with LLAMA3 before fine-tuning BERT models improves disease classification. The best setup, CAMeL-BERT with NER-extracted entities, reached 83% accuracy for disease type and 69% for severity, but the gains are modest and the experiments lack error bars.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported NER gain lacks statistical grounding because no split, dataset size, or run variance is given; the 4–6 point difference could be noise.","rationale":"The paper's central empirical claim is that NER augmentation with LLAMA3 preprocessing improves fine-tuned Arabic BERT performance. The reader's weakest assumption correctly identifies the lack of leakage control and missing split details as unverified. In my reading, the most load-bearing issue is even narrower: the observed gains are not accompanied by any measure of uncertainty or dataset scale, so they may simply reflect test-set sampling noise. The only evidence for the NER advantage is a single pair of accuracy numbers per condition; no confidence intervals, standard deviations, or repeated runs are reported. The ambiguity around which custom loss function was used for each condition further weakens the causal attribution. However, the underlying methodology is coherent, the trend is consistent with related work on preprocessing, and the results are not implausible. Therefore the appropriate verdict remains CONDITIONAL rather than ACCEPT: the claim is plausible but requires the authors to release the split, dataset size, and either repeated runs or a bootstrap analysis. This does not change the reader's verdict, so I mark it UNCHANGED.","tokens_in":7127,"tokens_out":7180,"duration_ms":72794,"concrete_test":"Obtain or reconstruct the exact train/test split and dataset size, then fine-tune CAMeL-BERT under identical hyperparameters on normal versus NER-augmented text with at least 10 random seeds, evaluating on the same held-out test set. Compute the paired difference in accuracy (NER minus normal) with a bootstrap 95% confidence interval; if the CI includes zero or the Type gain is less than the standard error, the reported NER advantage should be treated as unverified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on a small, unreplicated accuracy gap: CAMeL-BERT with NER improves Type from 79% to 83% and Severity from 63% to 69% (Tables 2 and 3). The paper never states the dataset size, the train/test split, or the number of runs; Section 4.5 only says evaluation uses 'accuracy and balanced accuracy,' yet only raw accuracy is reported. Because the same tables show a 62–75% spread across models and conditions, a 4–6 point difference is within the range one would expect from test-set sampling noise on a small corpus. Additionally, Section 4.4 says 'both balanced and accuracy-weighted custom loss functions have been used' without mapping loss functions to models or conditions, so the NER result could be confounded by a different training objective. Without a fixed split or error bars, the claim that NER augmentation causes the improvement is not supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-layered framework for disease type and severity classification from Arabic social telehealth posts. The pipeline uses LLAMA3 for three preprocessing operations (text refinement, summarization, and named entity recognition), augments the original text with these outputs, and then fine-tunes three Arabic BERT variants (CAMeL-BERT, AraBERT, and Asafaya-BERT) with LoRA. The central empirical claim is that NER-augmented text combined with CAMeL-BERT fine-tuning yields the best results, with 83% type classification accuracy and 69% severity assessment accuracy, and that non-fine-tuned models perform much worse (13%-20% type, 40%-49% severity). The paper presents four evaluation tables and concludes that NER is the most effective preprocessing method.","tokens_in":1500,"tokens_out":3514,"duration_ms":53101,"significance":"If the result is robust, the framework offers a practical recipe for combining generative LLM preprocessing with efficient fine-tuning of smaller Arabic models, which is a plausible and potentially useful direction for social telehealth applications. The paper's strength is that the comparison spans three models and four preprocessing conditions, and the NER effect appears directionally consistent across model variants. However, the empirical support is currently too thin for the central claim: the manuscript does not report the dataset size, the train/test split, the number of runs, or any measure of variance, and the best configuration is selected post hoc. The NER gain of 4-6 percentage points could easily be within sampling noise on a small test set. With missing protocol details and no statistical grounding, the paper's main contribution is not yet established.","major_comments":[{"comment":"The central claim that NER augmentation improves accuracy from 79% to 83% (Type) and from 63% to 69% (Severity) is not statistically supported. The paper never states the dataset size, the train/test split, or the number of independent runs, and Tables 2-3 report only single accuracy values without confidence intervals or significance tests. Section 4.5 says evaluation uses 'accuracy and balanced accuracy,' yet only raw accuracy is reported. Given that the same tables show accuracy ranges of 62%-83% across models and conditions, a 4-6 point difference is within the range one would expect from test-set sampling noise on a small corpus. Please report the split, class counts, run-to-run variance, and the promised balanced accuracy.","section":"Section 4.5 and Tables 2-3"},{"comment":"The description of the loss function is a confound for the NER comparison. The paper states that 'both balanced and accuracy-weighted custom loss functions have been used' but never maps each loss function to a specific model or preprocessing condition. If different loss functions were used for the NER condition than for the baseline condition, the reported improvement could be due to the loss function rather than to NER augmentation. The authors must specify the loss function used in every reported row or demonstrate that the NER advantage persists under a single fixed loss.","section":"Section 4.4"},{"comment":"The best configuration is selected after comparing three models and four preprocessing methods, which is a post hoc selection. With no held-out validation set or correction for multiple comparisons, the 83%/69% result is an optimistic estimate of the framework's performance. The authors should either pre-register the comparison or validate the selected configuration on a separate test set that was not used during model selection.","section":"Section 5.5"},{"comment":"The dataset description is too vague to support reproducibility. Section 3 says the data was 'collected from user-generated posts on an online social platform' but does not give the platform, the collection period, the number of posts, the class distribution, the annotation guidelines, or the medical adviser's role. Without these details, and without a data availability statement or link, the empirical results cannot be verified or compared against future work.","section":"Section 3"},{"comment":"Table 1 is not usable as presented because the Arabic text is rendered as a corrupted character sequence (e.g., ' 65 AëY J...'). The table is meant to illustrate the output of the preprocessing steps, but the garbled rendering prevents the reader from checking the qualitative difference between text, refined, summarized, and NER outputs. Please re-render the table with a proper font/encoding.","section":"Table 1"}],"minor_comments":[{"comment":"The abstract and introduction mention GPT-3.5 Turbo as one of the LLMs, but the experiments use LLAMA3 for preprocessing and BERT variants for classification; GPT-3.5 Turbo is not used anywhere in the evaluation. Please align the abstract with the actual experimental scope.","section":"Abstract and Section 1"},{"comment":"The phrase 'deleting unmet requirements such as irrelevant information' is unclear; the intended meaning appears to be removing irrelevant or uninformative content. Please rephrase.","section":"Section 4.2"},{"comment":"There is a typographical error in 'to assess the the symptoms' and another in Section 5.4, 'summariza- rization.' Please proofread the manuscript.","section":"Section 4.3"},{"comment":"Figures 2 and 3 show distributions but do not include axis labels, numeric counts, or a legend, so the reader cannot determine the number of samples per condition type or severity level. Please add this information.","section":"Figures 2 and 3"},{"comment":"Reference [24] is cited for many heterogeneous claims and appears to be a catch-all source; some references also have formatting issues (e.g., [4] 'IEEE Access1', [5] missing venue details). Please standardize the bibliography.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's main weakness is the absence of a reproducible experimental protocol: no dataset size, no split, no error bars, and no loss-function mapping. The self-citations in the reference list are numerous but not central to the claim, so I would not weigh them heavily. The result may be salvageable if the authors are willing to rerun the experiments with a fixed, clearly described protocol, add variance and significance measures, and provide the data or a link to it. If the dataset is proprietary, that should be stated explicitly, but then the empirical claim will remain hard to evaluate externally."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe paper is a straightforward empirical application: LLAMA3 preprocesses Arabic health posts (refinement, summarization, NER) and then three Arabic BERT variants are LoRA-fine-tuned for disease type and severity classification. The best result is CAMeL-BERT with NER-augmented text at 83% / 69%. Nothing conceptually new here – every component is established – but the combination on this dataset has not been reported before, and the paper includes useful non-fine-tuned baselines that make the point that fine-tuning is essential.\n\nThe central claim (NER helps) is plausible and consistent across the three models in Table 3. I'm willing to believe that extracting entities as structured features gives the classifier a clearer signal. The paper also documents what it did in enough detail to see the pipeline, and the tables are readable.\n\nThe soft spots are in the evaluation. The paper never gives the dataset size, the train/test split, or the number of runs. Only raw accuracy is reported, despite Section 4.5 mentioning balanced accuracy. With a small corpus (likely a few thousand posts), a 4–6 point gap can easily be sampling noise. The stress-test note is right: the 79→83 and 63→69 differences are within the spread across models/conditions. There is also a confound: the method section says both balanced and accuracy-weighted custom loss functions were used, but doesn't say which model/condition got which. So the NER result might not be caused by the NER preprocessing alone. That's a real problem, not a nitpick.\n\nThe paper also overstates itself in the conclusion by calling its result \"a new benchmark\" – there is no shared dataset or comparison to prior work on this data, so that's a stretch.\n\nI also note the self-citations (refs 1,2,3,12,13,14,15,26 are from the same group). They are not load-bearing – the main result doesn't depend on them – so I don't see that as a serious issue.\n\nBottom line: this is honest progress in a narrow subfield, but the evidence for the headline claim needs more rigor before it can be trusted. A serious referee would want dataset details, a fixed split, error bars or significance tests, and an unambiguous mapping of loss functions to experiments. I would not cite the numbers as-is, but the paper is worth engaging with as a starting point for a more careful study.\n\nMy recommendation: treat this as a conditional accept if resubmitted with those details, and as a desk reject only if the authors won't share the data/code. It deserves a serious review, not because the result is proven, but because the question is legitimate and the preliminary evidence is suggestive.","headline":"Useful empirical application with a plausible but unproven NER gain; evaluation lacks the split/variance details needed to trust the headline numbers.","tokens_in":7807,"tokens_out":2429,"would_cite":false,"duration_ms":21108,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that preprocessing Arabic health posts with LLAMA3-based named entity recognition, then fine-tuning CAMeL-BERT with LoRA, achieves 83% accuracy on disease type and 69% on severity.","keywords":["Arabic NLP","disease classification","social telehealth","LLAMA3 preprocessing","named entity recognition","LoRA fine-tuning","CAMeL-BERT","severity assessment"],"falsifier":"A reproduction in which the test set is held out and unprocessed until after labels are assigned, and the NER augmentation is applied only to training features, would settle whether the 83% figure reflects leakage. A simpler variant: train the model on NER-augmented text but replace the NER output with random entities; if accuracy stays high, the entities themselves are not load-bearing.","tokens_in":6938,"feed_emoji":"🩺","tokens_out":4371,"duration_ms":39763,"temperature":0.7,"pith_summary":"The paper proposes a multi-layered framework for classifying Arabic social-media health posts into disease type and severity. It uses LLAMA3 to refine, summarize, and extract named entities from the posts, then fine-tunes three Arabic BERT models with LoRA on the enriched text. The central claim is that NER-augmented preprocessing gives the largest improvement, with CAMeL-BERT reaching 83% type accuracy and 69% severity accuracy. The paper also reports that without fine-tuning, the same models perform poorly, showing that preprocessing alone is not enough. A sympathetic reader would care because the framework suggests a practical way to turn noisy user-generated health text into structured signals for telehealth diagnostics.","feed_headline":"NER preprocessing lifts Arabic disease prediction to 83%","feed_subtitle":"LLAMA3 entity extraction beats summarization and refinement for classifying symptoms and severity.","key_machinery":"The key mechanism is a preprocessing pipeline in which LLAMA3 performs three operations on the raw Arabic posts: text refinement, text summarization, and named entity recognition (NER), the last of which extracts medical entities such as symptoms, conditions, and drugs. Each output is concatenated with the original text to create an augmented dataset, which is then used to fine-tune Arabic BERT models with LoRA (low-rank adaptation) under settings of dropout 5%, scale 8, rank 16, batch size 4, and 25 epochs. NER is the variant that carries the improvement, lifting CAMeL-BERT's type accuracy from 79% to 83% and severity from 63% to 69%.","core_discovery":"The central claim is that NER-augmented preprocessing with LLAMA3 combined with LoRA fine-tuning of CAMeL-BERT achieves the best accuracy, 83% for disease type and 69% for severity, on Arabic social telehealth posts. The paper states this result in the abstract and supports it with tables comparing normal, refined, summarized, and NER-enhanced text across three Arabic BERT variants. It further claims that non-fine-tuned models perform poorly (13%–20% type, 40%–49% severity), establishing that fine-tuning is required to exploit the benefits of preprocessing.","pith_inferences":["If the NER gain is causal, it may come from canonicalizing symptom mentions, which effectively reduces the input space for the classifier; a cheaper substitute might be entity masking or normalization without full LLAMA3 preprocessing.","The reported gap between fine-tuned and non-fine-tuned models is unusually large; a replication with random train/test splits and repeated runs would help rule out an artifact of a small or unrepresentative dataset.","The framework depends on a single LLAMA3 pass for all preprocessing; testing whether a smaller or distilled model yields the same gain would clarify whether the benefit is tied to the specific LLM or to the entity-extraction supervision itself."],"forward_implications":["NER-augmented preprocessing is the most effective of the three tested techniques: it raises CAMeL-BERT type accuracy from 79% to 83% and severity accuracy from 63% to 69%.","Text refinement gives a smaller gain (type 79% to 81%, severity nearly unchanged), and summarization gives little or no gain, so the choice of preprocessing matters more than the choice of Arabic BERT model.","Fine-tuning is essential: without it, even NER-enhanced text yields only 15%–20% type accuracy and 40%–42% severity accuracy, so the reported gains come from the combination of preprocessing and fine-tuning.","The same framework—LLM-based preprocessing followed by parameter-efficient fine-tuning—could be extended to other languages and other clinical classification tasks."],"supporting_citations":[{"why":"Provides the task context of evaluating large language models on health-related text classification from public social media data.","marker":"[11]"},{"why":"Motivates the preprocessing-augmentation approach for data-scarce contexts, reshaping inputs and refining outputs before classification.","marker":"[13]"},{"why":"Supports the LoRA fine-tuning strategy by discussing repeatability and memory efficiency of parameter-efficient fine-tuning.","marker":"[4]"},{"why":"Demonstrates medication detection in social health text with a fine-tuned transformer, grounding the NER preprocessing choice.","marker":"[7]"},{"why":"Provides the transfer-learning baseline for fine-tuning BERT variants on text classification tasks.","marker":"[24]"},{"why":"Establishes the importance of social media as a health information source, motivating the social telehealth dataset.","marker":"[8]"},{"why":"Offers a benchmark overview for health text mining tasks, giving the paper a reference point for classification targets.","marker":"[18]"}],"fun_headline_variants":["NER + LLAMA3 boosts Arabic disease prediction to 83%","With NER, CAMeL-BERT hits 83% on Arabic symptom posts","NER beats summarization for Arabic BERT fine-tuning","LLAMA3 NER text lifts Arabic BERT to 83% accuracy","CAMeL-BERT + LoRA: NER gives 83% on Arabic disease text"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The observed improvement from NER augmentation is causal and not an artifact of data leakage or random variation: the LLAMA3 preprocessing must not expose label information, and the evaluation split must be independent of the preprocessing step.","fun_headline_variants_meta":{"raw":{"variants":["NER + LLAMA3 boosts Arabic disease prediction to 83%","With NER, CAMeL-BERT hits 83% on Arabic symptom posts","NER beats summarization for Arabic BERT fine-tuning","LLAMA3 NER text lifts Arabic BERT to 83% accuracy","CAMeL-BERT + LoRA: NER gives 83% on Arabic disease text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000748,"raw_usage":{"total_tokens":3267,"prompt_tokens":817,"completion_tokens":2450,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":433,"completion_tokens_details":{"reasoning_tokens":2350}},"tokens_in":433,"tokens_out":2450,"duration_ms":15747,"temperature":1.0,"reasoning_tokens":2350,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T22:54:40.050271+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reproduction in which the test set is held out and unprocessed until after labels are assigned, and the NER augmentation is applied only to training features, would settle whether the 83% figure reflects leakage. A simpler variant: train the model on NER-augmented text but replace the NER output with random entities; if accuracy stays high, the entities themselves are not load-bearing.","supporting_citations":[{"cited_title":"Journal of the American Medical Informatics Association (2024) 10 M","cited_arxiv_id":null,"evidence_quote":"Provides the task context of evaluating large language models on health-related text classification from public social media data."},{"cited_title":"RIRO: Reshaping Inputs, Refining Outputs Unlocking the Potential of Large Language Models in Data-Scarce Contexts","cited_arxiv_id":"2412.15254","evidence_quote":"Motivates the preprocessing-augmentation approach for data-scarce contexts, reshaping inputs and refining outputs before classification."},{"cited_title":"IEEE Access1 (2024)","cited_arxiv_id":null,"evidence_quote":"Supports the LoRA fine-tuning strategy by discussing repeatability and memory efficiency of parameter-efficient fine-tuning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Demonstrates medication detection in social health text with a fine-tuned transformer, grounding the NER preprocessing choice."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the transfer-learning baseline for fine-tuning BERT variants on text classification tasks."},{"cited_title":"Jour- nal of Medical Internet Research23(5) (2020)","cited_arxiv_id":null,"evidence_quote":"Establishes the importance of social media as a health information source, motivating the social telehealth dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Offers a benchmark overview for health text mining tasks, giving the paper a reference point for classification targets."}],"review_version":1}