{"id":"d9316fad-630e-4c95-8234-da829ba01507","arxiv_id":"2412.17131","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"LoRA fine-tuning of Nemo-Instruct achieves the best F1 among four LLMs on Devanagari hate speech detection (90.05%) and target identification (71.47%), but without baseline comparisons the approach's efficacy is unproven.","lead":"This paper fine-tunes four large language models with LoRA to detect hate speech and to identify its targets in Hindi and Nepali text written in the Devanagari script. The best model, Nemo, reaches 90% F1 for hate detection, but the paper reports no comparison against simpler baselines, so the efficacy claim is not yet established.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim rests on weighted F1; class-wise F1 on hate (53.51%) and community (40.74%) is poor, so efficacy is unproven absent baseline comparison.","rationale":"I reviewed the central claim that PEFT/LoRA is effective for Devanagari hate speech detection and target identification. The most load-bearing concern is not the authors' stated assumption that base models have Devanagari knowledge, although that is untested, but the evaluation metric used to support the claim. The class-wise results the paper honestly reports show F1 of 53.51% on the hate class and 40.74% on the community target class, with recall of 45.68% for hate and 36.07% for community. On a test set where 88% of instances are non-hate, weighted F1 is dominated by the majority class and does not measure the core detection capability. No majority-class baseline, no standard fine-tuned baseline, and no variance analysis is provided, so the reader cannot assess whether 90.05% reflects meaningful hate detection or mainly prediction of the majority class. This is an internal consistency issue: the paper's own Tables 5 and 6 undercut the abstract's 'efficacy' claim. I therefore agree with the reader's CONDITIONAL verdict, though I locate the main weakness in metric choice and missing baselines rather than in the base-model-knowledge assumption; hence 'partial' agreement. The concrete test of adding macro-F1 and standard baselines would settle whether the concern lands.","tokens_in":8069,"tokens_out":6849,"duration_ms":63808,"concrete_test":"Use the released code and data to recompute Tables 3 and 4 with additional columns for hate-class F1, community-class F1, and macro-F1, and run a majority-class classifier and a fine-tuned multilingual BERT on the identical train/test splits. If Nemo's macro-F1 or hate-class F1 is not clearly above these two baselines, the paper's efficacy conclusion is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section 6 conclude that PEFT is 'efficacious' for Devanagari hate speech detection based on weighted F1 scores (Table 3: Nemo 90.05%; Table 4: 71.47%). Yet the paper's own class-wise analysis (Table 5) shows the hate class, which is the class the task is designed to detect, receives F1 53.51% with recall 45.68%, meaning more than half of hate instances are missed. Table 6 shows the community target class at F1 40.74%. Because the test set is 88.4% non-hate (3601/4076), weighted F1 is dominated by the majority class and can mask such failures. No majority-class baseline, no macro-F1, no fine-tuned small-model baseline, and no CHiPSAL official baseline is reported, so there is no way to tell whether 90.05% demonstrates genuine detection ability or is largely majority-class prediction. This is an internal evaluation issue, not a disagreement with external consensus.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a Parameter-Efficient Fine-Tuning (PEFT) system, specifically LoRA, for two shared-task subtasks on Devanagari-scripted data from Thapa et al. (2025): binary hate speech detection and three-way hate speech target identification in Hindi and Nepali. Four quantized LLMs (Llama-3.1-8B, Phi-3-medium, Qwen2.5-7B-Instruct, and Nemo-Instruct-2407) are fine-tuned with rank-16 LoRA on a 16GB T4 GPU, and the authors report that Nemo achieves the best weighted F1 scores of 90.05% and 71.47% on the two tasks. The paper includes class-wise results and confusion matrices showing much weaker performance on the minority classes (hate F1 53.51%; community target F1 40.74%), which the authors attribute to class imbalance. Code is made publicly available.","tokens_in":8269,"tokens_out":2276,"duration_ms":22731,"significance":"If the central claim is accepted, the paper would provide a useful data point: a lightweight, single-GPU LoRA recipe can produce competitive hate speech detectors for two under-resourced Devanagari languages. The evaluation is conducted on an externally curated shared-task test set, and the code release is a genuine practical contribution. However, the paper's central claim of 'efficacy' is not yet supported because no baseline comparisons are reported: there is no majority-class baseline, no full fine-tuning or in-context learning comparison, and no official CHiPSAL baseline or prior published result on the same data. Given the heavily imbalanced test set (88.4% non-hate), the reported weighted F1 can be dominated by majority-class performance, so the paper's contribution is currently a description of a reproducible system rather than an evidence-backed demonstration of effectiveness.","major_comments":[{"comment":"The central claim that PEFT is 'efficacious' rests entirely on weighted F1 scores, but no baseline is reported. The test set for hate speech detection is 88.4% non-hate (3601 of 4076 instances), so weighted F1 can be high even for a model that predicts the majority class most of the time. To support the efficacy claim, the paper needs at least a majority-class baseline, macro-F1, and preferably a comparison with full fine-tuning, in-context learning, a smaller fine-tuned model such as mBERT or XLM-R, and the official CHiPSAL baselines from Thapa et al. (2025). Without these, the reader cannot distinguish genuine detection ability from majority-class prediction.","section":"Section 6, Tables 3 and 4"},{"comment":"The class-wise results undermine the conclusion in Section 7 that the methodology 'demonstrates good performance.' For the very class the task is designed to detect, Nemo achieves F1 of only 53.51% with recall 45.68%, meaning more than half of hate instances are missed. Similarly, the community target class has F1 40.74% and recall 36.07%. These numbers are not merely a minor caveat: they show that the model largely fails on the classes of primary interest. The paper should either temper the efficacy claim, report a cost-sensitive or class-balanced variant, or provide explicit evidence that these minority-class results are competitive with existing systems on the same data.","section":"Section 6.1, Tables 5 and 6, Figures 1 and 2"},{"comment":"The experimental section reports a single run for each model and gives no information about random seeds, validation-based early stopping, or variance across runs. LoRA fine-tuning of a 7-8B model on only 484 or 475 validation instances is likely to be sensitive to initialization and data ordering. At minimum, the authors should report results over multiple seeds with standard deviations, or state that only one run was performed and that the reported numbers are therefore point estimates without a measure of stability.","section":"Section 5, Experiments"},{"comment":"The paper's stated assumption that 'the LLMs will have some knowledge of devanagari languages for PEFT to work' is untested. This matters because the authors themselves note in Section 4 that LLMs 'struggle on low resource languages.' A simple zero-shot or few-shot evaluation of the same models on the same test sets would directly probe this assumption and would also provide a natural lower-bound baseline for the LoRA results. As written, the possibility that the reported F1 scores are driven largely by the majority class plus shallow lexical cues is not excluded.","section":"Section 8, Limitation"}],"minor_comments":[{"comment":"The model name is misspelled as 'Lama-3.1' instead of 'Llama-3.1'.","section":"Table 4"},{"comment":"The sentence 'techniques such as in-context learning (ICL) are increase the cost and latency' contains a grammatical error and should be rewritten.","section":"Section 2"},{"comment":"\"lantency\" should be \"latency\".","section":"Section 4"},{"comment":"The citation to Cassano et al. (2024) concerns low-resource programming languages, not natural languages; a more directly relevant citation on LLM performance for low-resource natural languages would be appropriate.","section":"Section 4"},{"comment":"The hyperparameter choices (rank=16, alpha=16, no dropout, 2 or 4 epochs) are stated but not justified; a sentence explaining why these values were chosen or that they were not tuned would improve reproducibility.","section":"Section 5"},{"comment":"The claim that target identification is harder because it has three classes and hate speech detection has two is too brief; the three-class task also has a severe minority-class problem (61 community instances in the test set), which is the more relevant explanation for the lower overall F1.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"This is a shared-task system description paper. The main weakness is not that the numbers are low; it is that without any baseline comparison, the paper cannot support its headline efficacy claim. If the authors add baselines (especially official CHiPSAL baselines and a majority-class baseline) and macro-F1, the contribution would be a useful and reproducible system description. The paper may also be stronger if positioned explicitly as a system-report rather than as a general claim about PEFT for Devanagari hate speech detection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a standard shared-task system description, not a research contribution. What it does well: it runs LoRA fine-tuning of four 7-8B models on the CHiPSAL Devanagari hate speech detection and target identification datasets, reports per-class F1 and confusion matrices, releases code, and runs on a 16GB T4 GPU. The class-wise analysis is honest: the authors show hate class recall is 45.68% and community F1 is 40.74%, and attribute it to imbalance. The limitation statement explicitly assumes the base LLMs have some Devanagari knowledge, which is the right thing to flag.\n\nThe soft spot is that the abstract and Section 6 call the approach 'efficacious' based on weighted F1 — 90.05% for Nemo on detection, 71.47% on target ID. With a test set that is 88.4% non-hate, a constant 'not hate' predictor already gets high accuracy, and the weighted F1 is dominated by the majority class. Nemo's 90% is only about two points above the majority-class baseline, and the hate class, the one the task is about, gets F1 53.5% with recall under half. There is no macro-F1, no majority-class baseline, no fine-tuned BERT-style baseline, and no CHiPSAL official baseline reported. So the central claim is not supported by the reported metrics. Also, no seeds or error bars are given, which is typical for shared task papers but still a limitation.\n\nThe paper's real value is a reproducible recipe: LoRA with rank=alpha=16, 4-bit quantization, Unsloth, 2-4 epochs on one GPU. That is useful for practitioners in low-resource moderation. But it is an application note, not a new method.\n\nRecommendation: send it to peer review if the venue is a workshop or shared-task track, with the request that the authors add at least a majority-class baseline and preferably macro-F1 and the CHiPSAL baseline before the efficacy claim is accepted. It is not a strong candidate for a main conference, but it deserves a fair referee.","headline":"A reproducible shared-task system description whose 'efficacy' claim rests on weighted F1; the hate-class F1 is 53.5% and no baselines are reported.","tokens_in":8822,"tokens_out":2354,"would_cite":false,"duration_ms":20872,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LoRA-based parameter-efficient fine-tuning lets a single 16GB GPU reach 90.05% weighted F1 for hate speech detection and 71.47% for target identification in Hindi and Nepali Devanagari text, with a 4-bit quantized Nemo model beating…","keywords":["hate speech detection","Devanagari script","Hindi","Nepali","parameter-efficient fine-tuning","LoRA","low-resource NLP","large language models"],"falsifier":"Take a published LLM with no meaningful Devanagari training data, run the same LoRA recipe, and compare hate-class F1 against an always-predict-'not hate' baseline; if the hate-class F1 stays at or near the majority baseline, the reported scores are carried by base-model language knowledge rather than by PEFT.","tokens_in":7884,"feed_emoji":"🛡️","tokens_out":8522,"duration_ms":65322,"temperature":0.7,"pith_summary":"This paper argues that parameter-efficient fine-tuning with LoRA is a practical route to hate speech detection and target identification in Devanagari-scripted, low-resource languages. Instead of fully fine-tuning large models, it freezes the base LLM and trains only small low-rank matrices on a single 16GB T4 GPU after 4-bit quantization. On the CHiPSAL shared-task dataset of Hindi and Nepali Devanagari text, the best adapted model, Nemo-Instruct-2407, reaches weighted F1 of 90.05% for hate speech detection and 71.47% for target identification, outperforming the larger Llama-3.1-8B. A sympathetic reader would take this as evidence that lightweight adaptation can put LLM-based moderation within reach where full training is not.","feed_headline":"LoRA on one 16GB GPU hits 90% F1 for Devanagari hate speech","feed_subtitle":"Nemo, fine-tuned with LoRA, beats larger Llama on Hindi and Nepali hate detection and target identification.","key_machinery":"The central mechanism is LoRA (Low-Rank Adaptation), in which each weight update is decomposed into the product of two low-rank matrices, $W' = W + \\Delta W = W + AB^T$, inserted into the model's attention layers; only $A$ and $B$ are trained while the pretrained weights stay frozen. The paper combines this with 4-bit quantization and a 16GB NVIDIA T4 GPU, fine-tuning for 2 epochs on hate detection and 4 epochs on target identification. The experimental comparison across Llama-3.1-8B, Nemo-Instruct-2407, Qwen2.5-7B-Instruct, and Phi-3-medium-4k-Instruct is what carries the claim that the approach is effective.","core_discovery":"On the paper's own terms, the discovery is that LoRA, a parameter-efficient fine-tuning method, is sufficient to make general-purpose LLMs competitive on Devanagari hate speech tasks. For a frozen weight matrix $W$, LoRA learns a low-rank update $\\Delta W = AB^T$ and sets $W' = W + AB^T$, so the number of trainable parameters stays small and the update can later be folded back into the weights with no extra inference cost. Applying this recipe to four quantized LLMs, the paper reports that Nemo-Instruct-2407 performs best, with weighted F1 scores of 90.05% for hate detection and 71.47% for target identification. The class-wise results show the remaining weakness: hate class F1 is 53.51% against 94.86% for non-hate, and community-target F1 is 40.74% against 78.04% for individual targets.","pith_inferences":["The paper does not report base-model Devanagari language knowledge before adaptation, so a natural next test is to measure zero-shot Devanagari perplexity or classification accuracy on the same test set; that would separate what LoRA adds from what the pretrained model already knows.","Since the reported F1 values are weighted averages, macro-F1 or per-class AUC would give a fairer ranking across models when the test set is imbalanced.","The same recipe could be probed on code-mixed Hinglish or other Indic scripts; if transfer holds, PEFT could serve as a cheap multilingual moderation baseline.","Comparing these LoRA results against an always-majority-class baseline and against full fine-tuning at the same compute budget would sharpen the claim that PEFT, not just the base model, drives the gains."],"forward_implications":["A lightweight LoRA recipe on one 16GB GPU is enough to produce competitive Devanagari hate speech detectors, so deployment no longer requires the compute budget of full fine-tuning.","Bigger base models are not automatically better: Nemo-Instruct-2407 (6.97B) beats Llama-3.1-8B on both tasks, so model selection matters more than parameter count.","Class imbalance, not the adaptation method, is the main remaining bottleneck: hate-class F1 is 53.51% versus 94.86% for non-hate, and community-target F1 is 40.74% versus 78.04% for individual targets.","After training, LoRA updates are folded back into the model weights, so the fine-tuned system runs at the same inference latency as the base model.","The identical PEFT pipeline can be carried over to other Devanagari-scripted or low-resource languages, as the paper notes."],"supporting_citations":[{"why":"Defines the CHiPSAL shared task and supplies the Hindi and Nepali Devanagari dataset on which all results are reported.","marker":"Thapa et al. (2025)"},{"why":"Introduces LoRA, the low-rank weight-update mechanism that the whole approach is built on.","marker":"Hu et al. (2021)"},{"why":"Releases Nemo-Instruct-2407, the base model with the best F1 on both tasks.","marker":"AI and NVIDIA (2023)"},{"why":"Provides Llama-3.1-8B, the largest model compared and the main baseline Nemo outperforms.","marker":"Dubey et al. (2024)"},{"why":"Provides Qwen2.5-7B-Instruct, one of the four evaluated base models.","marker":"Yang et al. (2024)"},{"why":"Provides Phi-3-medium-4k-Instruct, one of the four evaluated base models.","marker":"Abdin et al. (2024)"},{"why":"Establishes the PEFT-for-low-resource-LLM classification direction that motivates using LoRA rather than full fine-tuning.","marker":"Patwa et al. (2024)"}],"fun_headline_variants":["LoRA fine-tunes Nemo for 90% F1 on Devanagari hate speech","Parameter-efficient LoRA beats full fine-tuning for Hindi and Nepali hate detection","Nemo with LoRA tops Devanagari hate speech, but weak on minority classes","Low-rank updates make LLMs competitive on Devanagari hate speech tasks","PEFT with LoRA: 90% F1 for Devanagari hate detection on a single GPU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole approach assumes the pretrained LLMs already know enough Devanagari for LoRA to make useful adjustments, and the paper never measures that prior knowledge.","fun_headline_variants_meta":{"raw":{"variants":["LoRA fine-tunes Nemo for 90% F1 on Devanagari hate speech","Parameter-efficient LoRA beats full fine-tuning for Hindi and Nepali hate detection","Nemo with LoRA tops Devanagari hate speech, but weak on minority classes","Low-rank updates make LLMs competitive on Devanagari hate speech tasks","PEFT with LoRA: 90% F1 for Devanagari hate detection on a single GPU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000677,"raw_usage":{"total_tokens":3063,"prompt_tokens":911,"completion_tokens":2152,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":2036}},"tokens_in":527,"tokens_out":2152,"duration_ms":13535,"temperature":1.0,"reasoning_tokens":2036,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:45:00.133299+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a published LLM with no meaningful Devanagari training data, run the same LoRA recipe, and compare hate-class F1 against an always-predict-'not hate' baseline; if the hate-class F1 stays at or near the majority baseline, the reported scores are carried by base-model language knowledge rather than by PEFT.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Releases Nemo-Instruct-2407, the base model with the best F1 on both tasks."},{"cited_title":"Enhancing Low-Resource LLMs Classification with PEFT and Synthetic Data","cited_arxiv_id":"2404.02422","evidence_quote":"Establishes the PEFT-for-low-resource-LLM classification direction that motivates using LoRA rather than full fine-tuning."}],"review_version":1}