{"id":"54b73566-72af-449f-b311-210466b2a8b0","arxiv_id":"2606.08770","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":3.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Standalone decoder-only transformer tops binary hate-speech detection while soft-voting ensemble improves macro F1 by 15.8% on three-class sentiment for Nepali meme text.","lead":"The paper tests six transformer models plus hard and soft voting ensembles on text extracted from Nepali memes for binary hate-speech detection and three-class sentiment analysis. A generalist reader might examine it to see how standard NLP toolkits perform on low-resource code-mixed social-media content.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"OCR text may not capture full meme semantics, as paper acknowledges visual+textual nature but uses text-only input for all models and claims.","rationale":"The reader's weakest_assumption directly identifies the same load-bearing premise; the missing full experimental details compound the issue but the modality choice itself is the core risk to the performance claims even if details were present.","tokens_in":1666,"tokens_out":357,"duration_ms":16702,"concrete_test":"Sample 100 memes from the dataset; have two annotators independently transcribe the embedded text and note any visual context needed for correct hate/sentiment label. Compute OCR vs. human transcription CER and the fraction of samples where visual cues change the label. If CER >15% or >20% of labels shift with visual context, re-evaluate the top models after correcting OCR or adding image embeddings and check whether the 15.8% gap and best-model assignment persist.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on performance differences (decoder-only best for binary hate detection; soft-voting ensemble +15.8% Macro F1 for 3-class sentiment) obtained from OCR-extracted text. The abstract states memes combine visual and textual elements yet explicitly adopts a text-centric OCR approach for both tasks. In code-mixed Nepali memes, visual elements (image content, layout, stylized text) frequently carry or alter hate/sentiment meaning; OCR errors on non-standard fonts or mixed scripts would corrupt the input representation fed to every transformer and ensemble. If this representation is incomplete or noisy, the reported model rankings and the conclusion about task-dependent ensemble behavior rest on a potentially flawed premise rather than on the true meme distribution.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript describes a text-centric study on Nepali memes that extracts embedded text via OCR and evaluates six transformer-based models plus hard/soft voting ensembles on two tasks: binary hate speech detection and three-class sentiment analysis. It reports that a standalone decoder-only model achieves the highest performance on the binary task, while a soft-voting ensemble yields the best results on the multi-class task with a 15.8% relative Macro F1 improvement over the strongest standalone baseline. The work concludes that ensemble strategies behave differently across binary versus multi-class objectives.","tokens_in":1815,"tokens_out":499,"duration_ms":19608,"significance":"If the performance differentials are reproducible, the results supply concrete evidence that aggregation method choice should be task-dependent in low-resource code-mixed settings. The empirical comparison of standalone transformers versus ensembles on the same Nepali OCR data is a useful data point for practitioners. The text-only framing, however, restricts the scope to a subset of meme semantics and therefore limits the broader applicability of the task-dependent-ensemble claim.","major_comments":[{"comment":"Abstract: comparative performance figures (including the 15.8% relative Macro F1 gain) are presented without any description of training protocol, hyperparameter search, cross-validation procedure, statistical significance tests, or error bars. These omissions render the reported model rankings and the central claim about task-dependent ensemble behavior unverifiable from the given text.","section":"Abstract"},{"comment":"Abstract: the paper explicitly acknowledges that memes combine visual and textual elements yet adopts an exclusively OCR-text input for all models and ensembles. In code-mixed Nepali memes, visual layout, image content, and stylized text frequently modify or carry hate/sentiment meaning; if OCR errors or missing visual signals systematically distort the input, the observed ranking (decoder-only best for binary, soft-voting best for multi-class) rests on a potentially incomplete representation rather than the true data distribution.","section":"Abstract"}],"minor_comments":[{"comment":"Provide explicit citations and version numbers for all transformer checkpoints and the OCR engine used.","section":null}],"recommendation":"major_revision","confidential_remarks":"The title indicates a shared-task submission (CHIPSAL 2026); the manuscript should clarify whether the reported numbers follow the official task evaluation protocol and whether any external test-set leakage is possible."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We respond point-by-point to the major comments below.","responses":[{"response":"We agree that the abstract, as a concise summary, omits these details and that this reduces immediate verifiability of the rankings and the task-dependent claim. The full manuscript details the training protocol, grid-search hyperparameter tuning, 5-fold cross-validation, and paired statistical significance tests in the Experimental Setup and Results sections. We will revise the abstract to add a brief clause noting the cross-validation procedure and that reported improvements were assessed for statistical significance. This makes the central claims more self-contained while keeping the abstract concise.","revision_made":"yes","referee_comment":"[Abstract] Abstract: comparative performance figures (including the 15.8% relative Macro F1 gain) are presented without any description of training protocol, hyperparameter search, cross-validation procedure, statistical significance tests, or error bars. These omissions render the reported model rankings and the central claim about task-dependent ensemble behavior unverifiable from the given text."},{"response":"The manuscript explicitly scopes the work to a text-centric OCR approach, as stated in the abstract and introduction, to enable a controlled comparison of transformer models and ensembles on Nepali code-mixed text. We acknowledge that visual elements and potential OCR errors could alter inputs and thus the observed rankings. We will add an expanded Limitations paragraph discussing these factors and their implications for generalizing the task-dependent ensemble finding beyond the text-only setting. The reported results and model comparisons remain valid for the text representation used.","revision_made":"partial","referee_comment":"[Abstract] Abstract: the paper explicitly acknowledges that memes combine visual and textual elements yet adopts an exclusively OCR-text input for all models and ensembles. In code-mixed Nepali memes, visual layout, image content, and stylized text frequently modify or carry hate/sentiment meaning; if OCR errors or missing visual signals systematically distort the input, the observed ranking (decoder-only best for binary, soft-voting best for multi-class) rests on a potentially incomplete representation rather than the true data distribution."}],"tokens_in":1359,"tokens_out":460,"duration_ms":23343,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that this is an application paper: they take existing transformer models, run them on OCR-extracted text from Nepali memes, and compare hard and soft voting ensembles on binary hate detection versus three-class sentiment. A decoder-only model comes out on top for the binary task while soft voting gives a 15.8% relative Macro F1 lift on sentiment. That difference in ensemble behavior across task types is the one concrete observation worth noting.\n\nThey do a serviceable job of framing the code-mixing problem in Nepali memes and showing that aggregation choices can matter differently for binary versus multi-class settings. For anyone already working on the CHIPSAL shared task data, these numbers could serve as a quick baseline reference.\n\nThe soft spots are substantial and visible even from the abstract. There are no training details, hyperparameter protocols, significance tests, or error bars, so the reported numbers cannot be checked or reproduced. More critically, the authors note that memes combine visual and textual elements yet feed only OCR text to every model. In code-mixed Nepali memes, image content, layout, and stylized text frequently carry or change the intended meaning; OCR errors on non-standard fonts would further corrupt the input. This makes the model rankings and the ensemble conclusion rest on a potentially incomplete representation rather than the actual meme distribution.\n\nThe paper is mainly useful to researchers already participating in that specific shared task or studying low-resource Nepali NLP. Readers looking for new methods or multimodal work will find little. It shows clear thinking in the task setup but the evidence is too thin to support strong claims.\n\nI would not cite it without the full methods section and some error analysis. It could reasonably go to peer review if the authors add the missing protocol details and address the visual limitation, though it would need substantial revision.","headline":"This paper applies standard transformers and ensembles to OCR text from Nepali memes and reports task-dependent results, but supplies almost no experimental details and relies on a text-only premise that the authors themselves flag as incomplete.","tokens_in":2279,"tokens_out":456,"would_cite":false,"duration_ms":17688,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"For Nepali memes, a standalone decoder-only transformer leads binary hate speech detection while soft-voting ensembles raise macro F1 by 15.8 percent on three-class sentiment analysis.","keywords":["Nepali memes","hate speech detection","sentiment analysis","transformer models","ensemble learning","OCR","code-mixing","binary vs multi-class"],"falsifier":"A controlled experiment that adds visual features from the meme images and shows the performance gap between soft voting and the best single model shrinks below statistical significance or reverses.","tokens_in":2578,"feed_emoji":"","tokens_out":675,"duration_ms":11081,"temperature":0.7,"pith_summary":"The paper extracts text from Nepali memes via OCR and feeds it into six transformer models to tackle two separate tasks: spotting hate speech in a binary setup and classifying sentiment into three categories. It directly compares the models alone against versions combined by hard voting and soft voting. The key pattern is that the best aggregation method flips depending on the task, with no ensemble winning on the binary problem and soft voting delivering the reported gain on the multi-class one. A reader would care because code-mixed low-resource language content on social media is growing fast and current moderation tools often fail on it. The work supplies concrete baselines that future systems can beat or extend.","feed_headline":"Soft voting lifts Nepali meme sentiment F1 by 15.8%","feed_subtitle":"Standalone decoder-only model wins on binary hate speech; ensemble strategy flips for three-class task.","key_machinery":"Transformer-based architectures applied to OCR-extracted text, combined via hard and soft voting ensembles.","core_discovery":"A standalone decoder-only model achieved the highest performance for binary classification, whereas the Soft Voting ensemble performed best for the multi-class sentiment task, yielding a 15.8% relative improvement in Macro F1-score over the strongest standalone baseline. These findings suggest that ensemble strategies behave differently across binary and multi-class tasks, highlighting the importance of selecting aggregation methods suited to the classification objective.","pith_inferences":["Adding image encoders could test whether the current text-only ceiling is an artifact of the OCR step.","The same per-task ensemble pattern might appear in other code-mixed languages if the same six-model suite is reused.","Error analysis on the OCR layer itself would reveal how much token noise is baked into the current scores.","The binary-versus-multi-class divergence suggests a general rule for choosing voting schemes in low-resource NLP."],"forward_implications":["Ensemble choice must be tuned to whether the target task is binary or multi-class.","Text-only OCR pipelines can produce usable baselines even when memes contain both visual and textual information.","Decoder-only transformers are competitive or superior for binary hate detection in this language setting.","Soft voting yields measurable macro-F1 gains specifically on the three-class sentiment problem.","The reported 15.8 percent relative lift supplies a concrete target for later Nepali meme systems."],"fun_headline_variants":["Decoder-only model highest for binary hate speech in Nepali memes","Soft voting ensemble yields 15.8% F1 gain for sentiment analysis","Ensembles differ in performance across binary and multi-class tasks","Ensemble choice depends on binary versus multi-class objective"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"OCR-extracted text alone is sufficient to represent the semantics of memes for hate speech and sentiment decisions.","fun_headline_variants_meta":{"raw":{"variants":["Decoder-only model highest for binary hate speech in Nepali memes","Soft voting ensemble yields 15.8% F1 gain for sentiment analysis","Ensembles differ in performance across binary and multi-class tasks","Ensemble choice depends on binary versus multi-class objective"]},"model":"grok-4.3","cost_usd":0.009774,"raw_usage":{"total_tokens":4240,"prompt_tokens":607,"num_sources_used":0,"completion_tokens":68,"cost_in_usd_ticks":97740500,"prompt_tokens_details":{"text_tokens":607,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3565,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":607,"tokens_out":68,"duration_ms":23618,"temperature":1.0,"reasoning_tokens":3565,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T18:30:40.032336+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A controlled experiment that adds visual features from the meme images and shows the performance gap between soft voting and the best single model shrinks below statistical significance or reverses.","supporting_citations":[],"review_version":1}