{"id":"468681d2-5b45-4698-95e2-e56f8d328c02","arxiv_id":"2505.10260","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"GPT-4 achieved the highest F1 and accuracy among five LLMs for zero-shot and few-shot annotation of human rights violation references in Russian and Ukrainian Telegram posts.","lead":"This paper tests five large language models on the task of identifying social media posts that mention human rights violations in Russian and Ukrainian. It reports that GPT-4 performs best and that prompt language alignment and few-shot examples help some models, but statistical support is missing and some claims conflict with the paper's own numbers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GPT-4's 'highest overall performance' is not statistically established: no confidence intervals or significance tests are reported, and the best result is selected from 20 conditions.","rationale":"The reader's weakest assumption concerns the validity of the human gold standard. That is a reasonable concern, but it is not the most directly load-bearing issue for the paper's strongest claim. Even assuming the adjudicated labels are correct, the paper provides no statistical evidence that GPT-4's top F1 of 0.84 is meaningfully better than the closest competitors. With 1000 samples and 20 evaluated conditions, point-estimate differences can reflect sampling noise and multiple-comparison selection. A concrete paired test would settle this. I therefore keep the reader's CONDITIONAL verdict: the study is valuable and plausible, but the headline ranking must be tempered until uncertainty is quantified. My concern partially overlaps with the reader's because both undercut the reliability of the comparison, but I emphasize sampling and selection effects rather than label validity.","tokens_in":9638,"tokens_out":4742,"duration_ms":49260,"concrete_test":"Use the per-sample predictions from the paper's GitHub repository to recompute GPT-4 zero-shot English against the next-best rival configuration (e.g., GPT-3.5 few-shot Russian) with McNemar's exact test on the 1000 paired labels, and bootstrap 95% confidence intervals for F1 stratified by class. Apply a multiple-comparison correction across all 20 conditions. If the p-value exceeds 0.05 or the bootstrap intervals overlap, the claim that GPT-4 is 'highest overall' should be downgraded to 'not significantly different from the best rivals.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that GPT-4.0 is the best model, with F1=0.84 in zero-shot English—rests on point estimates only. Section 4 and Figure 1 report no confidence intervals, standard errors, or significance tests. On 1000 paired samples, a 6–12 point F1 or accuracy gap can easily arise by chance; the binomial 95% CI for accuracy at n=1000 is roughly ±2.5–3 points, and F1 CIs are wider because they combine precision and recall. Since the 'best' result is selected from 20 conditions (5 models × 2 prompt types × 2 languages), the maximum observed score is inflated even under the null hypothesis. This concern is independent of the gold-standard validity issue: even if the adjudicated labels are perfect, the ranking is not demonstrated. The internal contradiction—§3.2 and §5 claim Russian prompts generally outperform English, while GPT-4's reported best is English zero-shot—suggests the headline cell was chosen post hoc. A paired McNemar test plus bootstrap CIs would settle whether GPT-4's advantage is real or sampling noise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a comparative evaluation of five large language models—GPT-4.0, GPT-3.5, LLaMA-3 (described as LLaMA-3.2-1B in the method), Mistral-7B, and Claude-2—on the binary classification task of identifying references to human rights violations in a 1000-post sample of Russian and Ukrainian social media content from Telegram. The models are tested under zero-shot and few-shot prompting with prompts written in English and Russian, and their outputs are compared against a human double-annotated gold standard with adjudication. The authors report that GPT-4.0 achieves the best overall performance, with an F1 score of 0.84 and accuracy of 0.82 in zero-shot English prompting, that Russian prompts generally improve performance, that open-source models benefit more from few-shot prompting, and that models struggle on ambiguous cases, mirroring human difficulty. The paper also includes an error analysis and an ablation study on the subset of samples where human annotators disagreed.","tokens_in":9840,"tokens_out":4372,"duration_ms":40435,"significance":"If the reported ranking is statistically robust, the paper provides practically useful evidence on LLM suitability for high-stakes, multilingual annotation tasks in the human-rights domain. Its strengths include a realistic and difficult dataset, double-annotation with adjudication, a clear experimental matrix (two prompt types, two prompt languages, five models), fixed random seeds, and prompts made available in a GitHub repository. The error analysis and the agreement/disagreement ablation are informative. However, the central performance claims are supported only by point estimates, with no uncertainty quantification or significance testing, and several internal inconsistencies about the effect of prompt language undermine the general conclusions.","major_comments":[{"comment":"The central claim that GPT-4.0 is the best model, with an F1 score of 0.84 in zero-shot English prompting, rests entirely on point estimates without confidence intervals, standard errors, or significance tests. With n=1000, the approximate binomial 95% confidence interval for accuracy is about ±3 percentage points, and the F1 confidence interval is wider because it combines precision and recall. Moreover, the top score is selected from 20 conditions (5 models × 2 prompt types × 2 languages), which inflates the observed maximum even under the null hypothesis. To support the ranking, the authors should report bootstrap confidence intervals for F1 and accuracy and perform paired tests (e.g., McNemar) for the GPT-4.0 vs. GPT-3.5 comparison and for the prompt-language comparisons.","section":"Section 4 and Figure 1"},{"comment":"The paper makes contradictory claims about the effect of prompt language. Section 3.2 states that 'Russian prompts generally outperformed English prompts across models,' and the Key Findings section asserts that 'Models consistently performed better with prompts in Russian.' However, Section 4 reports GPT-4.0's best result is in zero-shot English, and Section 5.1 explicitly notes that Claude-2 and LLaMA-3 'generally performing worse on Russian prompts.' These statements cannot all be true as written. The authors should present a per-model and per-condition breakdown of the language effect and temper or qualify the general claim accordingly.","section":"Sections 3.2, 4, 5.1, and Key Findings"},{"comment":"The gold standard reliability is a load-bearing assumption. With an initial Cohen's Kappa of 0.63 and 184 of 1000 labels resolved by a single adjudicator, the reference labels themselves carry substantial uncertainty. Treating these labels as ground truth without a sensitivity analysis can overstate model performance and affect the ranking. The ablation in Figure 2 examines only GPT-4.0 and LLaMA-3 on the agreement/disagreement subsets; the authors should either extend this analysis to all models or explicitly discuss how potential adjudicator bias would affect the reported rankings.","section":"Section 3.1"}],"minor_comments":[{"comment":"The model is referred to as both 'LLaMA-3.2-1B' in the method and 'LLaMA-3' in the abstract and figures; please use a single consistent name and specify the exact model version.","section":"Section 3.2 and Abstract"},{"comment":"The few-shot setting does not state the number of examples used or how they were selected; please specify k and the selection procedure for reproducibility.","section":"Section 3.2"},{"comment":"The statement that all experiments used fixed random seeds is ambiguous for API-based models; report the sampling parameters (temperature, top-p, etc.) actually used for each model.","section":"Section 3.2"},{"comment":"The figures are referenced but not included in the provided text; ensure they are legible and include all reported metrics, and consider adding error bars or per-cell sample counts.","section":"Figures 1 and 2"},{"comment":"The reference list contains Törnberg (2023) twice; please deduplicate the entries.","section":"References"},{"comment":"The sentence referencing 'Mistral-7B Few-Shot (Russian)' provides no numerical results; consider adding a summary table of all experimental conditions to make the paper self-contained.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The dataset originates from the authors' own prior work (Nemkova et al. 2023). This is not necessarily a problem, but the relationship should be disclosed more prominently in the paper. More importantly, the manuscript currently lacks any statistical inference or multiple-comparison correction, which is essential for a ranking claim derived from many conditions. The paper appears to be a solid empirical study that needs a substantial statistical revision before it can be considered for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a straightforward empirical benchmark that will be genuinely useful to practitioners choosing an LLM for human-rights-violation annotation in Russian and Ukrainian Telegram data. The task domain is not covered in prior work, and the design—five models, zero- and few-shot, English and Russian prompts, against a 1000-sample adjudicated gold standard—is solid. The ablation on the 816 agreed versus 184 disputed samples is a nice touch. The paper also ships prompts, seeds, and code, which is more than many comparable papers do. The error analysis is qualitatively sensible, and the recommendation to use few-shot prompting for open-source models is plausible.\n\nThe soft spots are real but fixable. No confidence intervals, standard errors, or significance tests appear anywhere. With n=1000, a six- to twelve-point F1 gap can easily be sampling noise, and selecting the best cell from 20 conditions inflates the maximum regardless. A McNemar test or bootstrap would settle whether GPT-4's edge is real. The paper also contradicts itself: Section 3.2 and parts of Section 5 say Russian prompts generally outperform English, yet Section 5 says Claude-2 and LLaMA-3 do worse with Russian prompts, and GPT-4's reported best is English zero-shot. That inconsistency suggests the headline cell was chosen after the fact rather than reflecting a consistent finding. The gold standard itself deserves more scrutiny: initial kappa was 0.63, 184 disagreements were resolved by a single adjudicator, and the adjudicated labels are treated as ground truth throughout. That is not fatal, but a sensitivity analysis or even a paragraph acknowledging the uncertainty would help.\n\nNone of this undermines the core contribution. The paper is not circular; the evaluation is an independent comparison against human labels. There are no invented entities, no fitted parameters, and the limitations section is honest about sample size and language coverage. The abstract and conclusion do overclaim, calling GPT-4 \"ideal for high-stakes tasks\" without statistical qualification, but that is a revision issue, not a desk-reject issue.\n\nThis paper deserves a serious referee. The right referee will ask for significance testing, a reconciled story on prompt language, and a brief handling of gold-standard uncertainty. After that, it becomes a citable benchmark for the human-rights-NLP community. I would not cite it in my own work in the next year, but I would point practitioners to it.","headline":"Useful domain-specific benchmark, but the GPT-4 ranking is not statistically supported and the paper contradicts itself on prompt-language effects.","tokens_in":10346,"tokens_out":1849,"would_cite":false,"duration_ms":19061,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GPT-4.0 is the most reliable off-the-shelf labeler of human rights violation references in Russian and Ukrainian Telegram posts, reaching an F1 of 0.84 in zero-shot English prompting.","keywords":["human rights violations","large language models","text annotation","zero-shot prompting","few-shot prompting","multilingual social media","Russian and Ukrainian","Telegram"],"falsifier":"Re-annotate the 184 adjudicated posts with several independent expert annotators and measure each model's F1 against each annotator and against the adjudicated labels; if the ranking of GPT-4.0 changes materially, or if the adjudicated labels themselves show low expert agreement, the paper's performance claims are benchmark-dependent.","tokens_in":9447,"feed_emoji":"⚖️","tokens_out":3351,"duration_ms":31429,"temperature":0.7,"pith_summary":"This paper asks whether off-the-shelf large language models can label social media posts in Russian and Ukrainian for references to human rights violations as reliably as trained human annotators. It compares five models under zero-shot and few-shot prompting in English and Russian against 1000 human-labeled Telegram posts. The central finding is that GPT-4.0 performs best, reaching an F1 of 0.84 and accuracy of 0.82 with zero-shot English prompts, and that Russian prompts generally improve all models. Open-source models lag behind but gain substantially from few-shot examples, though their precision remains too low for unmonitored use. The upshot is a practical ranking of models and prompting strategies for a sensitive, multilingual classification task.","feed_headline":"GPT-4 tops five-model test labeling human rights posts","feed_subtitle":"Zero-shot English prompts hit F1 0.84 on Russian and Ukrainian Telegram data; Russian prompts help all models.","key_machinery":"The central object is the human-adjudicated gold-standard dataset: 1000 Telegram posts from public channels covering the Russia-Ukraine conflict, each carrying a binary label for whether it references a human rights violation. The evaluation machinery is a pair of prompting configurations, zero-shot and few-shot, each run in English and Russian, scored by precision, recall, F1, and accuracy, with an ablation study on the 816 posts where human annotators fully agreed and the 184 posts requiring adjudication.","core_discovery":"The paper claims that GPT-4.0 is the strongest off-the-shelf annotator for references to human rights violations in Russian and Ukrainian social media, achieving its top F1 of 0.84 and accuracy of 0.82 in zero-shot English prompting, with a precision of 0.78 and recall of 0.92. It further claims that prompting in Russian consistently improves performance across model architectures, and that few-shot prompting mainly helps open-source models such as LLaMA-3 and Mistral-7B, which still remain below the reliability needed for autonomous deployment in sensitive applications. The benchmark is a sample of 1000 Telegram posts, 517 labeled positive, double-annotated by native speakers with disagreements resolved by a senior adjudicator.","pith_inferences":["The paper treats the adjudicated human labels as ground truth; an editorial extension is that the absolute F1 numbers therefore measure agreement with one particular human judgment, not true detection accuracy, and should be read as upper bounds on real-world performance.","Because models show larger precision drops on the 184 disputed posts, a natural next step is to compare models against each annotator's original label rather than only the adjudicated label, which could reveal whether models align with one annotator's bias.","A testable extension is to replace binary gold labels with soft or probabilistic labels reflecting annotator disagreement; models that reproduce human uncertainty may be more useful for triage than models that mimic a single consensus label.","The same protocol could be applied to other conflict-reporting languages, but performance differences will likely depend on how well each model's training data covers that language."],"forward_implications":["If GPT-4.0's F1 of 0.84 generalizes beyond this sample, human rights monitors could use it to pre-screen large volumes of Telegram posts and reserve human review for flagged cases.","In settings where recall matters more than precision, GPT-3.5 with Russian few-shot prompts (recall 0.92) offers a cheaper screening option that catches most true positives.","Organizations using open-source models should treat their annotations as draft labels requiring manual validation, since LLaMA-3's Russian few-shot run reached recall 0.99 but precision 0.52.","Prompt language should match the data language regardless of model choice, because Russian prompts outperformed English prompts across tested models.","The disagreement ablation suggests that models, like humans, find ambiguous cases harder, so deployment should include an explicit review step for low-confidence predictions."],"supporting_citations":[{"why":"Supplies the source Human Rights Violations dataset of Telegram posts used to build the 1000-sample benchmark.","marker":"(Nemkova et al. 2023)"},{"why":"Provides the Kappa statistic used to measure the 0.63 inter-annotator agreement between the two primary human annotators.","marker":"(Cohen 1960)"},{"why":"Grounds the adjudication procedure for resolving the 184 annotator disagreements and treating the result as gold-standard labels.","marker":"(Artstein and Poesio 2008)"},{"why":"Establishes the in-context learning paradigm that motivates the few-shot prompting configuration tested in the experiments.","marker":"(Brown and Others 2020)"},{"why":"Supplies prior evidence that GPT-4 can outperform human experts and crowd workers in zero-shot political text annotation, motivating the comparison here.","marker":"(Törnberg 2023)"},{"why":"Documents the GPT-4 model used as the top-performing proprietary system.","marker":"(OpenAI 2023)"},{"why":"Describes the LLaMA model family from which the open-source LLaMA-3 in the study is drawn.","marker":"(Touvron et al. 2023)"}],"fun_headline_variants":["GPT-4 tops zero-shot labeling of human rights posts","Russian prompts improve LLM annotation of rights posts","GPT-4 best among five LLMs for rights post labeling","Zero-shot GPT-4 wins on RU/UA rights posts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire comparison rests on treating the human double-annotated labels, with an initial Kappa of 0.63 and 184 disagreements resolved by a single adjudicator, as correct ground truth for all 1000 posts; if those labels are biased or noisy, every model ranking and error analysis is measured against an unreliable benchmark.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4 tops zero-shot labeling of human rights posts","Russian prompts improve LLM annotation of rights posts","GPT-4 best among five LLMs for rights post labeling","Zero-shot GPT-4 wins on RU/UA rights posts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000517,"raw_usage":{"total_tokens":2504,"prompt_tokens":936,"completion_tokens":1568,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":1510}},"tokens_in":552,"tokens_out":1568,"duration_ms":10475,"temperature":1.0,"reasoning_tokens":1510,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:12:01.650864+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the 184 adjudicated posts with several independent expert annotators and measure each model's F1 against each annotator and against the adjudicated labels; if the ranking of GPT-4.0 changes materially, or if the adjudicated labels themselves show low expert agreement, the paper's performance claims are benchmark-dependent.","supporting_citations":[],"review_version":1}