{"id":"f6d06674-bb77-4984-b98f-ddeefb64387f","arxiv_id":"2412.13098","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new dataset of 14,169 categorized and geotagged Kenyan election citizen reports, with benchmarks showing few-shot LLMs reach volunteer-level agreement on topic labeling.","lead":"This paper releases Uchaguzi-2022, about 14,000 citizen reports on Kenya's 2022 election that volunteers labeled with topics, tags, and map coordinates, and tests whether language models can do that labeling automatically. It is a candidate resource for groups working on election monitoring and AI for social good in Africa.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test set for topic benchmarks is topic-stratified, so reported micro-F1 and the 'competitive with human baseline' claim may not reflect the natural data distribution.","rationale":"The reader's weakest_assumption focused on the reliability of the expert gold standard, but the more concrete and load-bearing issue is the construction of the test set itself. The stratified sampling in §2.4 means the test set has a very different topic distribution from the full dataset, and micro-F1 is distribution-dependent. This affects every number in Table 5 and Table 7, including the human baseline, so the central benchmark claim could change if the natural distribution were used. This is a specific, verifiable methodological concern rather than a subjective annotation-quality issue. The reader's concern about a single in-house expert is also real, but it is secondary to the distributional mismatch; even with a perfect gold standard, the reported micro-F1 would still be tied to a non-representative test set. The verdict remains CONDITIONAL because the paper could address this by reporting weighted or representative test-set results, but the current evidence does not support rejecting the dataset contribution itself.","tokens_in":19460,"tokens_out":6803,"duration_ms":64981,"concrete_test":"Recompute micro and macro F1 for Table 5 and Table 7 using inverse-propensity weighting to approximate the natural topic distribution (or evaluate on a fresh random expert-labeled sample from the held-out split). If the model ranking or the comparison to the volunteer-expert baseline changes materially—e.g., if XLM-RoBERTa (FS) matches or beats llama-3.1-70b on micro-F1 once Opinions is weighted at its natural 66.6% prevalence—then the headline claim is not robust to the test-set construction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The 500-report expert test set used in §4.1 is not a random sample of Uchaguzi-2022. §2.4 describes a topic-stratified sampling scheme: for each tag-bearing topic it takes either 100 or 10% of reports (whichever is smaller), then fills the remainder uniformly from topics without tags. As a result, Opinions—which is 66.6% of the full dataset—drops to about 20% of the test set, while rare topics (e.g., Irrelevant Report, Political Rallies, Staffing Issues) are oversampled by an order of magnitude. Because micro-F1 is a prevalence-weighted metric, this changes the reported numbers for every model and for the human baseline. The paper presents these numbers without reporting the test-set class distribution or any reweighted results. The central benchmark conclusion—that few-shot models are competitive with fine-tuned models and that llama-3.1-70b matches or exceeds the human baseline—may therefore be an artifact of the stratification rather than a property of the models.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Uchaguzi-2022, a dataset of 14,169 citizen reports on the 2022 Kenyan general election, collected via the Ushahidi platform. Each report is annotated by volunteers with a topic, topic-specific tags, a title, and geographic coordinates; the paper analyzes the dataset's coverage, language, geography, and temporal trends. The authors then benchmark models for automated topic classification, tag prediction, location extraction, and geocoding, comparing a fine-tuned XLM-RoBERTa model with few-shot in-context learning using gpt-4o, mixtral-8x7b, and llama-3.1-70b. The main empirical claims are that few-shot models are competitive with fully fine-tuned models on categorization, that llama-3.1-70b achieves micro-F1 54.6 on topic prediction versus a volunteer-expert agreement baseline of 50.4, and that ICL approaches substantially outperform a Wikipedia-trained NER model on location extraction and geocoding. The dataset and code are released under a licensing agreement.","tokens_in":19584,"tokens_out":7573,"duration_ms":65416,"significance":"If the benchmark results are robust, this paper makes a meaningful contribution to low-resource NLP and AI-for-social-good. The dataset fills a clear gap: it provides citizen-contributed election-issue reports with fine-grained topic tags and geotags for an African election, whereas existing election datasets focus mainly on sentiment or are based on news articles rather than citizen reports. The authors are transparent about the moderate inter-annotator agreement (Cohen's kappa 0.425 for topics), which is a strength. The paper also ships code and a data-access mechanism, supporting reproducibility. The central empirical conclusion—that few-shot LLMs can match or exceed the volunteer-expert agreement level—is plausible but depends on the construction of the expert-labeled test set, which is the main risk to the paper's claims.","major_comments":[{"comment":"The expert-labeled test set of 500 reports is not a random sample of Uchaguzi-2022. The sampling strategy in Section 2.4 deliberately oversamples rare topics (e.g., Irrelevant Report, Political Rallies) and undersamples the dominant Opinions topic, which is 66.6% of the dataset but only about 20% of the test set. Because micro-F1 is a prevalence-weighted metric, the reported values in Table 5 and the resulting comparison between fine-tuned and few-shot models, as well as the comparison against the human baseline, are specific to the stratified distribution and may not reflect performance on the natural data distribution. The authors should report the test set class distribution, provide reweighted or naturally-distributed results (e.g., by evaluating on a random sample or by weighting per-class F1), and discuss whether the conclusion that few-shot models are competitive with fully fine-tuned models remains valid under the natural distribution.","section":"Section 2.4 and Section 4.1.1, Table 5"},{"comment":"The silver geocoding evaluation set of 2,681 reports is constructed by using zero-shot gpt-4o prompting to filter the set of reports that 'likely include a location mention.' Since gpt-4o is also one of the models evaluated on this silver set (Table 10), the selection process is not independent of the evaluated systems and may bias the geocoding results in favor of gpt-4o and similar models. The authors should either construct the silver set using a method that is independent of the evaluated models (e.g., a rule-based heuristic or human annotation on a random subset) or provide a sensitivity analysis showing that the ranking of models does not change when a different filtering approach is used. Without this, the claim that 'ICL methods are still able to correctly geocode around 40% of reports without explicit location annotations' may not generalize to the broader population of unannotated reports.","section":"Section 4.2.2 and Appendix C"},{"comment":"The human baseline in Table 5 is computed as agreement between volunteer annotations and a single expert annotator, but the expert is also employed by Ushahidi, the same organization that trains and coordinates the volunteers. The moderate Cohen's kappa of 0.425 indicates that the gold standard itself carries substantial subjectivity. The paper should explicitly frame this baseline as agreement with one expert's opinion rather than as a ground-truth benchmark, and it should provide per-class human agreement to help interpret the macro-F1 numbers. This is particularly important because the conclusion that llama-3.1-70b is 'competitive with the human baseline' is based on micro-F1 exceeding 50.4, while its macro-F1 (39.7) is well below the human baseline (48.6).","section":"Section 2.4 and Section 4.1.1"}],"minor_comments":[{"comment":"The phrase 'with lower macro-averaged F1 but higher overall accuracy' is ambiguous; the table reports micro-F1 and macro-F1, not accuracy. Please clarify whether 'overall accuracy' refers to micro-F1 or to a separate accuracy metric, and if the latter, report it.","section":"Section 4.1.1, paragraph after Table 5"},{"comment":"The row 'Human annotation (baseline)' does not specify how the F1 scores are computed from volunteer and expert annotations. Please state explicitly which annotation set is treated as predictions and which as ground truth, and whether the reported values are micro/macro F1 over the same label set used for the models.","section":"Table 5"},{"comment":"The sentence 'For each task we omit labels with < 20 observations' should specify the resulting label sets, since this threshold directly affects the tag prediction tasks and the set of classes in Table 7.","section":"Section 3.1"},{"comment":"The column '% Empty Pred.' is not defined in the text. Please state explicitly that it is the percentage of samples for which the system returned no location, and indicate whether the denominator is the 147-sample evaluation set.","section":"Section 4.2.1, Table 8"},{"comment":"The example outputs in the location extraction prompt use inconsistent JSON keys: some use 'label' and others use 'location'. Please standardize the schema to avoid confusion in reproducing the prompt.","section":"Appendix C, Figure C2"},{"comment":"The claim that this is 'the first dataset of citizen-contributed election issues in the African continent that contains this associated metadata' would benefit from a more thorough related-work comparison, including earlier Ushahidi deployments in other African elections and any other crowdsourced election-monitoring datasets from Africa.","section":"Section 1, contribution 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the journal's scope well as a dataset and benchmark paper in NLP for social good. The central concern is whether the stratified expert test set inflates the reported few-shot performance relative to the natural data distribution; this is fixable with additional analysis, so I recommend major revision rather than rejection. I also note that the collaboration with Ushahidi is a strength in terms of domain authenticity, but the use of an Ushahidi expert as the gold standard means the human baseline is not fully independent; this should be acknowledged more prominently."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Uchaguzi-2022 is a genuinely useful resource: 14k citizen reports from the 2022 Kenyan election with volunteer-assigned topics, tags, and coordinates, plus an honest quality analysis. The benchmark experiments are secondary, and their absolute numbers should be read with caution, but the corpus itself deserves attention.\n\nWhat's new: as far as I know, this is the first public dataset of citizen-contributed election-issue reports from Africa that bundles topic labels, tags, and geotags. Prior election datasets are mostly sentiment or misinformation; conflict datasets like ACLED are news/NGO-sourced. The paper also ships code, prompts, and an inter-annotator analysis (kappa 0.425 on topics), which is more than most dataset papers do.\n\nSoft spots, in proportion. The 500-report test set is topic-stratified, not a random sample. Opinions is about 20% of the test set instead of 67% of the full data, and rare topics are oversampled. That means the reported micro-F1 numbers and the 'competitive with human baseline' claim are for a synthetic distribution. The comparison between models is still fair on the same test set, but the absolute numbers don't reflect the natural stream. The paper should report the test class distribution and reweighted micro-F1. The silver geocoding set is filtered by gpt-4o, one of the evaluated models, which inflates the absolute geocoding numbers for that model. Same fix: report the selection step's effect or use a model-independent filter. And the gold standard is a single Ushahidi employee, with moderate agreement to volunteers. That is honestly reported, but it means the 'human baseline' is not an independent benchmark.\n\nNone of these are fatal for the dataset contribution. They affect the benchmark claims, which the paper frames as demonstrations rather than deep findings. Missing variance or significance across runs is minor for a resource paper.\n\nWho this is for: anyone working on election monitoring, crisis informatics, or low-resource NLP for African languages. The corpus is the product; the modeling is routine. I'd cite it as a resource and would send it to review, asking for the test-distribution analysis to be added.","headline":"A solid citizen-report corpus for African election monitoring, with benchmark numbers that need a distributional caveat.","tokens_in":20258,"tokens_out":3478,"would_cite":true,"duration_ms":28279,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper presents Uchaguzi-2022, a dataset of 14,169 categorized and geotagged citizen reports from the 2022 Kenyan General Election, and shows that few-shot large language models can categorize reports as well as volunteer annotators…","keywords":["Uchaguzi-2022","citizen reports","election monitoring","Kenya 2022 general election","text categorization","geotagging","few-shot in-context learning","AI for Social Good"],"falsifier":"Re-annotate the 500-report sample with several independent experts who are not affiliated with the original annotators; if their pairwise agreement on topics is no higher than the current volunteer-expert kappa of 0.425, the human baseline against which models are compared is too unstable to support the claim that few-shot models are competitive.","tokens_in":19149,"feed_emoji":"🗳️","tokens_out":7760,"duration_ms":61670,"temperature":0.7,"pith_summary":"The paper introduces Uchaguzi-2022, a dataset of 14,169 citizen reports submitted during Kenya's August 2022 general election, each labeled by volunteers with a topic, optional topic-specific tags, and geographic coordinates. It claims this is the first dataset of citizen-contributed election issues on the African continent that combines categorization with geotagging. Using this dataset, the paper asks whether language models can do the labeling work at scale that volunteers currently do by hand. The answer it defends is yes in large part: a few-shot prompted 70-billion-parameter language model reaches a micro-averaged F1 of 54.6 on topic classification, slightly above the 50.4 agreement between volunteer annotations and a single expert labeler, and few-shot models also beat a fine-tuned encoder on rare topic classes. The same model family outperforms a Wikipedia-trained named-entity recognizer at extracting Kenyan locations, and geocoding resolves coordinates for about 40 percent of reports that carry no explicit location label.","feed_headline":"Few-shot AI matches human labels on 14k Kenyan election reports","feed_subtitle":"A new benchmark shows large language models sort Kenya's citizen election reports as well as volunteers do.","key_machinery":"The load-bearing object is the dataset itself: 14,169 volunteer-annotated reports plus a 500-report subset re-annotated by a single expert and used as the test set. The mechanism that carries the benchmark comparison is the pairing of two learning setups on the same test set: fully supervised fine-tuning of XLM-RoBERTa-base, and 25-shot in-context learning with large language models. Evaluation uses micro- and macro-averaged F1 for topic and tag prediction, with Cohen's kappa quantifying volunteer-expert agreement, and for geotagging it uses location-extraction exact match and ROUGE-L followed by geocoding coverage, accuracy at 161 km, and area under the curve. The few-shot prompts include topic and tag descriptions, which is what lets a model with no fine-tuning adapt to the taxonomy.","core_discovery":"The central claim is that crowdsourced election monitoring can be made scalable without sacrificing classification quality. Uchaguzi-2022 contains 14,169 reports across ten topics, dominated by Opinions (66.6 percent), with tags such as Biometric voter registration issues and Counting irregularities, and coordinates for 98.4 percent of reports. On the 500-report expert-labeled test set, the best few-shot model, llama-3.1-70b, achieves a micro F1 of 54.6 and macro F1 of 39.7 on topic prediction, improving on the volunteer-expert agreement baseline of 50.4 and 48.6 respectively, while the fully fine-tuned XLM-RoBERTa-base reaches 49.4 and 27.9. This makes few-shot in-context learning, not fine-tuning, the strongest configuration for these tasks, especially on rare classes. For geotagging, the same few-shot models extract location mentions far more accurately than a Wikipedia-trained NER system, and combining them with OpenStreetMap geocoding obtains correct coordinates for 68.7 percent of reports with labeled locations and 38.2 percent of reports without them.","pith_inferences":["An implication the paper leaves implicit is that the moderate volunteer-expert agreement of kappa 0.425 reflects genuine label subjectivity, so a practical pipeline should treat model outputs as a triage that surfaces confident and uncertain cases for volunteer review rather than as a replacement for human labeling.","The dataset's code-switched English/Swahili texts and long-tailed topic distribution could serve as a lightweight stress test for multilingual models in low-resource civic settings, going beyond the paper's own benchmark.","The paper leaves unquantified the volunteer-hours saved by automation; a natural next experiment would measure the end-to-end cost of a human-plus-model workflow against a human-only one using the same 500-report test set.","Because only 2.4 percent of reports carry a location text field while 98.4 percent have coordinates, the silver-geocoding result suggests that retrofitting existing archives with location names and edges could improve map-based analyses of past elections."],"forward_implications":["Because the best few-shot model slightly exceeds the human agreement baseline on micro F1, the paper argues it could generate silver annotations for the roughly 86,000 incoming reports that volunteers never labeled.","Few-shot in-context learning outperforms fine-tuning on underrepresented topics such as Irrelevant Report and Political Rallies, so rare-class performance is not a blocker for automation.","Both approaches fail entirely on Polling Station Administration and Staffing Issues, so fully automatic categorization still needs human review for these safety-relevant topics.","ICL location extraction combined with OpenStreetMap geocoding recovers coordinates for about 40 percent of reports without explicit location labels, indicating that missing locations on the platform are partly an annotation gap rather than an absence of location mentions.","The topic taxonomy is Kenyan-election-agnostic, so the categorization approach transfers across regions, while the tag taxonomy is Kenya-specific and would require redefinition."],"supporting_citations":[{"why":"It supplies XLM-RoBERTa-base, the encoder model fine-tuned for the fully supervised categorization baseline.","marker":"(Conneau et al., 2020)"},{"why":"It defines few-shot in-context learning, the prompting paradigm used for the decoder-only models.","marker":"(Brown et al., 2020)"},{"why":"It introduces llama-3.1-70b, the best-performing model in both categorization and geotagging.","marker":"(Dubey et al., 2024)"},{"why":"It introduces mixtral-8x7b, one of the three few-shot models benchmarked.","marker":"(Jiang et al., 2024)"},{"why":"It introduces gpt-4o, the model used for few-shot categorization, location extraction, and filtering of the geocoding evaluation set.","marker":"(OpenAI et al., 2024)"},{"why":"It provides wikiNEuRal, the Wikipedia-trained NER baseline that the ICL location extraction outperforms.","marker":"(Tedeschi et al., 2021)"},{"why":"It supplies the AUC geocoding evaluation metric used to compare geocoding accuracy.","marker":"(Jurgens et al., 2015)"},{"why":"It provides the threshold interpretation of Cohen's kappa used to call volunteer-expert topic agreement 'moderate.'","marker":"(McHugh, 2012)"}],"fun_headline_variants":["Few-shot AI matches human labels on 14k Kenyan election reports","14k Kenyan citizen reports: few-shot LLMs match volunteer accuracy","Uchaguzi-2022: few-shot AI rivals human coders on 14k reports","Few-shot LLMs beat fine-tuning on 14k Kenyan election reports"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark comparison assumes that the 500-report expert-annotated test set is a reliable gold standard, but the expert belongs to the same organization as the volunteers and agrees with them only moderately on topics, so any noise in the expert labels would shift the model rankings.","fun_headline_variants_meta":{"raw":{"variants":["Few-shot AI matches human labels on 14k Kenyan election reports","14k Kenyan citizen reports: few-shot LLMs match volunteer accuracy","Uchaguzi-2022: few-shot AI rivals human coders on 14k reports","Few-shot LLMs beat fine-tuning on 14k Kenyan election reports"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001036,"raw_usage":{"total_tokens":4361,"prompt_tokens":943,"completion_tokens":3418,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":3334}},"tokens_in":559,"tokens_out":3418,"duration_ms":22427,"temperature":1.0,"reasoning_tokens":3334,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:27:14.429862+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the 500-report sample with several independent experts who are not affiliated with the original annotators; if their pairwise agreement on topics is no higher than the current volunteer-expert kappa of 0.425, the human baseline against which models are compared is too unstable to support the claim that few-shot models are competitive.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the AUC geocoding evaluation metric used to compare geocoding accuracy."}],"review_version":1}