{"id":"fd9d93fe-346e-4fd2-aa8a-d3ef714d7a12","arxiv_id":"2501.00029","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A breadth-first survey of recent South Asian language NLP, speech, and multimodal research, organized with LLM-based classification and clustering.","lead":"This paper catalogs 2022-2024 research on text, speech, and multimodal AI for South Asian languages, with a spotlight on 21 low-resource languages. It uses large language models to sort and cluster the literature, giving researchers a broad map of the field and its gaps.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quantitative findings rely on an unvalidated LLM classifier; the only reported precision/recall is low, so counts in Table 1 and Section 7 need independent verification before they can be cited.","rationale":"The reader identified the same load-bearing weakness: the survey's specific paper counts and trends rest on an LLM-based relevance classifier trained on only 20 titles, with no reported accuracy for the main categories and only low precision/recall reported for the spotlight variant. My independent reading of Sections 2, 7, and the Limitations section confirms this. The paper is transparent about the low precision/recall in its Limitations, which mitigates the concern but does not resolve it: the quantitative statements in Table 1 and Section 7 are still presented as definitive findings, not as rough estimates. For a breadth-first catalog, the qualitative list of themes and representative papers has some value even if counts are approximate, so a conditional acceptance requiring release of the data (queries, labels, predictions) and a documented accuracy evaluation is the right call. I would not move the verdict to REJECT because the paper's limitations are disclosed and the qualitative survey could still serve as a useful starting point. I also note that the spotlight counts use literal string matching, so they are not directly affected by the classifier weakness, but they have their own unvalidated false-positive/negative risks. Thus the reader's CONDITIONAL verdict remains appropriate.","tokens_in":30994,"tokens_out":6865,"duration_ms":57708,"concrete_test":"Manually audit a random sample of 50 papers from each predicted-relevant category in Table 1, recording whether each is a 2022-2024 paper focused on a South Asian language; also audit a sample of the predicted-irrelevant pool to estimate recall. If estimated precision or recall is below 0.8, the Table 1 counts should be presented as approximate ranges. Separately, run an independent Google Scholar query per spotlight language (e.g., 'Nepali' AND 'language model', 2022-2024) and compare the union with the paper's 63/58/57 counts; a discrepancy above 20% would undermine the 'most popular' ranking.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claims—Table 1's 369/84/52 relevant papers and Section 7's 'more than 50 papers each' for Nepali, Assamese, and Telugu—depend on a curation pipeline whose accuracy is not established. Section 2 describes a GPT-4o relevance classifier trained on only 20 hand-labeled titles; no precision/recall is reported for the text, multimodal, or speech classifiers that produced Table 1. The only numerical evaluation reported anywhere is in Section 7, which gives precision 0.33 and recall 0.7 for the spotlight classifier—so low that the author switched to literal string matching for the spotlight counts. If the main classifiers have similar precision, the 'relevant papers' counts could be inflated roughly threefold; if their recall is similar, they could miss a third of relevant work. The paper's Limitations section concedes 'low precision/recall persist,' yet the counts are still presented as findings. Neither the queries, the 20 labels, nor per-paper predictions are released, so the numbers cannot be independently checked. The spotlight counts, being string-match based, avoid the classifier problem but are still unvalidated against an independent search, leaving the 'most popular languages' claim with unknown error bars.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a breadth-first survey of NLP research in South Asian languages published between January 2022 and October 2024, covering text processing, multimodal models, and speech processing. The methodology uses Google Scholar searches, a GPT-4o-based relevance classifier trained on 20 hand-labeled titles, and an O1-based topic grouping, followed by a low-resource language spotlight on 21 languages using literal string matching. The paper reports overall relevant-paper counts (369 text, 84 multimodal, 52 speech) in Table 1 and per-language counts in Section 7 (e.g., Nepali 63, Assamese 58, Telugu 57), and identifies research themes, trends, challenges, and future directions.","tokens_in":31190,"tokens_out":4540,"duration_ms":36144,"significance":"If the quantitative findings were reliable, this survey would be a useful entry point for NLP researchers working on South Asian languages, particularly the low-resource spotlight that covers many under-served languages. The paper is transparent about its methodology and includes a Limitations section that acknowledges low precision/recall and topic cohesion issues, which is commendable. Its breadth-first perspective complements depth-first reviews such as Sankalp et al. (2024). However, the central counts and trends rest on an LLM relevance classifier whose accuracy is not reported for the main corpus, and the only reported metrics (precision 0.33, recall 0.7, in Section 7) are explicitly described as significantly low by the authors themselves, leading them to abandon the classifier for the spotlight section. The paper also contains a clear mis-citation in Section 5.2. These issues undermine confidence in the paper's main quantitative claims as they currently stand, although the survey's structure and curated bibliography retain value if the counts can be validated or appropriately caveated.","major_comments":[{"comment":"The main counts in Table 1 (369/84/52 relevant papers) and the theme-level trends in Sections 3–5 are produced by a GPT-4o relevance classifier, but no precision/recall or any other accuracy metric is reported for that classifier on the main corpus. The only numerical evaluation in the paper is in Section 7, where the spotlight relevance classifier is reported to have precision 0.33 and recall 0.7, a level the authors themselves call 'significantly low' and which motivated them to switch to literal string matching. Because the same classification approach underlies Table 1, the central quantitative claims are not established. The authors should provide per-category validation on a held-out labeled set (or at least a random sample) for the text, multimodal, and speech classifiers, or re-frame the counts as unverified with appropriate caveats and error bounds.","section":"Section 2 / Table 1 / Section 7"},{"comment":"The paper cites Spiesberger et al. (2023) as an example of 'speech-to-speech translation for English → {Hindi, Telugu, Gujarati, Marathi, Punjabi}' in Section 5.2. The cited reference, 'Abusive speech detection in indic languages using acoustic features' (INTERSPEECH 2023), is a paper about abusive speech detection, not speech-to-speech translation. This is a load-bearing error because it inflates the speech-to-speech translation theme and indicates that the relevance/topic classification pipeline can mis-assign papers to incorrect themes. The citation should be corrected, and the other papers listed in that theme should be manually re-verified.","section":"Section 5.2 / References"},{"comment":"The survey methodology is not reproducible. The Google Scholar queries, the 20 hand-labeled titles, and the per-paper relevance predictions from GPT-4o are not released, so the counts in Table 1 and Section 7 cannot be independently checked. The authors should provide these artifacts as supplementary material or a public repository. At a minimum, they should report the exact queries used for each language and category and the labeled examples used for in-context learning.","section":"Section 2"},{"comment":"The language-specific counts in Section 7 (e.g., 'Nepali, Assamese, and Telugu were the most popular languages, with more than 50 papers each since 2022') are based on literal string matching rather than the LLM classifier, which is an improvement, but the string-matching step is not validated against an independent search or manual review. Given that some language names (e.g., 'Burmese', 'Sindhi') can appear in non-linguistic or unrelated contexts, a manual verification of a sample, or a comparison with an alternative search strategy, is needed to support the 'most popular languages' claim with any confidence.","section":"Section 7"}],"minor_comments":[{"comment":"The reference list contains a duplicate entry for Majhi and Saha (2024): entries '2024a' and '2024b' have identical titles ('An automatic speech recognition system in odia language using attention mechanism and data augmentation') and appear to be the same paper. Please merge them and cite consistently, as both Section 6 and Section 7.7 cite these entries.","section":"References"},{"comment":"The reference 'Singh et al., 2024b' (MIMIC: Misogyny identification in multimodal internet content) is listed twice in the reference list, once as 2024a and once as 2024b, with identical titles. Additionally, Section 4.3 cites this same work as both 'multimodal misogyny identification' and 'multimodal hate speech detection'; if the paper is specifically about misogyny, the hate-speech attribution should be corrected or clarified.","section":"References / Section 4.3"},{"comment":"The heading 'Chattisgarhi, Dhivehi, Sairaki, Rajasthani, and Balochi' contains two spelling errors: 'Chattisgarhi' should be 'Chhattisgarhi' and 'Sairaki' should be 'Saraiki', matching the spellings used in the abstract and elsewhere in the paper.","section":"Section 7.17"},{"comment":"There is a typo in Section 7.7: 'We alsio found work' should read 'We also found work'.","section":"Section 7.7"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful scope and a candid Limitations section, but its central quantitative claims rest on classifiers whose accuracy is unreported in the main pipeline and explicitly poor in the one reported evaluation. The mis-citation of Spiesberger et al. (2023) in Section 5.2 is a concrete accuracy problem that also raises concerns about the reliability of the LLM-assisted theme assignment. I recommend major revision: the authors should validate (or substantially soften) the counts, correct the citation and reference duplication, and provide the queries/labels/predictions for reproducibility. If these are addressed, the survey could be acceptable for publication as a breadth-oriented resource."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a genuinely useful breadth-first survey of recent research in South Asian language processing, and it is more honest about its own limitations than most. The paper covers text, speech, and multimodal work from January 2022 to October 2024, and adds a spotlight on 21 low-resource languages that usually sit in the long tail. That alone is a service: anyone starting work in Nepali, Assamese, Bodo, or Sindhi can use the sections as a quick map of recent papers, themes, and gaps. The author also cites the closest prior survey (Sankalp et al. 2024) and positions this one as breadth-over-depth, which is the right framing. The main weak spot is quantitative. The 369/84/52 counts in Table 1 come from GPT-4o relevance classification trained on 20 labeled titles, with no precision or recall reported for those classifiers. The only numerical evaluation given anywhere is in Section 7, where the spotlight classifier got precision 0.33 and recall 0.7, and the author then switched to literal string matching for the spotlight counts. If the main classifiers behave anywhere near that badly, the Table 1 numbers could be substantially off. The string-match spotlight counts are more defensible but still not validated against an independent search, so the claim that Nepali, Assamese, and Telugu have more than 50 papers each has unknown error bars. The author does acknowledge low precision/recall in the Limitations section, but the counts are still presented as findings, and the underlying queries, labels, and model outputs are not released. That is the load-bearing soft spot: the survey's qualitative narrative survives, but its exact numbers should not be cited as fact. Second, there is a clear mis-citation. Section 5.2 cites Spiesberger et al. (2023) both for abusive speech detection and for speech-to-speech translation into multiple languages, but the reference is titled 'Abusive speech detection in Indic languages using acoustic features.' That needs a correction. Third, many reference entries are incomplete, with '...' placeholders scattered through titles and venues; this will annoy anyone trying to track down a paper. Who is this for? Researchers and students wanting a quick overview of recent South Asian NLP, speech, and multimodal work, and anyone interested in LLM-based literature review. It is not a deep scientific result. It deserves peer review rather than a desk reject, but the author should be asked to add precision/recall for the main classifiers, release query logs and labels, fix the citation error, and clean up the references. I would cite it as a finding aid, not for its numbers.","headline":"A useful, honest breadth-first survey of South Asian language work, but the headline counts rest on an unvalidated classifier and one clear citation error, so treat the numbers as approximate and send it to review with a request for reproducibility.","tokens_in":793,"tokens_out":1328,"would_cite":true,"duration_ms":53049,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A broad LLM-assisted survey maps 2022–2024 South Asian text, speech, and multimodal research and ranks low-resource languages by activity.","keywords":["South Asian languages","low-resource NLP","speech processing","multimodal models","LLM-based literature survey","Nepali language","Assamese language","Telugu language"],"falsifier":"Manually label a random sample of the papers the pipeline classified as relevant and irrelevant (for example, 200 from the 369 'relevant' language-model papers and 200 from the excluded set), then compare the labels with the classifier's decisions; if precision or recall on that sample is far below what the survey's counts assume, the field-level counts and the spotlight orderings would need to be revised. A simpler check: rerun the same search queries for the 21 spotlight languages using only exact language-name strings and see whether the counts reproduce.","tokens_in":30747,"feed_emoji":"📚","tokens_out":7217,"duration_ms":55770,"temperature":0.7,"pith_summary":"This paper is a broad survey of research published between January 2022 and October 2024 on text processing, speech processing, and multimodal models in South Asian languages. Its central claim is that the recent literature can be systematically counted and grouped into themes using an LLM-driven pipeline, and that doing so reveals where research effort is concentrated. The paper reports 369 relevant papers on language models, 84 on multimodal models, and 52 on speech processing, plus a spotlight on 21 low-resource languages in which Nepali, Assamese, and Telugu each have more than 50 papers since 2022. The value of the survey, if its counts are right, is a breadth-first map that helps new researchers see which tasks, languages, and benchmarks are active and which remain almost untouched.","feed_headline":"505 South Asian language tech papers tallied since 2022","feed_subtitle":"Nepali, Assamese, and Telugu top the low-resource list; gaps remain in Dari, Rohingya, Kurukh, Santali.","key_machinery":"The load-bearing mechanism is the paper's multi-stage curation pipeline: Google Scholar queries generated from a base prompt with GPT-4o, metadata compiled through Publish or Perish; a hand-labeled sample of 20 titles used as in-context examples for GPT-4o relevance classification of all titles; irrelevant or pre-2022 papers excluded; the O1 model groups the relevant papers into themes, after BERTopic produced less coherent clusters; and for the 21-language spotlight, literal string matching is used instead of the classifier because the classifier's precision (0.33) and recall (0.7) were too low. This pipeline is what turns a large, noisy literature into the paper's tables, counts, and trend claims.","core_discovery":"The paper claims to provide a systematic, breadth-first catalog of recent South Asian language technology research, covering text-based language models, multimodal models, and speech processing separately, and then focusing on 21 low-resource languages that are often overshadowed in multilingual work. It identifies major thematic clusters—machine translation and cross-lingual transfer, hate speech and sentiment, bias and fairness, LLM adaptation, datasets and benchmarks, image captioning, multimodal machine translation and hate speech, code-mixed speech, automatic speech recognition, and low-resource speech—and reports paper counts for each field. The spotlight section asserts that Nepali (63 papers), Assamese (58), and Telugu (57) were the most studied low-resource languages, while Kannada shows the most advanced speech-related work, and that no papers were found for Dari, Rohingya, Kurukh, and Santali. The accompanying trends and challenges sections argue that the field is growing quickly but that publication venues are fragmented, benchmarks are under-collected in multilingual LLM evaluations, code-mixing comparisons across languages are missing, and LLM-based meta-analysis still needs better precision and coherence.","pith_inferences":["My inference: because the spotlight counts rely on literal string matching and the paper found the classifier imprecise, the reported per-language counts are best read as lower bounds; work indexed under alternative names (the paper itself tried 'Rakhine' for Rohingya) could be missed.","My inference: the sharp gap between text papers (369) and speech or multimodal papers (84 and 52) may partly reflect how easily each type of work is discovered by title-based queries, so the true gap could be smaller than the counts suggest.","My inference: a direct replication using abstracts instead of titles, or using a larger hand-labeled sample, would test whether the relevance classifier's low precision changes the field-level trends; the paper's own limitation section invites this."],"forward_implications":["If the counts are right, the field's center of gravity is text-based language modeling: 369 relevant papers, versus 84 multimodal and 52 speech-processing papers.","Researchers looking for low-resource languages with enough literature to build on should start with Nepali, Assamese, and Telugu (more than 50 papers each), followed by Kannada (49), which the survey identifies as the most speech-advanced.","Dari, Rohingya, Kurukh, and Santali appear to have no dedicated papers in this window, signaling near-empty research space.","Aggregating the scattered South Asian benchmarks into a single hosted benchmark would help because, per the paper, many are left out of multilingual LLM evaluations.","LLM-based survey methods need better precision and recall and more coherent topics before their counts can be treated as stable."],"supporting_citations":[{"why":"Supplies the Publish or Perish software used to compile Google Scholar metadata for the initial candidate paper set.","marker":"(Harzing, 2010)"},{"why":"Provides GPT-4o, which generates search queries and predicts relevance labels for all candidate titles.","marker":"(OpenAI, 2024)"},{"why":"Provides the O1 model whose reasoning is used to cluster relevant papers into the survey's themes.","marker":"(Zhong et al., 2024)"},{"why":"Provides BERTopic, the alternative topic-modeling approach that was tried and rejected for producing less coherent topics.","marker":"(Grootendorst, 2022)"},{"why":"The closest prior review that prioritizes depth over breadth; the paper defines its own breadth-first contribution against it.","marker":"(Sankalp et al., 2024)"}],"fun_headline_variants":["505 South Asian language-tech papers, but 4 languages get none","Nepali, Assamese, Telugu top low-resource language AI papers","Systematic review catalogs 505 South Asian language tech papers","Catalog of South Asian language tech: 505 papers, gaps for 4 languages","505-paper review spots low-resource gaps in South Asian language tech"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The survey's counts and trends rest on the assumption that a GPT-4o relevance classifier trained on only 20 hand-labeled titles decides correctly which papers belong in the survey; the paper itself reports precision of 0.33 and recall of 0.7 for the spotlight classifier and gives no accuracy measures for the main field-level classifications.","fun_headline_variants_meta":{"raw":{"variants":["505 South Asian language-tech papers, but 4 languages get none","Nepali, Assamese, Telugu top low-resource language AI papers","Systematic review catalogs 505 South Asian language tech papers","Catalog of South Asian language tech: 505 papers, gaps for 4 languages","505-paper review spots low-resource gaps in South Asian language tech"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000778,"raw_usage":{"total_tokens":3440,"prompt_tokens":947,"completion_tokens":2493,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":2396}},"tokens_in":563,"tokens_out":2493,"duration_ms":16309,"temperature":1.0,"reasoning_tokens":2396,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:41:01.612415+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually label a random sample of the papers the pipeline classified as relevant and irrelevant (for example, 200 from the 369 'relevant' language-model papers and 200 from the excluded set), then compare the labels with the classifier's decisions; if precision or recall on that sample is far below what the survey's counts assume, the field-level counts and the spotlight orderings would need to be revised. A simpler check: rerun the same search queries for the 21 spotlight languages using only exact language-name strings and see whether the counts reproduce.","supporting_citations":[],"review_version":1}