REVIEW 3 major objections 4 minor 10 references
Long-context Non-factoid Question Answering in Indic Languages
T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Shrinking the context with an answer-paragraph retriever improves non-factoid QA in four Indic languages.
desk verdict The headline gains (4% semantic, 47% token-level) are real but measured only on examples where the full-context baseline fit in memory; the paper is a solid, transparent empirical study whose central claim is plausible, but the headline numbers need a fuller comparison before being taken at face value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Answer Paragraph Selection (APS) model, a cross-encoder that reads a question and a paragraph together and outputs a relevance score. The paper uses a fine-tuned XLM-R based checkpoint to score every paragraph in the long context, ranks them, and keeps the top five as the LLM's context. Around this core sit four retrieval variants: APS alone (A1), verbalized OpenIE triples re-ranked by APS (A2), coreference-linked paragraphs re-ranked by APS (A3), and coreference clusters of verbalized triples re-ranked by APS (A4). The evaluation side is carried by STS-MuTe, a new semantic metric that averages cosine similarities from three multilingual embedding models with BERTScore, and by standard ROUGE variants for token-level comparison.
What would settle it
Run the baseline B and the A1 pipeline on every one of the 1,100 sampled questions on hardware that avoids out-of-memory errors and compare average STS-MuTe and ROUGE; if the gains shrink, reverse, or vanish on the previously excluded examples, the paper's central claim fails. A second check is to compare B and A1 on the translated Natural-QA subset with contexts above 8,000 tokens, where the paper reports only topline results for A1.
Extended reading notes
Core claim
On its own terms, the discovery is that a multilingual paragraph-scoring model can serve as a question-specific context compressor for non-factoid QA in Hindi, Tamil, Telugu, and Urdu. The A1 pipeline, built from a fine-tuned XLM-R based APS model, outperforms the full-context baseline B on the common test examples for all three LLMs, with a 4% average gain in STS-MuTe (a composite of BERTScore, USE, LaBSE, and LASER similarities) and a 47% average gain in ROUGE scores when the LLMs are not fine-tuned; fine-tuning shrinks the gain to about 2%. The token-level gain is large but the semantic gain is modest, and the paper attributes the semantic gain to the retriever's ability to find the paragraphs that actually answer the question. In an ablation, fine-tuned APS beats both BM25 and an embedding-based vectorstore retriever, and 75% of its top-k paragraphs differ from BM25's, supporting the claim that the selection is semantic rather than lexical.
Load-bearing premise
The comparison that carries the paper's headline numbers was run only on the subset of a 1,100-question sample where the full-context baseline did not run out of memory, so the load-bearing premise is that those completed examples represent the full test distribution.
Editorial extensions
If this is right
- If the gain is real, long-context QA systems for low-resource languages can be improved and made cheaper at the same time by inserting a paragraph-selection retriever before generation.
- The result implies that semantic metrics and token-level metrics move together in this setting, so the improvement is not just a paraphrase effect.
- The explainability finding suggests the APS score can be used as a confidence signal: low-scoring paragraphs are likely to be irrelevant, so the retriever can safely discard them.
- Because verbalizing OIE triples did not help, triple extraction is probably not a useful context-shortening step for non-factoid Indic QA, despite its success in knowledge-grounded English tasks.
- The 75% disagreement with BM25 indicates that lexical retrieval alone is insufficient for non-factoid Indic QA, supporting the need for learned semantic retrieval.
Reading between the lines
- Editorial inference: the A1 pipeline should transfer to other low-resource languages with a cross-lingual encoder, because it depends on no OIE or coreference tools; the paper only tests Hindi, Tamil, Telugu, and Urdu.
- Editorial inference: the true improvement over the baseline is probably closer to the 4% semantic gain than the 47% token-level gain, because token-level metrics penalize legitimate paraphrase in LLM answers.
- Editorial inference: a direct stress test is to run the baseline on the OOM examples with larger GPUs; if the baseline wins there, the central comparison is an artifact of memory limits rather than a property of long contexts.
- Editorial inference: using the APS score as a reward model for preference tuning, which the paper names as future work, is a testable next step: it would show whether retriever confidence aligns with downstream answer quality.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper studies context-shortening for non-factoid question answering in Hindi, Tamil, Telugu, and Urdu. It builds on the authors' MuNfQuAD dataset and compares four retrieval-based pipelines (APS, OIE+APS, coref+APS, OIE+coref+APS) against a baseline that feeds the full long context to Gemma-2B, Gemma-7B, and Llama-3.1-8B, in both zero-shot and LoRA fine-tuned settings. The authors report that the APS-only pipeline (A1) improves the average semantic score (STS-MuTe) by about 4% and ROUGE by about 47% over the full-context baseline, while also reducing GPU memory and inference time. They also introduce a new semantic similarity metric, report an LLM-as-a-judge evaluation, and use LIME and SHAP to analyze the APS model's rationales. Code, checkpoints, and resources are released. The headline comparison is limited to a 1100-question subset of the 6k+ test set and, within that subset, to the examples on which the full-context baseline did not run out of memory.
Significance. If the reported improvement is robust, this is a useful practical contribution: it shows that a fine-tuned paragraph selector can reduce long contexts for non-factoid QA in low-resource languages while matching or exceeding full-context LLM performance at substantially lower computational cost. The paper is transparent in several important ways: the APS training data is excluded from evaluation (a proper held-out split), ablations against BM25 and dense retrieval are included, computational footprints are reported, and a contamination analysis is attempted. These strengths make the central claim worth taking seriously. The main weakness is that the central A1-vs-B comparison is restricted to the subset where the baseline did not OOM, and no uncertainty quantification is provided; the headline numbers are therefore conditional on the hardware-restricted subset rather than unconditional claims about the test distribution.
major comments (3)
- [§5, Tables 1 and 3] The central claim that A1 outperforms B is evaluated only on the test examples common to both, because B OOMs on many long examples. The paper states: 'The resource intensive nature of B led to Out of Memory (OOM) errors on many test examples' and therefore B and A1 were compared on the 'test examples common to both' among the 1100. Since the OOM examples have average context lengths of 1238, 1189, and 1069 tokens by model (reported in §5), the comparison excludes exactly the longest contexts, where context-shortening is arguably most relevant. To support the abstract's 4% and 47% claims, the authors should either run B on the full 1100-item subset (e.g., with chunking, a longer-context model, or reduced batch size) or provide evidence that the common subset is representative in terms of context length, language, and question category distributions. As written, the reported gains are estimates conditional on the non-OOM subset.
- [§8, contamination analysis] The contamination filter shows that B's zero-shot scores drop by 13%, 2.5%, and 22% when test data that were publicly available before each model release are excluded, but the paper does not report A1's filtered scores or the A1-vs-B comparison on the filtered examples. This matters because the relative gain could change if contamination inflates B and A1 differently. The filtered sets are small (78, 197, and 18 examples), but the manuscript should still report the filtered comparison for at least STS-MuTe and ROUGE, with the small sample sizes acknowledged, so that readers can see whether the relative improvement survives the filter.
- [§5, Table 1] Several per-model differences are small (e.g., gemma-7b-it STS-MuTe: B 0.60 vs A1 0.61; BERTScore 0.66 vs 0.69 for gemma-7b-it), and no confidence intervals, bootstrap estimates, or significance tests are provided. Since the headline is an average over three models and four languages, the 4% semantic improvement may be within run-to-run noise. Please report per-language, per-model results with bootstrap confidence intervals or pairwise significance tests on the common subset, or explicitly state that the observed differences are not tested for significance.
minor comments (4)
- [§5, LLM-as-a-judge] The LLM-as-a-judge protocol should state whether the order of option1 and option2 is randomized across examples, and should report agreement with human judgments if available; the current text only gives preference percentages.
- [§3 and §4, data splits] Please clarify the relation between the 'over 40K' question-answer pairs after excluding the APS training data, the 'more than 29K' training questions, and the 'more than 6k' test questions; these numbers do not obviously sum, and a precise split description would help reproducibility.
- [§2 and References] There are minor typos (e.g., 'The A1 approach does not only shows better performance' in §5) and duplicate bibliography entries (Kolluru et al. 2022a/2022b and Ribeiro et al. 2016a/2016b); please clean these before publication.
- [§6, Figure 6] The text describing Figure 6 gives a concrete example of 7 out of 10 tokens above a threshold, but the figure caption does not explain the bucketing clearly; please make the caption self-contained and state which languages and how many examples the 8.5k question-paragraph rationale computation covers.
Circularity Check
No significant circularity: the central A1-vs-B comparison is an empirical, held-out evaluation that could have favored either pipeline.
full rationale
The paper's headline claim is that context-shortening with an Answer Paragraph Selection retriever (A1) improves LLM non-factoid QA over a full-context baseline (B) on MuNfQuAD. This is an empirical comparison on held-out test questions: B is fed the whole context, A1 receives the top-5 APS paragraphs, and both are scored with ROUGE and the paper's own STS-MuTe. The outcome is not forced by construction; B could have won, and indeed Table 1 shows B at or above A1 on some individual metrics (e.g., gemma-7b-it USE/LASER). The fine-tuned APS model, MuNfQuAD, Transmucores, and IndIE come from the authors' prior work, but they are used as components rather than as premises that entail the conclusion; the paper even ablates A1 against LangChain and BM25 retrievers, providing internal evidence for the APS retriever's contribution. STS-MuTe is an evaluation metric (a mean of existing embedding similarities and BERTScore), not a predicted quantity that is defined in terms of the systems' outputs, so using it to measure improvement is not a definitional reduction. The restriction of A1-vs-B to a 1100-question subset and to test examples where B did not OOM is a coverage limitation that affects generalizability and statistical strength, but it is not a circularity. Consequently, no self-definitional, fitted-input, or self-citation-chain reduction is present.
Assumptions & free parameters
free parameters (5)
- APS top-k =
5
- OIE triple top-k =
10
- LoRA rank and alpha =
32
- Sampling temperature =
0.001
- Long-context threshold =
512 tokens
assumptions (5)
- domain assumption MuNfQuAD silver answers are reliable enough to evaluate generated answers.
- domain assumption The fine-tuned APS model from prior work generalizes to held-out MuNfQuAD splits.
- ad hoc to paper Gen2OIE checkpoints transfer zero-shot from Hindi/Telugu to Urdu/Tamil.
- domain assumption GPT-4o as a judge provides reliable qualitative evaluation.
- ad hoc to paper NLLB 1.3B translations of NaturalQA questions are adequate for Indic evaluation.
Cite this review
Pith. "Pith review of Long-context Non-factoid Question Answering in Indic Languages." pith.science (2026). https://pith.science/paper/QCXYBCYW
@misc{pith2026250413615,
author = {Pith},
title = {Pith review of: Long-context Non-factoid Question Answering in Indic Languages},
year = {2026},
howpublished = {\url{https://pith.science/paper/QCXYBCYW}},
note = {Machine review of arXiv:2504.13615}
}
read the original abstract
Question Answering (QA) tasks, which involve extracting answers from a given context, are relatively straightforward for modern Large Language Models (LLMs) when the context is short. However, long contexts pose challenges due to the quadratic complexity of the self-attention mechanism. This challenge is compounded in Indic languages, which are often low-resource. This study explores context-shortening techniques, including Open Information Extraction (OIE), coreference resolution, Answer Paragraph Selection (APS), and their combinations, to improve QA performance. Compared to the baseline of unshortened (long) contexts, our experiments on four Indic languages (Hindi, Tamil, Telugu, and Urdu) demonstrate that context-shortening techniques yield an average improvement of 4\% in semantic scores and 47\% in token-level scores when evaluated on three popular LLMs without fine-tuning. Furthermore, with fine-tuning, we achieve an average increase of 2\% in both semantic and token-level scores. Additionally, context-shortening reduces computational overhead. Explainability techniques like LIME and SHAP reveal that when the APS model confidently identifies the paragraph containing the answer, nearly all tokens within the selected text receive high relevance scores. However, the study also highlights the limitations of LLM-based QA systems in addressing non-factoid questions, particularly those requiring reasoning or debate. Moreover, verbalizing OIE-generated triples does not enhance system performance. These findings emphasize the potential of context-shortening techniques to improve the efficiency and effectiveness of LLM-based QA systems, especially for low-resource languages. The source code and resources are available at https://github.com/ritwikmishra/IndicGenQA.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[8]
Multilingual Non-Factoid Question Answering with Answer Paragraph Selection
Indie: A multilingual open information ex- traction tool for indic languages. InFindings of the Association for Computational Linguistics: IJCNLP- AACL 2023 (Findings), pages 312–326. Ritwik Mishra, Sreeram V ennam, Rajiv Ratn Shah, and Ponnurangam Kumaraguru. 2024b. Multilingual non-factoid question answering with silver answers. arXiv preprint arXiv:240...
work page Pith review arXiv 2023
-
[10]
In2024 Conference on AI, Sci- ence, Engineering, and Technology (AIxSET), pages 276–289
Llm-based localization in the context of low- resource languages. In2024 Conference on AI, Sci- ence, Engineering, and Technology (AIxSET), pages 276–289. IEEE. David Uthus, Santiago Ontanon, Joshua Ainslie, and Mandy Guo. 2023. mlongt5: A multilingual and efficient text-to-text transformer for longer sequences. InThe 2023 Conference on Empirical Methods i...
work page 2023
-
[2004]
Learning tofind answers to questions on the web.ACM Transactions on Internet Technology (TOIT), 4(2):129–162. Kabir Ahuja, Harshita Diddee, Rishav Hada, Milli- cent Ochieng, Krithika Ramesh, Prachi Jain, Ak- shay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, and 1 others. 2023. Mega: Multilingual evaluation of generative ai. InProceedings of the 2023 Co...
work page 2023
-
[2014]
Open question answering over curated and ex- tracted knowledge bases. InProceedings of the 20th ACM SIGKDD international conference on Knowl- edge discovery and data mining, pages 1156–1165. Haytham M Fayek and Justin Johnson. 2020. Temporal reasoning via audio question answering.IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, 28:2283–2...
work page 2020
-
[2015]
Iiit-h system submission forfire2014 shared task on transliterated search . InProceedings of the F orum for Information Retrieval Evaluation, FIRE ’14, pages 48–53, New Y ork, NY , USA. ACM. V aleriia Bolotova, Vladislav Blinov, Falk Scholer, W Bruce Croft, and Mark Sanderson. 2022. A non- factoid question-answering taxonomy. InProceed- ings of the 45th In...
arXiv 2022
-
[2020]
In Proceedings of the AAAI conference on articial in- telligence, volume 34, pages 7780–7788
Tanda: Transfer and adapt pre-trained trans- former models for answer sentence selection. In Proceedings of the AAAI conference on articial in- telligence, volume 34, pages 7780–7788. Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Naik, Pengshan Cai, and Alfio Gliozzo. 2022. Re2G: Retrieve, rerank, generate . InProceedings of the 202...
arXiv 2022
-
[2021]
Nlquad: A non-factoid long question answer- ing data set. InProceedings of the 16th Conference of the European Chapter of the Association for Com- putational Linguistics: Main V olume, pages 1245– 1255. Radu Soricut and Eric Brill. 2006. Automatic ques- tion answering using the web: Beyond the factoid. Information Retrieval, 9:191–206. Mihai Surdeanu, Mas...
arXiv 2006
-
[2023]
Retrieving supporting evidence for generative question answering. InProceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacic Region, pages 11–20. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radfor...
arXiv 2024
Show all 10 references
-
[2024]
InProceedings of the 18th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 358– 368
Dynamic task-oriented dialogue: A compara- tive study of llama-2 and bert in slot value generation. InProceedings of the 18th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 358– 368. Tianle Li, Ge Zhang, ...
2024 arXiv
-
[2025]
Avinash Anand, Raj Jaiswal, Abhishek Dharmadhikari, Atharva Marathe, Harsh Popat, Harshil Mital, Ash- win R Nair, Kritarth Prasad, Sidharth Kumar, Astha V erma, and 1 others
Evaluation of open and closed-source llms for low-resource language with zero-shot, few-shot, and chain-of-thought prompting.Natural Language Processing Journal, page 100124. Avinash Anand, Raj Jaiswal, Abhishek Dharmadhikari, Atharva Marathe, Harsh Popat, Harshil Mital, Ash- ...
2019 arXiv
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.