REVIEW 2 major objections 5 minor 34 references
PolicyStory: Leveraging Large Language Models to Generate Comprehensible Summaries of Policy-News in India
T0 review · 2 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read PolicyStory shows that grouping Indian policy news by topic and year and summarizing it at three depths helps readers follow how policy issues evolve.
desk verdict A coherent pilot of an LLM-based policy-news summarizer; the user study is thin and summary fidelity is unproven, but the paper is honestly described and worth engaging. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a two-stage, prompt-driven summarization chain operating within a fixed context window. Articles from an open news archive are parsed, dated, and assigned by an LLM-based classifier to topic clusters; individual articles are then summarized to 5-8 sentences, and groups of twenty such summaries per cluster per year are combined into a story-like L2 narrative. L1 briefs and numeric key-value pairs are distilled from the L2 layer. The interface presents these by year and topic, with jargon definitions, so that the longitudinal structure—not just isolated headlines—is the primary object the user sees.
What would settle it
Take a random sample of articles from each topic-year cluster, extract the corresponding L1, L2, and numeric outputs, and have independent annotators check every factual claim and every number against the source articles; a nontrivial rate of hallucinated figures or misattributed facts would show that the pilot's positive user feedback rests on unchecked content.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that a compact open-source LLM (Llama-3.2-1B) can be staged into a zero-shot multi-document summarization chain that preserves topical structure and chronological progression. For each policy issue, news articles are clustered by theme (5 clusters for the Farmers' Protests, 8 for the Union Budget), each article is condensed to a 5-8 sentence intermediate summary to fit context limits, and batches of twenty per-topic summaries are then fused into yearly L2 narratives. L1 one-paragraph briefs and numeric key-value summaries are derived from the L2 stories. The paper reports inter-annotator agreement of 0.89 on cluster quality and a user study in which
Load-bearing premise
The whole tool's value depends on the LLM summaries staying faithful to the source articles at scale, yet the authors manually checked only 20 articles per topic; if the unvalidated summaries contain substantial invented or distorted facts, the promised trustworthy overview collapses.
Editorial extensions
If this is right
- Readers of PolicyStory can track how a policy topic evolves year by year, choosing a one-minute brief, a detailed narrative, or numeric facts.
- The 22-person user study suggests the three-level format suits both regular and occasional news consumers, with 95.5% reporting they learned something new from L1 briefs.
- Because the pipeline is zero-shot and open-source, it can be re-run on new policy issues without task-specific training data.
- The authors position the tool against current-affairs aggregators such as Inshorts, arguing that longitudinal, topic-wise summaries fill a gap that short bulletins do not.
- Trust-related feedback—requests for source citations and expert comments—marks the next features needed before the tool can serve as a reliable civic resource.
Reading between the lines
- The same clustering-plus-three-level-summary pipeline could plausibly transfer to other long-running national debates, such as public health emergencies or electoral cycles, but the paper only tests two economic and agrarian policy issues.
- A direct A/B comprehension test—comparing PolicyStory against reading the same articles in a single-level news feed—would quantify the claimed advantage, which the current qualitative user study does not.
- Because numeric summaries are extracted from LLM-generated L2 stories rather than from official government documents, the figures inherit any errors from earlier summarization stages; grounding them in the original articles or budget documents would be a natural robustness check.
- The topic-clustering step is prompt-driven, so the reported benefits may be sensitive to how clusters are defined; testing alternate prompt formulations would show how much of the tool's clarity comes from clustering versus summarization.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents PolicyStory, a pilot information tool that collects Indian policy-related news articles from Media Cloud for two cases (Union Budget 2019–2024 and Farmers' Protests 2020–2024), clusters them by topic and year, and uses an open-source LLM (Llama-3.2-1B) to generate three levels of summaries: short L1 overviews, detailed story-like L2 summaries, and numeric key-value pairs. A web interface presents these summaries chronologically and by topic. The authors report a small user study (N=22) indicating that participants found the tool usable, comprehensible, and helpful for following policy developments over time. The paper positions PolicyStory as a civic-information tool that addresses information overload and provides longitudinal, topic-structured policy summaries.
Significance. If the factual fidelity of the LLM-generated summaries is sound, PolicyStory is a useful and reproducible demonstration of how open-source LLMs can make longitudinal policy news more accessible. The paper's strengths include a deployed public pilot, a clearly described pipeline, manual validation of topic clusters with an inter-tagger agreement of 0.89, and a user study that collects qualitative feedback beyond simple rating scales. However, the central value proposition depends on the accuracy of summaries at scale, and that premise is not adequately established. The reported user satisfaction does not substitute for factual-fidelity evaluation, because users cannot reliably detect hallucinated policy facts they do not already know. The contribution is therefore conditional: the system design and initial usability evidence are promising, but the central claim of providing trustworthy and comprehensible policy overviews remains only partially supported.
major comments (2)
- [§3.2 (Content Summarization) and §5 (Discussion)] Factual fidelity of the generated summaries is a load-bearing premise that is not adequately verified. The authors state that LLM summaries were manually checked on "20 articles from each topic," out of thousands of articles (Table 2), and the paper explicitly defers formal fidelity evaluation to future work in §5. The pipeline compounds the risk: each article is summarized individually; batches of 20 per-article summaries are then re-summarized into L2 stories; L1 summaries and numeric key-value pairs are derived from L2 summaries. Errors can be introduced and amplified at each stage, and Llama-3.2-1B is a small model with known hallucination tendencies. Because the user study measures subjective comprehension, it cannot detect or correct such errors. I recommend adding a faithfulness evaluation on a representative sample of final L2/L1/numeric outputs, with human fact-checking against
- [§4 (User Study)] The user study provides only modest evidence for the claim that PolicyStory "effectively aided users in grasping policy developments over time." The study has N=22, participants were self-selected from two academic institutes, and there was no baseline or control condition comparing PolicyStory with, for example, reading original articles or using an existing aggregator. The outcome measures are self-reports: perceived clarity, self-reported learning, and stated intention to use the tool. No objective comprehension test was administered, and positive ratings do not establish that the tool improved users' actual understanding of policy facts or their temporal evolution. The paper acknowledges the scale limitation in §5, but the missing baseline is a more serious threat to the effectiveness claim. Adding a within-subjects comparison task with factual comprehension questions would substanti
minor comments (5)
- [§1, Introduction] There is a date typo: "Union Budget (2019-2014)" should almost certainly be "2019-2024" (as used in Table 2 and elsewhere). Also, the Farmers' Protests are described as "2021-2024" in the introduction but "2020-2024" in §3.1; the dates should be made consistent.
- [§3.2, Content Summarization] The sentence "The LLM generated summaries were still comprehensive, containing all the essential information from each article when checked manually (20 articles from each topic were checked by the authors)" would benefit from specifying how the 20 articles were selected and what criteria defined "all the essential information." As written, the reader cannot judge the strength of this spot check.
- [§3.3, Numeric Summary] The numeric summaries are extracted from L2 summaries with a "specialized prompt," but the paper does not report any evaluation of the accuracy of these key-value pairs. Since numbers are particularly prone to hallucination and are presented as authoritative facts, a small validation set for numeric extractions would be a valuable addition, even in a pilot paper.
- [§4, User Study] Several figures (e.g., Figure 4a-4f) are referenced but not fully described in the text. Adding brief in-text descriptions of the distributions would help readers who cannot access the figures or the supplementary material.
- [§5, Discussion and Conclusion] The phrase "open-source LLM" is used for Llama-3.2-1B; while the model weights are publicly available, the Llama license is not OSI-certified open source. A more precise term such as "open-weight" or "publicly available" would avoid terminological objections.
Circularity Check
No significant circularity: the central claim is evaluated by an external user study, and there is no fitted-input-as-prediction or self-citation chain.
full rationale
PolicyStory is a systems-and-user-study paper, not a derivation paper. The central claim—that the tool helped users grasp policy developments—is supported by a user study with 22 participants who were unrelated to the authors and responded anonymously. There is no fitted parameter that is later renamed as a prediction, and no formal result is derived from an input that already contains the conclusion. The only self-references are [5] (the deployed tool URL) and [30] (supplementary material); these are pointers, not load-bearing evidence for any claim. The choice of Llama-3.2-1B cites the external Meta model paper [29]. The acknowledged limitation that only 20 articles per topic were manually checked for summary fidelity is a correctness/robustness risk, not circularity: even if the unvalidated summaries contain hallucinations, this would undermine factual trustworthiness but would not make the user-study evaluation equivalent to its inputs by construction. No equation, fitted value, or self-citation chain is used to force the paper's conclusions. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- articles_per_year =
2000
- per_article_summary_length =
5-8 sentences
- topic_cluster_counts =
5 (Farmers), 8 (Budget)
- keyword_queries =
Table 1
assumptions (4)
- domain assumption Media Cloud provides a sufficiently comprehensive and representative collection of Indian policy news for the studied events.
- domain assumption The LLM-based topic classifier produces accurate, coherent topic clusters for all articles, not just the 100-article validation sample.
- domain assumption Llama-3.2-1B generates faithful, non-hallucinated summaries that preserve the factual content of source articles.
- domain assumption The 22 survey participants, recruited from two academic institutes and described as generally knowledgeable about the policies, provide feedback generalizable to the target user population.
Cite this review
Pith. "Pith review of PolicyStory: Leveraging Large Language Models to Generate Comprehensible Summaries of Policy-News in India." pith.science (2026). https://pith.science/paper/7KE3JM5G
@misc{pith2026250908218,
author = {Pith},
title = {Pith review of: PolicyStory: Leveraging Large Language Models to Generate Comprehensible Summaries of Policy-News in India},
year = {2026},
howpublished = {\url{https://pith.science/paper/7KE3JM5G}},
note = {Machine review of arXiv:2509.08218}
}
read the original abstract
In the era of information overload, traditional news consumption through both online and print media often fails to provide a structured and longitudinal understanding of complex sociopolitical issues. To address this gap, we present PolicyStory, an information tool designed to offer lucid, chronological, and summarized insights into Indian policy issues. PolicyStory collects news articles from diverse sources, clusters them by topic, and generates three levels of summaries from longitudinal media discourse on policies, leveraging open source large language models. A user study around the tool indicated that PolicyStory effectively aided users in grasping policy developments over time, with positive feedback highlighting its usability and clarity of summaries. By providing users a birds' eye view of complex policy topics, PolicyStory serves as a valuable resource.
Reference graph
Works this paper leans on
-
[1]
International Journal of Indian Psychology12(3) (2024)
Das, A., Mishra, M.H.: The effect of reels on attention among young and middle- aged adults. International Journal of Indian Psychology12(3) (2024)
work page 2024
-
[2]
Mark, G.: Attention Span: A Groundbreaking Way to Restore Balance, Happiness and Productivity. Harlequin, ??? (2023)
work page 2023
-
[3]
Review of Agrarian Studies12(1), 161–173 (2022) 14
Singh, S.: The farmers’ movement against three agricultural laws in india: A study of organisation. Review of Agrarian Studies12(1), 161–173 (2022) 14
work page 2022
-
[5]
https://neural-times.vercel.app/
Dar, A.N., Singh, A., Sen, A.: PolicyStory. https://neural-times.vercel.app/. Online; accessed 11 June 2025
work page 2025
-
[6]
In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp
Park, S., Kang, S., Chung, S., Song, J.: Newscube: delivering multiple aspects of news to mitigate media bias. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 443–452 (2009)
work page 2009
-
[7]
In: Proceedings of the International AAAI Conference on Web and Social Media, vol
Munson, S., Lee, S., Resnick, P.: Encouraging reading of diverse political viewpoints with a browser widget. In: Proceedings of the International AAAI Conference on Web and Social Media, vol. 7, pp. 419–428 (2013)
work page 2013
-
[8]
In: Proceedings of the 15th International Conference on Intelligent User Interfaces, pp
Park, S., Lee, S., Song, J.: Aspect-level news browsing: Understanding news events from multiple viewpoints. In: Proceedings of the 15th International Conference on Intelligent User Interfaces, pp. 41–50 (2010)
work page 2010
-
[9]
In: Handbook of Digital Journalism: Perspectives from South Asia, pp
Reddy, M.: Smart news for the smartphones in the era of data journalism. In: Handbook of Digital Journalism: Perspectives from South Asia, pp. 341–349. Springer, ??? (2024)
work page 2024
Show all 34 references
-
[10]
Journal of biomedical informatics151, 104606 (2024)
Chuang, Y.-N., Tang, R., Jiang, X., Hu, X.: Spec: a soft prompt-based cal- ibration on performance variability of large language model in clinical notes summarization. Journal of biomedical informatics151, 104606 (2024)
2024
-
[11]
arXiv preprint arXiv:2204.01841 (2022)
Hartl, P., Kruschwitz, U.: Applying automatic text summarization for fake news detection. arXiv preprint arXiv:2204.01841 (2022)
2022 arXiv
-
[12]
In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp
Zogan, H., Razzak, I., Jameel, S., Xu, G.: Depressionnet: learning multi- modalities with user post summarization for depression detection on social media. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. ...
2021
-
[13]
Expert Systems with Applications192, 116292 (2022)
Ghadimi, A., Beigy, H.: Hybrid multi-document summarization using pre-trained language models. Expert Systems with Applications192, 116292 (2022)
2022
-
[14]
In: 2024 IEEE International Conference on Artificial Intelligence Testing (AITest), pp
Ding, J., Nguyen, H., Chen, H.: Evaluation of question-answering based text summarization using llm invited paper. In: 2024 IEEE International Conference on Artificial Intelligence Testing (AITest), pp. 142–149 (2024). https://doi.org/ 10.1109/AITest62860.2024.00025
2024
-
[15]
arXiv preprint arXiv:2412.15487 (2024) 15
Fang, J., Liu, C.-T., Kim, J., Bhedaru, Y., Liu, E., Singh, N., Lipka, N., Mathur, P., Ahmed, N.K., Dernoncourt, F., et al.: Multi-llm text summarization. arXiv preprint arXiv:2412.15487 (2024) 15
2024 arXiv
-
[16]
Authorea Preprints (2025)
Shah, S., Chandrasekaran, D., Ryali, S., Venkatesh, R.: Topic driven text summarization with defragmentation using llms. Authorea Preprints (2025)
2025
-
[17]
arXiv preprint arXiv:2407.07341 (2024)
Sahu, G., Laradji, I.H.: Mixsumm: Topic-based data augmentation using llms for low-resource extractive text summarization. arXiv preprint arXiv:2407.07341 (2024)
2024 arXiv
-
[18]
arXiv preprint arXiv:2407.00908 (2024)
Song, H., Su, H., Shalyminov, I., Cai, J., Mansour, S.: Finesure: Fine-grained summarization evaluation using llms. arXiv preprint arXiv:2407.00908 (2024)
2024 arXiv
-
[19]
In: Pacific Rim International Conference on Artificial Intelligence, pp
Jiang, Z., Yang, J., Rao, D.: An empirical study of leveraging plms and llms for long-text summarization. In: Pacific Rim International Conference on Artificial Intelligence, pp. 424–435 (2024). Springer
2024
-
[20]
Journalism Practice3(1), 1–12 (2009)
Dalecki, L., Lasorsa, D.L., Lewis, S.C.: The news readability problem. Journalism Practice3(1), 1–12 (2009)
2009
-
[21]
European journal of communication6(4), 499–522 (1991)
Kleinnijenhuis, J.: Newspaper complexity and the knowledge gap. European journal of communication6(4), 499–522 (1991)
1991
-
[22]
Journalism Quarterly69(2), 436–446 (1992)
Danielson, W.A., Lasorsa, D.L., Im, D.S.: Journalists and novelists: A study of diverging styles. Journalism Quarterly69(2), 436–446 (1992)
1992
-
[23]
American Journal of Industrial and Business Management04, 567–572 (2014)
Patoko, N., Yazdanifard, R.: The impact of using many jargon words, while com- municating with the organization employees. American Journal of Industrial and Business Management04, 567–572 (2014)
2014
-
[24]
International Journal of Human-Computer Interaction30, 343–368 (2014)
Khawaja, M.A., Chen, F., Marcus, N.: Measuring cognitive load using linguistic features: Implications for usability evaluation and adaptive interaction design. International Journal of Human-Computer Interaction30, 343–368 (2014)
2014
-
[25]
Social Science Quarterly101(2), 811–824 (2020)
Fix, M.P., Fairbanks, B.R.: The effect of opinion readability on the impact of us supreme court precedents in state high courts. Social Science Quarterly101(2), 811–824 (2020)
2020
-
[26]
In: Proceedings of the International AAAI Conference on Web and Social Media, vol
Roberts, H., Bhargava, R., Valiukas, L., Jen, D., Malik, M.M., Bishop, C.S., Ndu- lue, E.B., Dave, A., Clark, J., Etling, B.,et al.: Media cloud: Massive open source collection of global news on the open web. In: Proceedings of the International AAAI Conference on Web and Soci...
2021
-
[27]
Reuters (2024)
Bhardwaj, M., Jadhav, R.: Why are indian farmers protesting again? demands for government explained. Reuters (2024). Accessed: 2025-04-29
2024
-
[28]
arXiv preprint arXiv:1910.03771 (2019)
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al.: Huggingface’s transformers: State-of- the-art natural language processing. arXiv preprint arXiv:1910.03771 (2019)
1910 arXiv
-
[29]
arXiv preprint arXiv:2407.21783 (2024)
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., 16 Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al.: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)
2024 arXiv
-
[30]
https://drive.google
Dar, A.N., Singh, A., Sen, A.: Supplementary Material. https://drive.google. com/file/d/1frihWhA9thS3ntUMhuODb6TJbKKSyt68/view?usp=sharing. Online; accessed 11 June 2025
2025
-
[31]
Newspaper Research Journal42(2), 162–181 (2021)
Diel, S., Roberts, C.: News story aggregation and perceived credibility. Newspaper Research Journal42(2), 162–181 (2021)
2021
-
[32]
Humanities and Social Sciences Communications11(1), 1–12 (2024)
Huang, H., Zhu, H., Liu, W., Gao, H., Jin, H., Liu, B.: Uncovering the essence of diverse media biases from the semantic embedding space. Humanities and Social Sciences Communications11(1), 1–12 (2024)
2024
-
[33]
In: Proceedings of the 2024 8th International Conference on Natural Language Processing and Information Retrieval, pp
Shah, B.S., Shah, D.S., Attar, V.: Decoding news bias: Multi bias detection in news articles. In: Proceedings of the 2024 8th International Conference on Natural Language Processing and Information Retrieval, pp. 97–104 (2024)
2024
-
[34]
In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp
Schaik, T.A., Pugh, B.: A field guide to automatic evaluation of llm-generated summaries. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2832–2836 (2024)
2024
-
[35]
Reading in a foreign language 23(1), 84–101 (2011) 17
Crossley, S.A., Allen, D.B., McNamara, D.S.: Text readability and intuitive sim- plification: A comparison of readability formulas. Reading in a foreign language 23(1), 84–101 (2011) 17
2011
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.