Civil Society in the Loop: Feedback-Driven Adaptation of (L)LM-Assisted Classification in an Open-Source Telegram Monitoring Tool

Elisabeth Steffen; Grischa Stanjek; Helena Mihaljevi\'c; Milena Pustet; Yannis Illies

REVIEW 4 major objections 4 minor 18 references

Civil Society in the Loop: Feedback-Driven Adaptation of (L)LM-Assisted Classification in an Open-Source Telegram Monitoring Tool

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A feedback loop turns civil society analysts' label corrections into a growing gold-standard dataset that retrains fine-tuned models or revises prompts, so an open-source Telegram monitoring tool improves as CSO users do their regular…

desk verdict Honest, clearly written work-in-progress design paper whose central feedback-loop claim rests on an untested implicit-agreement assumption; worth workshop referee time, not a full paper yet. read the letter →

arxiv 2507.06734 v1 pith:UUCCVZRR submitted 2025-07-09 cs.HC cs.AIcs.CLcs.CY

Milena Pustet , Elisabeth Steffen , Helena Mihaljević , Grischa Stanjek , Yannis Illies This is my paper · ORCID

classification cs.HCcs.AIcs.CLcs.CY

keywords civilsocietyorganizationscontentmoderationTelegrammonitoringconspiracytheoryclassificationhuman-in-the-loopfeedback-drivenadaptationfine-tuningpromptengineering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This work-in-progress paper proposes that an open-source Telegram monitoring tool can improve its conspiracy-theory classification by letting civil society analysts correct the machine's labels while they work. Each correction is stored with the original classification and feedback metadata, building a growing gold-standard dataset that is then used to retrain a fine-tuned model or revise a prompt. The paper argues this feedback loop makes model evaluation part of routine monitoring, not a separate task, and lets non-technical users steer AI-assisted content moderation in their own secure environment. If the loop works, civil society organizations gain a practical, transparent alternative to closed-source platform moderation.

What carries the argument

The load-bearing mechanism is the feedback loop itself, centered on the growing gold-standard dataset: labels that users correct while monitoring, stored together with the original prediction and feedback metadata, become the resource for retraining, prompt revision, validation, and drift-triggered updates. It is what turns routine human judgment into a reusable evaluation and adaptation signal, and it is what both the fine-tuning and prompting pathways feed on.

What would settle it

Take a random sample of Telegram posts from the same channels, have independent annotators label the full sample, and compare those labels with the dataset accumulated from ordinary monitoring feedback. If the feedback-based dataset is systematically unrepresentative—for example, if unmarked posts contain many false negatives or false positives—then model evaluations and updates built on it are unreliable, and the paper's central claim fails.

Watch

Extended reading notes

Core claim

The central proposal is a workflow in which each Telegram post is shown to a user with a binary conspiracy-theory label and a confidence score, and the user can correct that label directly in the monitoring feed. Those corrections accumulate into a growing gold-standard dataset—the stored original classification, feedback, and metadata—that serves two purposes: evaluating updated models and supplying training or prompting material. In the fine-tuning pathway, concept drift detected by comparing new data with the training corpus can trigger retraining; in the prompting pathway, feedback guides prompt revision and few-shot example selection. Updated models or prompts are re-evaluated on the accumulated feedback and iteratively deployed. The paper treats fine-tuning and prompting as complementary approaches with different trade-offs, not as a single winner.

Load-bearing premise

The load-bearing premise is that feedback left behind during ordinary monitoring, including inferred implicit agreement from unmarked posts, produces a representative gold-standard dataset; if that premise fails, the retraining, prompting updates, and evaluations in the loop are all built on skewed data.

Editorial extensions

If this is right

An open-source Telegram monitor can embed model evaluation inside routine monitoring rather than requiring separate annotation tasks, because the feed itself collects corrections.
Both fine-tuning and prompting can be steered by the same growing gold-standard dataset, so organizations can choose a pathway by available hardware and technical skill.
The accumulated feedback supports concept-drift detection and retraining triggers in the fine-tuning pathway, keeping the classifier current as online discourse shifts.
Deployment-time validation can be done with held-out parts of the accumulated feedback, and prompt changes can be tested before applying them at scale.
The privacy and cost constraints point toward smaller or distilled models running on local infrastructure, avoiding third-party APIs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper, the implicit-agreement strategy—treating unmarked seen posts as agreement—is the most promising fix for negativity bias, but it is also the clearest point where the loop could silently break; a field trial with explicit seen-tracking would settle it.
If the feedback loop works for conspiracy-theory classification, the same infrastructure could extend to other CSO monitoring tasks such as summarization or question answering, though prompt updates for those tasks would need equally careful validation.
The growing gold-standard dataset could eventually be pooled or federated across CSOs, but divergent labeling practices would require governance before such sharing becomes reliable.
A natural quantitative test the paper does not run: compare how quickly classification accuracy improves per unit of feedback in the fine-tuning versus prompting pathways, since the optimal choice may shift as feedback accumulates.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick read: this is a design proposal, not a demonstrated result. What it does well is lay out why CSO Telegram monitoring needs AI integration and why feedback-driven adaptation is the right lens. The related work is current and useful, and the authors are transparent that the tool is not implemented. The characterization of the gap—existing open-source Telegram tools offer extraction and search but not integrated, user-steerable classification—is fair.

The genuinely useful part is Section 3: the two-pathway design (fine-tuning vs prompting) and the explicit discussion of negativity bias. They state clearly that users over-report disagreements, and they float the idea of treating unmarked seen posts as implicit agreement. They also note this needs experimental validation and raises privacy issues. That is intellectually honest.

Now the soft spots. The central claim—that accumulated user feedback can refine and evaluate both model types—rests entirely on that untested implicit-agreement signal. In a monitoring feed, inaction often means inattention or fatigue, not agreement. If the signal is wrong, fine-tuning gets biased negative examples and the evaluation on the same feedback cannot detect the bias. The paper gestures at validation splits and A/B testing but offers no concrete protocol. Table 1 is an unsubstantiated comparison; there is no data behind it. So as submitted, the paper has no empirical content and its foundation is conditional.

That said, the authors are not overclaiming: they call it work in progress and list the open problems. The design is coherent and grounded in prior human-in-the-loop work, not circular with their own citations. The citation pattern looks fine.

Who is this for? People building monitoring tools for civil society, and researchers working on feedback loops for low-resource social-media classification. For that audience, this is a useful design note. It deserves a workshop referee or a short-paper track, not a main-conference submission in its current form. If they build the prototype and pilot the implicit-agreement signal with even a small user study, the next version would merit serious review.

Referee Report

4 major / 4 minor

Summary. This work-in-progress paper proposes a feedback-driven workflow for integrating fine-tuned and prompt-based (L)LM classifiers into an open-source Telegram monitoring tool used by civil society organizations (CSOs). The authors describe two pathways: fine-tuning (FT) a BERT-based conspiracy-theory classifier and prompting (P) an LLM, with user feedback on binary labels accumulated into a growing gold-standard dataset that is used for both model refinement and evaluation. They discuss key challenges, including negativity bias in feedback, implicit affirmation from unmarked posts, conflict resolution among users, retraining schedules, and privacy constraints, and they compare the FT and P approaches in Table 1. No implementation, simulation, or empirical evaluation is presented; the paper is explicitly framed as ongoing work.

Significance. If validated, this design would address a real gap in open-source social-media monitoring tools by giving CSOs an active role in continuously adapting AI classifiers to their monitoring needs. The paper's strengths include a clearly motivated problem, an explicit focus on stakeholder involvement, and transparent acknowledgment of its main assumptions and limitations, particularly the need to validate implicit-affirmation signals and to use proper validation splits. The proposed workflow is internally coherent and grounded in prior work on fact-checking and Telegram monitoring. However, the paper currently provides no empirical evidence for its central value proposition, and Table 1 presents a comparative rating without supporting evaluation. The contribution is therefore best seen as a design proposal whose load-bearing assumptions remain unverified.

major comments (4)

[§3, implicit affirmation] The feedback loop's value depends on treating posts that were seen or clicked but left unmarked as implicit signals of agreement. The paper itself notes that this 'would require not only experimental validation' and raises privacy concerns, but this assumption is load-bearing: if unmarked posts reflect inattention, fatigue, or uncertainty rather than agreement, then fine-tuning receives systematically biased negative examples and evaluation on the same feedback cannot detect the bias. Please either provide validation evidence for this signal or redesign the feedback collection so that representative explicit feedback (e.g., through dedicated full-sample rating views) is the primary mechanism, with implicit signals used only as auxiliary information.
[§3, re-evaluation] The sentence 'Updated models or prompts are then re-evaluated with accumulated feedback and iteratively deployed' is circular if the same accumulated feedback is used both for refinement and for evaluation. The later mention of using part of the gold-standard dataset for test/validation sets and A/B testing is only in passing; the workflow should make independent held-out validation an explicit, non-optional component rather than an afterthought. Without this, the claim that feedback-driven adaptation improves models is not defensible even in principle.
[Table 1] The ratings in Table 1 (++, +, 0) are presented as a formal comparison of FT and P, but no methodology, data, or references are given to support these ratings. As written, the table asserts, for example, that FT has '++' for model behavior control while P has '0', and that P has '++' for adaptability to task variations while FT has '0', without any evidence. Please either reframe Table 1 as an explicitly qualitative assessment based on the authors' experience and the cited literature, or provide a systematic evaluation that justifies the ratings.
[§1 and §4] The paper's central value proposition is that feedback-driven adaptation can meaningfully improve AI-assisted monitoring for CSOs, yet no empirical evidence, simulation, or pilot study is reported. The paper is transparently labeled work in progress, which is appropriate for a workshop or design-track submission, but for a full journal the central claim remains unverified. A small-scale evaluation, even with simulated user feedback or on a small set of Telegram posts, would substantially strengthen the contribution and allow the proposed workflow to be assessed.

minor comments (4)

[§3] The phrase 'concept drift, detected by comparing new data to the training corpus' is underspecified; please clarify what comparison method is intended (e.g., embedding distances, classifier confidence distributions, or statistical tests).
[§2] The Pustet et al. (2024) BERT-based model is described as 'well-performing,' but no performance figures are provided; citing a specific metric (e.g., F1 or accuracy) would make the starting point clearer.
[Table 1] The table legend defines symbols as ++ = very good, + = acceptable, 0 = limited, but some cells contain compound entries such as '0/+'; please clarify whether this means a range or an intermediate rating.
[§4] The future work section is very brief; expanding it with concrete evaluation plans (e.g., how the implicit-affirmation signal would be tested, or how hybrid pre-labeling strategies would be measured) would help readers understand the next steps.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the design proposal uses feedback for refinement and evaluation only with specified validation splits, and the sole self-citation is a non-load-bearing starting point.

full rationale

This is a work-in-progress design proposal rather than a derivation of empirical results. It makes no fitted-parameter predictions whose outputs could reduce to their inputs. The only self-citation, Pustet et al. (2024), is explicitly used as a 'solid starting point' for fine-tuning, not as evidence for the paper's contribution. The feedback loop is described as requiring validation before deployment: 'Both approaches require validation before deployment, using part of the growing gold-standard dataset for test and validation sets. In FT, staged rollouts or A/B testing may be needed.' This directly addresses the potential circularity of using the same feedback for refinement and evaluation. The paper also flags the implicit-affirmation alternative as needing 'experimental validation' and notes privacy challenges, so the concern about unmarked posts being treated as agreement is an acknowledged validity risk rather than a hidden circular assumption. No load-bearing step in the text reduces to its own inputs by construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. It relies on several domain assumptions about user feedback reliability, the validity of implicit signals, the quality of a previously trained model, and CSO deployment constraints. These assumptions are central to the proposed feedback loop and are untested.

assumptions (4)

domain assumption User feedback provides a reliable gold standard for model refinement and evaluation.
Section 3: feedback is stored as a growing gold-standard dataset; the paper acknowledges negativity bias, so reliability is questionable.
ad hoc to paper Implicit affirmation can be inferred from unmarked seen or clicked posts.
Section 3: proposed as an alternative to explicit feedback; the paper says it requires experimental validation and poses privacy challenges.
domain assumption The BERT-based CT model from Pustet et al. (2024) is a solid starting point for fine-tuning.
Section 3: the authors rely on their own prior trained model without independent evaluation in this paper.
domain assumption CSOs can deploy and run the tool in their own environments despite hardware constraints.
Section 3: notes GPU requirements but assumes localized deployment, suggesting smaller models to mitigate.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Civil Society in the Loop: Feedback-Driven Adaptation of (L)LM-Assisted Classification in an Open-Source Telegram Monitoring Tool." pith.science (2026). https://pith.science/paper/UUCCVZRR

@misc{pith2026250706734,
  author       = {Pith},
  title        = {Pith review of: Civil Society in the Loop: Feedback-Driven Adaptation of (L)LM-Assisted Classification in an Open-Source Telegram Monitoring Tool},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UUCCVZRR}},
  note         = {Machine review of arXiv:2507.06734}
}

read the original abstract

The role of civil society organizations (CSOs) in monitoring harmful online content is increasingly crucial, especially as platform providers reduce their investment in content moderation. AI tools can assist in detecting and monitoring harmful content at scale. However, few open-source tools offer seamless integration of AI models and social media monitoring infrastructures. Given their thematic expertise and contextual understanding of harmful content, CSOs should be active partners in co-developing technological tools, providing feedback, helping to improve models, and ensuring alignment with stakeholder needs and values, rather than as passive 'consumers'. However, collaborations between the open source community, academia, and civil society remain rare, and research on harmful content seldom translates into practical tools usable by civil society actors. This work in progress explores how CSOs can be meaningfully involved in an AI-assisted open-source monitoring tool of anti-democratic movements on Telegram, which we are currently developing in collaboration with CSO stakeholders.

Figures

Figures reproduced from arXiv: 2507.06734 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 11 canonical work pages

[1]

Jason Baumgartner, Savvas Zannettou, Megan Squire, and Jeremy Blackburn. 2020. https://doi.org/10.1609/icwsm.v14i1.7348 The Pushshift Telegram Dataset . Proceedings of the International AAAI Conference on Web and Social Media, 14:840--847

work page doi:10.1609/icwsm.v14i1.7348 2020
[2]

Youngjin Chae and Thomas Davidson. 2025. https://doi.org/10.1177/00491241251325243 Large Language Models for Text Classification : From Zero - Shot Learning to Instruction - Tuning . Sociological Methods & Research, page 00491241251325243

doi:10.1177/00491241251325243 2025
[3]

Andrea Hrckova, Robert Moro, Ivan Srba, Jakub Simko, and Maria Bielikova. 2024. https://doi.org/10.48550/arXiv.2211.12143 Autonomation, not Automation : Activities and Needs of Fact -checkers as a Basis for Designing Human - Centered AI Systems . arXiv preprint. ArXiv:2211.12143

work page Pith review arXiv doi:10.48550/arxiv.2211.12143 2024
[4]

Prerna Juneja and Tanushree Mitra. 2022. https://doi.org/10.1145/3555143 Human and Technological Infrastructures of Fact -checking . Proc. ACM Hum.-Comput. Interact., 6(CSCW2):418:1--418:36

doi:10.1145/3555143 2022
[5]

Manoel J\' u nior, Philipe Melo, Daniel Kansaon, Vitor Mafra, Kaio Sa, and Fabricio Benevenuto. 2022. https://doi.org/10.1145/3511095.3536375 Telegram monitor: Monitoring brazilian political groups and channels on telegram . In Proceedings of the 33rd ACM Conference on Hypertext and Social Media, HT '22, page 228–231, New York, NY, USA. Association for Co...

work page arXiv 2022
[6]

Dia Kayyali. 2025. https://techpolicy.press/metas-content-moderation-changes-are-going-to-have-a-real-world-impact-its-not-going-to-be-good Meta's Content Moderation Changes are Going to Have a Real World Impact . It 's Not Going to be Good . TechPolicy . Press

work page 2025
[7]

Stijn Peeters and Sal Hagen. 2022. https://doi.org/10.5117/CCR2022.2.007.HAGE The 4CAT Capture and Analysis Toolkit : A Modular Tool for Transparent and Traceable Social Media Research . Computational Communication Research, 4(2):571--589

work page doi:10.5117/ccr2022.2.007.hage 2022
[8]

Florian Primig and Fabian Fröschl. 2024. https://doi.org/10.1177/20501579241244973 Introducing the FROG tool for gathering Telegram data . Mobile Media & Communication, 12(2):449--453

work page doi:10.1177/20501579241244973 2024

Show all 18 references

[9]

Rob Procter, Miguel Arana-Catania, Yulan He, Maria Liakata, Arkaitz Zubiaga, Elena Kochkina, and Runcong Zhao. 2023. https://doi.org/10.48550/arXiv.2305.02224 Some Observations on Fact - Checking Work with Implications for Computational Support . arXiv preprint. ArXiv:2305.02224

2023 doi arXiv
[10]

Milena Pustet, Elisabeth Steffen, and Helena Mihaljevic. 2024. https://doi.org/10.18653/v1/2024.woah-1.2 Detection of conspiracy theories beyond keyword bias in G erman-language telegram using large language models . In Proceedings of the 8th Workshop on Online Abuse and Harms...

2024 doi
[11]

Giosuè Ruscica, Giulia Tucci, and Bia Carneiro. 2025. https://doi.org/10.1016/j.simpa.2024.100736 TeleCatch : An open-access software for visualizing, filtering and extracting Telegram messages data . Software Impacts, 23:100736

2025
[12]

Manos Schinas, Symeon Papadopoulos, Lazaros Apostolidis, Yiannis Kompatsiaris, and Pericles A. Mitkas. 2017. https://doi.org/10.1007/978-3-319-70284-1_28 Open- Source Monitoring , Search and Analytics Over Social Media . In Internet Science , pages 361--369, Cham. Springer Int...

2017 doi
[13]

Ergon Cugler de Moraes Silva. 2024. https://doi.org/10.48550/arXiv.2412.16786 TelegramScrap : A comprehensive tool for scraping Telegram data . arXiv preprint. ArXiv:2412.16786

2024 doi arXiv
[14]

Aleksandra Urman and Stefan Katz. 2022. https://doi.org/10.1080/1369118X.2020.1803946 What they do in the shadows: examining the far-right networks on Telegram . Information, Communication & Society, 25(7):904--923

2022
[15]

Robert Wolfe and Tanushree Mitra. 2024. https://doi.org/10.1145/3630106.3658987 The Impact and Opportunities of Generative AI in Fact - Checking . In The 2024 ACM Conference on Fairness , Accountability , and Transparency , pages 1531--1543

2024
[16]

Lui Yoshida. 2024. https://doi.org/10.1007/978-3-031-64315-6_5 The Impact of Example Selection in Few - Shot Prompting on Automated Essay Scoring Using GPT Models . In Andrew M. Olney, Irene-Angelica Chounta, Zitao Liu, Olga C. Santos, and Ig Ibert Bittencourt, editors, Artifi...

2024 doi
[17]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[18]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Jason Baumgartner, Savvas Zannettou, Megan Squire, and Jeremy Blackburn. 2020. https://doi.org/10.1609/icwsm.v14i1.7348 The Pushshift Telegram Dataset . Proceedings of the International AAAI Conference on Web and Social Media, 14:840--847

work page doi:10.1609/icwsm.v14i1.7348 2020

[2] [2]

Youngjin Chae and Thomas Davidson. 2025. https://doi.org/10.1177/00491241251325243 Large Language Models for Text Classification : From Zero - Shot Learning to Instruction - Tuning . Sociological Methods & Research, page 00491241251325243

doi:10.1177/00491241251325243 2025

[3] [3]

Andrea Hrckova, Robert Moro, Ivan Srba, Jakub Simko, and Maria Bielikova. 2024. https://doi.org/10.48550/arXiv.2211.12143 Autonomation, not Automation : Activities and Needs of Fact -checkers as a Basis for Designing Human - Centered AI Systems . arXiv preprint. ArXiv:2211.12143

work page Pith review arXiv doi:10.48550/arxiv.2211.12143 2024

[4] [4]

Prerna Juneja and Tanushree Mitra. 2022. https://doi.org/10.1145/3555143 Human and Technological Infrastructures of Fact -checking . Proc. ACM Hum.-Comput. Interact., 6(CSCW2):418:1--418:36

doi:10.1145/3555143 2022

[5] [5]

Manoel J\' u nior, Philipe Melo, Daniel Kansaon, Vitor Mafra, Kaio Sa, and Fabricio Benevenuto. 2022. https://doi.org/10.1145/3511095.3536375 Telegram monitor: Monitoring brazilian political groups and channels on telegram . In Proceedings of the 33rd ACM Conference on Hypertext and Social Media, HT '22, page 228–231, New York, NY, USA. Association for Co...

work page arXiv 2022

[6] [6]

Dia Kayyali. 2025. https://techpolicy.press/metas-content-moderation-changes-are-going-to-have-a-real-world-impact-its-not-going-to-be-good Meta's Content Moderation Changes are Going to Have a Real World Impact . It 's Not Going to be Good . TechPolicy . Press

work page 2025

[7] [7]

Stijn Peeters and Sal Hagen. 2022. https://doi.org/10.5117/CCR2022.2.007.HAGE The 4CAT Capture and Analysis Toolkit : A Modular Tool for Transparent and Traceable Social Media Research . Computational Communication Research, 4(2):571--589

work page doi:10.5117/ccr2022.2.007.hage 2022

[8] [8]

Florian Primig and Fabian Fröschl. 2024. https://doi.org/10.1177/20501579241244973 Introducing the FROG tool for gathering Telegram data . Mobile Media & Communication, 12(2):449--453

work page doi:10.1177/20501579241244973 2024