{"id":"8ec37d03-6fd4-4e78-8798-59956338fac8","arxiv_id":"2507.06734","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A design proposal for an open-source Telegram monitoring tool that lets civil society users correct AI classification labels, with the corrections used to retrain or reprompt the model.","lead":"This paper outlines a planned open-source tool for civil society groups to monitor harmful content on Telegram, with user feedback on AI classifications used to continually improve the models. It compares fine-tuning a BERT model versus prompting a large language model, and discusses challenges like feedback bias, privacy, and limited computing resources.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Feedback-loop value hinges on an untested implicit-agreement signal; if unmarked seen posts are not predominantly true agreements, both model refinement and evaluation are biased.","rationale":"The paper is an honest work-in-progress design proposal and should be evaluated as such: it does not claim a deployed system or empirical results, and it explicitly flags the implicit-agreement strategy as needing experimental validation. I therefore do not object to the paper's existence or to its framing. However, the central claim quoted by the reader—that updated models or prompts are re-evaluated with accumulated feedback and iteratively deployed—rests on the assumption that the accumulated feedback is a reliable gold standard. That assumption is load-bearing for both the fine-tuning and prompting pathways: fine-tuning on biased labels would encode user negativity bias into the model, and evaluating on the same biased feedback would mask the degradation. The paper's own caveats in Section 3 ('requires not only experimental validation but also continuous tracking of which users have seen which posts, posing both technical and data privacy challenges') confirm this is known, but the concern is not resolved. My proposed pilot test is feasible and would settle whether unmarked-seen posts can serve as implicit agreement. If the test fails, the workflow needs a different feedback-acquisition mechanism (e.g., dedicated sampling views) before the central claim can be supported. Since the reader already assigned CONDITIONAL and identified this assumption, my stress-test does not change the verdict; it sharpens the condition under which the paper would become a supported contribution.","tokens_in":3806,"tokens_out":4121,"duration_ms":47688,"concrete_test":"Run a small pilot with 5–10 CSO users on a fixed corpus of 500 Telegram posts with expert labels (e.g., the dataset from Pustet et al., 2024). Log every post displayed, clicks, and explicit corrections. Compute precision of implicit agreement: among posts shown but left unmarked, what fraction are expert-labeled conspiracy-theory posts? Also compute the disagreement/agreement ratio for explicit feedback. If implicit-agreement precision is below roughly 80% or explicit feedback is heavily skewed toward disagreement, then the Section 3 feedback mechanism cannot yield a representative gold standard. As a complementary check, simulate the same negative-bias sampling on the existing labeled corpus and compare a model fine-tuned on the simulated feedback versus one trained on unbiased labels; if performance on a held-out test set drops materially, the loop's core assumption fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that accumulated user feedback can continuously refine both fine-tuned and prompt-based classifiers depends on the feedback forming a representative gold-standard dataset. Section 3 explicitly concedes the main threat: 'due to negativity bias, users of the software are more likely to provide feedback when they disagree with a label, while agreement often goes unmarked.' The proposed remedy, treating 'posts that were seen or clicked but left unmarked as signals of agreement,' is not validated and is not obviously valid: in a monitoring feed, inaction can reflect inattention, uncertainty, fatigue, or lack of time rather than agreement. If that signal is wrong, then fine-tuning receives systematically mislabeled negative examples, and evaluation on the same accumulated feedback cannot detect the resulting bias. The paper also says updated models are 're-evaluated with accumulated feedback' without specifying an independent held-out split, so there is a risk of circular evaluation. Because no implementation or pilot data is presented, the feedback loop's foundational assumption is entirely unsupported. This is not a mismatch with consensus; it is an internal risk acknowledged in the text. The paper remains a useful design proposal, but the central claim should be treated as conditional on validating the implicit-agreement signal and on using independent validation for model updates.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This work-in-progress paper proposes a feedback-driven workflow for integrating fine-tuned and prompt-based (L)LM classifiers into an open-source Telegram monitoring tool used by civil society organizations (CSOs). The authors describe two pathways: fine-tuning (FT) a BERT-based conspiracy-theory classifier and prompting (P) an LLM, with user feedback on binary labels accumulated into a growing gold-standard dataset that is used for both model refinement and evaluation. They discuss key challenges, including negativity bias in feedback, implicit affirmation from unmarked posts, conflict resolution among users, retraining schedules, and privacy constraints, and they compare the FT and P approaches in Table 1. No implementation, simulation, or empirical evaluation is presented; the paper is explicitly framed as ongoing work.","tokens_in":4027,"tokens_out":4521,"duration_ms":53531,"significance":"If validated, this design would address a real gap in open-source social-media monitoring tools by giving CSOs an active role in continuously adapting AI classifiers to their monitoring needs. The paper's strengths include a clearly motivated problem, an explicit focus on stakeholder involvement, and transparent acknowledgment of its main assumptions and limitations, particularly the need to validate implicit-affirmation signals and to use proper validation splits. The proposed workflow is internally coherent and grounded in prior work on fact-checking and Telegram monitoring. However, the paper currently provides no empirical evidence for its central value proposition, and Table 1 presents a comparative rating without supporting evaluation. The contribution is therefore best seen as a design proposal whose load-bearing assumptions remain unverified.","major_comments":[{"comment":"The feedback loop's value depends on treating posts that were seen or clicked but left unmarked as implicit signals of agreement. The paper itself notes that this 'would require not only experimental validation' and raises privacy concerns, but this assumption is load-bearing: if unmarked posts reflect inattention, fatigue, or uncertainty rather than agreement, then fine-tuning receives systematically biased negative examples and evaluation on the same feedback cannot detect the bias. Please either provide validation evidence for this signal or redesign the feedback collection so that representative explicit feedback (e.g., through dedicated full-sample rating views) is the primary mechanism, with implicit signals used only as auxiliary information.","section":"§3, implicit affirmation"},{"comment":"The sentence 'Updated models or prompts are then re-evaluated with accumulated feedback and iteratively deployed' is circular if the same accumulated feedback is used both for refinement and for evaluation. The later mention of using part of the gold-standard dataset for test/validation sets and A/B testing is only in passing; the workflow should make independent held-out validation an explicit, non-optional component rather than an afterthought. Without this, the claim that feedback-driven adaptation improves models is not defensible even in principle.","section":"§3, re-evaluation"},{"comment":"The ratings in Table 1 (++, +, 0) are presented as a formal comparison of FT and P, but no methodology, data, or references are given to support these ratings. As written, the table asserts, for example, that FT has '++' for model behavior control while P has '0', and that P has '++' for adaptability to task variations while FT has '0', without any evidence. Please either reframe Table 1 as an explicitly qualitative assessment based on the authors' experience and the cited literature, or provide a systematic evaluation that justifies the ratings.","section":"Table 1"},{"comment":"The paper's central value proposition is that feedback-driven adaptation can meaningfully improve AI-assisted monitoring for CSOs, yet no empirical evidence, simulation, or pilot study is reported. The paper is transparently labeled work in progress, which is appropriate for a workshop or design-track submission, but for a full journal the central claim remains unverified. A small-scale evaluation, even with simulated user feedback or on a small set of Telegram posts, would substantially strengthen the contribution and allow the proposed workflow to be assessed.","section":"§1 and §4"}],"minor_comments":[{"comment":"The phrase 'concept drift, detected by comparing new data to the training corpus' is underspecified; please clarify what comparison method is intended (e.g., embedding distances, classifier confidence distributions, or statistical tests).","section":"§3"},{"comment":"The Pustet et al. (2024) BERT-based model is described as 'well-performing,' but no performance figures are provided; citing a specific metric (e.g., F1 or accuracy) would make the starting point clearer.","section":"§2"},{"comment":"The table legend defines symbols as ++ = very good, + = acceptable, 0 = limited, but some cells contain compound entries such as '0/+'; please clarify whether this means a range or an intermediate rating.","section":"Table 1"},{"comment":"The future work section is very brief; expanding it with concrete evaluation plans (e.g., how the implicit-affirmation signal would be tested, or how hybrid pre-labeling strategies would be measured) would help readers understand the next steps.","section":"§4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as a workshop-style position paper: it is short, largely qualitative, and explicitly work in progress. For a full journal, the contribution is currently thin, with no data and a central assumption (implicit affirmation) that is acknowledged as unvalidated. The topic is timely and the authors are transparent about limitations, but the paper would need at least a proof-of-concept evaluation or a substantially more detailed and falsifiable design before it could meet the standard for journal acceptance. The authors may also consider submitting to a venue that explicitly welcomes work-in-progress or design proposals."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: this is a design proposal, not a demonstrated result. What it does well is lay out why CSO Telegram monitoring needs AI integration and why feedback-driven adaptation is the right lens. The related work is current and useful, and the authors are transparent that the tool is not implemented. The characterization of the gap—existing open-source Telegram tools offer extraction and search but not integrated, user-steerable classification—is fair.\n\nThe genuinely useful part is Section 3: the two-pathway design (fine-tuning vs prompting) and the explicit discussion of negativity bias. They state clearly that users over-report disagreements, and they float the idea of treating unmarked seen posts as implicit agreement. They also note this needs experimental validation and raises privacy issues. That is intellectually honest.\n\nNow the soft spots. The central claim—that accumulated user feedback can refine and evaluate both model types—rests entirely on that untested implicit-agreement signal. In a monitoring feed, inaction often means inattention or fatigue, not agreement. If the signal is wrong, fine-tuning gets biased negative examples and the evaluation on the same feedback cannot detect the bias. The paper gestures at validation splits and A/B testing but offers no concrete protocol. Table 1 is an unsubstantiated comparison; there is no data behind it. So as submitted, the paper has no empirical content and its foundation is conditional.\n\nThat said, the authors are not overclaiming: they call it work in progress and list the open problems. The design is coherent and grounded in prior human-in-the-loop work, not circular with their own citations. The citation pattern looks fine.\n\nWho is this for? People building monitoring tools for civil society, and researchers working on feedback loops for low-resource social-media classification. For that audience, this is a useful design note. It deserves a workshop referee or a short-paper track, not a main-conference submission in its current form. If they build the prototype and pilot the implicit-agreement signal with even a small user study, the next version would merit serious review.","headline":"Honest, clearly written work-in-progress design paper whose central feedback-loop claim rests on an untested implicit-agreement assumption; worth workshop referee time, not a full paper yet.","tokens_in":4535,"tokens_out":2069,"would_cite":false,"duration_ms":24051,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A feedback loop turns civil society analysts' label corrections into a growing gold-standard dataset that retrains fine-tuned models or revises prompts, so an open-source Telegram monitoring tool improves as CSO users do their regular…","keywords":["civil society organizations","content moderation","Telegram monitoring","conspiracy theory classification","human-in-the-loop","feedback-driven adaptation","fine-tuning","prompt engineering"],"falsifier":"Take a random sample of Telegram posts from the same channels, have independent annotators label the full sample, and compare those labels with the dataset accumulated from ordinary monitoring feedback. If the feedback-based dataset is systematically unrepresentative—for example, if unmarked posts contain many false negatives or false positives—then model evaluations and updates built on it are unreliable, and the paper's central claim fails.","tokens_in":3624,"feed_emoji":"🔄","tokens_out":7508,"duration_ms":74465,"temperature":0.7,"pith_summary":"This work-in-progress paper proposes that an open-source Telegram monitoring tool can improve its conspiracy-theory classification by letting civil society analysts correct the machine's labels while they work. Each correction is stored with the original classification and feedback metadata, building a growing gold-standard dataset that is then used to retrain a fine-tuned model or revise a prompt. The paper argues this feedback loop makes model evaluation part of routine monitoring, not a separate task, and lets non-technical users steer AI-assisted content moderation in their own secure environment. If the loop works, civil society organizations gain a practical, transparent alternative to closed-source platform moderation.","feed_headline":"User corrections become training data for Telegram moderation AI","feed_subtitle":"A feedback loop turns analysts' label corrections into a gold-standard set that retrains models or revises prompts.","key_machinery":"The load-bearing mechanism is the feedback loop itself, centered on the growing gold-standard dataset: labels that users correct while monitoring, stored together with the original prediction and feedback metadata, become the resource for retraining, prompt revision, validation, and drift-triggered updates. It is what turns routine human judgment into a reusable evaluation and adaptation signal, and it is what both the fine-tuning and prompting pathways feed on.","core_discovery":"The central proposal is a workflow in which each Telegram post is shown to a user with a binary conspiracy-theory label and a confidence score, and the user can correct that label directly in the monitoring feed. Those corrections accumulate into a growing gold-standard dataset—the stored original classification, feedback, and metadata—that serves two purposes: evaluating updated models and supplying training or prompting material. In the fine-tuning pathway, concept drift detected by comparing new data with the training corpus can trigger retraining; in the prompting pathway, feedback guides prompt revision and few-shot example selection. Updated models or prompts are re-evaluated on the accumulated feedback and iteratively deployed. The paper treats fine-tuning and prompting as complementary approaches with different trade-offs, not as a single winner.","pith_inferences":["Beyond the paper, the implicit-agreement strategy—treating unmarked seen posts as agreement—is the most promising fix for negativity bias, but it is also the clearest point where the loop could silently break; a field trial with explicit seen-tracking would settle it.","If the feedback loop works for conspiracy-theory classification, the same infrastructure could extend to other CSO monitoring tasks such as summarization or question answering, though prompt updates for those tasks would need equally careful validation.","The growing gold-standard dataset could eventually be pooled or federated across CSOs, but divergent labeling practices would require governance before such sharing becomes reliable.","A natural quantitative test the paper does not run: compare how quickly classification accuracy improves per unit of feedback in the fine-tuning versus prompting pathways, since the optimal choice may shift as feedback accumulates."],"forward_implications":["An open-source Telegram monitor can embed model evaluation inside routine monitoring rather than requiring separate annotation tasks, because the feed itself collects corrections.","Both fine-tuning and prompting can be steered by the same growing gold-standard dataset, so organizations can choose a pathway by available hardware and technical skill.","The accumulated feedback supports concept-drift detection and retraining triggers in the fine-tuning pathway, keeping the classifier current as online discourse shifts.","Deployment-time validation can be done with held-out parts of the accumulated feedback, and prompt changes can be tested before applying them at scale.","The privacy and cost constraints point toward smaller or distilled models running on local infrastructure, avoiding third-party APIs."],"supporting_citations":[{"why":"Supplies the BERT-based German-language Telegram conspiracy-theory classifier that the fine-tuning pathway starts from.","marker":"Pustet et al., 2024"},{"why":"Shows that few-shot prompting results depend on example selection, motivating automated prompt experiments in the feedback loop.","marker":"Chae and Davidson, 2025"},{"why":"Documents fact-checkers' call for adaptive AI systems that incorporate user feedback, the motivation behind the loop.","marker":"Juneja and Mitra, 2022"},{"why":"Raises privacy and data-ownership concerns that constrain the design to local, secure deployment.","marker":"Wolfe and Mitra, 2024"},{"why":"An earlier open-source social-media monitoring system with relevance feedback, the direct predecessor this design extends.","marker":"Schinas et al., 2017"},{"why":"A transparent, NLP-capable capture and analysis toolkit that still lacks AI-based analysis and user feedback integration, defining the gap.","marker":"Peeters and Hagen, 2022"}],"fun_headline_variants":["Feed corrections retrain Telegram monitoring AI","User label fixes become training data for AI","Analyst corrections drive AI retraining loop","Feedback loop turns user fixes into training data","User corrections become gold-standard training set"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that feedback left behind during ordinary monitoring, including inferred implicit agreement from unmarked posts, produces a representative gold-standard dataset; if that premise fails, the retraining, prompting updates, and evaluations in the loop are all built on skewed data.","fun_headline_variants_meta":{"raw":{"variants":["Feed corrections retrain Telegram monitoring AI","User label fixes become training data for AI","Analyst corrections drive AI retraining loop","Feedback loop turns user fixes into training data","User corrections become gold-standard training set"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000451,"raw_usage":{"total_tokens":2228,"prompt_tokens":856,"completion_tokens":1372,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":1308}},"tokens_in":472,"tokens_out":1372,"duration_ms":49959,"temperature":1.0,"reasoning_tokens":1308,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:56:23.765657+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of Telegram posts from the same channels, have independent annotators label the full sample, and compare those labels with the dataset accumulated from ordinary monitoring feedback. If the feedback-based dataset is systematically unrepresentative—for example, if unmarked posts contain many false negatives or false positives—then model evaluations and updates built on it are unreliable, and the paper's central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"An earlier open-source social-media monitoring system with relevance feedback, the direct predecessor this design extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A transparent, NLP-capable capture and analysis toolkit that still lacks AI-based analysis and user feedback integration, defining the gap."}],"review_version":1}