{"id":"72511811-59ff-464e-ac18-683006b41024","arxiv_id":"2412.10576","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Agro-STAY is a working platform that gathers and analyzes French YouTube content on self-sufficient agriculture, with preliminary CamemBERT-based topic and controversy classifiers.","lead":"This paper presents Agro-STAY, a platform that collects YouTube videos and comments about self-sufficiency farming in France, transcribes them, and classifies them by topic. It is a tool for sociologists studying alternative agriculture communities online.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation splits comments from the same 12 videos across train/test, so reported macro-F1 may reflect video-specific cues rather than generalization to unseen videos.","rationale":"The paper's central claim is that Agro-STAY works as a platform collecting, processing, and visualizing YouTube data, and that its trained classifiers reach the F1 scores in Table 2. The weakest link in that claim is the evaluation protocol for the classifiers. The reader's weakest_assumption correctly identifies annotator agreement as a concern: no IAA is reported, and the two annotators contributed very different amounts of data. That is a real issue for label reliability. However, the more immediately load-bearing problem is the split strategy. All 1,400 training comments come from only 12 videos, and the 80/20 split appears to be done at the comment level. Comments on the same video are not independent: they are responses to the same content and share vocabulary. Thus the model can exploit video identity as a shortcut, and the held-out comments are not representative of new videos. The deployment goal, as stated in Section 4, is to use the model on newly collected comments and transcripts, so the evaluation should simulate unseen videos. A leave-one-video-out check is concrete and would settle whether the reported numbers reflect real generalization. Both the annotation and split issues are fixable, but the split issue directly undermines the quantitative headline. The platform itself is described with screenshots and appears plausible as a system contribution, so conditional acceptance remains appropriate; the verdict does not need to change, only the required revision does.","tokens_in":7808,"tokens_out":5396,"duration_ms":48548,"concrete_test":"Group the 1,400 comments by video and run leave-one-video-out cross-validation (train on comments of 11 videos, test on the remaining video; repeat for all 12). Report macro-F1 averaged over folds for both tasks. If the average is materially lower than Table 2 (e.g., >5 points), the original comment-level split overstates generalization to unseen videos; if it is comparable, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The only quantitative support for the platform's NLP component is Table 2, obtained by an 80/20 split of 1,400 comments (Section 4.2). Section 3.3 states these comments are associated with 12 videos. The split is described at the comment level, so comments from the same video are likely in both training and test sets. YouTube comments under a single video are strongly correlated: they discuss the same content, use the same video-specific vocabulary, and often quote the video. A classifier trained on those comments can therefore memorize per-video cues, and the held-out 20% is not an independent test of generalization to new videos — which is exactly the deployment scenario stated in Section 4 ('sélectionner les commentaires d'intérêt' for newly collected videos). Consequently, the headline macro-F1 values (72.01 controversy, 40.57 information-type) may substantially overstate performance on unseen videos. No statement in the paper indicates that the split was stratified by video, and the release of code/data would be required to check this. This is load-bearing because the paper's claim of a usable classification tool rests entirely on these numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents Agro-STAY, a web platform for collecting, processing, classifying, and visualizing YouTube video transcripts and comments in the domain of alternative agriculture and self-sufficiency. The platform integrates the YouTube API, a transcript library, punctuation restoration, spaCy-based preprocessing, and a CamemBERT-based classifier. Using 1,400 expert-annotated comments from 12 videos, the authors train two classifiers: a binary controversy detector and a six-class information-type classifier. They report macro F1-scores of 72.01 and 40.57, respectively, on a single 80/20 train/test split. The paper argues that these results are encouraging for supporting sociologists in analyzing large-scale YouTube data, while also acknowledging that the platform is still in its first version and several improvements are planned.","tokens_in":7912,"tokens_out":2319,"duration_ms":21335,"significance":"If the quantitative claims are reliable, the platform would be a useful contribution for social science research on YouTube-based knowledge circulation, particularly for studying alternative agriculture and self-sufficiency movements. The paper is explicit about the platform being a first version and describes the data collection and processing pipeline in detail, which aids reproducibility of the system design. The use of CamemBERT, a well-established French language model, and the inclusion of concrete class definitions with examples are also strengths. However, the evaluation methodology as reported does not yet substantiate the claim that the classifiers generalize to new YouTube videos, and the absence of baseline comparisons and inter-annotator agreement measures leaves the main quantitative contribution insufficiently supported.","major_comments":[{"comment":"The 80/20 split is performed at the comment level, but Section 3.3 states that the 1,400 annotated comments come from only 12 videos. As a result, comments from the same videos appear in both the training and test portions, allowing the model to exploit video-specific vocabulary and content rather than learning generalizable patterns. The deployment scenario described in Section 4 ('sélectionner les commentaires d'intérêt' for newly collected videos) requires generalization to unseen videos, which this evaluation does not measure. Please either stratify the split by video, use leave-one-video-out evaluation, or report whether the current split is actually video-disjoint. Without this, the reported macro F1 values of 72.01 and 40.57 may substantially overstate performance on new videos.","section":"Section 4.2, Table 2"},{"comment":"The claim that the information-type classifier's macro F1 of 40.57 'dépasse largement celles d'un classifieur aléatoire' is not supported by any baseline numbers. For a six-class problem with heavy class imbalance (Section 4.1 reports about 90% non-relevant comments), the expected macro F1 of a random or majority-class classifier is nontrivial and should be quantified. Similarly, for the binary controversy task, a majority-class baseline would achieve high weighted F1 and accuracy. The authors should report at least a majority-class baseline and a random baseline (with appropriate class priors) for macro F1, precision, recall, and accuracy. The absence of such baselines makes it difficult to interpret the reported scores as evidence of model competence.","section":"Section 4.3, Table 2"},{"comment":"No inter-annotator agreement measure (e.g., Cohen's kappa) is reported, even though the gold labels were produced by two experts who annotated significantly different amounts of data (100 comments vs. 1,400 comments). The text mentions that the experts discussed disagreements and refined class definitions after the first annotation round, but it is not stated how the final labels were consolidated or whether the final dataset predominantly reflects a single annotator's judgments. Without a measure of label consistency, the reliability of the training and evaluation data is unknown, and the reported F1 scores in Table 2 may reflect annotation noise rather than model quality.","section":"Section 3.3"}],"minor_comments":[{"comment":"The text reads 'environ90%' without a space; it should be 'environ 90%'.","section":"Section 4.1"},{"comment":"The sentence 'Nous l'Agence nationale française de la recherche...' is grammatically incomplete; it should likely be 'Nous remercions l'Agence...' or similar.","section":"Remerciements"},{"comment":"The text in Section 3.1 references 'figure 2b' for the phrase-list view, but Figure 2b in the caption is described as 'Annotation manuelle d'un article dans la classe Controverse.' Please verify the figure numbering and the in-text references.","section":"Figures"},{"comment":"The information-type classification is described as having 'six classes' (including 'Non-pertinent'), but Table 1 lists only five named classes (Maladies/Ravageurs, Eau, Sol, Adventices, Récolte) plus the implicit 'Non-pertinent' in the task description. Please clarify the class set explicitly to avoid ambiguity.","section":"Section 4, classification tasks"}],"recommendation":"major_revision","confidential_remarks":"The paper is essentially a system description with a preliminary evaluation. The evaluation shortcomings identified in the major comments are addressable (video-level split, baselines, agreement measures) within the scope of the manuscript, so I do not recommend rejection. However, the current quantitative claims are not yet sufficiently supported, and the revision should either strengthen the evaluation or temper the conclusions to match the evidence. There is also a question of whether the journal's audience would benefit from more detail on the platform's architecture and usability; as it stands, the balance between system description and NLP evaluation is somewhat uneven."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a legitimate platform paper with a genuinely new domain application, but the classifier evaluation is too thin to support the reported F1 scores, and the comment-level split likely overstates generalization to unseen videos.\n\nWhat's new: Agro-STAY is the first described platform for collecting and analyzing French YouTube content on self-sufficiency agriculture. That fills a real gap—sociologists of the autonomist movement haven't looked at its online presence. The platform itself seems real: YouTube API for metadata and comments, transcript scraping, punctuation restoration, spaCy processing, and a working web interface with screenshots. The annotated corpus of 1,400 comments, with a six-class scheme for agricultural practices and controversy labels, is a useful asset. The authors are honest that this is a first version.\n\nSoft spots: the evaluation is the weak link. One 80/20 split, no cross-validation, no baselines, no confidence intervals. The paper reports macro F1 of 72.01 for controversy and 40.57 for info-type, but without a majority-class baseline (which would give near-zero macro F1), the claim that 40.57 beats random is not the right comparison. More seriously, the 1,400 comments come from only 12 videos, and the split appears to be at the comment level, not stratified by video. Comments under one video share vocabulary and content, so random comment splitting leaks video-specific cues into the test set. The stated deployment scenario is classifying new videos, so this matters. The paper does not say the split was stratified, and no code or data is released to check. That's a load-bearing weakness for the numerical claims.\n\nAlso, no inter-annotator agreement is reported, and the two experts annotated very different amounts (100 vs 1,400 comments). This would be fine if kappa were given; it isn't.\n\nThese are not fatal for a demonstration paper, but the numbers should be labeled preliminary. The platform and corpus are the contribution; the classifiers are a feature.\n\nWho's this for? Computational social scientists studying alternative agriculture or niche French YouTube communities, and NLP folks needing a domain benchmark. It deserves serious refereeing, but not unconditional acceptance. My recommendation: send it to review, require the authors to either release code/data and clarify the split (preferably video-level stratified or at least report leakage), or significantly soften the performance claims.\n\nI wouldn't cite the F1 numbers, but I might cite the platform/corpus if the artifacts are released. Bring to reading group: maybe, as an example of an honest platform paper whose evaluation is too weak for its conclusion.","headline":"A worthwhile platform paper with a real gap and honest framing, but the classifier evaluation is too thin and the comment-level split likely inflates the reported F1 scores.","tokens_in":8530,"tokens_out":2955,"would_cite":false,"duration_ms":25172,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a purpose-built platform, Agro-STAY, makes YouTube videos and comments about self-sufficient agriculture searchable and classifiable, with baseline classifiers reaching macro F1 scores of 72.01 for controversy and…","keywords":["YouTube","alternative agriculture","self-sufficiency","comment classification","French NLP","CamemBERT","controversy detection","information-type classification"],"falsifier":"Have a third expert independently annotate a random sample of the same 1,400 comments, then compute agreement (e.g., Cohen's kappa) between the original labels and the new ones; if agreement is low, the reported macro F1 values cannot be taken as true model quality. A simpler check: run the trained classifiers on a fresh set of comments from different YouTube channels and see whether performance holds.","tokens_in":7545,"feed_emoji":"🌱","tokens_out":5309,"duration_ms":44330,"temperature":0.7,"pith_summary":"The paper is trying to establish that a dedicated platform, Agro-STAY, can collect YouTube videos and comments about self-sufficient (alternative) agriculture and make them analyzable at scale for social scientists. It reports a first working version that has gathered 1,423 video transcriptions and nearly 45,000 comments, and has trained French-language classifiers on 1,400 expert-annotated comments. On a held-out 20% of those comments, the controversy classifier reaches a macro F1 of 72.01, while the six-way information-type classifier reaches 40.57, both well above random performance despite strong class imbalance. The intended payoff is that sociologists can systematically study the technical knowledge shared by 'autonomist' gardening and farming communities on YouTube instead of relying on small ethnographic samples.","feed_headline":"NLP platform sorts YouTube comments on alternative farming","feed_subtitle":"Agro-STAY sifts 45,000 comments into pest, water, soil, weed, harvest, and controversy classes.","key_machinery":"The load-bearing object is the Agro-STAY processing pipeline, built around the fine-tuned CamemBERT French language model. CamemBERT is a RoBERTa-style transformer pretrained on the OSCAR French corpus; after fine-tuning on 1,400 expert-annotated YouTube comments, it produces the two classifiers (binary controversy, six-way information type) that the paper evaluates. Around this sit the collection and indexing components: the YouTube API and youtube-transcript-api for data acquisition, Punctuator for restoring punctuation to noisy transcripts, and spaCy for sentence segmentation, named-entity indexing, and keyword search. The pipeline is the mechanism that turns raw, poorly formatted YouTube text into the structured, queryable corpus that sociologists analyze.","core_discovery":"The central claim is that NLP and language models, applied to YouTube transcripts and comments, can support fine-grained analysis of alternative agricultural practices. The paper demonstrates this with Agro-STAY, whose pipeline collects transcripts via YouTube's API plus a third-party transcript library, restores punctuation with Punctuator, segments and indexes text with spaCy, and then classifies comments using a fine-tuned CamemBERT model. The classification experiments target two tasks: detecting whether a comment is controversial, and assigning it to one of six information categories (pest/disease management, water management, soil management, weed management, harvest, or non-relevant). The reported results, macro F1 of 72.01 for controversy and 40.57 for information type, are presented as encouraging baselines rather than final performance, with imbalance and informal comment text identified as the main obstacles.","pith_inferences":["The absence of any inter-annotator agreement measure means the reported F1 scores should be read as upper bounds; if the two experts label inconsistently, the model may be learning noise. A shared, independently adjudicated test set would make the numbers interpretable.","A macro F1 of 40.57 across six classes is modest in absolute terms, but in a corpus with roughly 90% non-relevant comments, even this filter could cut manual reading effort substantially if used as a ranking tool rather than a final label.","The platform's real test will come when transcript classification is added, because transcripts are longer, less grammatical, and contain more technical detail than comments; performance there may differ sharply from comment performance.","Connecting comment-level controversy to video-level topics could reveal which techniques generate disagreement, a signal that bears directly on the sociology of knowledge circulation the STAY project wants to study."],"forward_implications":["With Agro-STAY, a researcher can query 1,423 transcriptions and 45,000 comments by topic, keyword, entity, or channel, and then pull up the exact video and comment where a practice is discussed.","The trained comment classifiers give an automatic first pass that separates relevant from non-relevant comments, reducing the manual annotation burden for social science studies.","The same classifiers are intended to be applied to sentence-level transcript fragments, so relevant passages inside long videos can be located without watching them in full.","The class-imbalance gap between macro and weighted F1 makes clear that rebalancing, better text normalization, and hyperparameter tuning are the next concrete improvements.","Because the pipeline is generic, the same collection-to-classification workflow can be pointed at other YouTube topics beyond alternative agriculture."],"supporting_citations":[{"why":"Supplies CamemBERT, the pretrained French language model that is fine-tuned for the two classification tasks.","marker":"Martin et al., 2020"},{"why":"Provides the RoBERTa pretraining architecture on which CamemBERT is based.","marker":"Liu et al., 2019"},{"why":"Provides the OSCAR multilingual crawled corpus used to pretrain CamemBERT on French text.","marker":"Abadji et al., 2022"}],"fun_headline_variants":["Agro-STAY mines YouTube comments to classify alternative farming talk","NLP tool sorts 45K YouTube comments on alternative agriculture","CamemBERT classifies YouTube comments into farming topics","NLP model detects controversy in alternative farming comments","Agro-STAY analyzes YouTube data for self-sufficiency insights"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole evaluation rests on the two experts' labels being consistent, but the paper never measures how much the two annotators agree and they labeled very different amounts of data (100 versus 1,400 comments); if their labels are inconsistent, the reported F1 scores are not a reliable measure of the classifiers.","fun_headline_variants_meta":{"raw":{"variants":["Agro-STAY mines YouTube comments to classify alternative farming talk","NLP tool sorts 45K YouTube comments on alternative agriculture","CamemBERT classifies YouTube comments into farming topics","NLP model detects controversy in alternative farming comments","Agro-STAY analyzes YouTube data for self-sufficiency insights"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000901,"raw_usage":{"total_tokens":3927,"prompt_tokens":1039,"completion_tokens":2888,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":655,"completion_tokens_details":{"reasoning_tokens":2807}},"tokens_in":655,"tokens_out":2888,"duration_ms":22043,"temperature":1.0,"reasoning_tokens":2807,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:49:40.074513+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a third expert independently annotate a random sample of the same 1,400 comments, then compute agreement (e.g., Cohen's kappa) between the original labels and the new ones; if agreement is low, the reported macro F1 values cannot be taken as true model quality. A simpler check: run the trained classifiers on a fresh set of comments from different YouTube channels and see whether performance holds.","supporting_citations":[{"cited_title":"Muller, P","cited_arxiv_id":null,"evidence_quote":"Supplies CamemBERT, the pretrained French language model that is fine-tuned for the two classification tasks."}],"review_version":1}