REVIEW 3 major objections 4 minor 10 references
Agro-STAY : Collecte de donn\'ees et analyse des informations en agriculture alternative issues de YouTube
T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper claims that a purpose-built platform, Agro-STAY, makes YouTube videos and comments about self-sufficient agriculture searchable and classifiable, with baseline classifiers reaching macro F1 scores of 72.01 for controversy and…
desk verdict A worthwhile platform paper with a real gap and honest framing, but the classifier evaluation is too thin and the comment-level split likely inflates the reported F1 scores. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Agro-STAY processing pipeline, built around the fine-tuned CamemBERT French language model. CamemBERT is a RoBERTa-style transformer pretrained on the OSCAR French corpus; after fine-tuning on 1,400 expert-annotated YouTube comments, it produces the two classifiers (binary controversy, six-way information type) that the paper evaluates. Around this sit the collection and indexing components: the YouTube API and youtube-transcript-api for data acquisition, Punctuator for restoring punctuation to noisy transcripts, and spaCy for sentence segmentation, named-entity indexing, and keyword search. The pipeline is the mechanism that turns raw, poorly formatted YouTube text into the structured, queryable corpus that sociologists analyze.
What would settle it
Have a third expert independently annotate a random sample of the same 1,400 comments, then compute agreement (e.g., Cohen's kappa) between the original labels and the new ones; if agreement is low, the reported macro F1 values cannot be taken as true model quality. A simpler check: run the trained classifiers on a fresh set of comments from different YouTube channels and see whether performance holds.
Extended reading notes
Core claim
The central claim is that NLP and language models, applied to YouTube transcripts and comments, can support fine-grained analysis of alternative agricultural practices. The paper demonstrates this with Agro-STAY, whose pipeline collects transcripts via YouTube's API plus a third-party transcript library, restores punctuation with Punctuator, segments and indexes text with spaCy, and then classifies comments using a fine-tuned CamemBERT model. The classification experiments target two tasks: detecting whether a comment is controversial, and assigning it to one of six information categories (pest/disease management, water management, soil management, weed management, harvest, or non-relevant). The reported results, macro F1 of 72.01 for controversy and 40.57 for information type, are presented as encouraging baselines rather than final performance, with imbalance and informal comment text identified as the main obstacles.
Load-bearing premise
The whole evaluation rests on the two experts' labels being consistent, but the paper never measures how much the two annotators agree and they labeled very different amounts of data (100 versus 1,400 comments); if their labels are inconsistent, the reported F1 scores are not a reliable measure of the classifiers.
Editorial extensions
If this is right
- With Agro-STAY, a researcher can query 1,423 transcriptions and 45,000 comments by topic, keyword, entity, or channel, and then pull up the exact video and comment where a practice is discussed.
- The trained comment classifiers give an automatic first pass that separates relevant from non-relevant comments, reducing the manual annotation burden for social science studies.
- The same classifiers are intended to be applied to sentence-level transcript fragments, so relevant passages inside long videos can be located without watching them in full.
- The class-imbalance gap between macro and weighted F1 makes clear that rebalancing, better text normalization, and hyperparameter tuning are the next concrete improvements.
- Because the pipeline is generic, the same collection-to-classification workflow can be pointed at other YouTube topics beyond alternative agriculture.
Reading between the lines
- The absence of any inter-annotator agreement measure means the reported F1 scores should be read as upper bounds; if the two experts label inconsistently, the model may be learning noise. A shared, independently adjudicated test set would make the numbers interpretable.
- A macro F1 of 40.57 across six classes is modest in absolute terms, but in a corpus with roughly 90% non-relevant comments, even this filter could cut manual reading effort substantially if used as a ranking tool rather than a final label.
- The platform's real test will come when transcript classification is added, because transcripts are longer, less grammatical, and contain more technical detail than comments; performance there may differ sharply from comment performance.
- Connecting comment-level controversy to video-level topics could reveal which techniques generate disagreement, a signal that bears directly on the sociology of knowledge circulation the STAY project wants to study.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents Agro-STAY, a web platform for collecting, processing, classifying, and visualizing YouTube video transcripts and comments in the domain of alternative agriculture and self-sufficiency. The platform integrates the YouTube API, a transcript library, punctuation restoration, spaCy-based preprocessing, and a CamemBERT-based classifier. Using 1,400 expert-annotated comments from 12 videos, the authors train two classifiers: a binary controversy detector and a six-class information-type classifier. They report macro F1-scores of 72.01 and 40.57, respectively, on a single 80/20 train/test split. The paper argues that these results are encouraging for supporting sociologists in analyzing large-scale YouTube data, while also acknowledging that the platform is still in its first version and several improvements are planned.
Significance. If the quantitative claims are reliable, the platform would be a useful contribution for social science research on YouTube-based knowledge circulation, particularly for studying alternative agriculture and self-sufficiency movements. The paper is explicit about the platform being a first version and describes the data collection and processing pipeline in detail, which aids reproducibility of the system design. The use of CamemBERT, a well-established French language model, and the inclusion of concrete class definitions with examples are also strengths. However, the evaluation methodology as reported does not yet substantiate the claim that the classifiers generalize to new YouTube videos, and the absence of baseline comparisons and inter-annotator agreement measures leaves the main quantitative contribution insufficiently supported.
major comments (3)
- [Section 4.2, Table 2] The 80/20 split is performed at the comment level, but Section 3.3 states that the 1,400 annotated comments come from only 12 videos. As a result, comments from the same videos appear in both the training and test portions, allowing the model to exploit video-specific vocabulary and content rather than learning generalizable patterns. The deployment scenario described in Section 4 ('sélectionner les commentaires d'intérêt' for newly collected videos) requires generalization to unseen videos, which this evaluation does not measure. Please either stratify the split by video, use leave-one-video-out evaluation, or report whether the current split is actually video-disjoint. Without this, the reported macro F1 values of 72.01 and 40.57 may substantially overstate performance on new videos.
- [Section 4.3, Table 2] The claim that the information-type classifier's macro F1 of 40.57 'dépasse largement celles d'un classifieur aléatoire' is not supported by any baseline numbers. For a six-class problem with heavy class imbalance (Section 4.1 reports about 90% non-relevant comments), the expected macro F1 of a random or majority-class classifier is nontrivial and should be quantified. Similarly, for the binary controversy task, a majority-class baseline would achieve high weighted F1 and accuracy. The authors should report at least a majority-class baseline and a random baseline (with appropriate class priors) for macro F1, precision, recall, and accuracy. The absence of such baselines makes it difficult to interpret the reported scores as evidence of model competence.
- [Section 3.3] No inter-annotator agreement measure (e.g., Cohen's kappa) is reported, even though the gold labels were produced by two experts who annotated significantly different amounts of data (100 comments vs. 1,400 comments). The text mentions that the experts discussed disagreements and refined class definitions after the first annotation round, but it is not stated how the final labels were consolidated or whether the final dataset predominantly reflects a single annotator's judgments. Without a measure of label consistency, the reliability of the training and evaluation data is unknown, and the reported F1 scores in Table 2 may reflect annotation noise rather than model quality.
minor comments (4)
- [Section 4.1] The text reads 'environ90%' without a space; it should be 'environ 90%'.
- [Remerciements] The sentence 'Nous l'Agence nationale française de la recherche...' is grammatically incomplete; it should likely be 'Nous remercions l'Agence...' or similar.
- [Figures] The text in Section 3.1 references 'figure 2b' for the phrase-list view, but Figure 2b in the caption is described as 'Annotation manuelle d'un article dans la classe Controverse.' Please verify the figure numbering and the in-text references.
- [Section 4, classification tasks] The information-type classification is described as having 'six classes' (including 'Non-pertinent'), but Table 1 lists only five named classes (Maladies/Ravageurs, Eau, Sol, Adventices, Récolte) plus the implicit 'Non-pertinent' in the task description. Please clarify the class set explicitly to avoid ambiguity.
Circularity Check
No circularity: the platform and classifier evaluation are self-contained; reported scores rest on a train/test split, not on fitted inputs renamed as predictions.
full rationale
The paper's derivation chain is empirical rather than analytic. Agro-STAY is presented as a data collection, processing and visualization platform, and the only quantitative support for its NLP component is the fine-tuning of CamemBERT on 1,400 manually annotated comments followed by evaluation on a disjoint 20% split (Section 4.2: 'Les données sont segmentées en deux sous-ensembles : 80% pour l’apprentissage, 20% l’évaluation'). The reported macro-F1 scores (72.01 for controversy, 40.57 for information type) are measurements on held-out comments, not predictions of quantities that were used as training inputs. The class taxonomy was defined by the same experts who produced the labels, but this is standard corpus construction rather than a self-definitional loop: the classifiers are not defined in terms of their test labels, and the evaluation is separate from the training set. There are no self-citations in the reference list, no imported uniqueness theorems, and no ansatz smuggled in via citation. The potential methodological weaknesses raised by a skeptical reading—lack of reported inter-annotator agreement and the possibility that an 80/20 comment-level split shares videos between train and test, limiting evidence for generalization to unseen videos—bear on external validity of the F1 numbers, but they do not make any result equivalent to its input by construction. No circular step can be exhibited from the paper's own text, so the honest finding is no significant circularity.
Assumptions & free parameters
assumptions (3)
- domain assumption The six predefined thematic classes (Maladies/Ravageurs, Eau, Sol, Adventices, Récolte, Non-pertinent) are the correct and sufficient categories for the content.
- domain assumption Automatic YouTube transcripts, after punctuation restoration, are accurate enough for meaningful NLP analysis.
- domain assumption The six expert-selected YouTube channels are representative of the French self-sufficiency movement.
Cite this review
Pith. "Pith review of Agro-STAY : Collecte de donn\'ees et analyse des informations en agriculture alternative issues de YouTube." pith.science (2026). https://pith.science/paper/WFY4D7RD
@misc{pith2026241210576,
author = {Pith},
title = {Pith review of: Agro-STAY : Collecte de donn\'ees et analyse des informations en agriculture alternative issues de YouTube},
year = {2026},
howpublished = {\url{https://pith.science/paper/WFY4D7RD}},
note = {Machine review of arXiv:2412.10576}
}
read the original abstract
To address the current crises (climatic, social, economic), the self-sufficiency -- a set of practices that combine energy sobriety, self-production of food and energy, and self-construction - arouses an increasing interest. The CNRS STAY project (Savoirs Techniques pour l'Auto-suffisance, sur YouTube) explores this topic by analyzing techniques shared on YouTube. We present Agro-STAY, a platform designed for the collection, processing, and visualization of data from YouTube videos and their comments. We use Natural Language Processing (NLP) techniques and language models, which enable a fine-grained analysis of alternative agricultural practice described online. -- Face aux crises actuelles (climatiques, sociales, \'economiques), l'auto-suffisance -- ensemble de pratiques combinant sobri\'et\'e \'energ\'etique, autoproduction alimentaire et \'energ\'etique et autoconstruction - suscite un int\'er\^et croissant. Le projet CNRS STAY (Savoirs Techniques pour l'Auto-suffisance, sur YouTube) s'inscrit dans ce domaine en analysant les savoirs techniques diffus\'es sur YouTube. Nous pr\'esentons Agro-STAY, une plateforme d\'edi\'ee \`a la collecte, au traitement et \`a la visualisation de donn\'ees issues de vid\'eos YouTube et de leurs commentaires. En mobilisant des techniques de traitement automatique des langues (TAL) et des mod\`eles de langues, ce travail permet une analyse fine des pratiques agricoles alternatives d\'ecrites en ligne.
Reference graph
Works this paper leans on
- [1]
-
[2]
Bruhl, G. (2023). État des lieux de la vulgarisation scientifique vétérinaire francophone sur Youtube . Thèse de doctorat, Université Claude Bernard Lyon 1 (Médecine – Pharmacie)
work page 2023
-
[3]
Dill, S., S. Z. Li, M. Rohr, M. Sharbafi, C. H. Antink (2023). Automatic generation of labeled data for video-based human pose analysis via nlp applied to youtube subtitles. In Int. Conf. of the IEEE Engineering in Medicine and Biology Society (EMBC) , pp.\ 1--6. IEEE
work page 2023
-
[4]
The Media Leader (2023). Audience internet global février 2023 : près de la moitié des hommes sur les sites et applications de mode / beauté. Technical report, The Media Leader
work page 2023
-
[5]
Abadji , J., P. Ortiz Suarez , L. Romary , B. Sagot (2022). Towards a Cleaner Document-Oriented Multilingual Crawled Corpus . arXiv e-prints\/ , arXiv:2201.06642
arXiv 2022
-
[6]
Cervety, M. (2021). Audiences, création, parcours d’achat : ce qui change sur youtube en 2021. Think with Google\/ Sep
work page 2021
-
[7]
Martin, L., B. Muller, P. J. Ortiz Su \'a rez, Y. Dupont, L. Romary, \'E . de la Clergerie, D. Seddah, B. Sagot (2020). C amem BERT : a tasty F rench language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , Online, pp.\ 7203--7219. Association for Computational Linguistics
work page 2020
-
[8]
Liu, Y., M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov (2019). Roberta: A robustly optimized BERT pretraining approach. CoRR\/ abs/1907.11692
arXiv 2019
Show all 10 references
-
[9]
Sevigne, P. R. Stevens (2015). Comment tout peut s’effondrer. Petit manuel de collapsologie à l'usage des générations présentes . Paris: Seuil
2015
-
[10]
write newline
" write newline "" before.all 'output.state := FUNCTION article output.bibitem format.authors "author" output.check author format.key output output.year.check new.block format.title "title" output.check new.block crossref missing format.jour.vol output format.article.crossref ...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.