{"id":"7fadac08-17c3-4c85-8b42-d2e31508299a","arxiv_id":"2506.01468","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A graph neural network that combines YOLOv8n sheep face landmark detections estimates total facial pain with 92.71% accuracy on a new SPFES-labeled dataset.","lead":"This paper trains a graph neural network to combine sheep facial landmark detections (eyes, ears, nose, cheeks, lips, jaw) into a single pain score, reporting 92.71% accuracy with a YOLOv8n detector. It also introduces a new SPFES-annotated sheep facial landmark dataset and benchmarks eight detectors, aiming at automated animal welfare monitoring from drones or phones.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 92.71%/91.96% accuracy claim is not evidence of pain-assessment skill unless the ground-truth pain labels are independent of the SPFES part scores and weights that the WGNN itself aggregates; the paper never shows this.","rationale":"For the strongest claim to hold, three conditions are needed: the YOLOv8n part-level detections must be accurate enough, the SPFES weights and clustering must be meaningful, and the evaluation must compare against ground-truth pain labels that are not derived from the same formula the model applies. The paper's Section IV.7 already concedes the first condition is fragile, so the reader concentrated on p_i and w_i. I think the more fundamental, and less discussed, issue is the third condition: circularity between label construction and the WGNN's output. If the ground-truth pain level is generated by applying the same SPFES weighting to the same p_i annotations, then the reported 92.71% training and 91.96% testing accuracy are largely a measure of how well YOLOv8n reproduces those annotations, not of the WGNN's pain-assessment ability. The paper's own Eq. (2) graph inference is also not shown to be implemented as written, and the final formula in Eqs. (5)-(7) mirrors the SPFES aggregation, which makes the circularity concern concrete rather than speculative. The reader's rationale did mention circularity, so we partially agree, but the weakest-assumption statement emphasized detector error and hand-assigned weights rather than the label-construction tautology. The proposed no-GNN baseline check would settle whether the fixed formula alone already achieves the reported accuracy. If it does, the WGNN contribution is unestablished and the rejection stands; if it does not, the authors would still need to release artifacts and independent pain labels before the accuracy claim can be interpreted. The paper does provide a real YOLOv8n mAP result and a comparison across detectors, but that evidence bears on landmark detection, not on the pain-assessment accuracy that is the headline claim.","tokens_in":10816,"tokens_out":3734,"duration_ms":42494,"concrete_test":"On the held-out test split, disclose the image-level pain ground-truth labels and the exact mapping from the nine p_i part scores to pain levels. Then compute a no-GNN baseline: for each image, take the YOLOv8n detected p_i values, apply Eqs. (5)-(7) using the reported SPFES weights w_i and w_j (or, if clustering is integral, use the SPFES-defined clusters), and measure agreement with the same test labels under the same accuracy rule. If this fixed-formula baseline equals or exceeds 91.96%, the WGNN adds no demonstrable value; if the ground-truth pain labels are generated from p_i by that same formula, the comparison is circular and must be redone with independently annotated pain levels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy claim can only support pain assessment if the test labels are independent of the SPFES per-part scores p_i and weights w_i that the WGNN itself aggregates. The paper never establishes this. Section IV.1 says the dataset is labeled according to SPFES parameters, while Section III Eqs. (5)-(7) define the final pain score as a weighted average of p_i with SPFES weights w_i over clusters, then a weighted sum over clusters and normalization. If the image-level ground truth is computed from the same p_i annotations and weights, then applying Eqs. (5)-(7) to YOLOv8n detections will be scored as correct whenever the detector labels are correct, regardless of whether the GNN's clustering or learned message passing contributes anything. No ground-truth labeling protocol, no confusion matrix, no dataset size, and no w_i values are reported, so the reader cannot exclude this tautological evaluation. Consequently, the SOTA comparison in Table II is not yet a comparison of pain-assessment ability; it may be only a comparison of how accurately each pipeline reproduces the annotation formula. This concern is load-bearing because it attacks the meaning of the headline accuracy, not a tuning detail.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Weighted Graph Neural Network (WGNN) for assessing pain levels in sheep from facial landmarks. The pipeline combines a lightweight YOLOv8n detector that localizes nine facial part expressions (ears, eyes, nose, etc.) with a graph-based model that clusters these parts and aggregates per-part pain scores using weights derived from the Sheep Facial Expression Scale (SPFES). The authors introduce a new sheep facial landmark dataset and report that WGNN achieves 92.71% training accuracy and 91.96% testing accuracy, outperforming SVM, EfficientNetB5, CCVT, and CNN. The paper also reports detector mAP values for eight object detectors, with YOLOv8n giving the best mAP of 59.30%.","tokens_in":11070,"tokens_out":5536,"duration_ms":61242,"significance":"If the reported accuracy were valid, the paper would offer a practical application of GNNs for automated animal pain assessment, a topic of increasing interest for precision livestock farming. The proposed dataset could also be a useful resource. However, the current manuscript does not provide sufficient experimental detail or validation to support the central accuracy claim: the ground-truth labeling protocol is unspecified, the graph inference in Eq. (2) is not implemented or described, the dataset size is missing, and the evaluation lacks statistical rigor. These gaps make the claimed 92.71% accuracy not interpretable as a measure of pain-assessment ability. The significance is therefore conditional: the idea is promising, but the evidence presented is inadequate.","major_comments":[{"comment":"The ground-truth labeling protocol for the overall pain level is never described. Section IV.1 states that the dataset is labeled according to SPFES parameters, where each facial part expression is categorized as 0, 1, or 2. In Section III, Eqs. (5)-(7) compute the final pain score as a weighted average of these same p_i values using SPFES weights w_i, followed by a weighted sum over clusters and normalization. If the image-level ground-truth pain label was generated by applying the same SPFES weighted-aggregation rule to the manually annotated part scores, then the reported 92.71% accuracy would only measure how accurately YOLOv8n reproduces the part-level scores; the WGNN's clustering and message passing would be irrelevant. The paper must specify the complete annotation protocol, report the numerical values of w_i and w_j, and demonstrate that the ground-truth pain labels are independent of the aggregation formula in Eqs. (5)-(7). Without this, the headline accuracy cannot be taken as evidence of pain-assessment skill.","section":"§IV.1 and §III, Eqs. (5)-(7)"},{"comment":"The parse-graph inference in Eq. (2) is presented as the core of the proposed method, but the experiments do not describe how g* is computed or whether it is optimized at all. There is no explanation of the probability distribution Pd, the number of clusters O, the edge-construction rule (physical proximity vs. pain-score similarity), or the GNN architecture (number of layers, hidden dimensions, training loss). The experimental section only states that cross-entropy loss is used to cluster facial landmarks. As a result, the method is not reproducible, and it is unclear whether the learned graph component contributes anything beyond the weighted-average aggregation in Eqs. (5)-(7). Please provide the implementation details, or if Eq. (2) was not actually used in the experiments, revise the method description accordingly.","section":"§III, Eq. (2) and §IV"},{"comment":"The evaluation is missing basic statistical information. The dataset size is never reported; only a single 80/20 split is used, with no repeated runs, confidence intervals, or per-class results. Training and testing accuracies are given without a confusion matrix or class-wise breakdown. This is particularly problematic because the detector mAP is only 59.30% (Table I); the paper does not explain how an overall pain-level accuracy of 91.96% is achieved when part detection has low mAP, nor does it define the accuracy metric (exact match of the normalized score? thresholded pain class?). The SOTA comparison in Table II also lacks essential details: were SVM, EfficientNetB5, CCVT, and CNN evaluated on the same detected part-level features, on raw images, or with the same post-processing? Without matched input representations and evaluation protocols, the comparison does not support the claim of superiority.","section":"§IV, Table II and Fig. 5"},{"comment":"The total pain score (TPS) over five days is presented in Fig. 7 without specifying the number of animals, number of images, error bars, or statistical tests. The relationship between the TPS computed by the proposed model and the TPS defined in McLennan et al. [13] is not explained, so the reader cannot tell whether the plotted trend is the model's output or the ground truth. Please clarify what is shown and how it supports the claim of long-term pain monitoring.","section":"§IV.6 and Fig. 7"}],"minor_comments":[{"comment":"There are several typographical errors and inconsistencies, e.g., 'EfficeintNet' should be 'EfficientNet', 'UA Vs' should be 'UAVs', and 'Rethinking transformation' appears to be augmentation but is not clearly defined. Please proofread carefully.","section":"Throughout"},{"comment":"The dataset description does not state the total number of images, the number of sheep, or the class distribution for the three pain categories. This information is necessary for interpreting the reported accuracies and for anyone wishing to use the dataset.","section":"§IV.1"},{"comment":"The feature vector ψ_i^(0) in Eq. (1) is described as [p_i, FPT_i], but FPT (facial part expression type) is not defined clearly, and it is unclear how this feature is encoded (one-hot? scalar?). Please clarify.","section":"§III, Eq. (1)"},{"comment":"Table I reports precision and recall values, but the definitions of precision and recall for multi-class landmark detection are not specified. Given the low mAP values, it would be helpful to show per-class results and the confidence threshold used.","section":"§IV.3 and Table I"},{"comment":"The ablation study in Fig. 5 reports 'accuracy' for four YOLO models combined with WGNN, but the caption and text do not define whether this is training accuracy, validation accuracy, or the final pain-level accuracy. Please provide a consistent metric definition.","section":"§IV.4"}],"recommendation":"major_revision","confidential_remarks":"The reader's report and the stress-test concern about circularity are well grounded: the paper never shows that the ground-truth pain labels are independent of the SPFES aggregation rule used in Eqs. (5)-(7). I am recommending major revision rather than rejection because the authors could address the concern by reporting the annotation protocol and running a controlled ablation, but if it turns out that the labels were generated by the same weighted formula, the central claim collapses. The missing implementation of Eq. (2) and the lack of dataset statistics are also severe and require substantial additional experiments, not just text edits."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the sheep facial landmark dataset with SPFES-aligned part-level annotations is a real contribution, and the idea of aggregating per-part detector scores with a graph model is sensible for this application. Second, the central accuracy claim is currently uninterpretable. The paper never describes how image-level ground-truth pain labels were created, and Eqs. (5)–(7) define total pain as a weighted average of the same per-part scores that the detector predicts, using SPFES weights. If the labels were derived from those same manual part scores and weights, then the WGNN's 92.71% is mostly measuring how well YOLOv8n reproduces the manual annotations; the GNN's clustering and message passing could be contributing nothing. The paper does not rule this out, and that is a load-bearing gap, not a presentation nit.\n\nWhat the paper does well: the problem is real and the dataset fills a gap. The comparison of eight detectors and the ablation with four YOLO variants is useful work. The limitations section is honest about detection-dependent failures and the need for more diverse data. The graph formulation, while standard message passing, is applied to a domain where it has not been tried.\n\nSoft spots beyond the circularity issue: Eq. (2) describes an optimal parse graph inference, but the experiments never show it being implemented; the weights w_i and w_j are never reported; there is no dataset size, no error bars, no confidence intervals, and no confusion matrix. The detector mAP of 59.3% makes the 91.96% test accuracy look suspicious without knowing how many images were tested and how robust the final score is to individual landmark errors. The SOTA comparison in Table II does not state what inputs the baselines received, so the comparison is not fully interpretable.\n\nWho is this for: researchers in precision livestock farming and animal-computer interaction, particularly those working on pain assessment from facial landmarks. The dataset, once released, would be the main value. The method as described needs rework before the accuracy claim can be taken seriously.\n\nMy recommendation: send it to peer review, because the dataset and application deserve scrutiny, but expect major revision. If the authors release the data and annotation protocol and show the ground-truth labels are independent of the aggregation formula, the accuracy becomes meaningful. As it stands, I would not cite the 92.71% figure.","headline":"The new sheep facial landmark dataset is a genuine asset, but the headline 92.71% accuracy is not interpretable because the paper never shows that the ground-truth labels are independent of the SPFES aggregation formula the WGNN reproduces.","tokens_in":11596,"tokens_out":2578,"would_cite":false,"duration_ms":29611,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a weighted graph neural network over nine sheep facial landmarks, fed by a lightweight YOLOv8n detector, can assess pain level with 92.71% training and 91.96% testing accuracy.","keywords":["sheep facial pain assessment","weighted graph neural network","facial landmark detection","Sheep Facial Expression Scale","YOLOv8n","pain level prediction","animal welfare monitoring","object detection"],"falsifier":"Take a held-out collection of sheep face images from several farms with different breeds, lighting, and camera angles, have veterinarians assign SPFES pain scores, and compare WGNN+YOLOv8n predictions against those ground-truth scores. A large accuracy drop, or a finding that changing the hand-assigned SPFES weights in Eq. (5) changes the reported accuracy by more than a few points, would show the result depends on the particular split and weight choices.","tokens_in":10574,"feed_emoji":"🐑","tokens_out":3760,"duration_ms":35811,"temperature":0.7,"pith_summary":"This paper tries to establish that sheep pain can be read automatically from facial landmarks by linking nine part-level expressions (ears, eyes, nose, cheeks, lips and jaw) with a weighted graph neural network that clusters related parts and aggregates their pain scores. The proposed model is paired with a lightweight YOLOv8n detector, and the authors report 92.71% training and 91.96% testing accuracy on a new sheep facial landmarks dataset built on the Sheep Facial Expression Scale (SPFES). If these numbers hold beyond the paper's single 80/20 split, the approach would give farmers and veterinarians a cheap, non-invasive way to monitor pain continuously in natural settings.","feed_headline":"Weighted graph network reads sheep pain from faces at 92%","feed_subtitle":"It clusters ear, eye, nose, lip and cheek landmarks into a pain score, beating four baselines on a new sheep dataset.","key_machinery":"The central object is a weighted graph neural network (WGNN) whose nodes are the nine SPFES facial part expressions and whose edges connect parts that are anatomically close or share similar pain scores. The carried mechanism is message passing ($M_i^{(k)} = \\sum_{j \\in N(i)} \\psi_j^{(k-1)}$; $\\psi_i^{(k)} = F(\\sigma)(W^{(k)} M_i^{(k)} + b^{(k)})$) followed by weighted-average clustering within each cluster ($S_j = \\sum_{i \\in C_j} w_i p_i / \\sum_{i \\in C_j} w_i$) and a weighted sum over clusters ($T_p = \\sum_{j=1}^o w_j S_j$), normalized to a 0-100% pain score. The YOLOv8n detector supplies the per-part pain scores $p_i$ that initialize the node features, so the graph's accuracy is bounded by detector quality.","core_discovery":"The central claim is that pain-level assessment improves when the face is treated not as one whole image but as a graph of nine facial part expressions, where nodes carry per-part pain scores from a detector and edges encode anatomical proximity and similarity of pain. The WGNN infers the optimal parse graph, passes messages between neighboring nodes, clusters parts into pain-related groups, and computes a weighted total pain score normalized to 0-100%. Compared with SVM, EfficientNetB5, CCVT, and CNN on the new sheep facial landmarks dataset, the WGNN with YOLOv8n reaches the highest accuracy (92.71% train, 91.96% test).","pith_inferences":["If the same weighted-cluster aggregation were applied to other livestock species with standardized facial expression scales, the approach might transfer without retraining the graph, provided per-part pain scores are available.","Because the weighted average in Eq. (5) is transparent, the model could be extended to output per-facial-part pain contributions to a veterinarian, not just a single score.","The single 80/20 split and five-day monitoring period leave open whether the 92% accuracy generalizes across breeds, farms, and lighting; a cross-farm evaluation with multiple annotators would be the natural next test.","A weaker detector than YOLOv8n would likely lower the ceiling, since the paper itself states that detection performance determines the WGNN model's performance."],"forward_implications":["Farmers and veterinarians could obtain an automated, continuous pain score from ordinary face images or UAV footage without handling the sheep.","The new sheep facial landmarks dataset, with nine facial expressions across five facial parts, provides a reusable benchmark for later detection and pain-assessment work.","The lightweight YOLOv8n detector (6.3M parameters, 1.9 ms inference per image) makes on-board or embedded deployment feasible.","Clustering facial parts by shared pain scores yields an interpretable total pain score in the 0-100% range, rather than just a categorical label.","The WGNN framework exceeds the reported accuracy of SVM, EfficientNetB5, CCVT, and CNN on this dataset."],"supporting_citations":[{"why":"Defines the Sheep Facial Expression Scale (SPFES) with five facial parts and pain levels, providing the standard that the new dataset's labels and weights follow.","marker":"[13]"},{"why":"Earlier sheep facial action unit pain estimation work that the authors extend, and the source of the SVM baseline in the comparison table.","marker":"[12]"},{"why":"Provides the sheep facial expression images from which the new facial landmarks dataset is built.","marker":"[16]"},{"why":"Introduces graph parsing neural networks with probability-based parse graph inference, which the WGNN adapts for optimal graph structure.","marker":"[18]"},{"why":"Serves as the CNN baseline for sheep face detection and disease prevention used in the comparison.","marker":"[3]"},{"why":"Serves as the EfficientNet baseline for sheep breed identification in low-resolution images.","marker":"[8]"},{"why":"Serves as the CCVT baseline combining convolutional and vision transformer structures for sheep face recognition.","marker":"[11]"}],"fun_headline_variants":["Graph neural network scores sheep pain from facial landmarks","Sheep pain assessed via weighted graph on facial landmarks","WGNN achieves 92% accuracy in sheep facial pain detection","Facial landmark graph predicts sheep pain levels effectively"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the detector's per-face-part pain scores are accurate enough to serve as graph node features and that the hand-assigned SPFES weights correctly reflect each part's contribution to pain; the authors acknowledge that detection errors propagate directly into the WGNN's predictions.","fun_headline_variants_meta":{"raw":{"variants":["Graph neural network scores sheep pain from facial landmarks","Sheep pain assessed via weighted graph on facial landmarks","WGNN achieves 92% accuracy in sheep facial pain detection","Facial landmark graph predicts sheep pain levels effectively"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000514,"raw_usage":{"total_tokens":2472,"prompt_tokens":896,"completion_tokens":1576,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":1513}},"tokens_in":512,"tokens_out":1576,"duration_ms":11106,"temperature":1.0,"reasoning_tokens":1513,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:40:32.243544+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out collection of sheep face images from several farms with different breeds, lighting, and camera angles, have veterinarians assign SPFES pain scores, and compare WGNN+YOLOv8n predictions against those ground-truth scores. A large accuracy drop, or a finding that changing the hand-assigned SPFES weights in Eq. (5) changes the reported accuracy by more than a few points, would show the result depends on the particular split and weight choices.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Sheep Facial Expression Scale (SPFES) with five facial parts and pain levels, providing the standard that the new dataset's labels and weights follow."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier sheep facial action unit pain estimation work that the authors extend, and the source of the SVM baseline in the comparison table."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the sheep facial expression images from which the new facial landmarks dataset is built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces graph parsing neural networks with probability-based parse graph inference, which the WGNN adapts for optimal graph structure."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as the CNN baseline for sheep face detection and disease prevention used in the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as the EfficientNet baseline for sheep breed identification in low-resolution images."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as the CCVT baseline combining convolutional and vision transformer structures for sheep face recognition."}],"review_version":1}