REVIEW 3 major objections 5 minor 1 cited by
Multi-Channel Neural Network for Assessing Neonatal Pain from Videos
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A landmark-free multi-channel CNN that fuses facial expression and body movement with LSTM temporal modeling can classify neonatal pain from NICU videos, reaching 92.48% video-level accuracy and 0.90 AUC.
desk verdict A credible first face+body CNN/LSTM system for neonatal pain, but its central component-wise claims are underdetermined by confounded comparisons; still deserves a serious referee. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is a two-branch convolutional network: one branch is a VGG16 pretrained on face images, the other a VGG16 pretrained on generic images, and the downsampled features from each branch are concatenated into a shared representation before being merged with the individual face and body vectors. Two YOLO detectors, one trained on WIDER FACE for faces and one trained on COCO for bodies, supply the input crops. Two LSTM layers with 16 units each model temporal dynamics across 16-frame sequences, and the final dense layer performs binary pain/no-pain classification at both frame level and video level.
What would settle it
Take the same dataset and replace YOLO-detected crops with manually annotated face and body crops; if classification accuracy changes materially, detector errors are the active variable. Alternatively, compute per-frame detection precision and recall on the NICU videos and compare those numbers with the reported end-to-end accuracy.
Extended reading notes
Core claim
The central claim is that a landmark-free, multi-channel CNN that fuses face and body features and then passes the combined representation through two LSTM layers classifies neonatal pain from videos more accurately than existing methods on the same NICU dataset. The authors state this is the first exploration of neonatal pain assessment using a multi-channel CNN network, and that the shared face-plus-body representation with temporal modeling yields 92.48% video-level accuracy and 0.90 AUC, beating geometric, LBP-TOP, motion, and single-channel CNN baselines. They further argue that the approach removes the dependence on fragile facial landmark detectors and is therefore better suited to NICU conditions with occlusion and illumination variation.
Load-bearing premise
The pipeline depends on the two pre-trained object detectors finding the neonate's face and body correctly in every frame; the paper does not report detection accuracy on NICU footage, and face detection there is acknowledged to be hard under occlusion and lighting changes.
Editorial extensions
If this is right
- Continuous, automated neonatal pain monitoring in the NICU becomes feasible without a human observer scoring pain scales.
- Facial landmark annotation, a bottleneck for prior methods, is no longer required for pain feature extraction.
- Combining face and body provides resilience when one channel is occluded or partially missing, a stated motivation for the shared representation.
- The temporal LSTM component captures the unfolding of pain over time, improving video-level accuracy over frame-level accuracy.
- The framework can be extended to additional pain modalities such as crying sound and vital signs, as the authors propose for future work.
Reading between the lines
- If object-detector failures are not random but correlate with pain behavior, for instance vigorous movement during painful procedures, the reported accuracy could either over- or understate true performance; measuring YOLO per-frame failure rates on neonatal data is the natural next experiment.
- A face-only variant of the proposed framework could be evaluated on public static face datasets to separate the contribution of the face channel from the body channel, since the joint architecture prevents that attribution.
- The same two-channel design could transfer to postoperative pain monitoring or to other pre-verbal patients, using the face branch alone where the body is not visible.
- An ablation comparing face-only, body-only, and shared branches would identify which channel carries most of the pain signal, a question the joint results leave open.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a multi-channel deep learning framework for automated neonatal pain assessment from videos. The framework uses two pre-trained VGG16 branches to extract features from neonate face and body regions detected by YOLO, concatenates face, body, and shared representations, and adds two LSTM layers to model temporal dynamics. The authors report frame-level classification accuracy of 91.41% with AUC 0.89 and video-level accuracy of 92.48% with AUC 0.90 on a private NICU dataset of 31 neonates, using leave-one-subject-out cross-validation. They compare these results with earlier landmark-based face-only CNN methods and handcrafted feature methods on the same dataset, and claim the first multi-channel CNN approach for neonatal pain assessment.
Significance. If the reported results are reproducible and the design choices are properly isolated, this work would be a meaningful step toward continuous, automated neonatal pain monitoring in the NICU. The paper's strengths include the use of leave-one-subject-out evaluation, comparison with prior results on the same private dataset, a landmark-free pipeline, and a substantial reduction in trainable parameters through feature freezing (from 29,474,818 to 45,442). The clinical motivation is well argued, and the integration of face and body channels together with LSTM temporal modeling is a sensible direction. However, the empirical evidence, as presented, does not yet establish that the specific novel components (multi-channel fusion and LSTM) are responsible for the observed improvements, and the private dataset with only 31 subjects limits the generality of the conclusions.
major comments (3)
- [Tables II and III; Section III] The comparisons in Tables II and III do not isolate the contributions of the proposed multi-channel fusion or the LSTM. The proposed system differs from the baselines simultaneously in detection preprocessing (YOLO versus facial landmarks), feature extractor (fine-tuned VGG16 versus handcrafted features or landmark-based CNNs), input channels (face+body versus face-only or body-only), and temporal modeling (LSTM versus frame-level or KNN). No ablation is reported for variants such as face-only, body-only, no-LSTM, or single-channel with the same detection and feature pipeline. Without these controlled comparisons, the accuracy differences (91.41% vs. 89.47% frame-level; 92.48% vs. 88.87% video-level) cannot be attributed to the claimed innovations. This is load-bearing because the contributions explicitly highlight multi-channel fusion and temporal modeling as the central novelties.
- [Section IV.C; Tables II and III] The leave-one-subject-out protocol is appropriate, but the paper reports only point estimates of accuracy and AUC with no confidence intervals, significance tests, or per-subject breakdowns. With 31 subjects, a small number of subjects can drive the reported differences; note also that the frame-level AUC of the proposed method (0.89) is identical to that of the VGG16 baseline in Table II, so the only advantage at frame level is a 1.94-point accuracy gain of unknown statistical reliability. The authors should report variance across folds, paired subject-level results, or a significance test (e.g., Wilcoxon signed-rank) to support the claim of superiority.
- [Section II.A; Section IV.B] The face and body detection stage is a critical part of the pipeline, but its accuracy on neonatal NICU frames is never evaluated. The authors themselves acknowledge that face detection in the NICU is challenging due to illumination variation and partial occlusion, and the detectors are only described as pre-trained on WIDER FACE and COCO. No detection rate, bounding-box quality metric, or failure analysis is reported. If the detectors fail on a substantial fraction of frames, the downstream CNN and LSTM inputs are corrupted, which directly affects the generalizability of the reported accuracy to new NICU settings. At minimum, the authors should report detection performance on their dataset and analyze how detection failures affect classification.
minor comments (5)
- [Section III.A] The heading 'Mutli-channel Shared Network' contains a typo; it should be 'Multi-channel Shared Network'.
- [Section IV.B] The text '244 x 224 x 3, RGB images' contains a typo; the input size should be 224 x 224 x 3.
- [Section III.B] The statement that 'sigmoid function works better with LSTM than the cross-entropy error' is imprecise: sigmoid is an activation function, not a loss function. The authors should clarify that they use binary cross-entropy loss with a sigmoid output activation.
- [Section III.B] The frame-level classification uses overlapping sequences (S0 = {f1,...,f16}, S1 = {f2,...,f17}), which means adjacent training samples share 15 frames. This temporal overlap can introduce autocorrelation and optimistic estimates of frame-level performance; the paper should discuss this or evaluate with non-overlapping sequences.
- [Section IV.A] The dataset description does not specify the class distribution (number of pain vs. no-pain periods per subject) or whether the two NICU caregivers' labels were reconciled before training. Reporting these details would improve reproducibility.
Circularity Check
No significant circularity: the paper is an empirical performance comparison, and its self-cited baselines are independent prior results rather than inputs to the proposed model.
full rationale
The paper makes no derivation from first principles; its central claim is an empirical accuracy comparison on a proprietary NICU dataset. The only self-citations that appear in the comparison tables ([28] and [29]) are prior independent works by overlapping authors that were evaluated on the same dataset before the proposed framework was introduced; they are baselines, not fitted inputs or components of the proposed model. Hyperparameters such as the 16-frame LSTM sequence length are described as determined empirically, not disguised as predictions. The pretrained YOLO and VGG16 components are standard external models with stated training datasets (WIDER FACE, COCO, VGGFace2, ImageNet), so they constitute independent support rather than self-referential justification. No equation or defined quantity reduces to another by construction, and no fitted parameter is renamed as a prediction. Therefore the circularity burden is low and the appropriate score is 0.
Assumptions & free parameters
free parameters (4)
- sequence_length =
16 frames
- LSTM_units =
16
- learning_rate =
0.0001
- batch_size =
16
assumptions (3)
- domain assumption NIPS pain scale labels provided by two caregivers are valid ground truth for neonatal pain.
- domain assumption YOLO detectors pretrained on WIDER FACE and COCO generalize to neonatal faces and bodies in NICU videos.
- domain assumption Facial expression and body movement together provide sufficient discriminative information for pain classification in video.
Cite this review
Pith. "Pith review of Multi-Channel Neural Network for Assessing Neonatal Pain from Videos." pith.science (2026). https://pith.science/paper/VU46NWR7
@misc{pith2026190809254,
author = {Pith},
title = {Pith review of: Multi-Channel Neural Network for Assessing Neonatal Pain from Videos},
year = {2026},
howpublished = {\url{https://pith.science/paper/VU46NWR7}},
note = {Machine review of arXiv:1908.09254}
}
read the original abstract
Neonates do not have the ability to either articulate pain or communicate it non-verbally by pointing. The current clinical standard for assessing neonatal pain is intermittent and highly subjective. This discontinuity and subjectivity can lead to inconsistent assessment, and therefore, inadequate treatment. In this paper, we propose a multi-channel deep learning framework for assessing neonatal pain from videos. The proposed framework integrates information from two pain indicators or channels, namely facial expression and body movement, using convolutional neural network (CNN). It also integrates temporal information using a recurrent neural network (LSTM). The experimental results prove the efficiency and superiority of the proposed temporal and multi-channel framework as compared to existing similar methods.
Figures
Forward citations
Cited by 1 Pith paper
-
Harnessing the Power of Deep Learning Methods in Healthcare: Neonatal Pain Assessment from Crying Sound
On a 31-neonate NICU dataset, a lightweight CNN (N-CNN) and VGG16 both reached 96.77% accuracy and 0.94 AUC for classifying pain versus no-pain from cry spectrograms.
Reference graph
Works this paper leans on
-
[28]
Convolutional neural networks for neonatal pain assessment,
G. Zamzmi, R. Paul, M. S. Salekin, D. Goldgof, R. Kasturi, T. Ho, and Y . Sun, “Convolutional neural networks for neonatal pain assessment,” IEEE Transactions on Biometrics, Behavior, and Identity Science , 2019, early access DOI:10.1109/TBIOM.2019.2918619
-
[29]
A comprehensive and context-sensitive neonatal pain as- sessment using computer vision,
G. Zamzmi, C.-Y . Pai, D. Goldgof, R. Kasturi, T. Ashmeade, and Y . Sun, “A comprehensive and context-sensitive neonatal pain as- sessment using computer vision,” IEEE Transactions on Affective Computing, 2019, early access DOI:10.1109/TAFFC.2019.2926710
arXiv 2019
-
[1]
M. Puchalski and P. Hummel, “The reality of neonatal pain.” Advances in neonatal care: official journal of the National Association of Neonatal Nurses, vol. 2, no. 5, pp. 233–44, 2002
work page 2002
-
[2]
Current controversies regarding pain assessment in neonates,
M. Ranger, C. C. Johnston, and K. Anand, “Current controversies regarding pain assessment in neonates,” in Seminars in perinatology , vol. 31, no. 5. Elsevier, 2007, pp. 283–288
work page 2007
-
[3]
Validation of the pain assessment in neonates (pain) scale with the neonatal infant pain scale (nips),
D. Hudson-Barr, B. Capper-Michel, S. Lambert, T. M. Palermo, K. Morbeto, and S. Lombardo, “Validation of the pain assessment in neonates (pain) scale with the neonatal infant pain scale (nips),” Neonatal Network, vol. 21, no. 6, pp. 15–22, 2002
2002
-
[4]
P. Hummel, M. Puchalski, S. Creech, and M. Weiss, “Clinical relia- bility and validity of the n-pass: neonatal pain, agitation and sedation scale with prolonged pain,” Journal of Perinatology , vol. 28, no. 1, p. 55, 2008
work page 2008
-
[5]
Pain expression in neonates: facial action and cry,
R. V . Grunau and K. D. Craig, “Pain expression in neonates: facial action and cry,” Pain, vol. 28, no. 3, pp. 395–410, 1987
work page 1987
-
[6]
J. W. Peters, H. M. Koot, R. E. Grunau, J. de Boer, M. J. van Druenen, D. Tibboel, and H. J. Duivenvoorden, “Neonatal facial coding system for assessing postoperative pain in infants: item reduction is valid and feasible,” The Clinical journal of pain , vol. 19, no. 6, pp. 353–363, 2003
work page 2003
Show all 29 references
-
[7]
Infants’ pain recognition based on facial expression: Dynamic hybrid descriptions,
R. Zhi, G. Zamzmi, D. Goldgof, T. Ashmeade, T. Li, and Y . Sun, “Infants’ pain recognition based on facial expression: Dynamic hybrid descriptions,” IEICE TRANSACTIONS on Information and Systems , vol. 101, no. 7, pp. 1860–1869, 2018
2018
-
[8]
Body movements: an important additional factor in discriminating pain from stress in preterm infants,
L. Holsti, R. E. Grunau, T. F. Oberlander, M. F. Whitfield, and J. Weinberg, “Body movements: an important additional factor in discriminating pain from stress in preterm infants,” The Clinical journal of pain , vol. 21, no. 6, p. 491, 2005
2005
-
[9]
Pain assessment in human fetus and infants,
C. V . Bellieni, “Pain assessment in human fetus and infants,” The AAPS journal, vol. 14, no. 3, pp. 456–461, 2012
2012
-
[10]
A review of automated pain assessment in infants: Features, classifi- cation tasks, and databases,
G. Zamzmi, R. Kasturi, D. Goldgof, R. Zhi, T. Ashmeade, and Y . Sun, “A review of automated pain assessment in infants: Features, classifi- cation tasks, and databases,” IEEE reviews in biomedical engineering , vol. 11, pp. 77–96, 2018
2018
-
[11]
Introduction to neonatal facial pain detection using common and advanced face classification techniques,
S. Brahnam, L. Nanni, and R. Sexton, “Introduction to neonatal facial pain detection using common and advanced face classification techniques,” in Advanced Computational Intelligence Paradigms in Healthcare–1. Springer, 2007, pp. 225–253
2007
-
[12]
A local approach based on a local binary patterns variant texture descriptor for classifying pain states,
L. Nanni, S. Brahnam, and A. Lumini, “A local approach based on a local binary patterns variant texture descriptor for classifying pain states,” Expert Systems with Applications , vol. 37, no. 12, pp. 7888– 7894, 2010
2010
-
[13]
Relevance vector machine learning for neonate pain intensity assessment using digital imaging,
B. Gholami, W. M. Haddad, and A. R. Tannenbaum, “Relevance vector machine learning for neonate pain intensity assessment using digital imaging,” IEEE Transactions on biomedical engineering , vol. 57, no. 6, pp. 1457–1466, 2010
2010
-
[14]
An approach for automated multimodal analysis of infants’ pain,
G. Zamzmi, C.-Y . Pai, D. Goldgof, R. Kasturi, T. Ashmeade, and Y . Sun, “An approach for automated multimodal analysis of infants’ pain,” in 2016 23rd International Conference on Pattern Recognition (ICPR). IEEE, 2016, pp. 4148–4153
2016
-
[15]
Deep pain: Exploiting long short- term memory networks for facial expression classification,
P. Rodriguez, G. Cucurull, J. Gonz `alez, J. M. Gonfaus, K. Nasrollahi, T. B. Moeslund, and F. X. Roca, “Deep pain: Exploiting long short- term memory networks for facial expression classification,” IEEE transactions on cybernetics , no. 99, pp. 1–11, 2017
2017
-
[16]
Spatio-temporal pain recognition in cnn-based super-resolved facial images,
M. Bellantonio, M. A. Haque, P. Rodriguez, K. Nasrollahi, T. Telve, S. Escalera, J. Gonzalez, T. B. Moeslund, P. Rasti, and G. Anbarjafari, “Spatio-temporal pain recognition in cnn-based super-resolved facial images,” in Video Analytics. Face and Facial Expression Recognition ...
2016
-
[17]
Neonatal facial pain assessment combining hand-crafted and deep features,
L. Celona and L. Manoni, “Neonatal facial pain assessment combining hand-crafted and deep features,” in International Conference on Image Analysis and Processing . Springer, 2017, pp. 197–204
2017
-
[18]
Yolov3: An incremental improvement,
J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” arXiv preprint arXiv:1804.02767 , 2018
2018 arXiv
-
[19]
Wider face: A face detection benchmark,
S. Yang, P. Luo, C.-C. Loy, and X. Tang, “Wider face: A face detection benchmark,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 5525–5533
2016
-
[20]
Microsoft coco: Common objects in context,
T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European conference on computer vision. Springer, 2014, pp. 740–755
2014
-
[21]
Very deep convolutional networks for large-scale image recognition,
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 , 2014
2014 arXiv
-
[22]
Imagenet classification with deep convolutional neural networks,
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems , 2012, pp. 1097–1105
2012
-
[23]
Vg- gface2: A dataset for recognising faces across pose and age,
Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman, “Vg- gface2: A dataset for recognising faces across pose and age,” in 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) . IEEE, 2018, pp. 67–74
2018
-
[24]
Deep face recognition
O. M. Parkhi, A. Vedaldi, A. Zisserman et al., “Deep face recognition.” in bmvc, vol. 1, no. 3, 2015, p. 6
2015
-
[25]
Imagenet: A large-scale hierarchical image database,
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition . Ieee, 2009, pp. 248–255
2009
-
[26]
Long short-term memory,
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997
1997
-
[27]
Adam: A method for stochastic optimiza- tion,
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimiza- tion,” arXiv preprint arXiv:1412.6980 , 2014
2014 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.