Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Multi-Channel Neural Network for Assessing Neonatal Pain from Videos

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read A landmark-free multi-channel CNN that fuses facial expression and body movement with LSTM temporal modeling can classify neonatal pain from NICU videos, reaching 92.48% video-level accuracy and 0.90 AUC.

desk verdict A credible first face+body CNN/LSTM system for neonatal pain, but its central component-wise claims are underdetermined by confounded comparisons; still deserves a serious referee. read the letter →

arxiv 1908.09254 v1 pith:VU46NWR7 submitted 2019-08-25 cs.CV

classification cs.CV
keywords neonatalpainassessmentmulti-channelCNNLSTMtemporalmodelingfacialexpressionbodymovementNICUvideoanalysislandmark-freedeeplearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that neonatal pain can be assessed automatically from video without facial landmark tracking, by combining facial expression and body movement in a single multi-channel neural network with temporal memory. The authors report that on a NICU video dataset, their frame-level model reaches 91.41% accuracy and their video-level model reaches 92.48% accuracy with 0.90 AUC, outperforming landmark-dependent and handcrafted baselines evaluated on the same data. If the result holds, neonatal pain monitoring could become continuous and objective rather than intermittent and subjective, addressing a real clinical gap.

What carries the argument

The mechanism is a two-branch convolutional network: one branch is a VGG16 pretrained on face images, the other a VGG16 pretrained on generic images, and the downsampled features from each branch are concatenated into a shared representation before being merged with the individual face and body vectors. Two YOLO detectors, one trained on WIDER FACE for faces and one trained on COCO for bodies, supply the input crops. Two LSTM layers with 16 units each model temporal dynamics across 16-frame sequences, and the final dense layer performs binary pain/no-pain classification at both frame level and video level.

What would settle it

Take the same dataset and replace YOLO-detected crops with manually annotated face and body crops; if classification accuracy changes materially, detector errors are the active variable. Alternatively, compute per-frame detection precision and recall on the NICU videos and compare those numbers with the reported end-to-end accuracy.

Watch

Extended reading notes

Core claim

The central claim is that a landmark-free, multi-channel CNN that fuses face and body features and then passes the combined representation through two LSTM layers classifies neonatal pain from videos more accurately than existing methods on the same NICU dataset. The authors state this is the first exploration of neonatal pain assessment using a multi-channel CNN network, and that the shared face-plus-body representation with temporal modeling yields 92.48% video-level accuracy and 0.90 AUC, beating geometric, LBP-TOP, motion, and single-channel CNN baselines. They further argue that the approach removes the dependence on fragile facial landmark detectors and is therefore better suited to NICU conditions with occlusion and illumination variation.

Load-bearing premise

The pipeline depends on the two pre-trained object detectors finding the neonate's face and body correctly in every frame; the paper does not report detection accuracy on NICU footage, and face detection there is acknowledged to be hard under occlusion and lighting changes.

Editorial extensions

If this is right

  • Continuous, automated neonatal pain monitoring in the NICU becomes feasible without a human observer scoring pain scales.
  • Facial landmark annotation, a bottleneck for prior methods, is no longer required for pain feature extraction.
  • Combining face and body provides resilience when one channel is occluded or partially missing, a stated motivation for the shared representation.
  • The temporal LSTM component captures the unfolding of pain over time, improving video-level accuracy over frame-level accuracy.
  • The framework can be extended to additional pain modalities such as crying sound and vital signs, as the authors propose for future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If object-detector failures are not random but correlate with pain behavior, for instance vigorous movement during painful procedures, the reported accuracy could either over- or understate true performance; measuring YOLO per-frame failure rates on neonatal data is the natural next experiment.
  • A face-only variant of the proposed framework could be evaluated on public static face datasets to separate the contribution of the face channel from the body channel, since the joint architecture prevents that attribution.
  • The same two-channel design could transfer to postoperative pain monitoring or to other pre-verbal patients, using the face branch alone where the body is not visible.
  • An ablation comparing face-only, body-only, and shared branches would identify which channel carries most of the pain signal, a question the joint results leave open.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a multi-channel deep learning framework for automated neonatal pain assessment from videos. The framework uses two pre-trained VGG16 branches to extract features from neonate face and body regions detected by YOLO, concatenates face, body, and shared representations, and adds two LSTM layers to model temporal dynamics. The authors report frame-level classification accuracy of 91.41% with AUC 0.89 and video-level accuracy of 92.48% with AUC 0.90 on a private NICU dataset of 31 neonates, using leave-one-subject-out cross-validation. They compare these results with earlier landmark-based face-only CNN methods and handcrafted feature methods on the same dataset, and claim the first multi-channel CNN approach for neonatal pain assessment.

Significance. If the reported results are reproducible and the design choices are properly isolated, this work would be a meaningful step toward continuous, automated neonatal pain monitoring in the NICU. The paper's strengths include the use of leave-one-subject-out evaluation, comparison with prior results on the same private dataset, a landmark-free pipeline, and a substantial reduction in trainable parameters through feature freezing (from 29,474,818 to 45,442). The clinical motivation is well argued, and the integration of face and body channels together with LSTM temporal modeling is a sensible direction. However, the empirical evidence, as presented, does not yet establish that the specific novel components (multi-channel fusion and LSTM) are responsible for the observed improvements, and the private dataset with only 31 subjects limits the generality of the conclusions.

major comments (3)
  1. [Tables II and III; Section III] The comparisons in Tables II and III do not isolate the contributions of the proposed multi-channel fusion or the LSTM. The proposed system differs from the baselines simultaneously in detection preprocessing (YOLO versus facial landmarks), feature extractor (fine-tuned VGG16 versus handcrafted features or landmark-based CNNs), input channels (face+body versus face-only or body-only), and temporal modeling (LSTM versus frame-level or KNN). No ablation is reported for variants such as face-only, body-only, no-LSTM, or single-channel with the same detection and feature pipeline. Without these controlled comparisons, the accuracy differences (91.41% vs. 89.47% frame-level; 92.48% vs. 88.87% video-level) cannot be attributed to the claimed innovations. This is load-bearing because the contributions explicitly highlight multi-channel fusion and temporal modeling as the central novelties.
  2. [Section IV.C; Tables II and III] The leave-one-subject-out protocol is appropriate, but the paper reports only point estimates of accuracy and AUC with no confidence intervals, significance tests, or per-subject breakdowns. With 31 subjects, a small number of subjects can drive the reported differences; note also that the frame-level AUC of the proposed method (0.89) is identical to that of the VGG16 baseline in Table II, so the only advantage at frame level is a 1.94-point accuracy gain of unknown statistical reliability. The authors should report variance across folds, paired subject-level results, or a significance test (e.g., Wilcoxon signed-rank) to support the claim of superiority.
  3. [Section II.A; Section IV.B] The face and body detection stage is a critical part of the pipeline, but its accuracy on neonatal NICU frames is never evaluated. The authors themselves acknowledge that face detection in the NICU is challenging due to illumination variation and partial occlusion, and the detectors are only described as pre-trained on WIDER FACE and COCO. No detection rate, bounding-box quality metric, or failure analysis is reported. If the detectors fail on a substantial fraction of frames, the downstream CNN and LSTM inputs are corrupted, which directly affects the generalizability of the reported accuracy to new NICU settings. At minimum, the authors should report detection performance on their dataset and analyze how detection failures affect classification.
minor comments (5)
  1. [Section III.A] The heading 'Mutli-channel Shared Network' contains a typo; it should be 'Multi-channel Shared Network'.
  2. [Section IV.B] The text '244 x 224 x 3, RGB images' contains a typo; the input size should be 224 x 224 x 3.
  3. [Section III.B] The statement that 'sigmoid function works better with LSTM than the cross-entropy error' is imprecise: sigmoid is an activation function, not a loss function. The authors should clarify that they use binary cross-entropy loss with a sigmoid output activation.
  4. [Section III.B] The frame-level classification uses overlapping sequences (S0 = {f1,...,f16}, S1 = {f2,...,f17}), which means adjacent training samples share 15 frames. This temporal overlap can introduce autocorrelation and optimistic estimates of frame-level performance; the paper should discuss this or evaluate with non-overlapping sequences.
  5. [Section IV.A] The dataset description does not specify the class distribution (number of pain vs. no-pain periods per subject) or whether the two NICU caregivers' labels were reconciled before training. Reporting these details would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical performance comparison, and its self-cited baselines are independent prior results rather than inputs to the proposed model.

full rationale

The paper makes no derivation from first principles; its central claim is an empirical accuracy comparison on a proprietary NICU dataset. The only self-citations that appear in the comparison tables ([28] and [29]) are prior independent works by overlapping authors that were evaluated on the same dataset before the proposed framework was introduced; they are baselines, not fitted inputs or components of the proposed model. Hyperparameters such as the 16-frame LSTM sequence length are described as determined empirically, not disguised as predictions. The pretrained YOLO and VGG16 components are standard external models with stated training datasets (WIDER FACE, COCO, VGGFace2, ImageNet), so they constitute independent support rather than self-referential justification. No equation or defined quantity reduces to another by construction, and no fitted parameter is renamed as a prediction. Therefore the circularity burden is low and the appropriate score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim is empirical and depends on training hyperparameters, clinical label validity, and transferability of pretrained detectors. These are the main assumptions that are not independently justified beyond the paper's own results.

free parameters (4)
  • sequence_length = 16 frames
    Section III.B states the window size 'was determined empirically'; it controls temporal context in the LSTM and affects accuracy.
  • LSTM_units = 16
    Table I specifies 16 LSTM units and two dense layers of 16 units; no ablation or justification is given.
  • learning_rate = 0.0001
    Used for Adam optimizer in fine-tuning and LSTM training; a standard choice not justified by experiments.
  • batch_size = 16
    Used for CNN fine-tuning and LSTM training; no sensitivity analysis is provided.
assumptions (3)
  • domain assumption NIPS pain scale labels provided by two caregivers are valid ground truth for neonatal pain.
    Section IV.A uses NIPS scores as labels; inter-rater Kappa is 0.85 and correlation 0.89, but NIPS is itself a subjective clinical scale.
  • domain assumption YOLO detectors pretrained on WIDER FACE and COCO generalize to neonatal faces and bodies in NICU videos.
    Section II.A and IV.B; no detector evaluation on neonates is reported, despite acknowledged occlusion and immature facial muscle challenges.
  • domain assumption Facial expression and body movement together provide sufficient discriminative information for pain classification in video.
    The multi-channel design in Section III.A relies on this; the paper cites correlation between pain indicators [8] and [9] but does not prove sufficiency.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Channel Neural Network for Assessing Neonatal Pain from Videos." pith.science (2026). https://pith.science/paper/VU46NWR7

@misc{pith2026190809254,
  author       = {Pith},
  title        = {Pith review of: Multi-Channel Neural Network for Assessing Neonatal Pain from Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VU46NWR7}},
  note         = {Machine review of arXiv:1908.09254}
}
read the original abstract

Neonates do not have the ability to either articulate pain or communicate it non-verbally by pointing. The current clinical standard for assessing neonatal pain is intermittent and highly subjective. This discontinuity and subjectivity can lead to inconsistent assessment, and therefore, inadequate treatment. In this paper, we propose a multi-channel deep learning framework for assessing neonatal pain from videos. The proposed framework integrates information from two pain indicators or channels, namely facial expression and body movement, using convolutional neural network (CNN). It also integrates temporal information using a recurrent neural network (LSTM). The experimental results prove the efficiency and superiority of the proposed temporal and multi-channel framework as compared to existing similar methods.

Figures

Figures reproduced from arXiv: 1908.09254 by the authors.

Figure 1
Figure 1. Facial and bodily pain stimulus of neonate [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed temporal and multi-channel network for assessing neonatal pain [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Overview of the multi-channel network for feature generation [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Harnessing the Power of Deep Learning Methods in Healthcare: Neonatal Pain Assessment from Crying Sound

    cs.CV 2019-09 conditional novelty 5.0 of 10

    On a 31-neonate NICU dataset, a lightweight CNN (N-CNN) and VGG16 both reached 96.77% accuracy and 0.94 AUC for classifying pain versus no-pain from cry spectrograms.

Reference graph

Works this paper leans on

29 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [28]

    Convolutional neural networks for neonatal pain assessment,

    G. Zamzmi, R. Paul, M. S. Salekin, D. Goldgof, R. Kasturi, T. Ho, and Y . Sun, “Convolutional neural networks for neonatal pain assessment,” IEEE Transactions on Biometrics, Behavior, and Identity Science , 2019, early access DOI:10.1109/TBIOM.2019.2918619

  2. [29]

    A comprehensive and context-sensitive neonatal pain as- sessment using computer vision,

    G. Zamzmi, C.-Y . Pai, D. Goldgof, R. Kasturi, T. Ashmeade, and Y . Sun, “A comprehensive and context-sensitive neonatal pain as- sessment using computer vision,” IEEE Transactions on Affective Computing, 2019, early access DOI:10.1109/TAFFC.2019.2926710

  3. [1]

    The reality of neonatal pain

    M. Puchalski and P. Hummel, “The reality of neonatal pain.” Advances in neonatal care: official journal of the National Association of Neonatal Nurses, vol. 2, no. 5, pp. 233–44, 2002

  4. [2]

    Current controversies regarding pain assessment in neonates,

    M. Ranger, C. C. Johnston, and K. Anand, “Current controversies regarding pain assessment in neonates,” in Seminars in perinatology , vol. 31, no. 5. Elsevier, 2007, pp. 283–288

  5. [3]

    Validation of the pain assessment in neonates (pain) scale with the neonatal infant pain scale (nips),

    D. Hudson-Barr, B. Capper-Michel, S. Lambert, T. M. Palermo, K. Morbeto, and S. Lombardo, “Validation of the pain assessment in neonates (pain) scale with the neonatal infant pain scale (nips),” Neonatal Network, vol. 21, no. 6, pp. 15–22, 2002

  6. [4]

    Clinical relia- bility and validity of the n-pass: neonatal pain, agitation and sedation scale with prolonged pain,

    P. Hummel, M. Puchalski, S. Creech, and M. Weiss, “Clinical relia- bility and validity of the n-pass: neonatal pain, agitation and sedation scale with prolonged pain,” Journal of Perinatology , vol. 28, no. 1, p. 55, 2008

  7. [5]

    Pain expression in neonates: facial action and cry,

    R. V . Grunau and K. D. Craig, “Pain expression in neonates: facial action and cry,” Pain, vol. 28, no. 3, pp. 395–410, 1987

  8. [6]

    Neonatal facial coding system for assessing postoperative pain in infants: item reduction is valid and feasible,

    J. W. Peters, H. M. Koot, R. E. Grunau, J. de Boer, M. J. van Druenen, D. Tibboel, and H. J. Duivenvoorden, “Neonatal facial coding system for assessing postoperative pain in infants: item reduction is valid and feasible,” The Clinical journal of pain , vol. 19, no. 6, pp. 353–363, 2003

Show all 29 references
  1. [7]

    Infants’ pain recognition based on facial expression: Dynamic hybrid descriptions,

    R. Zhi, G. Zamzmi, D. Goldgof, T. Ashmeade, T. Li, and Y . Sun, “Infants’ pain recognition based on facial expression: Dynamic hybrid descriptions,” IEICE TRANSACTIONS on Information and Systems , vol. 101, no. 7, pp. 1860–1869, 2018

  2. [8]

    Body movements: an important additional factor in discriminating pain from stress in preterm infants,

    L. Holsti, R. E. Grunau, T. F. Oberlander, M. F. Whitfield, and J. Weinberg, “Body movements: an important additional factor in discriminating pain from stress in preterm infants,” The Clinical journal of pain , vol. 21, no. 6, p. 491, 2005

  3. [9]

    Pain assessment in human fetus and infants,

    C. V . Bellieni, “Pain assessment in human fetus and infants,” The AAPS journal, vol. 14, no. 3, pp. 456–461, 2012

  4. [10]

    A review of automated pain assessment in infants: Features, classifi- cation tasks, and databases,

    G. Zamzmi, R. Kasturi, D. Goldgof, R. Zhi, T. Ashmeade, and Y . Sun, “A review of automated pain assessment in infants: Features, classifi- cation tasks, and databases,” IEEE reviews in biomedical engineering , vol. 11, pp. 77–96, 2018

  5. [11]

    Introduction to neonatal facial pain detection using common and advanced face classification techniques,

    S. Brahnam, L. Nanni, and R. Sexton, “Introduction to neonatal facial pain detection using common and advanced face classification techniques,” in Advanced Computational Intelligence Paradigms in Healthcare–1. Springer, 2007, pp. 225–253

  6. [12]

    A local approach based on a local binary patterns variant texture descriptor for classifying pain states,

    L. Nanni, S. Brahnam, and A. Lumini, “A local approach based on a local binary patterns variant texture descriptor for classifying pain states,” Expert Systems with Applications , vol. 37, no. 12, pp. 7888– 7894, 2010

  7. [13]

    Relevance vector machine learning for neonate pain intensity assessment using digital imaging,

    B. Gholami, W. M. Haddad, and A. R. Tannenbaum, “Relevance vector machine learning for neonate pain intensity assessment using digital imaging,” IEEE Transactions on biomedical engineering , vol. 57, no. 6, pp. 1457–1466, 2010

  8. [14]

    An approach for automated multimodal analysis of infants’ pain,

    G. Zamzmi, C.-Y . Pai, D. Goldgof, R. Kasturi, T. Ashmeade, and Y . Sun, “An approach for automated multimodal analysis of infants’ pain,” in 2016 23rd International Conference on Pattern Recognition (ICPR). IEEE, 2016, pp. 4148–4153

  9. [15]

    Deep pain: Exploiting long short- term memory networks for facial expression classification,

    P. Rodriguez, G. Cucurull, J. Gonz `alez, J. M. Gonfaus, K. Nasrollahi, T. B. Moeslund, and F. X. Roca, “Deep pain: Exploiting long short- term memory networks for facial expression classification,” IEEE transactions on cybernetics , no. 99, pp. 1–11, 2017

  10. [16]

    Spatio-temporal pain recognition in cnn-based super-resolved facial images,

    M. Bellantonio, M. A. Haque, P. Rodriguez, K. Nasrollahi, T. Telve, S. Escalera, J. Gonzalez, T. B. Moeslund, P. Rasti, and G. Anbarjafari, “Spatio-temporal pain recognition in cnn-based super-resolved facial images,” in Video Analytics. Face and Facial Expression Recognition ...

  11. [17]

    Neonatal facial pain assessment combining hand-crafted and deep features,

    L. Celona and L. Manoni, “Neonatal facial pain assessment combining hand-crafted and deep features,” in International Conference on Image Analysis and Processing . Springer, 2017, pp. 197–204

  12. [18]

    Yolov3: An incremental improvement,

    J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” arXiv preprint arXiv:1804.02767 , 2018

  13. [19]

    Wider face: A face detection benchmark,

    S. Yang, P. Luo, C.-C. Loy, and X. Tang, “Wider face: A face detection benchmark,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 5525–5533

  14. [20]

    Microsoft coco: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European conference on computer vision. Springer, 2014, pp. 740–755

  15. [21]

    Very deep convolutional networks for large-scale image recognition,

    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 , 2014

  16. [22]

    Imagenet classification with deep convolutional neural networks,

    A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems , 2012, pp. 1097–1105

  17. [23]

    Vg- gface2: A dataset for recognising faces across pose and age,

    Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman, “Vg- gface2: A dataset for recognising faces across pose and age,” in 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) . IEEE, 2018, pp. 67–74

  18. [24]

    Deep face recognition

    O. M. Parkhi, A. Vedaldi, A. Zisserman et al., “Deep face recognition.” in bmvc, vol. 1, no. 3, 2015, p. 6

  19. [25]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition . Ieee, 2009, pp. 248–255

  20. [26]

    Long short-term memory,

    S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997

  21. [27]

    Adam: A method for stochastic optimiza- tion,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimiza- tion,” arXiv preprint arXiv:1412.6980 , 2014

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.