Pith. sign in

REVIEW 3 major objections 4 minor 98 references

Image Captioning using Facial Expression and Attention

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Feeding facial-expression features into attention-based captioning models improves captions for photos containing faces.

desk verdict A real recipe for injecting face-based emotional cues into captioning, with a genuinely non-obvious verb-diversity finding, but DUAL-FACE-ATT's training and inference weights are reversed as written, so the headline result is not interpretable until that is fixed. read the letter →

arxiv 1908.02923 v3 pith:Z35KNMBB submitted 2019-08-08 cs.CV cs.CL

Omid Mohamad Nezami , Mark Dras , Stephen Wan , Cecile Paris This is my paper · ORCID
classification cs.CVcs.CL
keywords imagecaptioningfacialexpressionrecognitionattentionmechanismLSTMemotionFlickrFace11Kverbdiversityaffectivecomputing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that emotional information read from faces can make image captioning better, not just different. On a face-focused subset of Flickr 30K with about 11,000 images, captioning models that receive facial-expression features beat their non-face counterparts on all standard metrics. The strongest models attend to fine-grained convolutional facial features with a separate attention mechanism. A linguistic analysis shows the improvement does not come from added emotion adjectives; it comes from more diverse action verbs such as smiling, singing, and reading.

What carries the argument

The machinery is a facial expression recognition model trained on FER-2013, used as a frozen feature extractor. It supplies two kinds of features: a one-hot aggregate expression vector for the whole image, and 6x6x512 convolutional features for up to three detected faces, which are attended to by a separate LSTM at each caption time step. The attention over these facial features is what lets the model localize emotional content and combine it with visual attention.

What would settle it

Train the same DUAL-FACE-ATT architecture but replace the facial features with random vectors of the same shape, or with face-detection boxes without expression information. If captions improve as much as with real facial features, the claimed cause — emotional content from faces — is not doing the work.

Watch

Extended reading notes

Core claim

The paper's central claim is that facial expression features, when fed into an LSTM caption generator alongside ordinary visual features, produce more expressive and more semantically correlated captions for images containing faces. Two attention-based variants, DUAL-FACE-ATT and JOINT-FACE-ATT, outperform the Show-Attend-Tell and Up-Down baselines on BLEU-1 through BLEU-4, METEOR, ROUGE-L, CIDEr, and SPICE on the authors' FlickrFace11K test split. The authors trace the gain to the distribution of verbs: models with facial features have higher verb entropy and rank emotion-linked action verbs higher, while the set of adjectives they use is essentially unchanged.

Load-bearing premise

The load-bearing assumption is that the facial expression recognizer trained on FER-2013's Google-search face images produces reliable enough emotion features when applied to faces cropped from natural Flickr photos.

Editorial extensions

If this is right

  • Adding facial features to a strong attention-based captioner improves every standard metric on face-containing images, not just emotion-related ones.
  • The one-hot aggregate expression vector helps less than attended convolutional facial features, suggesting fine-grained per-face features carry more useful signal.
  • The improvement appears in action verbs rather than emotion adjectives, so the emotional signal is changing what activity the model describes.
  • The gain is robust to the choice of FER backbone: VGG, ResNet, and Inception features perform similarly within the same captioning architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable implication is that the facial features may be functioning partly as a face-location prior: a model given face bounding boxes without expression labels might capture a large share of the verb-diversity gain.
  • If the transfer of FER-2013 features to natural photos is the weak link, then an oracle experiment using human expression labels on FlickrFace11K would show whether the ceiling is higher than the reported gains.
  • The verb-entropy analysis suggests a cheap evaluation for future emotion-aware captioning: measure action-verb diversity, not just emotion lexicon hits.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes FACE-CAP and FACE-ATTEND, two families of image captioning models that inject facial-expression information into caption generation. FACE-CAP uses a one-hot aggregate facial-expression encoding derived from a FER-2013-trained model, while FACE-ATTEND uses fine-grained convolutional facial features and attention, with DUAL-FACE-ATT and JOINT-FACE-ATT variants. The models are evaluated on FlickrFace11K, a new subset of Flickr30K containing 11,696 images with faces. The paper reports that DUAL-FACE-ATT and JOINT-FACE-ATT outperform Show-Attend-Tell and Up-Down baselines on BLEU, METEOR, ROUGE-L, CIDEr, and SPICE, and it attributes the improvements mainly to greater verb diversity rather than to added emotion adjectives.

Significance. If the results hold, this is a useful and relatively novel contribution: it is, to my knowledge, the first image captioning study to use facial-expression features, it provides a new public dataset split, and it includes a reasonable linguistic analysis showing that the gains appear in verb diversity rather than emotion adjectives. The paper also ships dataset splits and labels publicly, and it compares three FER backbone architectures. However, the central quantitative claim is currently undermined by an internal inconsistency between the training and test objectives of DUAL-FACE-ATT, by hyperparameter differences between the proposed models and their baselines, and by the absence of any statistical significance or control experiments. These are fixable within the scope of the paper, but they must be addressed before the claim can be accepted.

major comments (3)
  1. [Section 3.3.2, Eqs. (14) and (16)] The training objective and the test-time objective of DUAL-FACE-ATT use opposite stream weights. In Eq. (14), with lambda=0.8 as reported in Section 4.3, the visual-stream log-likelihood receives weight lambda=0.8 and the facial-stream term receives weight (1-lambda)=0.2. In Eq. (16), however, the next-word probability is lambda * p_f + (1-lambda) * p_c, so at inference the facial stream receives weight 0.8 and the visual stream receives weight 0.2. The equations therefore reverse the roles of the two streams at the reported value of lambda. Since DUAL-FACE-ATT is the model behind the headline metric gains and the verb-diversity finding in Tables 1, 3, and 4, the reported results could be an artifact of decoding with a facial stream that was trained with lower weight. No code is released to show which equation the implementation actually follows. The authors should reconcile the weighting convention, retrain, and report the corrected results; this is necessary before the central claim can be evaluated.
  2. [Section 4.3 and Table 1] The experimental comparison is not adequately controlled. UP-DOWN and JOINT-FACE-ATT use mini-batch size 64 and initial learning rate 0.005, while all other models use mini-batch size 100 and learning rate 0.001. The paper states that using similar parameters led to worse results for all models, but gives no supporting numbers. The reported gains over the baselines are small (e.g., BLEU-4 17.7 vs. 17.3, CIDEr 24.8 vs. 24.4, SPICE 10.2 vs. 10.1), and no significance tests, confidence intervals, or multiple runs are reported. Under these conditions, the reader cannot distinguish a genuine effect of facial features from hyperparameter or random-seed effects. The authors should run the baselines and the proposed models under identical hyperparameter schedules, and report variance estimates or significance tests.
  3. [Sections 3.1 and 3.2] The load-bearing assumption that a FER model trained on FER-2013 transfers reliably to FlickrFace11K is not tested. The paper acknowledges that the FER model is not 100% accurate and that the two domains differ (Google search images versus natural Flickr photos), but it reports no FER accuracy on FlickrFace11K faces and no control experiment using non-emotional facial features. Because the only comparisons are against baselines without facial features, the observed gains could originate from the extra face-attention machinery rather than from emotional content. The authors should report FER accuracy on the captioning test set and include a control with randomly initialized or shuffled facial features, or with features from a face-identity network, to isolate the role of expression information.
minor comments (4)
  1. [Section 3.2] The paragraph contains the sentence fragment "DUAL-FACE-ATT FER information." in the middle of a discussion of facial features; this appears to be an editing artifact and should be removed or completed.
  2. [Section 3.3.2, Eq. (14)] The text calls lambda and beta1 "regularization constants," but lambda is a mixture weight between the two stream losses and beta1 is better described as an attention-regularization coefficient; please use consistent terminology.
  3. [Section 4.3] The sentence "We used different parameters for UP-DOWN and JOINT-FACE-ATT in comparison with other models because using similar parameters led to worse results for all models" is stated without any quantitative support; reporting the actual values would clarify the hyperparameter sensitivity.
  4. [Table 4] The table uses "n/a" for some model-verb combinations, but the text does not say whether those verbs were never generated or merely fell below a rank threshold; please clarify, since this affects the interpretation of the verb-diversity results.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the FER features are trained on external FER-2013 data and caption metrics are computed on a held-out FlickrFace11K split.

full rationale

The paper's derivation chain is self-contained. The facial expression features are produced by a FER model trained on the external FER-2013 dataset (Section 3.1), and the captioning models are trained and evaluated on disjoint splits of the author-constructed FlickrFace11K subset of Flickr30K (Sections 4.3-4.4). The baseline comparisons SHOW-ATT-TELL and UP-DOWN are trained on the same splits, so the reported gains are not forced by construction. The manually set constants lambda=0.8, beta1=0.2, and beta2=0.4 are tuning choices, not quantities derived from the target captions. The earlier FACE-CAP paper by the same authors is cited in a footnote as prior publication, but the empirical evaluation here is new and does not rely on that citation as evidence. The verb-diversity analysis (entropy, Top4, and verb ranks) is computed from generated captions and ground-truth references, with no fitted parameter renamed as a prediction. The only notable issue is an internal inconsistency in DUAL-FACE-ATT: the training objective (Eq. 14) weights the visual stream by lambda=0.8 while the decoding combination (Eq. 16) weights the facial stream by lambda=0.8; this is a correctness/implementation concern, not a circularity, because neither equation defines the result in terms of the claimed conclusion.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard ML assumptions: external FER labels, transferability of FER features, and the validity of automatic captioning metrics. No new physical or conceptual entities are postulated. The main empirical constants are the three regularization weights set by hand.

free parameters (3)
  • lambda = 0.8
    Regularization constant in DUAL-FACE-ATT objective (Eq. 14), weighting the visual versus facial caption prediction streams; set empirically.
  • beta1 = 0.2
    Regularization constant for attention over facial features in DUAL-FACE-ATT (Eq. 14); set empirically.
  • beta2 = 0.4
    Regularization constant for attention over facial features in JOINT-FACE-ATT (Eq. 20); set empirically.
assumptions (4)
  • domain assumption FER-2013 labels (seven basic emotions plus neutral) are valid ground truth for facial expressions.
    The FER model is trained on FER-2013 and used to extract features for FlickrFace11K; the paper does not validate recognition accuracy on the target images.
  • domain assumption Facial features trained on FER-2013 transfer to FlickrFace11K faces after grayscale and 48x48 resizing.
    Section 3.1 pre-processes FlickrFace11K faces to match FER-2013; the paper acknowledges the FER model is not 100% accurate, so noisy transfer is possible.
  • domain assumption Standard captioning metrics (BLEU, METEOR, ROUGE-L, CIDEr, SPICE) adequately measure caption quality for the claim.
    Metrics are used as the primary evidence that facial features improve captions; the paper itself notes SPICE can misjudge individual examples.
  • domain assumption Flickr30K human captions are correct references for FlickrFace11K images.
    Ground-truth captions are taken from Flickr30K without additional validation on the face subset.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Image Captioning using Facial Expression and Attention." pith.science (2026). https://pith.science/paper/Z35KNMBB

@misc{pith2026190802923,
  author       = {Pith},
  title        = {Pith review of: Image Captioning using Facial Expression and Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z35KNMBB}},
  note         = {Machine review of arXiv:1908.02923}
}
read the original abstract

Benefiting from advances in machine vision and natural language processing techniques, current image captioning systems are able to generate detailed visual descriptions. For the most part, these descriptions represent an objective characterisation of the image, although some models do incorporate subjective aspects related to the observer's view of the image, such as sentiment; current models, however, usually do not consider the emotional content of images during the caption generation process. This paper addresses this issue by proposing novel image captioning models which use facial expression features to generate image captions. The models generate image captions using long short-term memory networks applying facial features in addition to other visual features at different time steps. We compare a comprehensive collection of image captioning models with and without facial features using all standard evaluation metrics. The evaluation metrics indicate that applying facial features with an attention mechanism achieves the best performance, showing more expressive and more correlated image captions, on an image caption dataset extracted from the standard Flickr 30K dataset, consisting of around 11K images containing faces. An analysis of the generated captions finds that, perhaps unexpectedly, the improvement in caption quality appears to come not from the addition of adjectives linked to emotional aspects of the images, but from more variety in the actions described in the captions.

Figures

Figures reproduced from arXiv: 1908.02923 by the authors.

Figure 1
Figure 1. The examples of Flickr 30K dataset (Young et al., 2014) with emotional content. The green color indicates words with strong emotional values. However, current image captioning methods often overlook the emotional aspects of the image, which play an important role in generating captions that are more semantically correlated with the visual content. For example, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Examples from the Facial Expression Recognition 2013 dataset (Goodfellow et al., 2013) including seven standard facial expressions. CNNs are also used for feature extraction and transfer learning in this domain. Kahou, Bouthillier, Lamblin, Gulcehre, Michalski, Konda, Jean, Froumenty, Dauphin, and Boulanger-Lewandowski (2016) applied a CNN model to recognize facial expressions. Their approach uses a combination of d… view at source ↗
Figure 3
Figure 3. The frameworks of FACE-CAP-REPEAT (top), and FACE-CAP-MEMORY (bottom). Attend is our attention mechanism attending to the visual features, {c1, . . . , cK}. 3.3.2 FACE-ATTEND Here, we apply two LSTMs to attend to our more fine-grained facial features (f) explained in Section 3.2, in addition to our visual features (c). We propose two variant architectures for combining these features, DUAL-FACE-ATT and JOINT-FACE-AT… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: DUAL-FACE-ATT model enables generating image captions with both facial features {f1, . . . , fK? } and visual content {c1, . . . , cK}. Kz is K? for F-LSTM and K for C-LSTM. The initial LSTM’s hidden state (h0,z) is computed using a standard multilayer perceptron: h0,z…
Figure 5
Figure 5. Figure 5: JOINT-FACE-ATT model enables generating image captions with two LSTMs for learning attention weights and generating captions, separately. generated word. However, the recent state-of-the-art image captioning model of Anderson et al. (2018) achieved better performance b…
Figure 6
Figure 6. Figure 6: Example generated captions using SAT (SHOW-ATT-TELL), UD (UP-DOWN) SI (STEP￾INJECT), IF (INIT-FLOW), FR (FACE-CAP-REPEAT), FM (FACE-CAP-MEMORY), DFA (DUAL-FACE-ATT) and JFA (JOINT-FACE-ATT) models. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]
Figure 7
Figure 7. Figure 7: Example images where FACE-CAP and DUAL-FACE-ATT fail to generate better results than SHOW-ATT-TELL. • Our models that use a distributed representation of facial emotion (FACE-ATTEND) outper￾formed those that use a one-hot encoding (FACE-CAP). • For FACE-CAP models, inj…
Figure 8
Figure 8. Figure 8: An example image where the SHOW-ATT-TELL model expresses the size attribute in its generated caption. • A linguistic analysis of the generated captions showed that much of the improvement in our models was manifested through verbs. In particular, under measures of dive…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

98 extracted references · 67 canonical work pages

  1. [1]

    Anderson, P., Fernando, B., Johnson, M., \ Gould, S. 2016 . SPICE : Semantic propositional image caption evaluation \ In ECCV , \ 382--398. Springer

    2016

  2. [2]

    Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., \ Zhang, L. 2018 . Bottom-up and top-down attention for image captioning and visual question answering \ In CVPR , 3, 6

    2018

  3. [3]

    Ba, J., Mnih, V., \ Kavukcuoglu, K. 2014 . Multiple object recognition with visual attention \ arXiv preprint arXiv:1412.7755

    arXiv 2014

  4. [4]

    Bahdanau, D., Cho, K., \ Bengio, Y. 2014 . Neural machine translation by jointly learning to align and translate \ arXiv preprint arXiv:1409.0473

    arXiv 2014

  5. [5]

    Bernardi, R., Cakici, R., Elliott, D., Erdem, A., Erdem, E., Ikizler-Cinbis, N., Keller, F., Muscat, A., \ Plank, B. 2016 . Automatic description generation from images: A survey of models, datasets, and evaluation measures \ Journal of Artificial Intelligence Research , 55 , 409--442

    2016

  6. [6]

    Chen, L., Zhang, H., Xiao, J., Nie, L., Shao, J., Liu, W., \ Chua, T.-S. 2017 . Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning \ In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , \ 6298--6306. IEEE

    2017

  7. [7]

    "Factual" or "Emotional": Stylized Image Captioning with Adaptive Learning and Attention

    Chen, T., Zhang, Z., You, Q., Fang, C., Wang, Z., Jin, H., \ Luo, J. 2018 . “factual” or “emotional”: Stylized image captioning with adaptive learning and attention \ arXiv preprint arXiv:1807.03871

    work page Pith review arXiv 2018

  8. [8]

    \ \ Lawrence Zitnick, C

    Chen, X. \ \ Lawrence Zitnick, C. 2015 . Mind's eye: A recurrent visual representation for image caption generation \ In CVPR , \ 2422--2431. IEEE

    2015

Show all 98 references
  1. [9]

    A., Clark, V

    Coffman, B. A., Clark, V. P., \ Parasuraman, R. 2014 . Battery powered thought: enhancement of attention, learning, and memory in healthy adults using transcranial direct current stimulation \ Neuroimage , 85 , 895--908

    2014

  2. [10]

    \ \ Shulman, G

    Corbetta, M. \ \ Shulman, G. L. 2002 . Control of goal-directed and stimulus-driven attention in the brain \ Nature reviews neuroscience , 3\/ (3), 201

    2002

  3. [11]

    \ \ Lavie, A

    Denkowski, M. \ \ Lavie, A. 2014 . METEOR universal: Language specific translation evaluation for any target language \ In WMT , \ 376--380

    2014

  4. [12]

    \ \ Duncan, J

    Desimone, R. \ \ Duncan, J. 1995 . Neural mechanisms of selective visual attention \ Annual review of neuroscience , 18\/ (1), 193--222

    1995

  5. [13]

    Donahue, J., Anne Hendricks, L., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., \ Darrell, T. 2015 . Long-term recurrent convolutional networks for visual recognition and description \ In CVPR , \ 2625--2634. IEEE

    2015

  6. [14]

    Ekman, P. 2006 . Darwin and facial expression: A century of research in review . Ishk

    2006

  7. [15]

    \ \ Keller, F

    Elliott, D. \ \ Keller, F. 2013 . Image description using visual dependency representations \ In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , \ 1292--1302

    2013

  8. [16]

    Eriksen, C. W. \ \ James, J. D. S. 1986 . Visual attention within and around the field of focal attention: A zoom lens model \ Perception & psychophysics , 40\/ (4), 225--240

    1986

  9. [17]

    A., Young, P., Rashtchian, C., Hockenmaier, J., \ Forsyth, D

    Farhadi, A., Hejrati, M., Sadeghi, M. A., Young, P., Rashtchian, C., Hockenmaier, J., \ Forsyth, D. 2010 . Every picture tells a story: Generating sentences from images \ In ECCV , \ 15--29. Springer

    2010

  10. [18]

    \ \ Luettin, J

    Fasel, B. \ \ Luettin, J. 2003 . Automatic facial expression analysis: a survey \ Pattern recognition , 36\/ (1), 259--275

    2003

  11. [19]

    M., Woodson, R., Greenberg, R., \ Cohen, D

    Field, T. M., Woodson, R., Greenberg, R., \ Cohen, D. 1982 . Discrimination and imitation of facial expression by neonates \ Science , 218\/ (4568), 179--181

    1982

  12. [20]

    Fridlund, A. J. 2014 . Human facial expression: An evolutionary view . Academic Press

    2014

  13. [21]

    Gan, C., Gan, Z., He, X., Gao, J., \ Deng, L. 2017 . Stylenet: Generating attractive visual captions with styles \ In CVPR . IEEE

    2017

  14. [22]

    J., Erhan, D., Carrier, P

    Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., \ Lee, D.-H. 2013 . Challenges in representation learning: A report on three machine learning contests \ In ICONIP , \ 117--124. Springer

    2013

  15. [23]

    He, K., Zhang, X., Ren, S., \ Sun, J. 2016 . Deep Residual Learning for Image Recognition \ In IEEE Conference on Computer Vision and Pattern Recognition , \ 770--778

    2016

  16. [24]

    Hodosh, M., Young, P., \ Hockenmaier, J. 2013 . Framing image description as a ranking task: Data, models and evaluation metrics \ Journal of Artificial Intelligence Research , 47 , 853--899

    2013

  17. [25]

    F., \ Laga, H

    Hossain, M., Sohel, F., Shiratuddin, M. F., \ Laga, H. 2019 . A comprehensive survey of deep learning for image captioning \ ACM Computing Surveys (CSUR) , 51\/ (6), 118

    2019

  18. [26]

    Hu, Z., Yang, Z., Liang, X., Salakhutdinov, R., \ Xing, E. P. 2017 . Toward controlled generation of text \ In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , \ 1587--1596. JMLR. org

    2017

  19. [27]

    Huber, B., McDuff, D., Brockett, C., Galley, M., \ Dolan, B. 2018 . Emotional dialogue generation using image-grounded language models \ In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems , \ 277. ACM

    2018

  20. [28]

    \ \ Szegedy, C

    Ioffe, S. \ \ Szegedy, C. 2015 . Batch normalization: Accelerating deep network training by reducing internal covariate shift \ arXiv preprint arXiv:1502.03167

    2015 arXiv

  21. [29]

    Jaderberg, M., Simonyan, K., \ Zisserman, A. 2015 . Spatial transformer networks \ In Advances in neural information processing systems , \ 2017--2025

    2015

  22. [30]

    Jin, J., Fu, K., Cui, R., Sha, F., \ Zhang, C. 2015 . Aligning where to see and what to tell: image caption with region-based attention and scene factorization \ arXiv preprint arXiv:1506.06272

    2015 arXiv

  23. [31]

    Johnson, J., Karpathy, A., \ Fei-Fei, L. 2016 . Densecap: Fully convolutional localization networks for dense captioning \ In CVPR , \ 4565--4574. IEEE

    2016

  24. [32]

    E., Bouthillier, X., Lamblin, P., Gulcehre, C., Michalski, V., Konda, K., Jean, S., Froumenty, P., Dauphin, Y., \ Boulanger-Lewandowski, N

    Kahou, S. E., Bouthillier, X., Lamblin, P., Gulcehre, C., Michalski, V., Konda, K., Jean, S., Froumenty, P., Dauphin, Y., \ Boulanger-Lewandowski, N. 2016 . Emonets: Multimodal deep learning approaches for emotion recognition in video \ Journal on Multimodal User Interfaces , ...

    2016

  25. [33]

    F., \ Tian, Y

    Kanade, T., Cohn, J. F., \ Tian, Y. 2000 . Comprehensive database for facial expression analysis \ In Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580) , \ 46--53. IEEE

    2000

  26. [34]

    \ \ Fei-Fei, L

    Karpathy, A. \ \ Fei-Fei, L. 2015 . Deep visual-semantic alignments for generating image descriptions \ In CVPR , \ 3128--3137. IEEE

    2015

  27. [35]

    Kim, B.-K., Dong, S.-Y., Roh, J., Kim, G., \ Lee, S.-Y. 2016 . Fusing aligned and non-aligned face information for automatic affect recognition in the wild: A deep learning approach \ In CVPR Workshops , \ 48--57. IEEE

    2016

  28. [36]

    King, D. E. 2009 . Dlib-ml: A machine learning toolkit \ Journal of Machine Learning Research , 10\/ (Jul), 1755--1758

    2009

  29. [37]

    Kingma, D. P. \ \ Ba, J. 2014 . Adam: A method for stochastic optimization \ arXiv preprint arXiv:1412.6980

    2014 arXiv

  30. [38]

    Kiros, R., Salakhutdinov, R., \ Zemel, R. S. 2014 . Unifying visual-semantic embeddings with multimodal neural language models \ arXiv preprint arXiv:1411.2539

    2014 arXiv

  31. [39]

    \ \ Ullman, S

    Koch, C. \ \ Ullman, S. 1987 . Shifts in selective visual attention: towards the underlying neural circuitry \ In Matters of intelligence , \ 115--141. Springer

    1987

  32. [40]

    Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., \ Shamma, D. A. 2017 . Visual genome: Connecting language and vision using crowdsourced dense image annotations \ International Journal of Computer Vision , 123\/ (1), 32--73

    2017

  33. [41]

    C., \ Berg, T

    Kulkarni, G., Premraj, V., Ordonez, V., Dhar, S., Li, S., Choi, Y., Berg, A. C., \ Berg, T. L. 2013 . Baby talk: Understanding and generating simple image descriptions \ IEEE Transactions on Pattern Analysis and Machine Intelligence , 35\/ (12), 2891--2903

    2013

  34. [42]

    C., Berg, T

    Kuznetsova, P., Ordonez, V., Berg, A. C., Berg, T. L., \ Choi, Y. 2012 . Collective generation of natural image descriptions \ In ACL , \ 359--368. Association for Computational Linguistics

    2012

  35. [43]

    Li, Y., Yao, T., Mei, T., Chao, H., \ Rui, Y. 2016 . Share-and-chat: Achieving human-level video commenting by search and multi-view embedding \ In Proceedings of the 24th ACM international conference on Multimedia , \ 928--937. ACM

    2016

  36. [44]

    Lin, C.-Y. 2004 . ROUGE : A package for automatic evaluation of summaries \ Text Summarization Branches Out

    2004

  37. [45]

    Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollar, P., \ Zitnick, C. L. 2014 . Microsoft coco: Common objects in context \ In European conference on computer vision , \ 740--755. Springer

    2014

  38. [46]

    Lisetti, C. 1998 . Affective computing

    1998

  39. [47]

    Lu, J., Xiong, C., Parikh, D., \ Socher, R. 2017 . Knowing when to look: Adaptive attention via a visual sentinel for image captioning \ In CVPR , 6, 2

    2017

  40. [48]

    F., Kanade, T., Saragih, J., Ambadar, Z., \ Matthews, I

    Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., \ Matthews, I. 2010 . The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression \ In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition...

    2010

  41. [49]

    Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., \ Yuille, A. 2014 . Deep captioning with multimodal recurrent neural networks (m-rnn) \ arXiv preprint arXiv:1412.6632

    2014 arXiv

  42. [50]

    Mathews, A., Xie, L., \ He, X. 2018 . Semstyle: Learning to generate stylised image captions using unaligned text \ In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , \ 8591--8600

    2018

  43. [51]

    P., Xie, L., \ He, X

    Mathews, A. P., Xie, L., \ He, X. 2016 . Senticap: Generating image descriptions with sentiments. \ In AAAI , \ 3574--3580

    2016

  44. [52]

    Mi, H., Sankaran, B., Wang, Z., \ Ittycheriah, A. 2016 . Coverage Embedding Models for Neural Machine Translation \ In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , \ 955--960, Austin, Texas. Association for Computational Linguistics

    2016

  45. [53]

    Mnih, V., Heess, N., \ Graves, A. 2014 . Recurrent models of visual attention \ In Advances in neural information processing systems , \ 2204--2212

    2014

  46. [54]

    Mohammad, S. M. \ \ Turney, P. D. 2013 . Crowdsourcing a word-emotion association lexicon . 29\/ (3), 436--465

    2013

  47. [55]

    M., Dras, M., Anderson, P., \ Hamey, L

    Nezami, O. M., Dras, M., Anderson, P., \ Hamey, L. 2018a . Face-cap: Image captioning using facial expression analysis \ In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , \ 226--240. Springer

  48. [56]

    M., Dras, M., Hamey, L., Richards, D., Wan, S., \ Paris, C

    Nezami, O. M., Dras, M., Hamey, L., Richards, D., Wan, S., \ Paris, C. 2018b . Automatic recognition of student engagement using deep learning and facial expression \ arXiv preprint arXiv:1808.02324

    arXiv

  49. [57]

    M., Dras, M., Wan, S., \ Paris, C

    Nezami, O. M., Dras, M., Wan, S., \ Paris, C. 2018c . Senti-attend: Image captioning using sentiment and attention \ arXiv preprint arXiv:1811.09789

    arXiv

  50. [58]

    M., Dras, M., Wan, S., Paris, C., \ Hamey, L

    Nezami, O. M., Dras, M., Wan, S., Paris, C., \ Hamey, L. 2019a . Towards generating stylized image captions via adversarial training \ In Pacific Rim International Conference on Artificial Intelligence , \ 270--284. Springer

  51. [59]

    M., Lou, P

    Nezami, O. M., Lou, P. J., \ Karami, M. 2019b . Shemo: a large-scale validated database for persian speech emotion detection \ Language Resources and Evaluation , 53\/ (1), 1--16

  52. [60]

    M., Richards, D., \ Hamey, L

    Nezami, O. M., Richards, D., \ Hamey, L. 2017 . Semi-supervised detection of student engagement. \ In PACIS , \ 157

    2017

  53. [61]

    \ \ Lee, L

    Pang, B. \ \ Lee, L. 2008 . Opinion mining and sentiment analysis \ Found. Trends Inf. Retr. , 2\/ (1-2), 1--135

    2008

  54. [62]

    Papineni, K., Roukos, S., Ward, T., \ Zhu, W.-J. 2002 . BLEU : a method for automatic evaluation of machine translation \ In ACL , \ 311--318. Association for Computational Linguistics

    2002

  55. [63]

    Pennebaker, J. W. \ \ King, L. A. 1999 . Linguistic styles: Language use as an individual difference. \ Journal of personality and social psychology , 77\/ (6), 1296

    1999

  56. [64]

    \ \ Kampel, M

    Pramerdorfer, C. \ \ Kampel, M. 2016 . Facial expression recognition using convolutional neural networks: State of the art \ arXiv preprint arXiv:1612.02903

    2016 arXiv

  57. [65]

    Radford, A., Jozefowicz, R., \ Sutskever, I. 2017 . Learning to generate reviews and discovering sentiment \ arXiv preprint arXiv:1704.01444

    2017 arXiv

  58. [66]

    Ren, S., He, K., Girshick, R., \ Sun, J. 2017 . Faster r-cnn: towards real-time object detection with region proposal networks \ IEEE Transactions on Pattern Analysis & Machine Intelligence , \ 1137--1149

    2017

  59. [67]

    J., Marcheret, E., Mroueh, Y., Ross, J., \ Goel, V

    Rennie, S. J., Marcheret, E., Mroueh, Y., Ross, J., \ Goel, V. 2017 . Self-critical sequence training for image captioning \ In CVPR , 1, 3

    2017

  60. [68]

    Rensink, R. A. 2000 . The dynamic representation of scenes \ Visual cognition , 7\/ (1-3), 17--42

    2000

  61. [69]

    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., \ Bernstein, M. 2015 . Imagenet large scale visual recognition challenge \ International journal of computer vision , 115\/ (3), 211--252

    2015

  62. [70]

    Sariyanidi, E., Gunes, H., \ Cavallaro, A. 2015 . Automatic analysis of facial affect: A survey of registration, representation, and recognition \ IEEE transactions on pattern analysis and machine intelligence , 37\/ (6), 1113--1133

    2015

  63. [71]

    \ \ Zisserman, A

    Simonyan, K. \ \ Zisserman, A. 2014 . Very deep convolutional networks for large-scale image recognition \ arXiv preprint arXiv:1409.1556

    2014 arXiv

  64. [72]

    Spratling, M. W. \ \ Johnson, M. H. 2004 . A feedback model of visual attention \ Journal of cognitive neuroscience , 16\/ (2), 219--237

    2004

  65. [73]

    Sutskever, I., Vinyals, O., \ Le, Q. V. 2014 . Sequence to sequence learning with neural networks \ In NIPS , \ 3104--3112

    2014

  66. [74]

    Sutton, R. S. \ \ Barto, A. G. 1998 . Introduction to reinforcement learning , \ 135. MIT press Cambridge

    1998

  67. [75]

    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., \ Rabinovich, A. 2015 . Going deeper with convolutions \ In Proceedings of the IEEE conference on computer vision and pattern recognition , \ 1--9

    2015

  68. [76]

    Tang, Y. 2013 . Deep learning using linear support vector machines \ arXiv preprint arXiv:1306.0239

    2013 arXiv

  69. [77]

    Tian, Y.-I., Kanade, T., \ Cohn, J. F. 2001 . Recognizing action units for facial expression analysis \ IEEE Transactions on pattern analysis and machine intelligence , 23\/ (2), 97--115

    2001

  70. [78]

    Tian, Y., Wang, X., Wu, J., Wang, R., \ Yang, B. 2019 . Multi-scale hierarchical residual network for dense captioning \ Journal of Artificial Intelligence Research , 64 , 181--196

    2019

  71. [79]

    D., \ Singer, Y

    Toutanova, K., Klein, D., Manning, C. D., \ Singer, Y. 2003 . Feature-rich part-of-speech tagging with a cyclic dependency network \ In NAACL HLT , \ 173--180. Association for Computational Linguistics

    2003

  72. [80]

    R., Van De Sande, K

    Uijlings, J. R., Van De Sande, K. E., Gevers, T., \ Smeulders, A. W. 2013 . Selective search for object recognition \ International journal of computer vision , 104\/ (2), 154--171

    2013

  73. [81]

    N., Kaiser, ., \ Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., \ Polosukhin, I. 2017 . Attention is all you need \ In Advances in neural information processing systems , \ 5998--6008

    2017

  74. [82]

    Vedantam, R., Lawrence Zitnick, C., \ Parikh, D. 2015 . CIDEr : Consensus-based image description evaluation \ In CVPR , \ 4566--4575. IEEE

    2015

  75. [83]

    \ \ Le, Q

    Vinyals, O. \ \ Le, Q. 2015 . A neural conversational model \ arXiv preprint arXiv:1506.05869

    2015 arXiv

  76. [84]

    Vinyals, O., Toshev, A., Bengio, S., \ Erhan, D. 2015 . Show and tell: A neural image caption generator \ In CVPR , \ 3156--3164. IEEE

    2015

  77. [85]

    Williams, R. J. 1992 . Simple statistical gradient-following algorithms for connectionist reinforcement learning \ Machine learning , 8\/ (3-4), 229--256

    1992

  78. [86]

    C., Roelofs, R., Stern, M., Srebro, N., \ Recht, B

    Wilson, A. C., Roelofs, R., Stern, M., Srebro, N., \ Recht, B. 2017 . The marginal value of adaptive gradient methods in machine learning \ In NIPS , \ 4151--4161

    2017

  79. [87]

    Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., \ Bengio, Y. 2015 . Show, attend and tell: Neural image caption generation with visual attention \ In ICML , \ 2048--2057

    2015

  80. [88]

    Yin, L., Wei, X., Sun, Y., Wang, J., \ Rosato, M. J. 2006 . A 3d facial expression database for facial behavior research \ In 7th international conference on automatic face and gesture recognition (FGR06) , \ 211--216. IEEE

    2006

  81. [89]

    You, Q., Jin, H., \ Luo, J. 2018 . Image captioning at will: A versatile scheme for effectively injecting sentiments into image descriptions \ arXiv preprint arXiv:1801.10121

    2018 arXiv

  82. [90]

    You, Q., Jin, H., Wang, Z., Fang, C., \ Luo, J. 2016a . Image captioning with semantic attention \ In CVPR , \ 4651--4659. IEEE

  83. [91]

    You, Q., Luo, J., Jin, H., \ Yang, J. 2016b . Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmark \ In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence , \ 308--314

  84. [92]

    Young, P., Lai, A., Hodosh, M., \ Hockenmaier, J. 2014 . From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions \ Transactions of the Association for Computational Linguistics , 2 , 67--78

    2014

  85. [93]

    Yu, Y., Ko, H., Choi, J., \ Kim, G. 2017 . End-to-end concept word detection for video captioning, retrieval, and question answering \ In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , \ 3261--3269. IEEE

    2017

  86. [94]

    \ \ Zhang, C

    Yu, Z. \ \ Zhang, C. 2015 . Image based static facial expression recognition with multiple deep network learning \ In ICMI , \ 435--442. ACM

    2015

  87. [95]

    Zeng, N., Zhang, H., Song, B., Liu, W., Li, Y., \ Dobaie, A. M. 2018 . Facial expression recognition via learning deep sparse autoencoders \ Neurocomputing , 273 , 643--649

    2018

  88. [96]

    I., \ Huang, T

    Zeng, Z., Pantic, M., Roisman, G. I., \ Huang, T. S. 2008 . A survey of affect recognition methods: Audio, visual, and spontaneous expressions \ IEEE transactions on pattern analysis and machine intelligence , 31\/ (1), 39--58

    2008

  89. [97]

    Zhang, Z., Luo, P., Loy, C.-C., \ Tang, X. 2015 . Learning social relation traits from face images \ In Proceedings of the IEEE International Conference on Computer Vision , \ 3631--3639

    2015

  90. [98]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.