Building emotional machines: Recognizing image emotions through deep neural networks

Hye Rin Kim, Yeong Seok Kim, Seon Joo Kim, In Kwon Lee

Research output: Contribution to journalArticlepeer-review

19 Citations (Scopus)

Abstract

An image is a very effective tool for conveying emotions. Many researchers have investigated emotions in images by using various features extracted from images. In this paper, we focus on two high-level features, the object and the background, and assume that the semantic information in images is a good cue for predicting emotions. An object is one of the most important elements that define an image, and we discover through experiments that there is a high correlation between the objects and emotions in images in most cases. Even with the same object, there may be slight differences in emotion due to different backgrounds, and we use the semantic information of the background to improve the prediction performance. By combining the different levels of features, we build an emotion-based feedforward deep neural network that produces the emotion values of a given image. The output emotion values in our framework are continuous values in two-dimensional space (valence and arousal), which are more effective than using a small number of emotion categories to describe emotions. Experiments confirm the effectiveness of our network in predicting the emotions of images.

Original languageEnglish
Article number8344491
Pages (from-to)2980-2992
Number of pages13
JournalIEEE Transactions on Multimedia
Volume20
Issue number11
DOIs
Publication statusPublished - 2018 Nov

Bibliographical note

Funding Information:
Manuscript received July 3, 2017; revised October 15, 2017, January 5, 2018, and March 23, 2018; accepted March 23, 2018. Date of publication April 20, 2018; date of current version October 15, 2018. This work was supported by the Samsung Research Funding Center of Samsung Electronics under Project SRFC-IT1601-04. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Hantao Liu. (Corresponding author: In-Kwon Lee.) The authors are with the Department of Computer Science, Yonsei University, Seoul 03722, South Korea (e-mail:, rinee86@gmail.com; kys71015@ gmail.com; seonjookim@yonsei.ac.kr; iklee@yonsei.ac.kr).

All Science Journal Classification (ASJC) codes

  • Signal Processing
  • Media Technology
  • Computer Science Applications
  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'Building emotional machines: Recognizing image emotions through deep neural networks'. Together they form a unique fingerprint.

Cite this