Affective image understanding has been extensively studied in the last decade since more and more users express emotion via visual contents. While current algorithms based on convolutional neural networks aim to distinguish emotional categories in a discrete label space, the task is inherently ambiguous. This is mainly because emotional labels with the same polarity (i.e., positive or negative) are highly related, which is different from concrete object concepts such as cat, dog and bird. To the best of our knowledge, few methods focus on leveraging such characteristic of emotions for affective image understanding. In this work, we address the problem of understanding affective images via deep metric learning and propose a multi-task deep framework to optimize both retrieval and classification goals. We propose the sentiment constraints adapted from the triplet constraints, which are able to explore the hierarchical relation of emotion labels. We further exploit the sentiment vector as an effective representation to distinguish affective images utilizing the texture representation derived from convolutional layers. Extensive evaluations on four widely-used affective datasets, i.e., Flickr and Instagram, IAPSa, Art Photo, and Abstract Paintings, demonstrate that the proposed algorithm performs favorably against the state-of-the-art methods on both affective image retrieval and classification tasks.
|Title of host publication||32nd AAAI Conference on Artificial Intelligence, AAAI 2018|
|Number of pages||8|
|Publication status||Published - 2018|
|Event||32nd AAAI Conference on Artificial Intelligence, AAAI 2018 - New Orleans, United States|
Duration: 2018 Feb 2 → 2018 Feb 7
|Name||32nd AAAI Conference on Artificial Intelligence, AAAI 2018|
|Other||32nd AAAI Conference on Artificial Intelligence, AAAI 2018|
|Period||18/2/2 → 18/2/7|
Bibliographical noteFunding Information:
This research was sponsored by NSFC (61620106008, 61572264), CAST (YESS20150117), Huawei Innovation Research Program (HIRP), and IBM Global SUR award.
All Science Journal Classification (ASJC) codes
- Artificial Intelligence