This study proposes a semantic-aware style transfer method for resolving semantic mismatch problems in existing algorithms. As the primary focus of this study, the consideration of semantic matching is expected to improve the quality of artistic style transfer. Here, each image is partitioned into several semantic regions for both a target photograph and a source painting. All partitioned regions of the target are then associated with one of the partitioned regions in the source according to their semantic interpretation. Given a pair of target and source regions, style is learned from the source region whereas content is learned from the target region. By integrating both the style and content components, we can successfully generate a stylized output. Unlike previous approaches, we obtain the best semantic match between regions using word embeddings. Thus, we guarantee that semantic matching is always established between the target and source. Moreover, it is unreliable to partition a painting using existing algorithms because of statistical gaps between the real photographs and paintings. To bridge such gaps, we apply a domain adaptation technique on the source painting to extract its semantic regions. We evaluated the effectiveness of the proposed algorithm based on a thorough experimental analysis and comparison. Through a user study, it is confirmed that semantic information considerably influences the quality assessment of style transfer.
Bibliographical noteFunding Information:
This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the MSIP ( NRF-2019R1A2C2006123 ), the MSIT (Ministry of Science and ICT), Korea, under the “ICT Consilience Creative Program” ( IITP-2018-2017-0-01015 ) supervised by the IITP (Institute of Information & Communications Technology Planning & Evaluation), and the MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program ( IITP-2019-2016-0-00288 ) supervised by the IITP (Institute for Information & Communications Technology Planning & Evaluation).
All Science Journal Classification (ASJC) codes
- Signal Processing
- Computer Vision and Pattern Recognition