Sky is not the limit: Semantic-aware sky replacement

Yi Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Ming Hsuan Yang

Research output: Contribution to journalConference article

20 Citations (Scopus)

Abstract

Skies are common backgrounds in photos but are often less interesting due to the time of photographing. Professional photographers correct this by using sophisticated tools with painstaking efforts that are beyond the command of ordinary users. In this work, we propose an automatic background replacement algorithm that can generate realistic, artifact-free images with a diverse styles of skies. The key idea of our algorithm is to utilize visual semantics to guide the entire process including sky segmentation, search and replacement. First we train a deep convolutional neural network for semantic scene parsing, which is used as visual prior to segment sky regions in a coarse-to-fine manner. Second, in order to find proper skies for replacement, we propose a data-driven sky search scheme based on semantic layout of the input image. Finally, to re-compose the stylized sky with the original foreground naturally, an appearance transfer method is developed to match statistics locally and semantically. We show that the proposed algorithm can automatically generate a set of visually pleasing results. In addition, we demonstrate the effectiveness of the proposed algorithm with extensive user studies.

Original languageEnglish
Article numbera149
JournalACM Transactions on Graphics
Volume35
Issue number4
DOIs
Publication statusPublished - 2016 Jul 11
EventACM SIGGRAPH 2016 - Anaheim, United States
Duration: 2016 Jul 242016 Jul 28

Fingerprint

Semantics
Statistics
Neural networks

All Science Journal Classification (ASJC) codes

  • Computer Graphics and Computer-Aided Design

Cite this

Tsai, Yi Hsuan ; Shen, Xiaohui ; Lin, Zhe ; Sunkavalli, Kalyan ; Yang, Ming Hsuan. / Sky is not the limit : Semantic-aware sky replacement. In: ACM Transactions on Graphics. 2016 ; Vol. 35, No. 4.
@article{b77b9c4ff14a469fac96b7391395d437,
title = "Sky is not the limit: Semantic-aware sky replacement",
abstract = "Skies are common backgrounds in photos but are often less interesting due to the time of photographing. Professional photographers correct this by using sophisticated tools with painstaking efforts that are beyond the command of ordinary users. In this work, we propose an automatic background replacement algorithm that can generate realistic, artifact-free images with a diverse styles of skies. The key idea of our algorithm is to utilize visual semantics to guide the entire process including sky segmentation, search and replacement. First we train a deep convolutional neural network for semantic scene parsing, which is used as visual prior to segment sky regions in a coarse-to-fine manner. Second, in order to find proper skies for replacement, we propose a data-driven sky search scheme based on semantic layout of the input image. Finally, to re-compose the stylized sky with the original foreground naturally, an appearance transfer method is developed to match statistics locally and semantically. We show that the proposed algorithm can automatically generate a set of visually pleasing results. In addition, we demonstrate the effectiveness of the proposed algorithm with extensive user studies.",
author = "Tsai, {Yi Hsuan} and Xiaohui Shen and Zhe Lin and Kalyan Sunkavalli and Yang, {Ming Hsuan}",
year = "2016",
month = "7",
day = "11",
doi = "10.1145/2897824.2925942",
language = "English",
volume = "35",
journal = "ACM Transactions on Graphics",
issn = "0730-0301",
publisher = "Association for Computing Machinery (ACM)",
number = "4",

}

Sky is not the limit : Semantic-aware sky replacement. / Tsai, Yi Hsuan; Shen, Xiaohui; Lin, Zhe; Sunkavalli, Kalyan; Yang, Ming Hsuan.

In: ACM Transactions on Graphics, Vol. 35, No. 4, a149, 11.07.2016.

Research output: Contribution to journalConference article

TY - JOUR

T1 - Sky is not the limit

T2 - Semantic-aware sky replacement

AU - Tsai, Yi Hsuan

AU - Shen, Xiaohui

AU - Lin, Zhe

AU - Sunkavalli, Kalyan

AU - Yang, Ming Hsuan

PY - 2016/7/11

Y1 - 2016/7/11

N2 - Skies are common backgrounds in photos but are often less interesting due to the time of photographing. Professional photographers correct this by using sophisticated tools with painstaking efforts that are beyond the command of ordinary users. In this work, we propose an automatic background replacement algorithm that can generate realistic, artifact-free images with a diverse styles of skies. The key idea of our algorithm is to utilize visual semantics to guide the entire process including sky segmentation, search and replacement. First we train a deep convolutional neural network for semantic scene parsing, which is used as visual prior to segment sky regions in a coarse-to-fine manner. Second, in order to find proper skies for replacement, we propose a data-driven sky search scheme based on semantic layout of the input image. Finally, to re-compose the stylized sky with the original foreground naturally, an appearance transfer method is developed to match statistics locally and semantically. We show that the proposed algorithm can automatically generate a set of visually pleasing results. In addition, we demonstrate the effectiveness of the proposed algorithm with extensive user studies.

AB - Skies are common backgrounds in photos but are often less interesting due to the time of photographing. Professional photographers correct this by using sophisticated tools with painstaking efforts that are beyond the command of ordinary users. In this work, we propose an automatic background replacement algorithm that can generate realistic, artifact-free images with a diverse styles of skies. The key idea of our algorithm is to utilize visual semantics to guide the entire process including sky segmentation, search and replacement. First we train a deep convolutional neural network for semantic scene parsing, which is used as visual prior to segment sky regions in a coarse-to-fine manner. Second, in order to find proper skies for replacement, we propose a data-driven sky search scheme based on semantic layout of the input image. Finally, to re-compose the stylized sky with the original foreground naturally, an appearance transfer method is developed to match statistics locally and semantically. We show that the proposed algorithm can automatically generate a set of visually pleasing results. In addition, we demonstrate the effectiveness of the proposed algorithm with extensive user studies.

UR - http://www.scopus.com/inward/record.url?scp=84980011651&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84980011651&partnerID=8YFLogxK

U2 - 10.1145/2897824.2925942

DO - 10.1145/2897824.2925942

M3 - Conference article

AN - SCOPUS:84980011651

VL - 35

JO - ACM Transactions on Graphics

JF - ACM Transactions on Graphics

SN - 0730-0301

IS - 4

M1 - a149

ER -