Deep image harmonization

Yi Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Xin Lu, Ming Hsuan Yang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

23 Citations (Scopus)

Abstract

Compositing is one of the most common operations in photo editing. To generate realistic composites, the appearances of foreground and background need to be adjusted to make them compatible. Previous approaches to harmonize composites have focused on learning statistical relationships between hand-crafted appearance features of the foreground and background, which is unreliable especially when the contents in the two layers are vastly different. In this work, we propose an end-to-end deep convolutional neural network for image harmonization, which can capture both the context and semantic information of the composite images during harmonization. We also introduce an efficient way to collect large-scale and high-quality training data that can facilitate the training process. Experiments on the synthesized dataset and real composite images show that the proposed network outperforms previous stateof- the-art methods.

Original languageEnglish
Title of host publicationProceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages2799-2807
Number of pages9
ISBN (Electronic)9781538604571
DOIs
Publication statusPublished - 2017 Nov 6
Event30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 - Honolulu, United States
Duration: 2017 Jul 212017 Jul 26

Publication series

NameProceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017
Volume2017-January

Other

Other30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017
CountryUnited States
CityHonolulu
Period17/7/2117/7/26

Fingerprint

Composite materials
Semantics
Neural networks
Experiments

All Science Journal Classification (ASJC) codes

  • Signal Processing
  • Computer Vision and Pattern Recognition

Cite this

Tsai, Y. H., Shen, X., Lin, Z., Sunkavalli, K., Lu, X., & Yang, M. H. (2017). Deep image harmonization. In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 (pp. 2799-2807). (Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017; Vol. 2017-January). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CVPR.2017.299
Tsai, Yi Hsuan ; Shen, Xiaohui ; Lin, Zhe ; Sunkavalli, Kalyan ; Lu, Xin ; Yang, Ming Hsuan. / Deep image harmonization. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. Institute of Electrical and Electronics Engineers Inc., 2017. pp. 2799-2807 (Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017).
@inproceedings{bc96f8b9d2e74eca84c75584ee977d7f,
title = "Deep image harmonization",
abstract = "Compositing is one of the most common operations in photo editing. To generate realistic composites, the appearances of foreground and background need to be adjusted to make them compatible. Previous approaches to harmonize composites have focused on learning statistical relationships between hand-crafted appearance features of the foreground and background, which is unreliable especially when the contents in the two layers are vastly different. In this work, we propose an end-to-end deep convolutional neural network for image harmonization, which can capture both the context and semantic information of the composite images during harmonization. We also introduce an efficient way to collect large-scale and high-quality training data that can facilitate the training process. Experiments on the synthesized dataset and real composite images show that the proposed network outperforms previous stateof- the-art methods.",
author = "Tsai, {Yi Hsuan} and Xiaohui Shen and Zhe Lin and Kalyan Sunkavalli and Xin Lu and Yang, {Ming Hsuan}",
year = "2017",
month = "11",
day = "6",
doi = "10.1109/CVPR.2017.299",
language = "English",
series = "Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "2799--2807",
booktitle = "Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017",
address = "United States",

}

Tsai, YH, Shen, X, Lin, Z, Sunkavalli, K, Lu, X & Yang, MH 2017, Deep image harmonization. in Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, vol. 2017-January, Institute of Electrical and Electronics Engineers Inc., pp. 2799-2807, 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, United States, 17/7/21. https://doi.org/10.1109/CVPR.2017.299

Deep image harmonization. / Tsai, Yi Hsuan; Shen, Xiaohui; Lin, Zhe; Sunkavalli, Kalyan; Lu, Xin; Yang, Ming Hsuan.

Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. Institute of Electrical and Electronics Engineers Inc., 2017. p. 2799-2807 (Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017; Vol. 2017-January).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Deep image harmonization

AU - Tsai, Yi Hsuan

AU - Shen, Xiaohui

AU - Lin, Zhe

AU - Sunkavalli, Kalyan

AU - Lu, Xin

AU - Yang, Ming Hsuan

PY - 2017/11/6

Y1 - 2017/11/6

N2 - Compositing is one of the most common operations in photo editing. To generate realistic composites, the appearances of foreground and background need to be adjusted to make them compatible. Previous approaches to harmonize composites have focused on learning statistical relationships between hand-crafted appearance features of the foreground and background, which is unreliable especially when the contents in the two layers are vastly different. In this work, we propose an end-to-end deep convolutional neural network for image harmonization, which can capture both the context and semantic information of the composite images during harmonization. We also introduce an efficient way to collect large-scale and high-quality training data that can facilitate the training process. Experiments on the synthesized dataset and real composite images show that the proposed network outperforms previous stateof- the-art methods.

AB - Compositing is one of the most common operations in photo editing. To generate realistic composites, the appearances of foreground and background need to be adjusted to make them compatible. Previous approaches to harmonize composites have focused on learning statistical relationships between hand-crafted appearance features of the foreground and background, which is unreliable especially when the contents in the two layers are vastly different. In this work, we propose an end-to-end deep convolutional neural network for image harmonization, which can capture both the context and semantic information of the composite images during harmonization. We also introduce an efficient way to collect large-scale and high-quality training data that can facilitate the training process. Experiments on the synthesized dataset and real composite images show that the proposed network outperforms previous stateof- the-art methods.

UR - http://www.scopus.com/inward/record.url?scp=85044398650&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85044398650&partnerID=8YFLogxK

U2 - 10.1109/CVPR.2017.299

DO - 10.1109/CVPR.2017.299

M3 - Conference contribution

AN - SCOPUS:85044398650

T3 - Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017

SP - 2799

EP - 2807

BT - Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Tsai YH, Shen X, Lin Z, Sunkavalli K, Lu X, Yang MH. Deep image harmonization. In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. Institute of Electrical and Electronics Engineers Inc. 2017. p. 2799-2807. (Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017). https://doi.org/10.1109/CVPR.2017.299