Unsupervised Holistic Image Generation from Key Local Patches

Donghoon Lee, Sangdoo Yun, Sungjoon Choi, Hwiyeon Yoo, Ming Hsuan Yang, Songhwai Oh

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We introduce a new problem of generating an image based on a small number of key local patches without any geometric prior. In this work, key local patches are defined as informative regions of the target object or scene. This is a challenging problem since it requires generating realistic images and predicting locations of parts at the same time. We construct adversarial networks to tackle this problem. A generator network generates a fake image as well as a mask based on the encoder-decoder framework. On the other hand, a discriminator network aims to detect fake images. The network is trained with three losses to consider spatial, appearance, and adversarial information. The spatial loss determines whether the locations of predicted parts are correct. Input patches are restored in the output image without much modification due to the appearance loss. The adversarial loss ensures output images are realistic. The proposed network is trained without supervisory signals since no labels of key parts are required. Experimental results on seven datasets demonstrate that the proposed algorithm performs favorably on challenging objects and scenes.

Original languageEnglish
Title of host publicationComputer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings
EditorsVittorio Ferrari, Cristian Sminchisescu, Martial Hebert, Yair Weiss
PublisherSpringer Verlag
Pages21-37
Number of pages17
ISBN (Print)9783030012274
DOIs
Publication statusPublished - 2018 Jan 1
Event15th European Conference on Computer Vision, ECCV 2018 - Munich, Germany
Duration: 2018 Sep 82018 Sep 14

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11209 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other15th European Conference on Computer Vision, ECCV 2018
CountryGermany
CityMunich
Period18/9/818/9/14

Fingerprint

Patch
Discriminators
Labels
Masks
Output
Encoder
Mask
Generator
Target
Experimental Results
Demonstrate
Object

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Lee, D., Yun, S., Choi, S., Yoo, H., Yang, M. H., & Oh, S. (2018). Unsupervised Holistic Image Generation from Key Local Patches. In V. Ferrari, C. Sminchisescu, M. Hebert, & Y. Weiss (Eds.), Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings (pp. 21-37). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11209 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-030-01228-1_2
Lee, Donghoon ; Yun, Sangdoo ; Choi, Sungjoon ; Yoo, Hwiyeon ; Yang, Ming Hsuan ; Oh, Songhwai. / Unsupervised Holistic Image Generation from Key Local Patches. Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings. editor / Vittorio Ferrari ; Cristian Sminchisescu ; Martial Hebert ; Yair Weiss. Springer Verlag, 2018. pp. 21-37 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{f4334ff9d3dd4ae08de7c95040676d83,
title = "Unsupervised Holistic Image Generation from Key Local Patches",
abstract = "We introduce a new problem of generating an image based on a small number of key local patches without any geometric prior. In this work, key local patches are defined as informative regions of the target object or scene. This is a challenging problem since it requires generating realistic images and predicting locations of parts at the same time. We construct adversarial networks to tackle this problem. A generator network generates a fake image as well as a mask based on the encoder-decoder framework. On the other hand, a discriminator network aims to detect fake images. The network is trained with three losses to consider spatial, appearance, and adversarial information. The spatial loss determines whether the locations of predicted parts are correct. Input patches are restored in the output image without much modification due to the appearance loss. The adversarial loss ensures output images are realistic. The proposed network is trained without supervisory signals since no labels of key parts are required. Experimental results on seven datasets demonstrate that the proposed algorithm performs favorably on challenging objects and scenes.",
author = "Donghoon Lee and Sangdoo Yun and Sungjoon Choi and Hwiyeon Yoo and Yang, {Ming Hsuan} and Songhwai Oh",
year = "2018",
month = "1",
day = "1",
doi = "10.1007/978-3-030-01228-1_2",
language = "English",
isbn = "9783030012274",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "21--37",
editor = "Vittorio Ferrari and Cristian Sminchisescu and Martial Hebert and Yair Weiss",
booktitle = "Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings",
address = "Germany",

}

Lee, D, Yun, S, Choi, S, Yoo, H, Yang, MH & Oh, S 2018, Unsupervised Holistic Image Generation from Key Local Patches. in V Ferrari, C Sminchisescu, M Hebert & Y Weiss (eds), Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11209 LNCS, Springer Verlag, pp. 21-37, 15th European Conference on Computer Vision, ECCV 2018, Munich, Germany, 18/9/8. https://doi.org/10.1007/978-3-030-01228-1_2

Unsupervised Holistic Image Generation from Key Local Patches. / Lee, Donghoon; Yun, Sangdoo; Choi, Sungjoon; Yoo, Hwiyeon; Yang, Ming Hsuan; Oh, Songhwai.

Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings. ed. / Vittorio Ferrari; Cristian Sminchisescu; Martial Hebert; Yair Weiss. Springer Verlag, 2018. p. 21-37 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11209 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Unsupervised Holistic Image Generation from Key Local Patches

AU - Lee, Donghoon

AU - Yun, Sangdoo

AU - Choi, Sungjoon

AU - Yoo, Hwiyeon

AU - Yang, Ming Hsuan

AU - Oh, Songhwai

PY - 2018/1/1

Y1 - 2018/1/1

N2 - We introduce a new problem of generating an image based on a small number of key local patches without any geometric prior. In this work, key local patches are defined as informative regions of the target object or scene. This is a challenging problem since it requires generating realistic images and predicting locations of parts at the same time. We construct adversarial networks to tackle this problem. A generator network generates a fake image as well as a mask based on the encoder-decoder framework. On the other hand, a discriminator network aims to detect fake images. The network is trained with three losses to consider spatial, appearance, and adversarial information. The spatial loss determines whether the locations of predicted parts are correct. Input patches are restored in the output image without much modification due to the appearance loss. The adversarial loss ensures output images are realistic. The proposed network is trained without supervisory signals since no labels of key parts are required. Experimental results on seven datasets demonstrate that the proposed algorithm performs favorably on challenging objects and scenes.

AB - We introduce a new problem of generating an image based on a small number of key local patches without any geometric prior. In this work, key local patches are defined as informative regions of the target object or scene. This is a challenging problem since it requires generating realistic images and predicting locations of parts at the same time. We construct adversarial networks to tackle this problem. A generator network generates a fake image as well as a mask based on the encoder-decoder framework. On the other hand, a discriminator network aims to detect fake images. The network is trained with three losses to consider spatial, appearance, and adversarial information. The spatial loss determines whether the locations of predicted parts are correct. Input patches are restored in the output image without much modification due to the appearance loss. The adversarial loss ensures output images are realistic. The proposed network is trained without supervisory signals since no labels of key parts are required. Experimental results on seven datasets demonstrate that the proposed algorithm performs favorably on challenging objects and scenes.

UR - http://www.scopus.com/inward/record.url?scp=85055084000&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85055084000&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-01228-1_2

DO - 10.1007/978-3-030-01228-1_2

M3 - Conference contribution

AN - SCOPUS:85055084000

SN - 9783030012274

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 21

EP - 37

BT - Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings

A2 - Ferrari, Vittorio

A2 - Sminchisescu, Cristian

A2 - Hebert, Martial

A2 - Weiss, Yair

PB - Springer Verlag

ER -

Lee D, Yun S, Choi S, Yoo H, Yang MH, Oh S. Unsupervised Holistic Image Generation from Key Local Patches. In Ferrari V, Sminchisescu C, Hebert M, Weiss Y, editors, Computer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings. Springer Verlag. 2018. p. 21-37. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-01228-1_2