Context-aware synthesis and placement of object instances

Donghoon Lee, Sifei Liu, Jinwei Gu, Ming Yu Liu, Ming Hsuan Yang, Jan Kautz

Research output: Contribution to journalConference article

1 Citation (Scopus)

Abstract

Learning to insert an object instance into an image in a semantically coherent manner is a challenging and interesting problem. Solving it requires (a) determining a location to place an object in the scene and (b) determining its appearance at the location. Such an object insertion model can potentially facilitate numerous image editing and scene parsing applications. In this paper, we propose an end-to-end trainable neural network for the task of inserting an object instance mask of a specified class into the semantic label map of an image. Our network consists of two generative modules where one determines where the inserted object mask should be (i.e., location and scale) and the other determines what the object mask shape (and pose) should look like. The two modules are connected together via a spatial transformation network and jointly trained. We devise a learning procedure that leverage both supervised and unsupervised data and show our model can insert an object at diverse locations with various appearances. We conduct extensive experimental validations with comparisons to strong baselines to verify the effectiveness of the proposed network. Code is available at https://github.com/NVlabs/Instance_Insertion.

Original languageEnglish
Pages (from-to)10393-10403
Number of pages11
JournalAdvances in Neural Information Processing Systems
Volume2018-December
Publication statusPublished - 2018 Jan 1
Event32nd Conference on Neural Information Processing Systems, NeurIPS 2018 - Montreal, Canada
Duration: 2018 Dec 22018 Dec 8

Fingerprint

Masks
Labels
Semantics
Neural networks

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Information Systems
  • Signal Processing

Cite this

Lee, D., Liu, S., Gu, J., Liu, M. Y., Yang, M. H., & Kautz, J. (2018). Context-aware synthesis and placement of object instances. Advances in Neural Information Processing Systems, 2018-December, 10393-10403.
Lee, Donghoon ; Liu, Sifei ; Gu, Jinwei ; Liu, Ming Yu ; Yang, Ming Hsuan ; Kautz, Jan. / Context-aware synthesis and placement of object instances. In: Advances in Neural Information Processing Systems. 2018 ; Vol. 2018-December. pp. 10393-10403.
@article{0f37b53515e446b2a1d33bc2448f77b5,
title = "Context-aware synthesis and placement of object instances",
abstract = "Learning to insert an object instance into an image in a semantically coherent manner is a challenging and interesting problem. Solving it requires (a) determining a location to place an object in the scene and (b) determining its appearance at the location. Such an object insertion model can potentially facilitate numerous image editing and scene parsing applications. In this paper, we propose an end-to-end trainable neural network for the task of inserting an object instance mask of a specified class into the semantic label map of an image. Our network consists of two generative modules where one determines where the inserted object mask should be (i.e., location and scale) and the other determines what the object mask shape (and pose) should look like. The two modules are connected together via a spatial transformation network and jointly trained. We devise a learning procedure that leverage both supervised and unsupervised data and show our model can insert an object at diverse locations with various appearances. We conduct extensive experimental validations with comparisons to strong baselines to verify the effectiveness of the proposed network. Code is available at https://github.com/NVlabs/Instance_Insertion.",
author = "Donghoon Lee and Sifei Liu and Jinwei Gu and Liu, {Ming Yu} and Yang, {Ming Hsuan} and Jan Kautz",
year = "2018",
month = "1",
day = "1",
language = "English",
volume = "2018-December",
pages = "10393--10403",
journal = "Advances in Neural Information Processing Systems",
issn = "1049-5258",

}

Lee, D, Liu, S, Gu, J, Liu, MY, Yang, MH & Kautz, J 2018, 'Context-aware synthesis and placement of object instances', Advances in Neural Information Processing Systems, vol. 2018-December, pp. 10393-10403.

Context-aware synthesis and placement of object instances. / Lee, Donghoon; Liu, Sifei; Gu, Jinwei; Liu, Ming Yu; Yang, Ming Hsuan; Kautz, Jan.

In: Advances in Neural Information Processing Systems, Vol. 2018-December, 01.01.2018, p. 10393-10403.

Research output: Contribution to journalConference article

TY - JOUR

T1 - Context-aware synthesis and placement of object instances

AU - Lee, Donghoon

AU - Liu, Sifei

AU - Gu, Jinwei

AU - Liu, Ming Yu

AU - Yang, Ming Hsuan

AU - Kautz, Jan

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Learning to insert an object instance into an image in a semantically coherent manner is a challenging and interesting problem. Solving it requires (a) determining a location to place an object in the scene and (b) determining its appearance at the location. Such an object insertion model can potentially facilitate numerous image editing and scene parsing applications. In this paper, we propose an end-to-end trainable neural network for the task of inserting an object instance mask of a specified class into the semantic label map of an image. Our network consists of two generative modules where one determines where the inserted object mask should be (i.e., location and scale) and the other determines what the object mask shape (and pose) should look like. The two modules are connected together via a spatial transformation network and jointly trained. We devise a learning procedure that leverage both supervised and unsupervised data and show our model can insert an object at diverse locations with various appearances. We conduct extensive experimental validations with comparisons to strong baselines to verify the effectiveness of the proposed network. Code is available at https://github.com/NVlabs/Instance_Insertion.

AB - Learning to insert an object instance into an image in a semantically coherent manner is a challenging and interesting problem. Solving it requires (a) determining a location to place an object in the scene and (b) determining its appearance at the location. Such an object insertion model can potentially facilitate numerous image editing and scene parsing applications. In this paper, we propose an end-to-end trainable neural network for the task of inserting an object instance mask of a specified class into the semantic label map of an image. Our network consists of two generative modules where one determines where the inserted object mask should be (i.e., location and scale) and the other determines what the object mask shape (and pose) should look like. The two modules are connected together via a spatial transformation network and jointly trained. We devise a learning procedure that leverage both supervised and unsupervised data and show our model can insert an object at diverse locations with various appearances. We conduct extensive experimental validations with comparisons to strong baselines to verify the effectiveness of the proposed network. Code is available at https://github.com/NVlabs/Instance_Insertion.

UR - http://www.scopus.com/inward/record.url?scp=85064835258&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85064835258&partnerID=8YFLogxK

M3 - Conference article

AN - SCOPUS:85064835258

VL - 2018-December

SP - 10393

EP - 10403

JO - Advances in Neural Information Processing Systems

JF - Advances in Neural Information Processing Systems

SN - 1049-5258

ER -

Lee D, Liu S, Gu J, Liu MY, Yang MH, Kautz J. Context-aware synthesis and placement of object instances. Advances in Neural Information Processing Systems. 2018 Jan 1;2018-December:10393-10403.