Learning affinity via spatial propagation networks

Sifei Liu, Shalini De Mello, Jinwei Gu, Guangyu Zhong, Ming Hsuan Yang, Jan Kautz

Research output: Contribution to journalConference article

28 Citations (Scopus)

Abstract

In this paper, we propose spatial propagation networks for learning the affinity matrix for vision tasks. We show that by constructing a row/column linear propagation model, the spatially varying transformation matrix exactly constitutes an affinity matrix that models dense, global pairwise relationships of an image. Specifically, we develop a three-way connection for the linear propagation model, which (a) formulates a sparse transformation matrix, where all elements can be outputs from a deep CNN, but (b) results in a dense affinity matrix that effectively models any task-specific pairwise similarity matrix. Instead of designing the similarity kernels according to image features of two points, we can directly output all the similarities in a purely data-driven manner. The spatial propagation network is a generic framework that can be applied to many affinity-related tasks, such as image matting, segmentation and colorization, to name a few. Essentially, the model can learn semantically-aware affinity values for high-level vision tasks due to the powerful learning capability of deep CNNs. We validate the framework on the task of refinement of image segmentation boundaries. Experiments on the HELEN face parsing and PASCAL VOC-2012 semantic segmentation tasks show that the spatial propagation network provides a general, effective and efficient solution for generating high-quality segmentation results.

Original languageEnglish
Pages (from-to)1521-1531
Number of pages11
JournalAdvances in Neural Information Processing Systems
Volume2017-December
Publication statusPublished - 2017 Jan 1
Event31st Annual Conference on Neural Information Processing Systems, NIPS 2017 - Long Beach, United States
Duration: 2017 Dec 42017 Dec 9

Fingerprint

Image segmentation
Volatile organic compounds
Semantics
Experiments

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Information Systems
  • Signal Processing

Cite this

Liu, S., De Mello, S., Gu, J., Zhong, G., Yang, M. H., & Kautz, J. (2017). Learning affinity via spatial propagation networks. Advances in Neural Information Processing Systems, 2017-December, 1521-1531.
Liu, Sifei ; De Mello, Shalini ; Gu, Jinwei ; Zhong, Guangyu ; Yang, Ming Hsuan ; Kautz, Jan. / Learning affinity via spatial propagation networks. In: Advances in Neural Information Processing Systems. 2017 ; Vol. 2017-December. pp. 1521-1531.
@article{0b5f73794686450ca9282ef4863691d2,
title = "Learning affinity via spatial propagation networks",
abstract = "In this paper, we propose spatial propagation networks for learning the affinity matrix for vision tasks. We show that by constructing a row/column linear propagation model, the spatially varying transformation matrix exactly constitutes an affinity matrix that models dense, global pairwise relationships of an image. Specifically, we develop a three-way connection for the linear propagation model, which (a) formulates a sparse transformation matrix, where all elements can be outputs from a deep CNN, but (b) results in a dense affinity matrix that effectively models any task-specific pairwise similarity matrix. Instead of designing the similarity kernels according to image features of two points, we can directly output all the similarities in a purely data-driven manner. The spatial propagation network is a generic framework that can be applied to many affinity-related tasks, such as image matting, segmentation and colorization, to name a few. Essentially, the model can learn semantically-aware affinity values for high-level vision tasks due to the powerful learning capability of deep CNNs. We validate the framework on the task of refinement of image segmentation boundaries. Experiments on the HELEN face parsing and PASCAL VOC-2012 semantic segmentation tasks show that the spatial propagation network provides a general, effective and efficient solution for generating high-quality segmentation results.",
author = "Sifei Liu and {De Mello}, Shalini and Jinwei Gu and Guangyu Zhong and Yang, {Ming Hsuan} and Jan Kautz",
year = "2017",
month = "1",
day = "1",
language = "English",
volume = "2017-December",
pages = "1521--1531",
journal = "Advances in Neural Information Processing Systems",
issn = "1049-5258",

}

Liu, S, De Mello, S, Gu, J, Zhong, G, Yang, MH & Kautz, J 2017, 'Learning affinity via spatial propagation networks', Advances in Neural Information Processing Systems, vol. 2017-December, pp. 1521-1531.

Learning affinity via spatial propagation networks. / Liu, Sifei; De Mello, Shalini; Gu, Jinwei; Zhong, Guangyu; Yang, Ming Hsuan; Kautz, Jan.

In: Advances in Neural Information Processing Systems, Vol. 2017-December, 01.01.2017, p. 1521-1531.

Research output: Contribution to journalConference article

TY - JOUR

T1 - Learning affinity via spatial propagation networks

AU - Liu, Sifei

AU - De Mello, Shalini

AU - Gu, Jinwei

AU - Zhong, Guangyu

AU - Yang, Ming Hsuan

AU - Kautz, Jan

PY - 2017/1/1

Y1 - 2017/1/1

N2 - In this paper, we propose spatial propagation networks for learning the affinity matrix for vision tasks. We show that by constructing a row/column linear propagation model, the spatially varying transformation matrix exactly constitutes an affinity matrix that models dense, global pairwise relationships of an image. Specifically, we develop a three-way connection for the linear propagation model, which (a) formulates a sparse transformation matrix, where all elements can be outputs from a deep CNN, but (b) results in a dense affinity matrix that effectively models any task-specific pairwise similarity matrix. Instead of designing the similarity kernels according to image features of two points, we can directly output all the similarities in a purely data-driven manner. The spatial propagation network is a generic framework that can be applied to many affinity-related tasks, such as image matting, segmentation and colorization, to name a few. Essentially, the model can learn semantically-aware affinity values for high-level vision tasks due to the powerful learning capability of deep CNNs. We validate the framework on the task of refinement of image segmentation boundaries. Experiments on the HELEN face parsing and PASCAL VOC-2012 semantic segmentation tasks show that the spatial propagation network provides a general, effective and efficient solution for generating high-quality segmentation results.

AB - In this paper, we propose spatial propagation networks for learning the affinity matrix for vision tasks. We show that by constructing a row/column linear propagation model, the spatially varying transformation matrix exactly constitutes an affinity matrix that models dense, global pairwise relationships of an image. Specifically, we develop a three-way connection for the linear propagation model, which (a) formulates a sparse transformation matrix, where all elements can be outputs from a deep CNN, but (b) results in a dense affinity matrix that effectively models any task-specific pairwise similarity matrix. Instead of designing the similarity kernels according to image features of two points, we can directly output all the similarities in a purely data-driven manner. The spatial propagation network is a generic framework that can be applied to many affinity-related tasks, such as image matting, segmentation and colorization, to name a few. Essentially, the model can learn semantically-aware affinity values for high-level vision tasks due to the powerful learning capability of deep CNNs. We validate the framework on the task of refinement of image segmentation boundaries. Experiments on the HELEN face parsing and PASCAL VOC-2012 semantic segmentation tasks show that the spatial propagation network provides a general, effective and efficient solution for generating high-quality segmentation results.

UR - http://www.scopus.com/inward/record.url?scp=85046998756&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85046998756&partnerID=8YFLogxK

M3 - Conference article

AN - SCOPUS:85046998756

VL - 2017-December

SP - 1521

EP - 1531

JO - Advances in Neural Information Processing Systems

JF - Advances in Neural Information Processing Systems

SN - 1049-5258

ER -

Liu S, De Mello S, Gu J, Zhong G, Yang MH, Kautz J. Learning affinity via spatial propagation networks. Advances in Neural Information Processing Systems. 2017 Jan 1;2017-December:1521-1531.