Multispectral human co-segmentation via joint convolutional neural networks

Sungil Choi, Seungryong Kim, Kihong Park, Kwanghoon Sohn

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

We present a novel human body co-segmentation method for unregistered multispectral, color and thermal, images by leveraging CNNs. The main challenges for that tasks are no-alignment between color and thermal images and an absent of ground truth human segmentation labels. To solve these limitations, our key-insight is to formulate the segmentation network for each modality that solve two sub-tasks, correspondence and classification, in a joint and iterative manner. We formulate the learning framework between multispectral images in a way that training labels for one modality are used to learn the network for the other modality. We estimate dense correspondences between multispectral image pairs using intermediate convolutional activations of CNNs and perform human segmentation for each modality through the conditional random fields (CRF) optimization using unary and pairwise fusion. These two steps are formulated as an iterative framework, enables the network to converge on an optimal solution. Experimental results show that our proposed method outperforms conventional state-of-the-art methods on the VAP benchmark consisting of unregistered multispectral color and thermal images.

Original languageEnglish
Title of host publication2017 IEEE International Conference on Image Processing, ICIP 2017 - Proceedings
PublisherIEEE Computer Society
Pages3115-3119
Number of pages5
ISBN (Electronic)9781509021758
DOIs
Publication statusPublished - 2018 Feb 20
Event24th IEEE International Conference on Image Processing, ICIP 2017 - Beijing, China
Duration: 2017 Sep 172017 Sep 20

Publication series

NameProceedings - International Conference on Image Processing, ICIP
Volume2017-September
ISSN (Print)1522-4880

Other

Other24th IEEE International Conference on Image Processing, ICIP 2017
CountryChina
CityBeijing
Period17/9/1717/9/20

Fingerprint

Color
Neural networks
Labels
Fusion reactions
Chemical activation
Hot Temperature

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Vision and Pattern Recognition
  • Signal Processing

Cite this

Choi, S., Kim, S., Park, K., & Sohn, K. (2018). Multispectral human co-segmentation via joint convolutional neural networks. In 2017 IEEE International Conference on Image Processing, ICIP 2017 - Proceedings (pp. 3115-3119). (Proceedings - International Conference on Image Processing, ICIP; Vol. 2017-September). IEEE Computer Society. https://doi.org/10.1109/ICIP.2017.8296856
Choi, Sungil ; Kim, Seungryong ; Park, Kihong ; Sohn, Kwanghoon. / Multispectral human co-segmentation via joint convolutional neural networks. 2017 IEEE International Conference on Image Processing, ICIP 2017 - Proceedings. IEEE Computer Society, 2018. pp. 3115-3119 (Proceedings - International Conference on Image Processing, ICIP).
@inproceedings{4c6acb582c5049fa8532f83127f7cab4,
title = "Multispectral human co-segmentation via joint convolutional neural networks",
abstract = "We present a novel human body co-segmentation method for unregistered multispectral, color and thermal, images by leveraging CNNs. The main challenges for that tasks are no-alignment between color and thermal images and an absent of ground truth human segmentation labels. To solve these limitations, our key-insight is to formulate the segmentation network for each modality that solve two sub-tasks, correspondence and classification, in a joint and iterative manner. We formulate the learning framework between multispectral images in a way that training labels for one modality are used to learn the network for the other modality. We estimate dense correspondences between multispectral image pairs using intermediate convolutional activations of CNNs and perform human segmentation for each modality through the conditional random fields (CRF) optimization using unary and pairwise fusion. These two steps are formulated as an iterative framework, enables the network to converge on an optimal solution. Experimental results show that our proposed method outperforms conventional state-of-the-art methods on the VAP benchmark consisting of unregistered multispectral color and thermal images.",
author = "Sungil Choi and Seungryong Kim and Kihong Park and Kwanghoon Sohn",
year = "2018",
month = "2",
day = "20",
doi = "10.1109/ICIP.2017.8296856",
language = "English",
series = "Proceedings - International Conference on Image Processing, ICIP",
publisher = "IEEE Computer Society",
pages = "3115--3119",
booktitle = "2017 IEEE International Conference on Image Processing, ICIP 2017 - Proceedings",
address = "United States",

}

Choi, S, Kim, S, Park, K & Sohn, K 2018, Multispectral human co-segmentation via joint convolutional neural networks. in 2017 IEEE International Conference on Image Processing, ICIP 2017 - Proceedings. Proceedings - International Conference on Image Processing, ICIP, vol. 2017-September, IEEE Computer Society, pp. 3115-3119, 24th IEEE International Conference on Image Processing, ICIP 2017, Beijing, China, 17/9/17. https://doi.org/10.1109/ICIP.2017.8296856

Multispectral human co-segmentation via joint convolutional neural networks. / Choi, Sungil; Kim, Seungryong; Park, Kihong; Sohn, Kwanghoon.

2017 IEEE International Conference on Image Processing, ICIP 2017 - Proceedings. IEEE Computer Society, 2018. p. 3115-3119 (Proceedings - International Conference on Image Processing, ICIP; Vol. 2017-September).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Multispectral human co-segmentation via joint convolutional neural networks

AU - Choi, Sungil

AU - Kim, Seungryong

AU - Park, Kihong

AU - Sohn, Kwanghoon

PY - 2018/2/20

Y1 - 2018/2/20

N2 - We present a novel human body co-segmentation method for unregistered multispectral, color and thermal, images by leveraging CNNs. The main challenges for that tasks are no-alignment between color and thermal images and an absent of ground truth human segmentation labels. To solve these limitations, our key-insight is to formulate the segmentation network for each modality that solve two sub-tasks, correspondence and classification, in a joint and iterative manner. We formulate the learning framework between multispectral images in a way that training labels for one modality are used to learn the network for the other modality. We estimate dense correspondences between multispectral image pairs using intermediate convolutional activations of CNNs and perform human segmentation for each modality through the conditional random fields (CRF) optimization using unary and pairwise fusion. These two steps are formulated as an iterative framework, enables the network to converge on an optimal solution. Experimental results show that our proposed method outperforms conventional state-of-the-art methods on the VAP benchmark consisting of unregistered multispectral color and thermal images.

AB - We present a novel human body co-segmentation method for unregistered multispectral, color and thermal, images by leveraging CNNs. The main challenges for that tasks are no-alignment between color and thermal images and an absent of ground truth human segmentation labels. To solve these limitations, our key-insight is to formulate the segmentation network for each modality that solve two sub-tasks, correspondence and classification, in a joint and iterative manner. We formulate the learning framework between multispectral images in a way that training labels for one modality are used to learn the network for the other modality. We estimate dense correspondences between multispectral image pairs using intermediate convolutional activations of CNNs and perform human segmentation for each modality through the conditional random fields (CRF) optimization using unary and pairwise fusion. These two steps are formulated as an iterative framework, enables the network to converge on an optimal solution. Experimental results show that our proposed method outperforms conventional state-of-the-art methods on the VAP benchmark consisting of unregistered multispectral color and thermal images.

UR - http://www.scopus.com/inward/record.url?scp=85045326472&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85045326472&partnerID=8YFLogxK

U2 - 10.1109/ICIP.2017.8296856

DO - 10.1109/ICIP.2017.8296856

M3 - Conference contribution

T3 - Proceedings - International Conference on Image Processing, ICIP

SP - 3115

EP - 3119

BT - 2017 IEEE International Conference on Image Processing, ICIP 2017 - Proceedings

PB - IEEE Computer Society

ER -

Choi S, Kim S, Park K, Sohn K. Multispectral human co-segmentation via joint convolutional neural networks. In 2017 IEEE International Conference on Image Processing, ICIP 2017 - Proceedings. IEEE Computer Society. 2018. p. 3115-3119. (Proceedings - International Conference on Image Processing, ICIP). https://doi.org/10.1109/ICIP.2017.8296856