Modality-invariant image classification based on modality uniqueness and dictionary learning

Seungryong Kim, Rui Cai, Kihong Park, Sunok Kim, Kwanghoon Sohn

Research output: Contribution to journalArticle

3 Citations (Scopus)

Abstract

We present a unified framework for the image classification of image sets taken under varying modality conditions. Our method is motivated by a key observation that the image feature distribution is simultaneously influenced by the semantic-class and the modality category label, which limits the performance of conventional methods for that task. With this insight, we introduce modality uniqueness as a discriminative weight that divides each modality cluster from all other clusters. By leveraging the modality uniqueness, our framework is formulated as unsupervised modality clustering and classifier learning based on modality-invariant similarity kernel. Specifically, in the assignment step, each training image is first assigned to the most similar cluster according to its modality. In the update step, based on the current cluster hypothesis, the modality uniqueness and the sparse dictionary are updated. These two steps are formulated in an iterative manner. Based on the final clusters, a modality-invariant marginalized kernel is then computed, where the similarities between the reconstructed features of each modality are aggregated across all clusters. Our framework enables the reliable inference of semantic-class category for an image, even across large photometric variations. Experimental results show that our method outperforms conventional methods on various benchmarks, such as landmark identification under severely varying weather conditions, domain-adapting image classification, and RGB and near-infrared image classification.

Original languageEnglish
Article number7765067
Pages (from-to)844-899
Number of pages56
JournalIEEE Transactions on Image Processing
Volume26
Issue number2
DOIs
Publication statusPublished - 2017 Feb 1

Fingerprint

Image classification
Glossaries
Semantics
Labels
Classifiers
Infrared radiation

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Graphics and Computer-Aided Design

Cite this

Kim, Seungryong ; Cai, Rui ; Park, Kihong ; Kim, Sunok ; Sohn, Kwanghoon. / Modality-invariant image classification based on modality uniqueness and dictionary learning. In: IEEE Transactions on Image Processing. 2017 ; Vol. 26, No. 2. pp. 844-899.
@article{0cc7bd2b05254408a2749aee925628f0,
title = "Modality-invariant image classification based on modality uniqueness and dictionary learning",
abstract = "We present a unified framework for the image classification of image sets taken under varying modality conditions. Our method is motivated by a key observation that the image feature distribution is simultaneously influenced by the semantic-class and the modality category label, which limits the performance of conventional methods for that task. With this insight, we introduce modality uniqueness as a discriminative weight that divides each modality cluster from all other clusters. By leveraging the modality uniqueness, our framework is formulated as unsupervised modality clustering and classifier learning based on modality-invariant similarity kernel. Specifically, in the assignment step, each training image is first assigned to the most similar cluster according to its modality. In the update step, based on the current cluster hypothesis, the modality uniqueness and the sparse dictionary are updated. These two steps are formulated in an iterative manner. Based on the final clusters, a modality-invariant marginalized kernel is then computed, where the similarities between the reconstructed features of each modality are aggregated across all clusters. Our framework enables the reliable inference of semantic-class category for an image, even across large photometric variations. Experimental results show that our method outperforms conventional methods on various benchmarks, such as landmark identification under severely varying weather conditions, domain-adapting image classification, and RGB and near-infrared image classification.",
author = "Seungryong Kim and Rui Cai and Kihong Park and Sunok Kim and Kwanghoon Sohn",
year = "2017",
month = "2",
day = "1",
doi = "10.1109/TIP.2016.2635444",
language = "English",
volume = "26",
pages = "844--899",
journal = "IEEE Transactions on Image Processing",
issn = "1057-7149",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "2",

}

Modality-invariant image classification based on modality uniqueness and dictionary learning. / Kim, Seungryong; Cai, Rui; Park, Kihong; Kim, Sunok; Sohn, Kwanghoon.

In: IEEE Transactions on Image Processing, Vol. 26, No. 2, 7765067, 01.02.2017, p. 844-899.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Modality-invariant image classification based on modality uniqueness and dictionary learning

AU - Kim, Seungryong

AU - Cai, Rui

AU - Park, Kihong

AU - Kim, Sunok

AU - Sohn, Kwanghoon

PY - 2017/2/1

Y1 - 2017/2/1

N2 - We present a unified framework for the image classification of image sets taken under varying modality conditions. Our method is motivated by a key observation that the image feature distribution is simultaneously influenced by the semantic-class and the modality category label, which limits the performance of conventional methods for that task. With this insight, we introduce modality uniqueness as a discriminative weight that divides each modality cluster from all other clusters. By leveraging the modality uniqueness, our framework is formulated as unsupervised modality clustering and classifier learning based on modality-invariant similarity kernel. Specifically, in the assignment step, each training image is first assigned to the most similar cluster according to its modality. In the update step, based on the current cluster hypothesis, the modality uniqueness and the sparse dictionary are updated. These two steps are formulated in an iterative manner. Based on the final clusters, a modality-invariant marginalized kernel is then computed, where the similarities between the reconstructed features of each modality are aggregated across all clusters. Our framework enables the reliable inference of semantic-class category for an image, even across large photometric variations. Experimental results show that our method outperforms conventional methods on various benchmarks, such as landmark identification under severely varying weather conditions, domain-adapting image classification, and RGB and near-infrared image classification.

AB - We present a unified framework for the image classification of image sets taken under varying modality conditions. Our method is motivated by a key observation that the image feature distribution is simultaneously influenced by the semantic-class and the modality category label, which limits the performance of conventional methods for that task. With this insight, we introduce modality uniqueness as a discriminative weight that divides each modality cluster from all other clusters. By leveraging the modality uniqueness, our framework is formulated as unsupervised modality clustering and classifier learning based on modality-invariant similarity kernel. Specifically, in the assignment step, each training image is first assigned to the most similar cluster according to its modality. In the update step, based on the current cluster hypothesis, the modality uniqueness and the sparse dictionary are updated. These two steps are formulated in an iterative manner. Based on the final clusters, a modality-invariant marginalized kernel is then computed, where the similarities between the reconstructed features of each modality are aggregated across all clusters. Our framework enables the reliable inference of semantic-class category for an image, even across large photometric variations. Experimental results show that our method outperforms conventional methods on various benchmarks, such as landmark identification under severely varying weather conditions, domain-adapting image classification, and RGB and near-infrared image classification.

UR - http://www.scopus.com/inward/record.url?scp=85015145690&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85015145690&partnerID=8YFLogxK

U2 - 10.1109/TIP.2016.2635444

DO - 10.1109/TIP.2016.2635444

M3 - Article

VL - 26

SP - 844

EP - 899

JO - IEEE Transactions on Image Processing

JF - IEEE Transactions on Image Processing

SN - 1057-7149

IS - 2

M1 - 7765067

ER -