Dynamic Match Kernel with Deep Convolutional Features for Image Retrieval

Jufeng Yang, Jie Liang, Hui Shen, Kai Wang, Paul L. Rosin, Ming Hsuan Yang

Research output: Contribution to journalArticle

8 Citations (Scopus)

Abstract

For image retrieval methods based on bag of visual words, much attention has been paid to enhancing the discriminative powers of the local features. Although retrieved images are usually similar to a query in minutiae, they may be significantly different from a semantic perspective, which can be effectively distinguished by convolutional neural networks (CNN). Such images should not be considered as relevant pairs. To tackle this problem, we propose to construct a dynamic match kernel by adaptively calculating the matching thresholds between query and candidate images based on the pairwise distance among deep CNN features. In contrast to the typical static match kernel which is independent to the global appearance of retrieved images, the dynamic one leverages the semantical similarity as a constraint for determining the matches. Accordingly, we propose a semantic-constrained retrieval framework by incorporating the dynamic match kernel, which focuses on matched patches between relevant images and filters out the ones for irrelevant pairs. Furthermore, we demonstrate that the proposed kernel complements recent methods, such as hamming embedding, multiple assignment, local descriptors aggregation, and graph-based re-ranking, while it outperforms the static one under various settings on off-the-shelf evaluation metrics. We also propose to evaluate the matched patches both quantitatively and qualitatively. Extensive experiments on five benchmark data sets and large-scale distractors validate the merits of the proposed method against the state-of-the-art methods for image retrieval.

Original languageEnglish
Article number8374838
Pages (from-to)5288-5302
Number of pages15
JournalIEEE Transactions on Image Processing
Volume27
Issue number11
DOIs
Publication statusPublished - 2018 Nov

Fingerprint

Image retrieval
Semantics
Neural networks
Agglomeration
Experiments

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Graphics and Computer-Aided Design

Cite this

Yang, Jufeng ; Liang, Jie ; Shen, Hui ; Wang, Kai ; Rosin, Paul L. ; Yang, Ming Hsuan. / Dynamic Match Kernel with Deep Convolutional Features for Image Retrieval. In: IEEE Transactions on Image Processing. 2018 ; Vol. 27, No. 11. pp. 5288-5302.
@article{9dab9f246cc34570bdf73187c92c586c,
title = "Dynamic Match Kernel with Deep Convolutional Features for Image Retrieval",
abstract = "For image retrieval methods based on bag of visual words, much attention has been paid to enhancing the discriminative powers of the local features. Although retrieved images are usually similar to a query in minutiae, they may be significantly different from a semantic perspective, which can be effectively distinguished by convolutional neural networks (CNN). Such images should not be considered as relevant pairs. To tackle this problem, we propose to construct a dynamic match kernel by adaptively calculating the matching thresholds between query and candidate images based on the pairwise distance among deep CNN features. In contrast to the typical static match kernel which is independent to the global appearance of retrieved images, the dynamic one leverages the semantical similarity as a constraint for determining the matches. Accordingly, we propose a semantic-constrained retrieval framework by incorporating the dynamic match kernel, which focuses on matched patches between relevant images and filters out the ones for irrelevant pairs. Furthermore, we demonstrate that the proposed kernel complements recent methods, such as hamming embedding, multiple assignment, local descriptors aggregation, and graph-based re-ranking, while it outperforms the static one under various settings on off-the-shelf evaluation metrics. We also propose to evaluate the matched patches both quantitatively and qualitatively. Extensive experiments on five benchmark data sets and large-scale distractors validate the merits of the proposed method against the state-of-the-art methods for image retrieval.",
author = "Jufeng Yang and Jie Liang and Hui Shen and Kai Wang and Rosin, {Paul L.} and Yang, {Ming Hsuan}",
year = "2018",
month = "11",
doi = "10.1109/TIP.2018.2845136",
language = "English",
volume = "27",
pages = "5288--5302",
journal = "IEEE Transactions on Image Processing",
issn = "1057-7149",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "11",

}

Dynamic Match Kernel with Deep Convolutional Features for Image Retrieval. / Yang, Jufeng; Liang, Jie; Shen, Hui; Wang, Kai; Rosin, Paul L.; Yang, Ming Hsuan.

In: IEEE Transactions on Image Processing, Vol. 27, No. 11, 8374838, 11.2018, p. 5288-5302.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Dynamic Match Kernel with Deep Convolutional Features for Image Retrieval

AU - Yang, Jufeng

AU - Liang, Jie

AU - Shen, Hui

AU - Wang, Kai

AU - Rosin, Paul L.

AU - Yang, Ming Hsuan

PY - 2018/11

Y1 - 2018/11

N2 - For image retrieval methods based on bag of visual words, much attention has been paid to enhancing the discriminative powers of the local features. Although retrieved images are usually similar to a query in minutiae, they may be significantly different from a semantic perspective, which can be effectively distinguished by convolutional neural networks (CNN). Such images should not be considered as relevant pairs. To tackle this problem, we propose to construct a dynamic match kernel by adaptively calculating the matching thresholds between query and candidate images based on the pairwise distance among deep CNN features. In contrast to the typical static match kernel which is independent to the global appearance of retrieved images, the dynamic one leverages the semantical similarity as a constraint for determining the matches. Accordingly, we propose a semantic-constrained retrieval framework by incorporating the dynamic match kernel, which focuses on matched patches between relevant images and filters out the ones for irrelevant pairs. Furthermore, we demonstrate that the proposed kernel complements recent methods, such as hamming embedding, multiple assignment, local descriptors aggregation, and graph-based re-ranking, while it outperforms the static one under various settings on off-the-shelf evaluation metrics. We also propose to evaluate the matched patches both quantitatively and qualitatively. Extensive experiments on five benchmark data sets and large-scale distractors validate the merits of the proposed method against the state-of-the-art methods for image retrieval.

AB - For image retrieval methods based on bag of visual words, much attention has been paid to enhancing the discriminative powers of the local features. Although retrieved images are usually similar to a query in minutiae, they may be significantly different from a semantic perspective, which can be effectively distinguished by convolutional neural networks (CNN). Such images should not be considered as relevant pairs. To tackle this problem, we propose to construct a dynamic match kernel by adaptively calculating the matching thresholds between query and candidate images based on the pairwise distance among deep CNN features. In contrast to the typical static match kernel which is independent to the global appearance of retrieved images, the dynamic one leverages the semantical similarity as a constraint for determining the matches. Accordingly, we propose a semantic-constrained retrieval framework by incorporating the dynamic match kernel, which focuses on matched patches between relevant images and filters out the ones for irrelevant pairs. Furthermore, we demonstrate that the proposed kernel complements recent methods, such as hamming embedding, multiple assignment, local descriptors aggregation, and graph-based re-ranking, while it outperforms the static one under various settings on off-the-shelf evaluation metrics. We also propose to evaluate the matched patches both quantitatively and qualitatively. Extensive experiments on five benchmark data sets and large-scale distractors validate the merits of the proposed method against the state-of-the-art methods for image retrieval.

UR - http://www.scopus.com/inward/record.url?scp=85048157396&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85048157396&partnerID=8YFLogxK

U2 - 10.1109/TIP.2018.2845136

DO - 10.1109/TIP.2018.2845136

M3 - Article

AN - SCOPUS:85048157396

VL - 27

SP - 5288

EP - 5302

JO - IEEE Transactions on Image Processing

JF - IEEE Transactions on Image Processing

SN - 1057-7149

IS - 11

M1 - 8374838

ER -