Dynamic Match Kernel with Deep Convolutional Features for Image Retrieval

Jufeng Yang, Jie Liang, Hui Shen, Kai Wang, Paul L. Rosin, Ming Hsuan Yang

Research output: Contribution to journalArticlepeer-review

27 Citations (Scopus)

Abstract

For image retrieval methods based on bag of visual words, much attention has been paid to enhancing the discriminative powers of the local features. Although retrieved images are usually similar to a query in minutiae, they may be significantly different from a semantic perspective, which can be effectively distinguished by convolutional neural networks (CNN). Such images should not be considered as relevant pairs. To tackle this problem, we propose to construct a dynamic match kernel by adaptively calculating the matching thresholds between query and candidate images based on the pairwise distance among deep CNN features. In contrast to the typical static match kernel which is independent to the global appearance of retrieved images, the dynamic one leverages the semantical similarity as a constraint for determining the matches. Accordingly, we propose a semantic-constrained retrieval framework by incorporating the dynamic match kernel, which focuses on matched patches between relevant images and filters out the ones for irrelevant pairs. Furthermore, we demonstrate that the proposed kernel complements recent methods, such as hamming embedding, multiple assignment, local descriptors aggregation, and graph-based re-ranking, while it outperforms the static one under various settings on off-the-shelf evaluation metrics. We also propose to evaluate the matched patches both quantitatively and qualitatively. Extensive experiments on five benchmark data sets and large-scale distractors validate the merits of the proposed method against the state-of-the-art methods for image retrieval.

Original languageEnglish
Article number8374838
Pages (from-to)5288-5302
Number of pages15
JournalIEEE Transactions on Image Processing
Volume27
Issue number11
DOIs
Publication statusPublished - 2018 Nov

Bibliographical note

Funding Information:
This work was supported in part by NSFC under Grants 61620106008, 61572264, 61633021, 61525306, 61301238, and 61201424, in part by NSF CAREER under Grant 1149783, in part by the Open Project Program of the National Laboratory of Pattern Recognition, in part by the Natural Science Foundation of Tianjin, China, under Grant 18JCYBJC15400, and in part by the Fundamental Research Funds for the Central Universities.

Funding Information:
Manuscript received May 10, 2018; accepted May 26, 2018. Date of publication June 7, 2018; date of current version July 31, 2018. This work was supported in part by NSFC under Grants 61620106008, 61572264, 61633021, 61525306, 61301238, and 61201424, in part by NSF CAREER under Grant 1149783, in part by the Open Project Program of the National Laboratory of Pattern Recognition, in part by the Natural Science Foundation of Tianjin, China, under Grant 18JCYBJC15400, and in part by the Fundamental Research Funds for the Central Universities. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Jie Liang. (Corresponding author: Ming-Hsuan Yang.) J. Yang, J. Liang, H. Shen, and K. Wang are with the School of Computer Science and Control Engineering, Nankai University, Tianjin 300350, China (e-mail: yangjufeng@nankai.edu.cn; liang27jie@163.com; jhonjoe.c@ gmail.com; wangk@nankai.edu.cn).

Publisher Copyright:
© 1992-2012 IEEE.

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Graphics and Computer-Aided Design

Fingerprint

Dive into the research topics of 'Dynamic Match Kernel with Deep Convolutional Features for Image Retrieval'. Together they form a unique fingerprint.

Cite this