FCSS: Fully convolutional self-similarity for dense semantic correspondence

Seungryong Kim, Dongbo Min, Bumsub Ham, Sangryul Jeon, Stephen Lin, Kwanghoon Sohn

Research output: Chapter in Book/Report/Conference proceedingConference contribution

29 Citations (Scopus)

Abstract

We present a descriptor, called fully convolutional self-similarity (FCSS), for dense semantic correspondence. To robustly match points among different instances within the same object class, we formulate FCSS using local self-similarity (LSS) within a fully convolutional network. In contrast to existing CNN-based descriptors, FCSS is inherently insensitive to intra-class appearance variations because of its LSS-based structure, while maintaining the precise localization ability of deep neural networks. The sampling patterns of local structure and the self-similarity measure are jointly learned within the proposed network in an end-to-end and multi-scale manner. As training data for semantic correspondence is rather limited, we propose to leverage object candidate priors provided in existing image datasets and also correspondence consistency between object pairs to enable weakly-supervised learning. Experiments demonstrate that FCSS outperforms conventional handcrafted descriptors and CNN-based descriptors on various benchmarks.

Original languageEnglish
Title of host publicationProceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages616-625
Number of pages10
ISBN (Electronic)9781538604571
DOIs
Publication statusPublished - 2017 Nov 6
Event30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 - Honolulu, United States
Duration: 2017 Jul 212017 Jul 26

Publication series

NameProceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017
Volume2017-January

Other

Other30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017
CountryUnited States
CityHonolulu
Period17/7/2117/7/26

Fingerprint

Semantics
Supervised learning
Sampling
Experiments
Deep neural networks

All Science Journal Classification (ASJC) codes

  • Signal Processing
  • Computer Vision and Pattern Recognition

Cite this

Kim, S., Min, D., Ham, B., Jeon, S., Lin, S., & Sohn, K. (2017). FCSS: Fully convolutional self-similarity for dense semantic correspondence. In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 (pp. 616-625). (Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017; Vol. 2017-January). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CVPR.2017.73
Kim, Seungryong ; Min, Dongbo ; Ham, Bumsub ; Jeon, Sangryul ; Lin, Stephen ; Sohn, Kwanghoon. / FCSS : Fully convolutional self-similarity for dense semantic correspondence. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. Institute of Electrical and Electronics Engineers Inc., 2017. pp. 616-625 (Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017).
@inproceedings{e4575077d63f4fb9876dc3afe930241a,
title = "FCSS: Fully convolutional self-similarity for dense semantic correspondence",
abstract = "We present a descriptor, called fully convolutional self-similarity (FCSS), for dense semantic correspondence. To robustly match points among different instances within the same object class, we formulate FCSS using local self-similarity (LSS) within a fully convolutional network. In contrast to existing CNN-based descriptors, FCSS is inherently insensitive to intra-class appearance variations because of its LSS-based structure, while maintaining the precise localization ability of deep neural networks. The sampling patterns of local structure and the self-similarity measure are jointly learned within the proposed network in an end-to-end and multi-scale manner. As training data for semantic correspondence is rather limited, we propose to leverage object candidate priors provided in existing image datasets and also correspondence consistency between object pairs to enable weakly-supervised learning. Experiments demonstrate that FCSS outperforms conventional handcrafted descriptors and CNN-based descriptors on various benchmarks.",
author = "Seungryong Kim and Dongbo Min and Bumsub Ham and Sangryul Jeon and Stephen Lin and Kwanghoon Sohn",
year = "2017",
month = "11",
day = "6",
doi = "10.1109/CVPR.2017.73",
language = "English",
series = "Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "616--625",
booktitle = "Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017",
address = "United States",

}

Kim, S, Min, D, Ham, B, Jeon, S, Lin, S & Sohn, K 2017, FCSS: Fully convolutional self-similarity for dense semantic correspondence. in Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, vol. 2017-January, Institute of Electrical and Electronics Engineers Inc., pp. 616-625, 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, United States, 17/7/21. https://doi.org/10.1109/CVPR.2017.73

FCSS : Fully convolutional self-similarity for dense semantic correspondence. / Kim, Seungryong; Min, Dongbo; Ham, Bumsub; Jeon, Sangryul; Lin, Stephen; Sohn, Kwanghoon.

Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. Institute of Electrical and Electronics Engineers Inc., 2017. p. 616-625 (Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017; Vol. 2017-January).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - FCSS

T2 - Fully convolutional self-similarity for dense semantic correspondence

AU - Kim, Seungryong

AU - Min, Dongbo

AU - Ham, Bumsub

AU - Jeon, Sangryul

AU - Lin, Stephen

AU - Sohn, Kwanghoon

PY - 2017/11/6

Y1 - 2017/11/6

N2 - We present a descriptor, called fully convolutional self-similarity (FCSS), for dense semantic correspondence. To robustly match points among different instances within the same object class, we formulate FCSS using local self-similarity (LSS) within a fully convolutional network. In contrast to existing CNN-based descriptors, FCSS is inherently insensitive to intra-class appearance variations because of its LSS-based structure, while maintaining the precise localization ability of deep neural networks. The sampling patterns of local structure and the self-similarity measure are jointly learned within the proposed network in an end-to-end and multi-scale manner. As training data for semantic correspondence is rather limited, we propose to leverage object candidate priors provided in existing image datasets and also correspondence consistency between object pairs to enable weakly-supervised learning. Experiments demonstrate that FCSS outperforms conventional handcrafted descriptors and CNN-based descriptors on various benchmarks.

AB - We present a descriptor, called fully convolutional self-similarity (FCSS), for dense semantic correspondence. To robustly match points among different instances within the same object class, we formulate FCSS using local self-similarity (LSS) within a fully convolutional network. In contrast to existing CNN-based descriptors, FCSS is inherently insensitive to intra-class appearance variations because of its LSS-based structure, while maintaining the precise localization ability of deep neural networks. The sampling patterns of local structure and the self-similarity measure are jointly learned within the proposed network in an end-to-end and multi-scale manner. As training data for semantic correspondence is rather limited, we propose to leverage object candidate priors provided in existing image datasets and also correspondence consistency between object pairs to enable weakly-supervised learning. Experiments demonstrate that FCSS outperforms conventional handcrafted descriptors and CNN-based descriptors on various benchmarks.

UR - http://www.scopus.com/inward/record.url?scp=85041917113&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85041917113&partnerID=8YFLogxK

U2 - 10.1109/CVPR.2017.73

DO - 10.1109/CVPR.2017.73

M3 - Conference contribution

AN - SCOPUS:85041917113

T3 - Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017

SP - 616

EP - 625

BT - Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Kim S, Min D, Ham B, Jeon S, Lin S, Sohn K. FCSS: Fully convolutional self-similarity for dense semantic correspondence. In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. Institute of Electrical and Electronics Engineers Inc. 2017. p. 616-625. (Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017). https://doi.org/10.1109/CVPR.2017.73