Multi-Task Structure-Aware Context Modeling for Robust Keypoint-Based Object Tracking

Xi Li, Liming Zhao, Wei Ji, Yiming Wu, Fei Wu, Ming Hsuan Yang, Dacheng Tao, Ian Reid

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

In the fields of computer vision and graphics, keypoint-based object tracking is a fundamental and challenging problem, which is typically formulated in a spatio-temporal context modeling framework. However, many existing keypoint trackers are incapable of effectively modeling and balancing the following three aspects in a simultaneous manner: temporal model coherence across frames, spatial model consistency within frames, and discriminative feature construction. To address this problem, we propose a robust keypoint tracker based on spatio-temporal multi-task structured output optimization driven by discriminative metric learning. Consequently, temporal model coherence is characterized by multi-task structured keypoint model learning over several adjacent frames; spatial model consistency is modeled by solving a geometric verification based structured learning problem; discriminative feature construction is enabled by metric learning to ensure the intra-class compactness and inter-class separability. To achieve the goal of effective object tracking, we jointly optimize the above three modules in a spatio-temporal multi-task learning scheme. Furthermore, we incorporate this joint learning scheme into both single-object and multi-object tracking scenarios, resulting in robust tracking results. Experiments over several challenging datasets have justified the effectiveness of our single-object and multi-object trackers against the state-of-the-art.

Original languageEnglish
Article number8322285
Pages (from-to)915-927
Number of pages13
JournalIEEE transactions on pattern analysis and machine intelligence
Volume41
Issue number4
DOIs
Publication statusPublished - 2019 Apr 1

Fingerprint

Context Modeling
Object Tracking
Spatial Model
Spatio-temporal Modeling
Multi-task Learning
Metric
Separability
Computer graphics
Balancing
Computer Vision
Computer vision
Compactness
Adjacent
Optimise
Learning
Model
Module
Scenarios
Optimization
Output

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Vision and Pattern Recognition
  • Computational Theory and Mathematics
  • Artificial Intelligence
  • Applied Mathematics

Cite this

Li, Xi ; Zhao, Liming ; Ji, Wei ; Wu, Yiming ; Wu, Fei ; Yang, Ming Hsuan ; Tao, Dacheng ; Reid, Ian. / Multi-Task Structure-Aware Context Modeling for Robust Keypoint-Based Object Tracking. In: IEEE transactions on pattern analysis and machine intelligence. 2019 ; Vol. 41, No. 4. pp. 915-927.
@article{74acb1a4d18448c597b14f54178b3242,
title = "Multi-Task Structure-Aware Context Modeling for Robust Keypoint-Based Object Tracking",
abstract = "In the fields of computer vision and graphics, keypoint-based object tracking is a fundamental and challenging problem, which is typically formulated in a spatio-temporal context modeling framework. However, many existing keypoint trackers are incapable of effectively modeling and balancing the following three aspects in a simultaneous manner: temporal model coherence across frames, spatial model consistency within frames, and discriminative feature construction. To address this problem, we propose a robust keypoint tracker based on spatio-temporal multi-task structured output optimization driven by discriminative metric learning. Consequently, temporal model coherence is characterized by multi-task structured keypoint model learning over several adjacent frames; spatial model consistency is modeled by solving a geometric verification based structured learning problem; discriminative feature construction is enabled by metric learning to ensure the intra-class compactness and inter-class separability. To achieve the goal of effective object tracking, we jointly optimize the above three modules in a spatio-temporal multi-task learning scheme. Furthermore, we incorporate this joint learning scheme into both single-object and multi-object tracking scenarios, resulting in robust tracking results. Experiments over several challenging datasets have justified the effectiveness of our single-object and multi-object trackers against the state-of-the-art.",
author = "Xi Li and Liming Zhao and Wei Ji and Yiming Wu and Fei Wu and Yang, {Ming Hsuan} and Dacheng Tao and Ian Reid",
year = "2019",
month = "4",
day = "1",
doi = "10.1109/TPAMI.2018.2818132",
language = "English",
volume = "41",
pages = "915--927",
journal = "IEEE Transactions on Pattern Analysis and Machine Intelligence",
issn = "0162-8828",
publisher = "IEEE Computer Society",
number = "4",

}

Multi-Task Structure-Aware Context Modeling for Robust Keypoint-Based Object Tracking. / Li, Xi; Zhao, Liming; Ji, Wei; Wu, Yiming; Wu, Fei; Yang, Ming Hsuan; Tao, Dacheng; Reid, Ian.

In: IEEE transactions on pattern analysis and machine intelligence, Vol. 41, No. 4, 8322285, 01.04.2019, p. 915-927.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Multi-Task Structure-Aware Context Modeling for Robust Keypoint-Based Object Tracking

AU - Li, Xi

AU - Zhao, Liming

AU - Ji, Wei

AU - Wu, Yiming

AU - Wu, Fei

AU - Yang, Ming Hsuan

AU - Tao, Dacheng

AU - Reid, Ian

PY - 2019/4/1

Y1 - 2019/4/1

N2 - In the fields of computer vision and graphics, keypoint-based object tracking is a fundamental and challenging problem, which is typically formulated in a spatio-temporal context modeling framework. However, many existing keypoint trackers are incapable of effectively modeling and balancing the following three aspects in a simultaneous manner: temporal model coherence across frames, spatial model consistency within frames, and discriminative feature construction. To address this problem, we propose a robust keypoint tracker based on spatio-temporal multi-task structured output optimization driven by discriminative metric learning. Consequently, temporal model coherence is characterized by multi-task structured keypoint model learning over several adjacent frames; spatial model consistency is modeled by solving a geometric verification based structured learning problem; discriminative feature construction is enabled by metric learning to ensure the intra-class compactness and inter-class separability. To achieve the goal of effective object tracking, we jointly optimize the above three modules in a spatio-temporal multi-task learning scheme. Furthermore, we incorporate this joint learning scheme into both single-object and multi-object tracking scenarios, resulting in robust tracking results. Experiments over several challenging datasets have justified the effectiveness of our single-object and multi-object trackers against the state-of-the-art.

AB - In the fields of computer vision and graphics, keypoint-based object tracking is a fundamental and challenging problem, which is typically formulated in a spatio-temporal context modeling framework. However, many existing keypoint trackers are incapable of effectively modeling and balancing the following three aspects in a simultaneous manner: temporal model coherence across frames, spatial model consistency within frames, and discriminative feature construction. To address this problem, we propose a robust keypoint tracker based on spatio-temporal multi-task structured output optimization driven by discriminative metric learning. Consequently, temporal model coherence is characterized by multi-task structured keypoint model learning over several adjacent frames; spatial model consistency is modeled by solving a geometric verification based structured learning problem; discriminative feature construction is enabled by metric learning to ensure the intra-class compactness and inter-class separability. To achieve the goal of effective object tracking, we jointly optimize the above three modules in a spatio-temporal multi-task learning scheme. Furthermore, we incorporate this joint learning scheme into both single-object and multi-object tracking scenarios, resulting in robust tracking results. Experiments over several challenging datasets have justified the effectiveness of our single-object and multi-object trackers against the state-of-the-art.

UR - http://www.scopus.com/inward/record.url?scp=85044253167&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85044253167&partnerID=8YFLogxK

U2 - 10.1109/TPAMI.2018.2818132

DO - 10.1109/TPAMI.2018.2818132

M3 - Article

C2 - 29993768

AN - SCOPUS:85044253167

VL - 41

SP - 915

EP - 927

JO - IEEE Transactions on Pattern Analysis and Machine Intelligence

JF - IEEE Transactions on Pattern Analysis and Machine Intelligence

SN - 0162-8828

IS - 4

M1 - 8322285

ER -