TapLab: A Fast Framework for Semantic Video Segmentation Tapping into Compressed-Domain Knowledge

Junyi Feng, Songyuan Li, Xi Li, Fei Wu, Qi Tian, Ming Hsuan Yang, Haibin Ling

Research output: Contribution to journalArticlepeer-review

1 Citation (Scopus)


Real-time semantic video segmentation is a challenging task due to the strict requirements of inference speed. Recent approaches mainly devote great efforts to reducing the model size for high efficiency. In this paper, we rethink this problem from a different viewpoint: using knowledge contained in compressed videos. We propose a simple and effective framework, dubbed TapLab, to tap into resources from the compressed domain. Specifically, we design a fast feature warping module using motion vectors for acceleration. To reduce the noise introduced by motion vectors, we design a residual-guided correction module and a residual-guided frame selection module using residuals. TapLab significantly reduces redundant computations of the state-of-the-art fast semantic image segmentation models, running 3 to 10 times faster with controllable accuracy degradation. The experimental results show that TapLab achieves 70.6 percent mIoU on the Cityscapes dataset at 99.8 FPS with a single GPU card for the 1024×2048 videos. A high-speed version even reaches the speed of 160+ FPS. Code will be available soon at https://github.com/Sixkplus/TapLab.

Original languageEnglish
Pages (from-to)1591-1603
Number of pages13
JournalIEEE transactions on pattern analysis and machine intelligence
Issue number3
Publication statusPublished - 2022 Mar 1

Bibliographical note

Funding Information:
This work was supported by a key scientific and technological innovation research project by the Ministry of Education, Zhejiang Provincial Natural Science Foundation of China under Grant LR19F020004, Baidu AI Frontier Technology Joint Research Program, and Zhejiang University K.P. Chao s High Technology Development Foundation. Junyi Feng and Songyuan Li contributed equally to this work.

Publisher Copyright:
© 1979-2012 IEEE.

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Vision and Pattern Recognition
  • Computational Theory and Mathematics
  • Artificial Intelligence
  • Applied Mathematics


Dive into the research topics of 'TapLab: A Fast Framework for Semantic Video Segmentation Tapping into Compressed-Domain Knowledge'. Together they form a unique fingerprint.

Cite this