Sequence feature generation with temporal unrolling network for zero-shot action recognition

Jewook Lee, Hoseong Kim, Hyeran Byun

Research output: Contribution to journalArticlepeer-review

Abstract

Zero-Shot Action Recognition (ZSAR) aims to recognize unseen action classes not included in the training dataset. Existing generative methods for ZSAR synthesize a feature of unseen action from a class embedding to overcome the absence of training data. Specifically, previous methods synthesize a feature which is averaged along a time axis, even though a video is extracted as a sequence of feature vectors. They suffer from the ambiguity of temporal information, which leads to confusion among actions sharing similar subactions. To tackle the problem, we first propose to synthesize not an averaged feature but a sequence consisting of feature vectors along the time axis. Hence, we design Sequence Feature Generative Adversarial Network (SFGAN) with Temporal Unrolling NEtwork (TUNE), which unrolls a class embedding into a set of condition vectors for generating sequences of features. Also, we employ a sequence discriminator as the second teacher. Through extensive experiments on the three benchmarks, HMDB51, UCF101, and Olympic, we validate the efficacy of sequence generation for ZSAR, and our method achieves the state-of-the-art generalized zero-shot learning performances.

Original languageEnglish
Pages (from-to)313-323
Number of pages11
JournalNeurocomputing
Volume448
DOIs
Publication statusPublished - 2021 Aug 11

Bibliographical note

Funding Information:
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No.2019R1A2C2003760) and Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2020-0-01361, Artificial Intelligence Graduate School Program (YONSEI UNIVERSITY)).

Publisher Copyright:
© 2021 Elsevier B.V.

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Cognitive Neuroscience
  • Artificial Intelligence

Fingerprint Dive into the research topics of 'Sequence feature generation with temporal unrolling network for zero-shot action recognition'. Together they form a unique fingerprint.

Cite this