Self-Supervised Audio Spatialization with Correspondence Classifier

Yu Ding Lu; Hsin Ying Lee; Hung Yu Tseng; Ming Hsuan Yang

doi:10.1109/ICIP.2019.8803494

Self-Supervised Audio Spatialization with Correspondence Classifier

Yu Ding Lu, Hsin Ying Lee, Hung Yu Tseng, Ming Hsuan Yang

College of Computing

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

15 Citations (Scopus)

Abstract

Spatial audio is an essential medium to audiences for 3D visual and auditory experience. However, the recording devices and techniques are expensive or inaccessible to the general public. In this work, we propose a self-supervised audio spatialization network that can generate spatial audio given the corresponding video and monaural audio. To enhance spatialization performance, we use an auxiliary classifier to classify ground-truth videos and those with audio where the left and right channels are swapped. We collect a large-scale video dataset with spatial audio to validate the proposed method. Experimental results demonstrate the effectiveness of the proposed model on the audio spatialization task.

Original language	English
Title of host publication	2019 IEEE International Conference on Image Processing, ICIP 2019 - Proceedings
Publisher	IEEE Computer Society
Pages	3347-3351
Number of pages	5
ISBN (Electronic)	9781538662496
DOIs	https://doi.org/10.1109/ICIP.2019.8803494
Publication status	Published - 2019 Sept
Event	26th IEEE International Conference on Image Processing, ICIP 2019 - Taipei, Taiwan, Province of China Duration: 2019 Sept 22 → 2019 Sept 25

Publication series

Name	Proceedings - International Conference on Image Processing, ICIP
Volume	2019-September
ISSN (Print)	1522-4880

Conference

Conference	26th IEEE International Conference on Image Processing, ICIP 2019
Country/Territory	Taiwan, Province of China
City	Taipei
Period	19/9/22 → 19/9/25

Bibliographical note

Publisher Copyright:
© 2019 IEEE.

All Science Journal Classification (ASJC) codes

Software
Computer Vision and Pattern Recognition
Signal Processing

Access to Document

10.1109/ICIP.2019.8803494

Cite this

Lu, Y. D., Lee, H. Y., Tseng, H. Y., & Yang, M. H. (2019). Self-Supervised Audio Spatialization with Correspondence Classifier. In 2019 IEEE International Conference on Image Processing, ICIP 2019 - Proceedings (pp. 3347-3351). [8803494] (Proceedings - International Conference on Image Processing, ICIP; Vol. 2019-September). IEEE Computer Society. https://doi.org/10.1109/ICIP.2019.8803494

@inproceedings{3fe55d1dc19540e6b79fbbf6d6c66b17,

title = "Self-Supervised Audio Spatialization with Correspondence Classifier",

abstract = "Spatial audio is an essential medium to audiences for 3D visual and auditory experience. However, the recording devices and techniques are expensive or inaccessible to the general public. In this work, we propose a self-supervised audio spatialization network that can generate spatial audio given the corresponding video and monaural audio. To enhance spatialization performance, we use an auxiliary classifier to classify ground-truth videos and those with audio where the left and right channels are swapped. We collect a large-scale video dataset with spatial audio to validate the proposed method. Experimental results demonstrate the effectiveness of the proposed model on the audio spatialization task.",

author = "Lu, {Yu Ding} and Lee, {Hsin Ying} and Tseng, {Hung Yu} and Yang, {Ming Hsuan}",

note = "Publisher Copyright: {\textcopyright} 2019 IEEE.; 26th IEEE International Conference on Image Processing, ICIP 2019 ; Conference date: 22-09-2019 Through 25-09-2019",

year = "2019",

month = sep,

doi = "10.1109/ICIP.2019.8803494",

language = "English",

series = "Proceedings - International Conference on Image Processing, ICIP",

publisher = "IEEE Computer Society",

pages = "3347--3351",

booktitle = "2019 IEEE International Conference on Image Processing, ICIP 2019 - Proceedings",

address = "United States",

}

Lu, YD, Lee, HY, Tseng, HY & Yang, MH 2019, Self-Supervised Audio Spatialization with Correspondence Classifier. in 2019 IEEE International Conference on Image Processing, ICIP 2019 - Proceedings., 8803494, Proceedings - International Conference on Image Processing, ICIP, vol. 2019-September, IEEE Computer Society, pp. 3347-3351, 26th IEEE International Conference on Image Processing, ICIP 2019, Taipei, Taiwan, Province of China, 19/9/22. https://doi.org/10.1109/ICIP.2019.8803494

Self-Supervised Audio Spatialization with Correspondence Classifier. / Lu, Yu Ding; Lee, Hsin Ying; Tseng, Hung Yu et al.

2019 IEEE International Conference on Image Processing, ICIP 2019 - Proceedings. IEEE Computer Society, 2019. p. 3347-3351 8803494 (Proceedings - International Conference on Image Processing, ICIP; Vol. 2019-September).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Self-Supervised Audio Spatialization with Correspondence Classifier

AU - Lu, Yu Ding

AU - Lee, Hsin Ying

AU - Tseng, Hung Yu

AU - Yang, Ming Hsuan

PY - 2019/9

Y1 - 2019/9

N2 - Spatial audio is an essential medium to audiences for 3D visual and auditory experience. However, the recording devices and techniques are expensive or inaccessible to the general public. In this work, we propose a self-supervised audio spatialization network that can generate spatial audio given the corresponding video and monaural audio. To enhance spatialization performance, we use an auxiliary classifier to classify ground-truth videos and those with audio where the left and right channels are swapped. We collect a large-scale video dataset with spatial audio to validate the proposed method. Experimental results demonstrate the effectiveness of the proposed model on the audio spatialization task.

AB - Spatial audio is an essential medium to audiences for 3D visual and auditory experience. However, the recording devices and techniques are expensive or inaccessible to the general public. In this work, we propose a self-supervised audio spatialization network that can generate spatial audio given the corresponding video and monaural audio. To enhance spatialization performance, we use an auxiliary classifier to classify ground-truth videos and those with audio where the left and right channels are swapped. We collect a large-scale video dataset with spatial audio to validate the proposed method. Experimental results demonstrate the effectiveness of the proposed model on the audio spatialization task.

UR - http://www.scopus.com/inward/record.url?scp=85076801998&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85076801998&partnerID=8YFLogxK

U2 - 10.1109/ICIP.2019.8803494

DO - 10.1109/ICIP.2019.8803494

M3 - Conference contribution

AN - SCOPUS:85076801998

T3 - Proceedings - International Conference on Image Processing, ICIP

SP - 3347

EP - 3351

BT - 2019 IEEE International Conference on Image Processing, ICIP 2019 - Proceedings

PB - IEEE Computer Society

T2 - 26th IEEE International Conference on Image Processing, ICIP 2019

Y2 - 22 September 2019 through 25 September 2019

ER -

Self-Supervised Audio Spatialization with Correspondence Classifier

Abstract

Publication series

Conference

Bibliographical note

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this