A Fast and Lightweight Speech Synthesis Model based on FastSpeech2

Huu Kim Nguyen; Kihyuk Jeong; Hong Goo Kang

doi:10.1109/ITC-CSCC52171.2021.9501479

A Fast and Lightweight Speech Synthesis Model based on FastSpeech2

Huu Kim Nguyen, Kihyuk Jeong, Hong Goo Kang

Department of Electrical and Electronic Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

In this paper, we present a fast and lightweight speech synthesis model that is suitable for on-device applications. By leveraging the techniques of long-short range attention, depth-wise separable convolution, and linear attention, we significantly reduce the model size and complexity of the baseline FastSpeech2-based Transformer framework. Unlike the baseline model that requires O(N^{2}) to compute attention and convolution operations because of nested-loop computations, our proposed model only requires O(N) computations due to the modification of a nested-loop into two cascaded single loops. Experimental results show that our proposed model is able to generate speech with a real-time factor of 0.26 and requires only 10.4 million parameters. Despite the reduction in model size and complexity, still, the generated speech quality of our model is nearly close to the baseline.

Original language	English
Title of host publication	2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781665435536
DOIs	https://doi.org/10.1109/ITC-CSCC52171.2021.9501479
Publication status	Published - 2021 Jun 27
Event	36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021 - Jeju, Korea, Republic of Duration: 2021 Jun 27 → 2021 Jun 30

Publication series

Name	2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021

Conference

Conference	36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021
Country/Territory	Korea, Republic of
City	Jeju
Period	21/6/27 → 21/6/30

Bibliographical note

Funding Information:
ACKNOWLEDGMENT The work is supported by Clova Voice, NAVER Corp., Seongnam, Korea.

Publisher Copyright:
© 2021 IEEE.

All Science Journal Classification (ASJC) codes

Artificial Intelligence
Computer Networks and Communications
Hardware and Architecture
Information Systems
Electrical and Electronic Engineering

Access to Document

10.1109/ITC-CSCC52171.2021.9501479

Cite this

Nguyen, H. K., Jeong, K., & Kang, H. G. (2021). A Fast and Lightweight Speech Synthesis Model based on FastSpeech2. In 2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021 [9501479] (2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ITC-CSCC52171.2021.9501479

Nguyen, Huu Kim ; Jeong, Kihyuk ; Kang, Hong Goo. / A Fast and Lightweight Speech Synthesis Model based on FastSpeech2. 2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021. Institute of Electrical and Electronics Engineers Inc., 2021. (2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021).

@inproceedings{d998ac510ca8429582cd91f99fe34061,

title = "A Fast and Lightweight Speech Synthesis Model based on FastSpeech2",

abstract = "In this paper, we present a fast and lightweight speech synthesis model that is suitable for on-device applications. By leveraging the techniques of long-short range attention, depth-wise separable convolution, and linear attention, we significantly reduce the model size and complexity of the baseline FastSpeech2-based Transformer framework. Unlike the baseline model that requires O(N^{2}) to compute attention and convolution operations because of nested-loop computations, our proposed model only requires O(N) computations due to the modification of a nested-loop into two cascaded single loops. Experimental results show that our proposed model is able to generate speech with a real-time factor of 0.26 and requires only 10.4 million parameters. Despite the reduction in model size and complexity, still, the generated speech quality of our model is nearly close to the baseline.",

author = "Nguyen, {Huu Kim} and Kihyuk Jeong and Kang, {Hong Goo}",

note = "Funding Information: ACKNOWLEDGMENT The work is supported by Clova Voice, NAVER Corp., Seongnam, Korea. Publisher Copyright: {\textcopyright} 2021 IEEE.; 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021 ; Conference date: 27-06-2021 Through 30-06-2021",

year = "2021",

month = jun,

day = "27",

doi = "10.1109/ITC-CSCC52171.2021.9501479",

language = "English",

series = "2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021",

address = "United States",

}

Nguyen, HK, Jeong, K & Kang, HG 2021, A Fast and Lightweight Speech Synthesis Model based on FastSpeech2. in 2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021., 9501479, 2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021, Institute of Electrical and Electronics Engineers Inc., 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021, Jeju, Korea, Republic of, 21/6/27. https://doi.org/10.1109/ITC-CSCC52171.2021.9501479

A Fast and Lightweight Speech Synthesis Model based on FastSpeech2. / Nguyen, Huu Kim; Jeong, Kihyuk; Kang, Hong Goo.

2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021. Institute of Electrical and Electronics Engineers Inc., 2021. 9501479 (2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - A Fast and Lightweight Speech Synthesis Model based on FastSpeech2

AU - Nguyen, Huu Kim

AU - Jeong, Kihyuk

AU - Kang, Hong Goo

PY - 2021/6/27

Y1 - 2021/6/27

N2 - In this paper, we present a fast and lightweight speech synthesis model that is suitable for on-device applications. By leveraging the techniques of long-short range attention, depth-wise separable convolution, and linear attention, we significantly reduce the model size and complexity of the baseline FastSpeech2-based Transformer framework. Unlike the baseline model that requires O(N^{2}) to compute attention and convolution operations because of nested-loop computations, our proposed model only requires O(N) computations due to the modification of a nested-loop into two cascaded single loops. Experimental results show that our proposed model is able to generate speech with a real-time factor of 0.26 and requires only 10.4 million parameters. Despite the reduction in model size and complexity, still, the generated speech quality of our model is nearly close to the baseline.

AB - In this paper, we present a fast and lightweight speech synthesis model that is suitable for on-device applications. By leveraging the techniques of long-short range attention, depth-wise separable convolution, and linear attention, we significantly reduce the model size and complexity of the baseline FastSpeech2-based Transformer framework. Unlike the baseline model that requires O(N^{2}) to compute attention and convolution operations because of nested-loop computations, our proposed model only requires O(N) computations due to the modification of a nested-loop into two cascaded single loops. Experimental results show that our proposed model is able to generate speech with a real-time factor of 0.26 and requires only 10.4 million parameters. Despite the reduction in model size and complexity, still, the generated speech quality of our model is nearly close to the baseline.

UR - http://www.scopus.com/inward/record.url?scp=85113915143&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85113915143&partnerID=8YFLogxK

U2 - 10.1109/ITC-CSCC52171.2021.9501479

DO - 10.1109/ITC-CSCC52171.2021.9501479

M3 - Conference contribution

AN - SCOPUS:85113915143

T3 - 2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021

BT - 2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021

Y2 - 27 June 2021 through 30 June 2021

ER -

Nguyen HK, Jeong K, Kang HG. A Fast and Lightweight Speech Synthesis Model based on FastSpeech2. In 2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021. Institute of Electrical and Electronics Engineers Inc. 2021. 9501479. (2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021). doi: 10.1109/ITC-CSCC52171.2021.9501479

A Fast and Lightweight Speech Synthesis Model based on FastSpeech2

Abstract

Publication series

Conference

Bibliographical note

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this