Emotional Speech Synthesis with Rich and Granularized Control

Se Yun Um; Sangshin Oh; Kyungguen Byun; Inseon Jang; Chunghyun Ahn; Hong Goo Kang

doi:10.1109/ICASSP40776.2020.9053732

Emotional Speech Synthesis with Rich and Granularized Control

Se Yun Um, Sangshin Oh, Kyungguen Byun, Inseon Jang, Chunghyun Ahn, Hong Goo Kang

Department of Electrical and Electronic Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

43 Citations (Scopus)

Abstract

This paper proposes an effective emotion control method for an end-to-end text-to-speech (TTS) system. To flexibly control the distinct characteristic of a target emotion category, it is essential to determine embedding vectors representing the TTS input. We introduce an inter-to-intra emotional distance ratio algorithm to the embedding vectors that can minimize the distance to the target emotion category while maximizing its distance to the other emotion categories. To further enhance the expressiveness of a target speech, we also introduce an effective interpolation technique that enables the intensity of a target emotion to be gradually changed to that of neutral speech. Subjective evaluation results in terms of emotional expressiveness and controllability show the superiority of the proposed algorithm to the conventional methods.

Original language	English
Title of host publication	2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	7254-7258
Number of pages	5
ISBN (Electronic)	9781509066315
DOIs	https://doi.org/10.1109/ICASSP40776.2020.9053732
Publication status	Published - 2020 May
Event	2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Barcelona, Spain Duration: 2020 May 4 → 2020 May 8

Publication series

Name	ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
Volume	2020-May
ISSN (Print)	1520-6149

Conference

Conference	2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020
Country/Territory	Spain
City	Barcelona
Period	20/5/4 → 20/5/8

Bibliographical note

Funding Information:
This work was supported by Institute for Information communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (2019-0-00447,Development of emotional expression service to support hearing/visually impaired).

Publisher Copyright:
© 2020 IEEE.

All Science Journal Classification (ASJC) codes

Software
Signal Processing
Electrical and Electronic Engineering

Access to Document

10.1109/ICASSP40776.2020.9053732

Cite this

Um, S. Y., Oh, S., Byun, K., Jang, I., Ahn, C., & Kang, H. G. (2020). Emotional Speech Synthesis with Rich and Granularized Control. In 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings (pp. 7254-7258). [9053732] (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings; Vol. 2020-May). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICASSP40776.2020.9053732

Um, Se Yun ; Oh, Sangshin ; Byun, Kyungguen et al. / Emotional Speech Synthesis with Rich and Granularized Control. 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2020. pp. 7254-7258 (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings).

@inproceedings{51dfed22f5214520a2a08ba82f73ef5b,

title = "Emotional Speech Synthesis with Rich and Granularized Control",

abstract = "This paper proposes an effective emotion control method for an end-to-end text-to-speech (TTS) system. To flexibly control the distinct characteristic of a target emotion category, it is essential to determine embedding vectors representing the TTS input. We introduce an inter-to-intra emotional distance ratio algorithm to the embedding vectors that can minimize the distance to the target emotion category while maximizing its distance to the other emotion categories. To further enhance the expressiveness of a target speech, we also introduce an effective interpolation technique that enables the intensity of a target emotion to be gradually changed to that of neutral speech. Subjective evaluation results in terms of emotional expressiveness and controllability show the superiority of the proposed algorithm to the conventional methods.",

author = "Um, {Se Yun} and Sangshin Oh and Kyungguen Byun and Inseon Jang and Chunghyun Ahn and Kang, {Hong Goo}",

note = "Funding Information: This work was supported by Institute for Information communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (2019-0-00447,Development of emotional expression service to support hearing/visually impaired). Publisher Copyright: {\textcopyright} 2020 IEEE.; 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 ; Conference date: 04-05-2020 Through 08-05-2020",

year = "2020",

month = may,

doi = "10.1109/ICASSP40776.2020.9053732",

language = "English",

series = "ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "7254--7258",

booktitle = "2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings",

address = "United States",

}

Um, SY, Oh, S, Byun, K, Jang, I, Ahn, C & Kang, HG 2020, Emotional Speech Synthesis with Rich and Granularized Control. in 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings., 9053732, ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, vol. 2020-May, Institute of Electrical and Electronics Engineers Inc., pp. 7254-7258, 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020, Barcelona, Spain, 20/5/4. https://doi.org/10.1109/ICASSP40776.2020.9053732

Emotional Speech Synthesis with Rich and Granularized Control. / Um, Se Yun; Oh, Sangshin; Byun, Kyungguen et al.

2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2020. p. 7254-7258 9053732 (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings; Vol. 2020-May).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Emotional Speech Synthesis with Rich and Granularized Control

AU - Um, Se Yun

AU - Oh, Sangshin

AU - Byun, Kyungguen

AU - Jang, Inseon

AU - Ahn, Chunghyun

AU - Kang, Hong Goo

N1 - Funding Information: This work was supported by Institute for Information communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (2019-0-00447,Development of emotional expression service to support hearing/visually impaired). Publisher Copyright: © 2020 IEEE.

PY - 2020/5

Y1 - 2020/5

N2 - This paper proposes an effective emotion control method for an end-to-end text-to-speech (TTS) system. To flexibly control the distinct characteristic of a target emotion category, it is essential to determine embedding vectors representing the TTS input. We introduce an inter-to-intra emotional distance ratio algorithm to the embedding vectors that can minimize the distance to the target emotion category while maximizing its distance to the other emotion categories. To further enhance the expressiveness of a target speech, we also introduce an effective interpolation technique that enables the intensity of a target emotion to be gradually changed to that of neutral speech. Subjective evaluation results in terms of emotional expressiveness and controllability show the superiority of the proposed algorithm to the conventional methods.

AB - This paper proposes an effective emotion control method for an end-to-end text-to-speech (TTS) system. To flexibly control the distinct characteristic of a target emotion category, it is essential to determine embedding vectors representing the TTS input. We introduce an inter-to-intra emotional distance ratio algorithm to the embedding vectors that can minimize the distance to the target emotion category while maximizing its distance to the other emotion categories. To further enhance the expressiveness of a target speech, we also introduce an effective interpolation technique that enables the intensity of a target emotion to be gradually changed to that of neutral speech. Subjective evaluation results in terms of emotional expressiveness and controllability show the superiority of the proposed algorithm to the conventional methods.

UR - http://www.scopus.com/inward/record.url?scp=85089242733&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85089242733&partnerID=8YFLogxK

U2 - 10.1109/ICASSP40776.2020.9053732

DO - 10.1109/ICASSP40776.2020.9053732

M3 - Conference contribution

AN - SCOPUS:85089242733

T3 - ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

SP - 7254

EP - 7258

BT - 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020

Y2 - 4 May 2020 through 8 May 2020

ER -

Um SY, Oh S, Byun K, Jang I, Ahn C, Kang HG. Emotional Speech Synthesis with Rich and Granularized Control. In 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings. Institute of Electrical and Electronics Engineers Inc. 2020. p. 7254-7258. 9053732. (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings). doi: 10.1109/ICASSP40776.2020.9053732

Emotional Speech Synthesis with Rich and Granularized Control

Abstract

Publication series

Conference

Bibliographical note

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this