Privacy-Preserving Text Classification on BERT Embeddings with Homomorphic Encryption

Garam Lee; Minsoo Kim; Jai Hyun Park; Seung Won Hwang; Jung Hee Cheon

Privacy-Preserving Text Classification on BERT Embeddings with Homomorphic Encryption

Garam Lee, Minsoo Kim, Jai Hyun Park, Seung Won Hwang, Jung Hee Cheon

College of Computing

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Embeddings, which compress information in raw text into semantics-preserving low-dimensional vectors, have been widely adopted for their efficacy. However, recent research has shown that embeddings can potentially leak private information about sensitive attributes of the text, and in some cases, can be inverted to recover the original input text. To address these growing privacy challenges, we propose a privatization mechanism for embeddings based on homomorphic encryption, to prevent potential leakage of any piece of information in the process of text classification. In particular, our method performs text classification on the encryption of embeddings from state-of-the-art models like BERT, supported by an efficient GPU implementation of CKKS encryption scheme. We show that our method offers encrypted protection of BERT embeddings, while largely preserving their utility on downstream text classification tasks.

Original language	English
Title of host publication	NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics
Subtitle of host publication	Human Language Technologies, Proceedings of the Conference
Publisher	Association for Computational Linguistics (ACL)
Pages	3169-3175
Number of pages	7
ISBN (Electronic)	9781955917711
Publication status	Published - 2022
Event	2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022 - Seattle, United States Duration: 2022 Jul 10 → 2022 Jul 15

Publication series

Name	NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference

Conference

Conference	2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022
Country/Territory	United States
City	Seattle
Period	22/7/10 → 22/7/15

Bibliographical note

Funding Information:
Cheon’s team was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [NO.2020-0-00840, Development and Library Implementation of Fully Homomorphic Machine Learning Algorithms supporting Neural Network Learning over Encrypted Data, 50%]. Hwang’s team was supported by Microsoft Research Asia and IITP [(2022-00155958, High Potential Individuals Global Training Program) and (NO.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University), 50%] .

Publisher Copyright:
© 2022 Association for Computational Linguistics.

All Science Journal Classification (ASJC) codes

Computer Networks and Communications
Hardware and Architecture
Information Systems
Software

Cite this

Lee, G., Kim, M., Park, J. H., Hwang, S. W., & Cheon, J. H. (2022). Privacy-Preserving Text Classification on BERT Embeddings with Homomorphic Encryption. In NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference (pp. 3169-3175). (NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference). Association for Computational Linguistics (ACL).

Lee, Garam ; Kim, Minsoo ; Park, Jai Hyun et al. / Privacy-Preserving Text Classification on BERT Embeddings with Homomorphic Encryption. NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference. Association for Computational Linguistics (ACL), 2022. pp. 3169-3175 (NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference).

@inproceedings{bd4a708ae2754f7daf7d2e2662d3cb0e,

title = "Privacy-Preserving Text Classification on BERT Embeddings with Homomorphic Encryption",

abstract = "Embeddings, which compress information in raw text into semantics-preserving low-dimensional vectors, have been widely adopted for their efficacy. However, recent research has shown that embeddings can potentially leak private information about sensitive attributes of the text, and in some cases, can be inverted to recover the original input text. To address these growing privacy challenges, we propose a privatization mechanism for embeddings based on homomorphic encryption, to prevent potential leakage of any piece of information in the process of text classification. In particular, our method performs text classification on the encryption of embeddings from state-of-the-art models like BERT, supported by an efficient GPU implementation of CKKS encryption scheme. We show that our method offers encrypted protection of BERT embeddings, while largely preserving their utility on downstream text classification tasks.",

author = "Garam Lee and Minsoo Kim and Park, {Jai Hyun} and Hwang, {Seung Won} and Cheon, {Jung Hee}",

note = "Funding Information: Cheon{\textquoteright}s team was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [NO.2020-0-00840, Development and Library Implementation of Fully Homomorphic Machine Learning Algorithms supporting Neural Network Learning over Encrypted Data, 50%]. Hwang{\textquoteright}s team was supported by Microsoft Research Asia and IITP [(2022-00155958, High Potential Individuals Global Training Program) and (NO.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University), 50%] . Publisher Copyright: {\textcopyright} 2022 Association for Computational Linguistics.; 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022 ; Conference date: 10-07-2022 Through 15-07-2022",

year = "2022",

language = "English",

series = "NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference",

publisher = "Association for Computational Linguistics (ACL)",

pages = "3169--3175",

booktitle = "NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics",

}

Lee, G, Kim, M, Park, JH, Hwang, SW & Cheon, JH 2022, Privacy-Preserving Text Classification on BERT Embeddings with Homomorphic Encryption. in NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference. NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, Association for Computational Linguistics (ACL), pp. 3169-3175, 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, United States, 22/7/10.

Privacy-Preserving Text Classification on BERT Embeddings with Homomorphic Encryption. / Lee, Garam; Kim, Minsoo; Park, Jai Hyun et al.

NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference. Association for Computational Linguistics (ACL), 2022. p. 3169-3175 (NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Privacy-Preserving Text Classification on BERT Embeddings with Homomorphic Encryption

AU - Lee, Garam

AU - Kim, Minsoo

AU - Park, Jai Hyun

AU - Hwang, Seung Won

AU - Cheon, Jung Hee

N1 - Funding Information: Cheon’s team was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [NO.2020-0-00840, Development and Library Implementation of Fully Homomorphic Machine Learning Algorithms supporting Neural Network Learning over Encrypted Data, 50%]. Hwang’s team was supported by Microsoft Research Asia and IITP [(2022-00155958, High Potential Individuals Global Training Program) and (NO.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University), 50%] . Publisher Copyright: © 2022 Association for Computational Linguistics.

PY - 2022

Y1 - 2022

N2 - Embeddings, which compress information in raw text into semantics-preserving low-dimensional vectors, have been widely adopted for their efficacy. However, recent research has shown that embeddings can potentially leak private information about sensitive attributes of the text, and in some cases, can be inverted to recover the original input text. To address these growing privacy challenges, we propose a privatization mechanism for embeddings based on homomorphic encryption, to prevent potential leakage of any piece of information in the process of text classification. In particular, our method performs text classification on the encryption of embeddings from state-of-the-art models like BERT, supported by an efficient GPU implementation of CKKS encryption scheme. We show that our method offers encrypted protection of BERT embeddings, while largely preserving their utility on downstream text classification tasks.

AB - Embeddings, which compress information in raw text into semantics-preserving low-dimensional vectors, have been widely adopted for their efficacy. However, recent research has shown that embeddings can potentially leak private information about sensitive attributes of the text, and in some cases, can be inverted to recover the original input text. To address these growing privacy challenges, we propose a privatization mechanism for embeddings based on homomorphic encryption, to prevent potential leakage of any piece of information in the process of text classification. In particular, our method performs text classification on the encryption of embeddings from state-of-the-art models like BERT, supported by an efficient GPU implementation of CKKS encryption scheme. We show that our method offers encrypted protection of BERT embeddings, while largely preserving their utility on downstream text classification tasks.

UR - http://www.scopus.com/inward/record.url?scp=85138400468&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85138400468&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85138400468

T3 - NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference

SP - 3169

EP - 3175

BT - NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics

PB - Association for Computational Linguistics (ACL)

T2 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022

Y2 - 10 July 2022 through 15 July 2022

ER -

Lee G, Kim M, Park JH, Hwang SW, Cheon JH. Privacy-Preserving Text Classification on BERT Embeddings with Homomorphic Encryption. In NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference. Association for Computational Linguistics (ACL). 2022. p. 3169-3175. (NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference).

Privacy-Preserving Text Classification on BERT Embeddings with Homomorphic Encryption

Abstract

Publication series

Conference

Bibliographical note

All Science Journal Classification (ASJC) codes

Other files and links

Fingerprint

Cite this