DynaPAT: A dynamic pattern-aware encoding technique for robust MLC PCM-based deep neural networks

Thai Hoang Nguyen; Muhammad Imran; Joon Sung Yang

doi:10.1145/3508352.3549400

DynaPAT: A dynamic pattern-aware encoding technique for robust MLC PCM-based deep neural networks

Thai Hoang Nguyen, Muhammad Imran, Joon Sung Yang

College of Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

As the effectiveness of Deep Neural Networks (DNNs) is rising over time, so is the need for highly scalable and efficient hardware architectures to capitalize this effectiveness in many practical applications. Emerging non-volatile Phase Change Memory (PCM) technology has been found to be a promising candidate for future memory systems due to its better scalability, non-volatility and low leakage/dynamic power consumption, compared to conventional charged-based memories. Additionally, with its cell s wide resistance span, PCM also has the Flash-like Multi-Level Cell (MLC) capability, which has enhanced storage density, providing an opportunity for the deployment of data-intensive applications such as DNNs on resource-constrained edge devices. However, the practical deployment of MLC PCM is hampered by certain reliability challenges, among which, the resistance drift is considered to be a critical concern. In a DNN application, the presence of resistance drift in MLC PCM can cause a severe impact to DNN s accuracy if no drift-error-tolerance technique is utilized. This paper proposes DynaPAT, a low-cost and effective pattern-aware encoding technique to enhance the drift-error-tolerance of MLC PCM-based Deep Neural Networks. DynaPAT has been constructed on the insight into DNN s vulnerability against different data pattern switching. Based on this insight, DynaPAT efficiently maps the most-frequent data pattern in DNN s parameters to the least-drift-prone level of the MLC PCM, thus significantly enhancing the robustness of the system against drift errors. Various experiments on different DNN models and configurations demonstrate the effectiveness of DynaPAT. The experimental results indicate that DynaPAT can achieve up to 500× enhancement in the drift-errors-tolerance capability over the baseline MLC PCM based DNN while requiring only a negligible hardware overhead (below 1% storage overhead). Being orthogonal, DynaPAT can be integrated with existing drifttolerance schemes for even higher gains in reliability.

Original language	English
Title of host publication	Proceedings of the 41st IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2022
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781450392174
DOIs	https://doi.org/10.1145/3508352.3549400
Publication status	Published - 2022 Oct 30
Event	41st IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2022 - San Diego, United States Duration: 2022 Oct 30 → 2022 Nov 4

Publication series

Name	IEEE/ACM International Conference on Computer-Aided Design, Digest of Technical Papers, ICCAD
ISSN (Print)	1092-3152

Conference

Conference	41st IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2022
Country/Territory	United States
City	San Diego
Period	22/10/30 → 22/11/4

Bibliographical note

Funding Information:
This work was supported in part by the Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korea Government (MSIT) under Grant 2022-0-00971; in part by the Next Generation Intelligent Semiconductor Development by the Ministry of Trade, Industry and Energy (MOTIE) under Grant 20011074; and in part by the Basic Science Research Program through the National Research Foundation of Korea funded by the Ministry of Education under Grant NRF-2020M3F3A2A01082326

Funding Information:
The performance of DynaPAT compared with other methods when evaluated on ImageNet dataset using ResNet-50, DensenNet-169 and Inception-v3, is shown in Fig. 12. Note that ImageNet is a more difficult challenge compared with the CIFAR datasets since it contains 1000 classes as opposed to 10/100 classes in CIFAR datasets. Thus, the performance of the DynaPAT as well as other fault-tolerance techniques is expected to be lower than in case of CIFAR-10. Additionally, in our evaluation, the models evaluated on ImageNet dataset also have more layers (deeper) than the models evaluated on CIFAR datasets, hence, the fault-tolerance methods tend to be less effective. This observation is consistent with previous works [7, 18] that evaluate DNNs considering the presence of hardware faults. Nevertheless, DynaPAT still shows a significant robustness improvement over the baseline as well as other techniques. For example, for 8-bit quantized DenseNet-169 network, DynaPAT enhances the robustness over the baseline and Flipcy by 32× and 8×, respectively. Being orthogonal, if combined with any encoding scheme to reduce the frequency of error-prone patterns at cost of additional storage, DynaPAT can perform even better. 5 Conclusion Resistance drift severely impacts the accuracy of MLC PCM-based DNN systems. This paper analyzes the DNN’s behavior when different data pattern switching due to resistance drift errors are considered. Based on this analysis, this paper presents DynaPAT, a dynamic pattern-aware encoding technique for robust MLC PCM-based Deep Neural Networks. DynaPAT is lightweight yet effective in reducing the impact of drift errors in MLC PCM-based DNN systems. By efficiently mapping the most-frequent patterns to the least-error-prone levels of MLC PCM in the order of the criticality in DNNs, DynaPAT is able to significantly enhance the robustness of the system by orders of magnitude. DynaPAT is evaluated and compared with other relevant state-of-the-art encoding techniques on DNN models. It is found that DynaPAT can achieve up to 500 × robustness over the baseline models (without any fault-tolerance technique). The results are consistent even when using different DNN architectures, configurations and datasets. DynaPAT adds negligible area, storage and latency overhead compared to the existing techniques, while still being more effective in tolerating drift errors. Lastly, being orthogonal to other encoding methods, DynaPAT can be integrated with them to further enhance the robustness of MLC PCM-based DNN systems. Acknowledgments This work was supported in part by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea Government (MSIT) under Grant 2022-0-00971; in part by the Next Generation Intelligent Semiconductor Development by the Ministry of Trade, Industry and Energy (MOTIE) under Grant 20011074; and in part by the Basic Science Research Program through the National Research Foundation of Korea funded by the Ministry of Education under Grant NRF-2020M3F3A2A01082326.

Publisher Copyright:
© 2022 Association for Computing Machinery.

All Science Journal Classification (ASJC) codes

Software
Computer Science Applications
Computer Graphics and Computer-Aided Design

Access to Document

10.1145/3508352.3549400

Cite this

Nguyen, T. H., Imran, M., & Yang, J. S. (2022). DynaPAT: A dynamic pattern-aware encoding technique for robust MLC PCM-based deep neural networks. In Proceedings of the 41st IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2022 [138] (IEEE/ACM International Conference on Computer-Aided Design, Digest of Technical Papers, ICCAD). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1145/3508352.3549400

Nguyen, Thai Hoang ; Imran, Muhammad ; Yang, Joon Sung. / DynaPAT : A dynamic pattern-aware encoding technique for robust MLC PCM-based deep neural networks. Proceedings of the 41st IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2022. Institute of Electrical and Electronics Engineers Inc., 2022. (IEEE/ACM International Conference on Computer-Aided Design, Digest of Technical Papers, ICCAD).

@inproceedings{60bdd19686034999be6f08b05520933d,

title = "DynaPAT: A dynamic pattern-aware encoding technique for robust MLC PCM-based deep neural networks",

abstract = "As the effectiveness of Deep Neural Networks (DNNs) is rising over time, so is the need for highly scalable and efficient hardware architectures to capitalize this effectiveness in many practical applications. Emerging non-volatile Phase Change Memory (PCM) technology has been found to be a promising candidate for future memory systems due to its better scalability, non-volatility and low leakage/dynamic power consumption, compared to conventional charged-based memories. Additionally, with its cell s wide resistance span, PCM also has the Flash-like Multi-Level Cell (MLC) capability, which has enhanced storage density, providing an opportunity for the deployment of data-intensive applications such as DNNs on resource-constrained edge devices. However, the practical deployment of MLC PCM is hampered by certain reliability challenges, among which, the resistance drift is considered to be a critical concern. In a DNN application, the presence of resistance drift in MLC PCM can cause a severe impact to DNN s accuracy if no drift-error-tolerance technique is utilized. This paper proposes DynaPAT, a low-cost and effective pattern-aware encoding technique to enhance the drift-error-tolerance of MLC PCM-based Deep Neural Networks. DynaPAT has been constructed on the insight into DNN s vulnerability against different data pattern switching. Based on this insight, DynaPAT efficiently maps the most-frequent data pattern in DNN s parameters to the least-drift-prone level of the MLC PCM, thus significantly enhancing the robustness of the system against drift errors. Various experiments on different DNN models and configurations demonstrate the effectiveness of DynaPAT. The experimental results indicate that DynaPAT can achieve up to 500× enhancement in the drift-errors-tolerance capability over the baseline MLC PCM based DNN while requiring only a negligible hardware overhead (below 1% storage overhead). Being orthogonal, DynaPAT can be integrated with existing drifttolerance schemes for even higher gains in reliability.",

author = "Nguyen, {Thai Hoang} and Muhammad Imran and Yang, {Joon Sung}",

note = "Funding Information: This work was supported in part by the Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korea Government (MSIT) under Grant 2022-0-00971; in part by the Next Generation Intelligent Semiconductor Development by the Ministry of Trade, Industry and Energy (MOTIE) under Grant 20011074; and in part by the Basic Science Research Program through the National Research Foundation of Korea funded by the Ministry of Education under Grant NRF-2020M3F3A2A01082326 Funding Information: The performance of DynaPAT compared with other methods when evaluated on ImageNet dataset using ResNet-50, DensenNet-169 and Inception-v3, is shown in Fig. 12. Note that ImageNet is a more difficult challenge compared with the CIFAR datasets since it contains 1000 classes as opposed to 10/100 classes in CIFAR datasets. Thus, the performance of the DynaPAT as well as other fault-tolerance techniques is expected to be lower than in case of CIFAR-10. Additionally, in our evaluation, the models evaluated on ImageNet dataset also have more layers (deeper) than the models evaluated on CIFAR datasets, hence, the fault-tolerance methods tend to be less effective. This observation is consistent with previous works [7, 18] that evaluate DNNs considering the presence of hardware faults. Nevertheless, DynaPAT still shows a significant robustness improvement over the baseline as well as other techniques. For example, for 8-bit quantized DenseNet-169 network, DynaPAT enhances the robustness over the baseline and Flipcy by 32× and 8×, respectively. Being orthogonal, if combined with any encoding scheme to reduce the frequency of error-prone patterns at cost of additional storage, DynaPAT can perform even better. 5 Conclusion Resistance drift severely impacts the accuracy of MLC PCM-based DNN systems. This paper analyzes the DNN{\textquoteright}s behavior when different data pattern switching due to resistance drift errors are considered. Based on this analysis, this paper presents DynaPAT, a dynamic pattern-aware encoding technique for robust MLC PCM-based Deep Neural Networks. DynaPAT is lightweight yet effective in reducing the impact of drift errors in MLC PCM-based DNN systems. By efficiently mapping the most-frequent patterns to the least-error-prone levels of MLC PCM in the order of the criticality in DNNs, DynaPAT is able to significantly enhance the robustness of the system by orders of magnitude. DynaPAT is evaluated and compared with other relevant state-of-the-art encoding techniques on DNN models. It is found that DynaPAT can achieve up to 500 × robustness over the baseline models (without any fault-tolerance technique). The results are consistent even when using different DNN architectures, configurations and datasets. DynaPAT adds negligible area, storage and latency overhead compared to the existing techniques, while still being more effective in tolerating drift errors. Lastly, being orthogonal to other encoding methods, DynaPAT can be integrated with them to further enhance the robustness of MLC PCM-based DNN systems. Acknowledgments This work was supported in part by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea Government (MSIT) under Grant 2022-0-00971; in part by the Next Generation Intelligent Semiconductor Development by the Ministry of Trade, Industry and Energy (MOTIE) under Grant 20011074; and in part by the Basic Science Research Program through the National Research Foundation of Korea funded by the Ministry of Education under Grant NRF-2020M3F3A2A01082326. Publisher Copyright: {\textcopyright} 2022 Association for Computing Machinery.; 41st IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2022 ; Conference date: 30-10-2022 Through 04-11-2022",

year = "2022",

month = oct,

day = "30",

doi = "10.1145/3508352.3549400",

language = "English",

series = "IEEE/ACM International Conference on Computer-Aided Design, Digest of Technical Papers, ICCAD",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "Proceedings of the 41st IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2022",

address = "United States",

}

Nguyen, TH, Imran, M & Yang, JS 2022, DynaPAT: A dynamic pattern-aware encoding technique for robust MLC PCM-based deep neural networks. in Proceedings of the 41st IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2022., 138, IEEE/ACM International Conference on Computer-Aided Design, Digest of Technical Papers, ICCAD, Institute of Electrical and Electronics Engineers Inc., 41st IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2022, San Diego, United States, 22/10/30. https://doi.org/10.1145/3508352.3549400

DynaPAT : A dynamic pattern-aware encoding technique for robust MLC PCM-based deep neural networks. / Nguyen, Thai Hoang; Imran, Muhammad; Yang, Joon Sung.

Proceedings of the 41st IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2022. Institute of Electrical and Electronics Engineers Inc., 2022. 138 (IEEE/ACM International Conference on Computer-Aided Design, Digest of Technical Papers, ICCAD).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - DynaPAT

T2 - 41st IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2022

AU - Nguyen, Thai Hoang

AU - Imran, Muhammad

AU - Yang, Joon Sung

N1 - Funding Information: This work was supported in part by the Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korea Government (MSIT) under Grant 2022-0-00971; in part by the Next Generation Intelligent Semiconductor Development by the Ministry of Trade, Industry and Energy (MOTIE) under Grant 20011074; and in part by the Basic Science Research Program through the National Research Foundation of Korea funded by the Ministry of Education under Grant NRF-2020M3F3A2A01082326 Funding Information: The performance of DynaPAT compared with other methods when evaluated on ImageNet dataset using ResNet-50, DensenNet-169 and Inception-v3, is shown in Fig. 12. Note that ImageNet is a more difficult challenge compared with the CIFAR datasets since it contains 1000 classes as opposed to 10/100 classes in CIFAR datasets. Thus, the performance of the DynaPAT as well as other fault-tolerance techniques is expected to be lower than in case of CIFAR-10. Additionally, in our evaluation, the models evaluated on ImageNet dataset also have more layers (deeper) than the models evaluated on CIFAR datasets, hence, the fault-tolerance methods tend to be less effective. This observation is consistent with previous works [7, 18] that evaluate DNNs considering the presence of hardware faults. Nevertheless, DynaPAT still shows a significant robustness improvement over the baseline as well as other techniques. For example, for 8-bit quantized DenseNet-169 network, DynaPAT enhances the robustness over the baseline and Flipcy by 32× and 8×, respectively. Being orthogonal, if combined with any encoding scheme to reduce the frequency of error-prone patterns at cost of additional storage, DynaPAT can perform even better. 5 Conclusion Resistance drift severely impacts the accuracy of MLC PCM-based DNN systems. This paper analyzes the DNN’s behavior when different data pattern switching due to resistance drift errors are considered. Based on this analysis, this paper presents DynaPAT, a dynamic pattern-aware encoding technique for robust MLC PCM-based Deep Neural Networks. DynaPAT is lightweight yet effective in reducing the impact of drift errors in MLC PCM-based DNN systems. By efficiently mapping the most-frequent patterns to the least-error-prone levels of MLC PCM in the order of the criticality in DNNs, DynaPAT is able to significantly enhance the robustness of the system by orders of magnitude. DynaPAT is evaluated and compared with other relevant state-of-the-art encoding techniques on DNN models. It is found that DynaPAT can achieve up to 500 × robustness over the baseline models (without any fault-tolerance technique). The results are consistent even when using different DNN architectures, configurations and datasets. DynaPAT adds negligible area, storage and latency overhead compared to the existing techniques, while still being more effective in tolerating drift errors. Lastly, being orthogonal to other encoding methods, DynaPAT can be integrated with them to further enhance the robustness of MLC PCM-based DNN systems. Acknowledgments This work was supported in part by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea Government (MSIT) under Grant 2022-0-00971; in part by the Next Generation Intelligent Semiconductor Development by the Ministry of Trade, Industry and Energy (MOTIE) under Grant 20011074; and in part by the Basic Science Research Program through the National Research Foundation of Korea funded by the Ministry of Education under Grant NRF-2020M3F3A2A01082326. Publisher Copyright: © 2022 Association for Computing Machinery.

PY - 2022/10/30

Y1 - 2022/10/30

N2 - As the effectiveness of Deep Neural Networks (DNNs) is rising over time, so is the need for highly scalable and efficient hardware architectures to capitalize this effectiveness in many practical applications. Emerging non-volatile Phase Change Memory (PCM) technology has been found to be a promising candidate for future memory systems due to its better scalability, non-volatility and low leakage/dynamic power consumption, compared to conventional charged-based memories. Additionally, with its cell s wide resistance span, PCM also has the Flash-like Multi-Level Cell (MLC) capability, which has enhanced storage density, providing an opportunity for the deployment of data-intensive applications such as DNNs on resource-constrained edge devices. However, the practical deployment of MLC PCM is hampered by certain reliability challenges, among which, the resistance drift is considered to be a critical concern. In a DNN application, the presence of resistance drift in MLC PCM can cause a severe impact to DNN s accuracy if no drift-error-tolerance technique is utilized. This paper proposes DynaPAT, a low-cost and effective pattern-aware encoding technique to enhance the drift-error-tolerance of MLC PCM-based Deep Neural Networks. DynaPAT has been constructed on the insight into DNN s vulnerability against different data pattern switching. Based on this insight, DynaPAT efficiently maps the most-frequent data pattern in DNN s parameters to the least-drift-prone level of the MLC PCM, thus significantly enhancing the robustness of the system against drift errors. Various experiments on different DNN models and configurations demonstrate the effectiveness of DynaPAT. The experimental results indicate that DynaPAT can achieve up to 500× enhancement in the drift-errors-tolerance capability over the baseline MLC PCM based DNN while requiring only a negligible hardware overhead (below 1% storage overhead). Being orthogonal, DynaPAT can be integrated with existing drifttolerance schemes for even higher gains in reliability.

AB - As the effectiveness of Deep Neural Networks (DNNs) is rising over time, so is the need for highly scalable and efficient hardware architectures to capitalize this effectiveness in many practical applications. Emerging non-volatile Phase Change Memory (PCM) technology has been found to be a promising candidate for future memory systems due to its better scalability, non-volatility and low leakage/dynamic power consumption, compared to conventional charged-based memories. Additionally, with its cell s wide resistance span, PCM also has the Flash-like Multi-Level Cell (MLC) capability, which has enhanced storage density, providing an opportunity for the deployment of data-intensive applications such as DNNs on resource-constrained edge devices. However, the practical deployment of MLC PCM is hampered by certain reliability challenges, among which, the resistance drift is considered to be a critical concern. In a DNN application, the presence of resistance drift in MLC PCM can cause a severe impact to DNN s accuracy if no drift-error-tolerance technique is utilized. This paper proposes DynaPAT, a low-cost and effective pattern-aware encoding technique to enhance the drift-error-tolerance of MLC PCM-based Deep Neural Networks. DynaPAT has been constructed on the insight into DNN s vulnerability against different data pattern switching. Based on this insight, DynaPAT efficiently maps the most-frequent data pattern in DNN s parameters to the least-drift-prone level of the MLC PCM, thus significantly enhancing the robustness of the system against drift errors. Various experiments on different DNN models and configurations demonstrate the effectiveness of DynaPAT. The experimental results indicate that DynaPAT can achieve up to 500× enhancement in the drift-errors-tolerance capability over the baseline MLC PCM based DNN while requiring only a negligible hardware overhead (below 1% storage overhead). Being orthogonal, DynaPAT can be integrated with existing drifttolerance schemes for even higher gains in reliability.

UR - http://www.scopus.com/inward/record.url?scp=85145647039&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85145647039&partnerID=8YFLogxK

U2 - 10.1145/3508352.3549400

DO - 10.1145/3508352.3549400

M3 - Conference contribution

AN - SCOPUS:85145647039

T3 - IEEE/ACM International Conference on Computer-Aided Design, Digest of Technical Papers, ICCAD

BT - Proceedings of the 41st IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2022

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 30 October 2022 through 4 November 2022

ER -

Nguyen TH, Imran M, Yang JS. DynaPAT: A dynamic pattern-aware encoding technique for robust MLC PCM-based deep neural networks. In Proceedings of the 41st IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2022. Institute of Electrical and Electronics Engineers Inc. 2022. 138. (IEEE/ACM International Conference on Computer-Aided Design, Digest of Technical Papers, ICCAD). doi: 10.1145/3508352.3549400

DynaPAT: A dynamic pattern-aware encoding technique for robust MLC PCM-based deep neural networks

Abstract

Publication series

Conference

Bibliographical note

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this