Enhancing network I/o performance for a virtualized Hadoop cluster

Jinkyu Jeong; Dong Hoon Choi; Heeseung Jo

doi:10.1002/cpe.3974

Enhancing network I/o performance for a virtualized Hadoop cluster

Jinkyu Jeong, Dong Hoon Choi, Heeseung Jo

Research output: Contribution to journal › Article › peer-review

Abstract

A MapReduce programming model is proposed to process big data using Hadoop, one of the major cloud computing frameworks. With the increasing adoption of cloud computing, running a Hadoop framework on a virtualized cluster is a compelling approach to reducing costs and increasing efficiency. In this paper, we measure the performance of a virtualized network and analyze the impact of network performance on Hadoop workloads running on a virtualized cluster. Then, we propose a virtualized network I/O architecture as a novel optimization for a virtualized Hadoop cluster for a public/private cloud provider. The proposed network architecture combines traditional network configurations and achieves better performance for Hadoop workloads. We also show a better way to utilize the rack awareness feature of the Hadoop framework in the proposed computing environment. The evaluation demonstrates that the proposed network architecture and mechanisms improve performance by up to 4.1 times compared with a bridge network architecture. This novel architecture can even virtually match the performance of the expensive, hardware-based single root I/O virtualization network architecture.

Original language	English
Article number	e3974
Journal	Concurrency Computation Practice and Experience
Volume	29
Issue number	8
DOIs	https://doi.org/10.1002/cpe.3974
Publication status	Published - 2017 Apr 25

Bibliographical note

Funding Information:
This work was supported by the ICT R&D program of MSIP/IITP (B0101-15-0644, The research project on High Performance and Scalable Manycore Operating System) and the Institute for Information and Communications Technology Promotion (IITP) grant funded by the Korean government (MSIP) (no. B0101-15-0104, The Development of Supercomputing System for the Genome Analysis). This paper was supported by research funds of Chonbuk National University in 2015. †Correction added on 17 February 2017, after first online publication: this sentence was moved from the Conclusion and Future Work section to the Acknowledgements.

Publisher Copyright:
Copyright © 2016 John Wiley & Sons, Ltd.

All Science Journal Classification (ASJC) codes

Theoretical Computer Science
Software
Computer Science Applications
Computer Networks and Communications
Computational Theory and Mathematics

Access to Document

10.1002/cpe.3974

Cite this

@article{185fb524c38a41709b12a97c3e4d2430,

title = "Enhancing network I/o performance for a virtualized Hadoop cluster",

abstract = "A MapReduce programming model is proposed to process big data using Hadoop, one of the major cloud computing frameworks. With the increasing adoption of cloud computing, running a Hadoop framework on a virtualized cluster is a compelling approach to reducing costs and increasing efficiency. In this paper, we measure the performance of a virtualized network and analyze the impact of network performance on Hadoop workloads running on a virtualized cluster. Then, we propose a virtualized network I/O architecture as a novel optimization for a virtualized Hadoop cluster for a public/private cloud provider. The proposed network architecture combines traditional network configurations and achieves better performance for Hadoop workloads. We also show a better way to utilize the rack awareness feature of the Hadoop framework in the proposed computing environment. The evaluation demonstrates that the proposed network architecture and mechanisms improve performance by up to 4.1 times compared with a bridge network architecture. This novel architecture can even virtually match the performance of the expensive, hardware-based single root I/O virtualization network architecture.",

author = "Jinkyu Jeong and Choi, {Dong Hoon} and Heeseung Jo",

note = "Funding Information: This work was supported by the ICT R&D program of MSIP/IITP (B0101-15-0644, The research project on High Performance and Scalable Manycore Operating System) and the Institute for Information and Communications Technology Promotion (IITP) grant funded by the Korean government (MSIP) (no. B0101-15-0104, The Development of Supercomputing System for the Genome Analysis). This paper was supported by research funds of Chonbuk National University in 2015. †Correction added on 17 February 2017, after first online publication: this sentence was moved from the Conclusion and Future Work section to the Acknowledgements. Publisher Copyright: Copyright {\textcopyright} 2016 John Wiley & Sons, Ltd.",

year = "2017",

month = apr,

day = "25",

doi = "10.1002/cpe.3974",

language = "English",

volume = "29",

journal = "Concurrency Computation Practice and Experience",

issn = "1532-0626",

publisher = "John Wiley and Sons Ltd",

number = "8",

}

TY - JOUR

T1 - Enhancing network I/o performance for a virtualized Hadoop cluster

AU - Jeong, Jinkyu

AU - Choi, Dong Hoon

AU - Jo, Heeseung

N1 - Funding Information: This work was supported by the ICT R&D program of MSIP/IITP (B0101-15-0644, The research project on High Performance and Scalable Manycore Operating System) and the Institute for Information and Communications Technology Promotion (IITP) grant funded by the Korean government (MSIP) (no. B0101-15-0104, The Development of Supercomputing System for the Genome Analysis). This paper was supported by research funds of Chonbuk National University in 2015. †Correction added on 17 February 2017, after first online publication: this sentence was moved from the Conclusion and Future Work section to the Acknowledgements. Publisher Copyright: Copyright © 2016 John Wiley & Sons, Ltd.

PY - 2017/4/25

Y1 - 2017/4/25

N2 - A MapReduce programming model is proposed to process big data using Hadoop, one of the major cloud computing frameworks. With the increasing adoption of cloud computing, running a Hadoop framework on a virtualized cluster is a compelling approach to reducing costs and increasing efficiency. In this paper, we measure the performance of a virtualized network and analyze the impact of network performance on Hadoop workloads running on a virtualized cluster. Then, we propose a virtualized network I/O architecture as a novel optimization for a virtualized Hadoop cluster for a public/private cloud provider. The proposed network architecture combines traditional network configurations and achieves better performance for Hadoop workloads. We also show a better way to utilize the rack awareness feature of the Hadoop framework in the proposed computing environment. The evaluation demonstrates that the proposed network architecture and mechanisms improve performance by up to 4.1 times compared with a bridge network architecture. This novel architecture can even virtually match the performance of the expensive, hardware-based single root I/O virtualization network architecture.

AB - A MapReduce programming model is proposed to process big data using Hadoop, one of the major cloud computing frameworks. With the increasing adoption of cloud computing, running a Hadoop framework on a virtualized cluster is a compelling approach to reducing costs and increasing efficiency. In this paper, we measure the performance of a virtualized network and analyze the impact of network performance on Hadoop workloads running on a virtualized cluster. Then, we propose a virtualized network I/O architecture as a novel optimization for a virtualized Hadoop cluster for a public/private cloud provider. The proposed network architecture combines traditional network configurations and achieves better performance for Hadoop workloads. We also show a better way to utilize the rack awareness feature of the Hadoop framework in the proposed computing environment. The evaluation demonstrates that the proposed network architecture and mechanisms improve performance by up to 4.1 times compared with a bridge network architecture. This novel architecture can even virtually match the performance of the expensive, hardware-based single root I/O virtualization network architecture.

UR - http://www.scopus.com/inward/record.url?scp=84992343457&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84992343457&partnerID=8YFLogxK

U2 - 10.1002/cpe.3974

DO - 10.1002/cpe.3974

M3 - Article

AN - SCOPUS:84992343457

SN - 1532-0626

VL - 29

JO - Concurrency Computation Practice and Experience

JF - Concurrency Computation Practice and Experience

IS - 8

M1 - e3974

ER -

Enhancing network I/o performance for a virtualized Hadoop cluster

Abstract

Bibliographical note

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this