Time Estimation and Resource Minimization Scheme for Apache Spark and Hadoop Big Data Systems with Failures

Jinbae Lee; Bobae Kim; Jong-Moon Chung

doi:10.1109/ACCESS.2019.2891001

Time Estimation and Resource Minimization Scheme for Apache Spark and Hadoop Big Data Systems with Failures

Jinbae Lee, Bobae Kim, Jong-Moon Chung

Department of Electrical and Electronic Engineering

Research output: Contribution to journal › Article

2 Citations (Scopus)

Abstract

Apache Spark and Hadoop are open source frameworks for big data processing, which have been adopted by many companies. In order to implement a reliable big data system that can satisfy processing target completion times, accurate resource provisioning and job execution time estimations are needed. In this paper, time estimation and resource minimization schemes for Spark and Hadoop systems are presented. The proposed models use the probability of failure in the estimations to more accurately formulate the characteristics of real big data operations. The experimental results show that the proposed Spark adaptive failure-compensation and Hadoop adaptive failure-compensation schemes improve the accuracy of resource provisions by considering failure events, which improves the scheduling success rate of big data processing tasks.

Original language	English
Article number	8605312
Pages (from-to)	9658-9666
Number of pages	9
Journal	IEEE Access
Volume	7
DOIs	https://doi.org/10.1109/ACCESS.2019.2891001
Publication status	Published - 2019 Jan 1

Fingerprint

All Science Journal Classification (ASJC) codes

Computer Science(all)
Materials Science(all)
Engineering(all)

Cite this

Lee, J., Kim, B., & Chung, J-M. (2019). Time Estimation and Resource Minimization Scheme for Apache Spark and Hadoop Big Data Systems with Failures. IEEE Access, 7, 9658-9666. [8605312]. https://doi.org/10.1109/ACCESS.2019.2891001

@article{030203a3248348d49d686837a6c1992e,

title = "Time Estimation and Resource Minimization Scheme for Apache Spark and Hadoop Big Data Systems with Failures",

abstract = "Apache Spark and Hadoop are open source frameworks for big data processing, which have been adopted by many companies. In order to implement a reliable big data system that can satisfy processing target completion times, accurate resource provisioning and job execution time estimations are needed. In this paper, time estimation and resource minimization schemes for Spark and Hadoop systems are presented. The proposed models use the probability of failure in the estimations to more accurately formulate the characteristics of real big data operations. The experimental results show that the proposed Spark adaptive failure-compensation and Hadoop adaptive failure-compensation schemes improve the accuracy of resource provisions by considering failure events, which improves the scheduling success rate of big data processing tasks.",

author = "Jinbae Lee and Bobae Kim and Jong-Moon Chung",

year = "2019",

month = jan

day = "1",

doi = "10.1109/ACCESS.2019.2891001",

language = "English",

volume = "7",

pages = "9658--9666",

journal = "IEEE Access",

issn = "2169-3536",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - JOUR

T1 - Time Estimation and Resource Minimization Scheme for Apache Spark and Hadoop Big Data Systems with Failures

AU - Lee, Jinbae

AU - Kim, Bobae

AU - Chung, Jong-Moon

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Apache Spark and Hadoop are open source frameworks for big data processing, which have been adopted by many companies. In order to implement a reliable big data system that can satisfy processing target completion times, accurate resource provisioning and job execution time estimations are needed. In this paper, time estimation and resource minimization schemes for Spark and Hadoop systems are presented. The proposed models use the probability of failure in the estimations to more accurately formulate the characteristics of real big data operations. The experimental results show that the proposed Spark adaptive failure-compensation and Hadoop adaptive failure-compensation schemes improve the accuracy of resource provisions by considering failure events, which improves the scheduling success rate of big data processing tasks.

AB - Apache Spark and Hadoop are open source frameworks for big data processing, which have been adopted by many companies. In order to implement a reliable big data system that can satisfy processing target completion times, accurate resource provisioning and job execution time estimations are needed. In this paper, time estimation and resource minimization schemes for Spark and Hadoop systems are presented. The proposed models use the probability of failure in the estimations to more accurately formulate the characteristics of real big data operations. The experimental results show that the proposed Spark adaptive failure-compensation and Hadoop adaptive failure-compensation schemes improve the accuracy of resource provisions by considering failure events, which improves the scheduling success rate of big data processing tasks.

UR - http://www.scopus.com/inward/record.url?scp=85061182412&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85061182412&partnerID=8YFLogxK

U2 - 10.1109/ACCESS.2019.2891001

DO - 10.1109/ACCESS.2019.2891001

M3 - Article

AN - SCOPUS:85061182412

VL - 7

SP - 9658

EP - 9666

JO - IEEE Access

JF - IEEE Access

SN - 2169-3536

M1 - 8605312

ER -

Access to Document

10.1109/ACCESS.2019.2891001