Time Estimation and Resource Minimization Scheme for Apache Spark and Hadoop Big Data Systems with Failures

Jinbae Lee, Bobae Kim, Jong Moon Chung

Research output: Contribution to journalArticlepeer-review

7 Citations (Scopus)

Abstract

Apache Spark and Hadoop are open source frameworks for big data processing, which have been adopted by many companies. In order to implement a reliable big data system that can satisfy processing target completion times, accurate resource provisioning and job execution time estimations are needed. In this paper, time estimation and resource minimization schemes for Spark and Hadoop systems are presented. The proposed models use the probability of failure in the estimations to more accurately formulate the characteristics of real big data operations. The experimental results show that the proposed Spark adaptive failure-compensation and Hadoop adaptive failure-compensation schemes improve the accuracy of resource provisions by considering failure events, which improves the scheduling success rate of big data processing tasks.

Original languageEnglish
Article number8605312
Pages (from-to)9658-9666
Number of pages9
JournalIEEE Access
Volume7
DOIs
Publication statusPublished - 2019

Bibliographical note

Funding Information:
This research was supported by the grant MOIS-DP-2015-10 through the Disaster and Safety Management Institute, which is funded by the Ministry of Interior and Safety of the Republic of Korea Government.

All Science Journal Classification (ASJC) codes

  • Computer Science(all)
  • Materials Science(all)
  • Engineering(all)

Fingerprint Dive into the research topics of 'Time Estimation and Resource Minimization Scheme for Apache Spark and Hadoop Big Data Systems with Failures'. Together they form a unique fingerprint.

Cite this