Adaptive Trust Management and Data Process Time Optimization for Real-Time Spark Big Data Systems

Seungwoo Seo, Jong Moon Chung

Research output: Contribution to journalArticlepeer-review


Applications supporting businesses, smart systems, social networks, and advanced video applications such as eXtended Reality (XR) require large amounts of data processing to be provided in real-time. Therefore, the processing speed of big data systems is more important than ever. On the other hand, protecting a big data system is not easy, as various types of nodes and clusters are supported by various wired and wireless networks. Commonly security procedures slow down the response time of big data networks, and therefore, enhanced security and performance speed techniques need to be co-designed into the system. In this paper, a trusted streaming adaptive failure-compensation (TSAF) scheme is proposed that uses a trust management scheme to identify malicious nodes in Spark big data systems, exclude them from job/task processing, and calculate the number of nodes that can satisfy the process's object completion time. The TSAF scheme shows an improved processing performance when there are attacks on the big data system compared to other existing real-time big data processing schemes. For the case of no security attack, the results show that the processing time of TSAF is faster by about 1 2% compared to the existing big data processing schemes when the process completion object time is set to 0.5 s. Even when the ratio of malicious nodes performing security attacks on worker nodes reaches 0.5, the results show that TSAF can satisfy over 75% of the tasks within the object time, which is significantly higher compared to the existing big data processing schemes.

Original languageEnglish
Pages (from-to)156372-156379
Number of pages8
JournalIEEE Access
Publication statusPublished - 2021

Bibliographical note

Funding Information:
This work was supported by the Ministry of Science and ICT (MSIT), South Korea, under the Information Technology Research Center (ITRC) support Program through the Institute for Information and Communications Technology Planning and Evaluation (IITP) under Grant IITP-2021-2018-0-01799.

Publisher Copyright:
© 2013 IEEE.

All Science Journal Classification (ASJC) codes

  • Computer Science(all)
  • Materials Science(all)
  • Engineering(all)


Dive into the research topics of 'Adaptive Trust Management and Data Process Time Optimization for Real-Time Spark Big Data Systems'. Together they form a unique fingerprint.

Cite this