A write-friendly approach to manage namespace of Hadoop distributed file system by utilizing nonvolatile memory

Won Gi Choi, Sang Hyun Park

Research output: Contribution to journalArticle

Abstract

With the emergence of the big data era, various technologies have been proposed to cope with the exascale of data. For a considerably large volume of data, a single machine does not comprise enough resources to store the complete data. Hadoop distributed file system (HDFS) enables large datasets to be stored across the big data environment consisting of several machines. Although Hadoop has become a crucial part of the big data industry, because of its simple architecture which composed of master and slaves several problems such as scalability and performance bottleneck has been remained to solve. New storage technologies offer an opportunity to solve the problems and improve HDFS. We propose a novel management scheme for namespace metadata of HDFS by utilizing nonvolatile memory which has been mentioned as the next-generation device since flash memory devices. Nonvolatile memory, which can guarantee data persistence and high performance with byte-address access, alleviates Namenode bottlenecks resulting from journaling processes performed to preserve the file system’s metadata. Our proposed methods show significant improvement compared with block devices such as hard disk drive, solid-state drive in terms of NameNode performance.

Original languageEnglish
JournalJournal of Supercomputing
DOIs
Publication statusPublished - 2019 Jan 1

Fingerprint

Distributed File System
Metadata
Data storage equipment
Flash memory
Hard disk storage
Scalability
Flash Memory
File System
Single Machine
Big data
Industry
Large Data Sets
Persistence
High Performance
Resources

All Science Journal Classification (ASJC) codes

  • Software
  • Theoretical Computer Science
  • Information Systems
  • Hardware and Architecture

Cite this

@article{4ad12fe697aa4583bdb62c39cdaaace4,
title = "A write-friendly approach to manage namespace of Hadoop distributed file system by utilizing nonvolatile memory",
abstract = "With the emergence of the big data era, various technologies have been proposed to cope with the exascale of data. For a considerably large volume of data, a single machine does not comprise enough resources to store the complete data. Hadoop distributed file system (HDFS) enables large datasets to be stored across the big data environment consisting of several machines. Although Hadoop has become a crucial part of the big data industry, because of its simple architecture which composed of master and slaves several problems such as scalability and performance bottleneck has been remained to solve. New storage technologies offer an opportunity to solve the problems and improve HDFS. We propose a novel management scheme for namespace metadata of HDFS by utilizing nonvolatile memory which has been mentioned as the next-generation device since flash memory devices. Nonvolatile memory, which can guarantee data persistence and high performance with byte-address access, alleviates Namenode bottlenecks resulting from journaling processes performed to preserve the file system’s metadata. Our proposed methods show significant improvement compared with block devices such as hard disk drive, solid-state drive in terms of NameNode performance.",
author = "Choi, {Won Gi} and Park, {Sang Hyun}",
year = "2019",
month = "1",
day = "1",
doi = "10.1007/s11227-019-02876-9",
language = "English",
journal = "Journal of Supercomputing",
issn = "0920-8542",
publisher = "Springer Netherlands",

}

TY - JOUR

T1 - A write-friendly approach to manage namespace of Hadoop distributed file system by utilizing nonvolatile memory

AU - Choi, Won Gi

AU - Park, Sang Hyun

PY - 2019/1/1

Y1 - 2019/1/1

N2 - With the emergence of the big data era, various technologies have been proposed to cope with the exascale of data. For a considerably large volume of data, a single machine does not comprise enough resources to store the complete data. Hadoop distributed file system (HDFS) enables large datasets to be stored across the big data environment consisting of several machines. Although Hadoop has become a crucial part of the big data industry, because of its simple architecture which composed of master and slaves several problems such as scalability and performance bottleneck has been remained to solve. New storage technologies offer an opportunity to solve the problems and improve HDFS. We propose a novel management scheme for namespace metadata of HDFS by utilizing nonvolatile memory which has been mentioned as the next-generation device since flash memory devices. Nonvolatile memory, which can guarantee data persistence and high performance with byte-address access, alleviates Namenode bottlenecks resulting from journaling processes performed to preserve the file system’s metadata. Our proposed methods show significant improvement compared with block devices such as hard disk drive, solid-state drive in terms of NameNode performance.

AB - With the emergence of the big data era, various technologies have been proposed to cope with the exascale of data. For a considerably large volume of data, a single machine does not comprise enough resources to store the complete data. Hadoop distributed file system (HDFS) enables large datasets to be stored across the big data environment consisting of several machines. Although Hadoop has become a crucial part of the big data industry, because of its simple architecture which composed of master and slaves several problems such as scalability and performance bottleneck has been remained to solve. New storage technologies offer an opportunity to solve the problems and improve HDFS. We propose a novel management scheme for namespace metadata of HDFS by utilizing nonvolatile memory which has been mentioned as the next-generation device since flash memory devices. Nonvolatile memory, which can guarantee data persistence and high performance with byte-address access, alleviates Namenode bottlenecks resulting from journaling processes performed to preserve the file system’s metadata. Our proposed methods show significant improvement compared with block devices such as hard disk drive, solid-state drive in terms of NameNode performance.

UR - http://www.scopus.com/inward/record.url?scp=85065700386&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85065700386&partnerID=8YFLogxK

U2 - 10.1007/s11227-019-02876-9

DO - 10.1007/s11227-019-02876-9

M3 - Article

JO - Journal of Supercomputing

JF - Journal of Supercomputing

SN - 0920-8542

ER -