Design of global data deduplication for a scale-out distributed storage system

Myoungwon Oh, Sejin Park, Jungyeon Yoon, Sangjae Kim, Kang Won Lee, Sage Weil, Heon Y. Yeom, Myoungsoo Jung

Research output: Chapter in Book/Report/Conference proceedingConference contribution

12 Citations (Scopus)


Scale-out distributed storage systems can uphold balanced data growth in terms of capacity and performance on an on-demand basis. However, it is a challenge to store and manage large sets of contents being generated by the explosion of data. One of the promising solutions to mitigate big data issues is data deduplication, which removes redundant data across many nodes of the storage system. Nevertheless, it is non-trivial to apply a conventional deduplication design to the scale-out storage due to the following root causes. First, chunk-lookup for deduplication is not as scalable and extendable as the underlying storage system supports. Second, managing the metadata associated to deduplication requires a huge amount of design and implementation modifications of the existing distributed storage system. Lastly, the data processing and additional I/O traffic imposed by deduplication can significantly degrade performance of the scale-out storage. To address these challenges, we propose a new deduplication method, which is highly scalable and compatible with the existing scale-out storage. Specifically, our deduplication method employs a double hashing algorithm that leverages hashes used by the underlying scale-out storage, which addresses the limits of current fingerprint hashing. In addition, our design integrates the meta-information of file system and deduplication into a single object, and it controls the deduplication ratio at online by being aware of system demands based on post-processing. We implemented the proposed deduplication method on an open source scale-out storage. The experimental results show that our design can save more than 90% of the total amount of storage space, under the execution of diverse standard storage workloads, while offering the same or similar performance, compared to the conventional scale-out storage.

Original languageEnglish
Title of host publicationProceedings - 2018 IEEE 38th International Conference on Distributed Computing Systems, ICDCS 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages11
ISBN (Electronic)9781538668719
Publication statusPublished - 2018 Jul 19
Event38th IEEE International Conference on Distributed Computing Systems, ICDCS 2018 - Vienna, Austria
Duration: 2018 Jul 22018 Jul 5

Publication series

NameProceedings - International Conference on Distributed Computing Systems


Other38th IEEE International Conference on Distributed Computing Systems, ICDCS 2018

Bibliographical note

Funding Information:
This research was partly supported by National Research Foundation of Korea (2015M3C4A7065646).

Publisher Copyright:
© 2018 IEEE.

All Science Journal Classification (ASJC) codes

  • Software
  • Hardware and Architecture
  • Computer Networks and Communications


Dive into the research topics of 'Design of global data deduplication for a scale-out distributed storage system'. Together they form a unique fingerprint.

Cite this