Efficient attribute-based data access in astronomy analysis

B. Ma, A. Shoshani, A. Sim, K. Wu, Y. Byun, J. Hahm, M. S. Shin

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Citations (Scopus)

Abstract

Large experiments and high-performance computer models generate many petabytes of data. While Cloud Computing systems may meet the needs for analyzing these petabytes by harnessing the computing power of many distributed computers, the key challenge in effectively utilizing such a distributed system is the data management process, including storage, indexing, searching, accessing, and transferring data. Most analysis tasks perform computations on a subset of a large data records satisfying some user specified constraints on attribute (variable) values. This subsetting procedure is extremely important in that it reduces the network traffic to and from the cloud facilities. However, selected data records often span many different data files, and extracting the values out these files can be time-consuming especially if the number of files is large. This work addresses this challenge of working with a large number of files. We use a set of astronomical data set as an example and use an efficient database indexing technique, called FastBit, to significantly speed up the subsetting and thus optimize network usage. Overall, we aim to provide transparent and highly efficient attribute-based data access to scientists through a web-based Astronomy Data Analysis Portal. We will discuss the system design, and options for managing an extremely large number of files while minimizing network usage and latency.

Original languageEnglish
Title of host publicationProceedings - 2012 SC Companion
Subtitle of host publicationHigh Performance Computing, Networking Storage and Analysis, SCC 2012
Pages562-571
Number of pages10
DOIs
Publication statusPublished - 2012 Dec 1
Event2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012 - Salt Lake City, UT, United States
Duration: 2012 Nov 102012 Nov 16

Publication series

NameProceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012

Other

Other2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012
CountryUnited States
CitySalt Lake City, UT
Period12/11/1012/11/16

Fingerprint

Astronomy
Cloud computing
Set theory
World Wide Web
Information management
Systems analysis
Experiments

All Science Journal Classification (ASJC) codes

  • Computational Theory and Mathematics
  • Computer Science Applications
  • Software

Cite this

Ma, B., Shoshani, A., Sim, A., Wu, K., Byun, Y., Hahm, J., & Shin, M. S. (2012). Efficient attribute-based data access in astronomy analysis. In Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012 (pp. 562-571). [6495862] (Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012). https://doi.org/10.1109/SC.Companion.2012.80
Ma, B. ; Shoshani, A. ; Sim, A. ; Wu, K. ; Byun, Y. ; Hahm, J. ; Shin, M. S. / Efficient attribute-based data access in astronomy analysis. Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012. 2012. pp. 562-571 (Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012).
@inproceedings{3c5ce061705c4b96bbd8ff8f64a28cb0,
title = "Efficient attribute-based data access in astronomy analysis",
abstract = "Large experiments and high-performance computer models generate many petabytes of data. While Cloud Computing systems may meet the needs for analyzing these petabytes by harnessing the computing power of many distributed computers, the key challenge in effectively utilizing such a distributed system is the data management process, including storage, indexing, searching, accessing, and transferring data. Most analysis tasks perform computations on a subset of a large data records satisfying some user specified constraints on attribute (variable) values. This subsetting procedure is extremely important in that it reduces the network traffic to and from the cloud facilities. However, selected data records often span many different data files, and extracting the values out these files can be time-consuming especially if the number of files is large. This work addresses this challenge of working with a large number of files. We use a set of astronomical data set as an example and use an efficient database indexing technique, called FastBit, to significantly speed up the subsetting and thus optimize network usage. Overall, we aim to provide transparent and highly efficient attribute-based data access to scientists through a web-based Astronomy Data Analysis Portal. We will discuss the system design, and options for managing an extremely large number of files while minimizing network usage and latency.",
author = "B. Ma and A. Shoshani and A. Sim and K. Wu and Y. Byun and J. Hahm and Shin, {M. S.}",
year = "2012",
month = "12",
day = "1",
doi = "10.1109/SC.Companion.2012.80",
language = "English",
isbn = "9780769549569",
series = "Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012",
pages = "562--571",
booktitle = "Proceedings - 2012 SC Companion",

}

Ma, B, Shoshani, A, Sim, A, Wu, K, Byun, Y, Hahm, J & Shin, MS 2012, Efficient attribute-based data access in astronomy analysis. in Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012., 6495862, Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012, pp. 562-571, 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012, Salt Lake City, UT, United States, 12/11/10. https://doi.org/10.1109/SC.Companion.2012.80

Efficient attribute-based data access in astronomy analysis. / Ma, B.; Shoshani, A.; Sim, A.; Wu, K.; Byun, Y.; Hahm, J.; Shin, M. S.

Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012. 2012. p. 562-571 6495862 (Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Efficient attribute-based data access in astronomy analysis

AU - Ma, B.

AU - Shoshani, A.

AU - Sim, A.

AU - Wu, K.

AU - Byun, Y.

AU - Hahm, J.

AU - Shin, M. S.

PY - 2012/12/1

Y1 - 2012/12/1

N2 - Large experiments and high-performance computer models generate many petabytes of data. While Cloud Computing systems may meet the needs for analyzing these petabytes by harnessing the computing power of many distributed computers, the key challenge in effectively utilizing such a distributed system is the data management process, including storage, indexing, searching, accessing, and transferring data. Most analysis tasks perform computations on a subset of a large data records satisfying some user specified constraints on attribute (variable) values. This subsetting procedure is extremely important in that it reduces the network traffic to and from the cloud facilities. However, selected data records often span many different data files, and extracting the values out these files can be time-consuming especially if the number of files is large. This work addresses this challenge of working with a large number of files. We use a set of astronomical data set as an example and use an efficient database indexing technique, called FastBit, to significantly speed up the subsetting and thus optimize network usage. Overall, we aim to provide transparent and highly efficient attribute-based data access to scientists through a web-based Astronomy Data Analysis Portal. We will discuss the system design, and options for managing an extremely large number of files while minimizing network usage and latency.

AB - Large experiments and high-performance computer models generate many petabytes of data. While Cloud Computing systems may meet the needs for analyzing these petabytes by harnessing the computing power of many distributed computers, the key challenge in effectively utilizing such a distributed system is the data management process, including storage, indexing, searching, accessing, and transferring data. Most analysis tasks perform computations on a subset of a large data records satisfying some user specified constraints on attribute (variable) values. This subsetting procedure is extremely important in that it reduces the network traffic to and from the cloud facilities. However, selected data records often span many different data files, and extracting the values out these files can be time-consuming especially if the number of files is large. This work addresses this challenge of working with a large number of files. We use a set of astronomical data set as an example and use an efficient database indexing technique, called FastBit, to significantly speed up the subsetting and thus optimize network usage. Overall, we aim to provide transparent and highly efficient attribute-based data access to scientists through a web-based Astronomy Data Analysis Portal. We will discuss the system design, and options for managing an extremely large number of files while minimizing network usage and latency.

UR - http://www.scopus.com/inward/record.url?scp=84876566269&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84876566269&partnerID=8YFLogxK

U2 - 10.1109/SC.Companion.2012.80

DO - 10.1109/SC.Companion.2012.80

M3 - Conference contribution

AN - SCOPUS:84876566269

SN - 9780769549569

T3 - Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012

SP - 562

EP - 571

BT - Proceedings - 2012 SC Companion

ER -

Ma B, Shoshani A, Sim A, Wu K, Byun Y, Hahm J et al. Efficient attribute-based data access in astronomy analysis. In Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012. 2012. p. 562-571. 6495862. (Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012). https://doi.org/10.1109/SC.Companion.2012.80