5th International ICST Conference on Collaborative Computing: Networking, Applications, Worksharing

Research Article

DiSK: A distributed shared disk cache for HPC environments

Download444 downloads
  • @INPROCEEDINGS{10.4108/ICST.COLLABORATECOM2009.8307 ,
        author={Brandon Szeliga and Tung Nguyen and Weisong Shi},
        title={DiSK: A distributed shared disk cache for HPC environments},
        proceedings={5th International ICST Conference on Collaborative Computing: Networking, Applications, Worksharing},
        proceedings_a={COLLABORATECOM},
        year={2009},
        month={12},
        keywords={Cache storage Computer networks Data analysis Delay effects Explosions Hardware Memory Monitoring Performance analysis Prototypes},
        doi={10.4108/ICST.COLLABORATECOM2009.8307 }
    }
    
  • Brandon Szeliga
    Tung Nguyen
    Weisong Shi
    Year: 2009
    DiSK: A distributed shared disk cache for HPC environments
    COLLABORATECOM
    ICST
    DOI: 10.4108/ICST.COLLABORATECOM2009.8307
Brandon Szeliga1,*, Tung Nguyen1,*, Weisong Shi1,*
  • 1: Wayne State University
*Contact email: aj3638@wayne.edu, nttung@wayne.edu, weisong@wayne.edu

Abstract

Data movement within high performance environments can be a large bottleneck to the overall performance of programs. With the addition of continuous storage and usage of older data, the back end storage is becoming a larger problem than the improving network and computational nodes. This has led us to develop a distributed shared disk cache, DiSK, to reduce the dependence on these back end storage systems. With DiSK requested files will be distributed across nodes in order to reduce the amount of requests directed to the archives. DiSK has two key components. One is a distributed metadata management, DIMM, scheme that allows a centralized manager to access what data is available in the system. This is accomplished through the use of a counter-based bloom filter with locality checks in order to reduce false positives and false negatives. The second component is a method of replication called differentiable replication, DiR. The novelty of DiR is that the requirements of the files and capabilities of underlying nodes are taken into consideration for replication. This allows for a varying degree of replication depending on the file. This customization of DiSK yields better performance than the conventional archive system.