Research Article
DiSK: A distributed shared disk cache for HPC environments
@INPROCEEDINGS{10.4108/ICST.COLLABORATECOM2009.8307 , author={Brandon Szeliga and Tung Nguyen and Weisong Shi}, title={DiSK: A distributed shared disk cache for HPC environments}, proceedings={5th International ICST Conference on Collaborative Computing: Networking, Applications, Worksharing}, proceedings_a={COLLABORATECOM}, year={2009}, month={12}, keywords={Cache storage Computer networks Data analysis Delay effects Explosions Hardware Memory Monitoring Performance analysis Prototypes}, doi={10.4108/ICST.COLLABORATECOM2009.8307 } }- Brandon Szeliga
Tung Nguyen
Weisong Shi
Year: 2009
DiSK: A distributed shared disk cache for HPC environments
COLLABORATECOM
ICST
DOI: 10.4108/ICST.COLLABORATECOM2009.8307
Abstract
Data movement within high performance environments can be a large bottleneck to the overall performance of programs. With the addition of continuous storage and usage of older data, the back end storage is becoming a larger problem than the improving network and computational nodes. This has led us to develop a distributed shared disk cache, DiSK, to reduce the dependence on these back end storage systems. With DiSK requested files will be distributed across nodes in order to reduce the amount of requests directed to the archives. DiSK has two key components. One is a distributed metadata management, DIMM, scheme that allows a centralized manager to access what data is available in the system. This is accomplished through the use of a counter-based bloom filter with locality checks in order to reduce false positives and false negatives. The second component is a method of replication called differentiable replication, DiR. The novelty of DiR is that the requirements of the files and capabilities of underlying nodes are taken into consideration for replication. This allows for a varying degree of replication depending on the file. This customization of DiSK yields better performance than the conventional archive system.
