1st International ICST Conference on Networks for Grid Applications

Research Article

A Peer-to-Peer Infrastructure for Distributing Large Scientific Data Sets over Wide Area High-Performance Networks: Experimental Studies Using Wide Area Layer 2 Services

Download629 downloads
  • @INPROCEEDINGS{10.4108/gridnets.2007.2166,
        author={Yunhong Gu and Robert L. Grossman and Joe Mambretti},
        title={A Peer-to-Peer Infrastructure for Distributing Large Scientific Data Sets over Wide Area High-Performance Networks: Experimental Studies Using Wide Area Layer 2 Services},
        proceedings={1st International ICST Conference on Networks for Grid Applications},
        publisher={ICST},
        proceedings_a={GRIDNETS},
        year={2007},
        month={10},
        keywords={Distributed Data Storage High Performance Networks},
        doi={10.4108/gridnets.2007.2166}
    }
    
  • Yunhong Gu
    Robert L. Grossman
    Joe Mambretti
    Year: 2007
    A Peer-to-Peer Infrastructure for Distributing Large Scientific Data Sets over Wide Area High-Performance Networks: Experimental Studies Using Wide Area Layer 2 Services
    GRIDNETS
    ICST
    DOI: 10.4108/gridnets.2007.2166
Yunhong Gu1, Robert L. Grossman1, Joe Mambretti2
  • 1: National Center for Data Mining, University of Illinois at Chicago. www.ncdm.uic.edu
  • 2: International Center for Advanced Internet Research, Northwestern University. www.icair.org

Abstract

This paper presents Sector, a distributed environment that was created specifically to address the challenges inherent in accessing, exploring, analyzing and transporting extremely large scientific datasets over high performance wide area networks. To date, the effective utilization of such datasets has been limited because accessing and transporting large remote data sets in traditional distributed computing environments is often a challenge. Sector was designed and developed to eliminate these barriers. The Sector architecture incorporates specialized communications services and specialized data services that are designed for high volume data flows over wide area high performance optical networks. This design employs several innovative techniques to ensure that data flows are maximized at all times and at all locations required regardless of distances. This architectural design has been implemented on a prototype international experimental testbed. The results and implications of these preliminary tests are described.