Testbeds and Research Infrastructures for the Development of Networks and Communications. 13th EAI International Conference, TridentCom 2018, Shanghai, China, December 1-3, 2018, Proceedings

Research Article

Throughput Analytics of Data Transfer Infrastructures

  • @INPROCEEDINGS{10.1007/978-3-030-12971-2_2,
        author={Nageswara Rao and Qiang Liu and Zhengchun Liu and Rajkumar Kettimuthu and Ian Foster},
        title={Throughput Analytics of Data Transfer Infrastructures},
        proceedings={Testbeds and Research Infrastructures for the Development of Networks and Communications. 13th EAI International Conference, TridentCom 2018, Shanghai, China, December 1-3, 2018, Proceedings},
        proceedings_a={TRIDENTCOM},
        year={2019},
        month={2},
        keywords={Data transfer Infrastructure Throughput profile},
        doi={10.1007/978-3-030-12971-2_2}
    }
    
  • Nageswara Rao
    Qiang Liu
    Zhengchun Liu
    Rajkumar Kettimuthu
    Ian Foster
    Year: 2019
    Throughput Analytics of Data Transfer Infrastructures
    TRIDENTCOM
    Springer
    DOI: 10.1007/978-3-030-12971-2_2
Nageswara Rao1,*, Qiang Liu1,*, Zhengchun Liu2,*, Rajkumar Kettimuthu2,*, Ian Foster2,*
  • 1: Oak Ridge National Laboratory
  • 2: Argonne National Laboratory
*Contact email: raons@ornl.gov, liuq1@ornl.gov, zhengchun.liu@anl.gov, kettimut@anl.gov, foster@anl.gov

Abstract

To support increasingly distributed scientific and big-data applications, powerful data transfer infrastructures are being built with dedicated networks and software frameworks customized to distributed file systems and data transfer nodes. The data transfer performance of such infrastructures critically depends on the combined choices of file, disk, and host systems as well as network protocols and file transfer software, all of which may vary across sites. The randomness of throughput measurements makes it challenging to assess the impact of these choices on the performance of infrastructure or its parts. We propose regression-based throughput profiles by aggregating measurements from sites of the infrastructure, with RTT as the independent variable. The peak values and convex-concave shape of a profile together determine the overall throughput performance of memory and file transfers, and its variations show the performance differences among the sites. We then present projection and difference operators, and coefficients of throughput profiles to characterize the performance of infrastructure and its parts, including sites and file transfer tools. In particular, the utilization-concavity coefficient provides a value in the range [0, 1] that reflects overall transfer effectiveness. We present results of measurements collected using (i) testbed experiments over dedicated 0–366 ms 10 Gbps connections with combinations of TCP versions, file systems, host systems and transfer tools, and (ii) Globus GridFTP transfers over production infrastructure with varying site configurations.