Research Article
Throughput Analytics of Data Transfer Infrastructures
@INPROCEEDINGS{10.1007/978-3-030-12971-2_2, author={Nageswara Rao and Qiang Liu and Zhengchun Liu and Rajkumar Kettimuthu and Ian Foster}, title={Throughput Analytics of Data Transfer Infrastructures}, proceedings={Testbeds and Research Infrastructures for the Development of Networks and Communications. 13th EAI International Conference, TridentCom 2018, Shanghai, China, December 1-3, 2018, Proceedings}, proceedings_a={TRIDENTCOM}, year={2019}, month={2}, keywords={Data transfer Infrastructure Throughput profile}, doi={10.1007/978-3-030-12971-2_2} }- Nageswara Rao
Qiang Liu
Zhengchun Liu
Rajkumar Kettimuthu
Ian Foster
Year: 2019
Throughput Analytics of Data Transfer Infrastructures
TRIDENTCOM
Springer
DOI: 10.1007/978-3-030-12971-2_2
Abstract
To support increasingly distributed scientific and big-data applications, powerful data transfer infrastructures are being built with dedicated networks and software frameworks customized to distributed file systems and data transfer nodes. The data transfer performance of such infrastructures critically depends on the combined choices of file, disk, and host systems as well as network protocols and file transfer software, all of which may vary across sites. The randomness of throughput measurements makes it challenging to assess the impact of these choices on the performance of infrastructure or its parts. We propose regression-based throughput profiles by aggregating measurements from sites of the infrastructure, with RTT as the independent variable. The peak values and convex-concave shape of a profile together determine the overall throughput performance of memory and file transfers, and its variations show the performance differences among the sites. We then present projection and difference operators, and coefficients of throughput profiles to characterize the performance of infrastructure and its parts, including sites and file transfer tools. In particular, the utilization-concavity coefficient provides a value in the range [0, 1] that reflects overall transfer effectiveness. We present results of measurements collected using (i) testbed experiments over dedicated 0–366 ms 10 Gbps connections with combinations of TCP versions, file systems, host systems and transfer tools, and (ii) Globus GridFTP transfers over production infrastructure with varying site configurations.