Research Article
End-to-End Dynamic Pipelining Tuning Strategy for Small Files Transfer
@INPROCEEDINGS{10.1007/978-3-030-93479-8_2, author={Shimin Wu and Dawei Sun and Shang Gao and Guangyan Zhang}, title={End-to-End Dynamic Pipelining Tuning Strategy for Small Files Transfer}, proceedings={Broadband Communications, Networks, and Systems. 12th EAI International Conference, BROADNETS 2021, Virtual Event, October 28--29, 2021, Proceedings}, proceedings_a={BROADNETS}, year={2022}, month={1}, keywords={GridFTP Throughput optimization Pipelining and concurrency Lots of small files Network protocols}, doi={10.1007/978-3-030-93479-8_2} }- Shimin Wu
Dawei Sun
Shang Gao
Guangyan Zhang
Year: 2022
End-to-End Dynamic Pipelining Tuning Strategy for Small Files Transfer
BROADNETS
Springer
DOI: 10.1007/978-3-030-93479-8_2
Abstract
Improving the transmission efficiency for small files over a wide area network is always challenging. Time may be wasted when waiting for transmission commands due to the design of transfer protocols, which in turn increases the Round-trip time (RTT). GridFTP is widely deployed as a transfer protocol in the grid era, where a concept of pipelining is proposed to improve the transmission efficiency for small files. Based on the GridFTP protocol, we design a smart data structure to classify files and propose a corresponding scheduling algorithm to tune the pipelining parameters, making them more reasonable and adaptive to different transmission scenarios. Bandwidth usage is optimized when a large number of small files are transferred with our strategy by combining the optimal pipelining and concurrency parameters. A method to optimizing the throughput for high-priority file transfer is also proposed. By adjusting the pipelining parameter dynamically, the throughput is increased by almost 10% compared with other methods. Moreover, our method achieves better performance even with a smaller concurrency setting. The favorable throughput is maintained when transferring high-priority files.
