EAI Endorsed Transactions on Future Internet 17(12): e3

Research Article

Implementation of Network Cards Optimizations in Hadoop Cluster Data Transmissions

Download18 downloads
  • @ARTICLE{10.4108/eai.21-12-2017.153506,
        author={Okta Nurika and Mohd Fadzil Hassan and Nordin Zakaria},
        title={Implementation of Network Cards Optimizations in Hadoop Cluster Data Transmissions},
        journal={EAI Endorsed Transactions on Future Internet},
        volume={17},
        number={12},
        publisher={EAI},
        journal_a={UE},
        year={2017},
        month={12},
        keywords={Hadoop DataTransferProtocol; HDFS; Network Card Optimization.},
        doi={10.4108/eai.21-12-2017.153506}
    }
    
  • Okta Nurika
    Mohd Fadzil Hassan
    Nordin Zakaria
    Year: 2017
    Implementation of Network Cards Optimizations in Hadoop Cluster Data Transmissions
    UE
    EAI
    DOI: 10.4108/eai.21-12-2017.153506
Okta Nurika1,*, Mohd Fadzil Hassan1, Nordin Zakaria1
  • 1: High Performance Cloud Computing Centre (HPC3), Universiti Teknologi PETRONAS
*Contact email: okta.rider@gmail.com

Abstract

In this paper, the previously invented new methods of network card optimization are applied in a Hadoop cluster, where data transfers occur from the Master to the slave node. The slave node's network card setting is optimized subjective to the characteristics of the incoming data transmissions, which are indicated by the overall transmission size and packet size. The throughput comparisons between the optimized network card settings and the default setting conclude that the optimized versions always generate higher throughputs. Synchronously, the optimized settings also minimize CPU cycles utilization as they deploy timer-based polling (passive wait mode), in order to process the received data packets. This novel practice within Hadoop cluster may be replicated by other data cluster vendors, thus improving their data transfer's throughput and efficiency.