IoT-F2CDM-LB: IoT Based Fog-to-Cloud and Data-in-Motion Architectures with Load Balancing

The work in this paper tries to enhance the performance of IoT by modifying the Cloud based architecture in terms of storage, processing, and Load Balancing (LB). The assumption is as follows: In a single Fog server, high traffic coming from Things may cause packet loss which in turn affect the overall IoT performance. To overcome such a situation, LB on Fog layer is proposed and implemented practically using virtualization technology. The proposed IoT based Fog-To-Cloud and Data-inMotion with LB (IoT-F2CDM-LB) architecture consists of two load balancers; HAProxy and Server Load Balancing (SLB), are used to distribute messages from Things to four virtualized Fog servers according to Least Connection technique. The implementation is carried out using VirtualBox and GNS3. Message Queue Telemetry Transport (MQTT) protocol is used to transfer messages between layers. Both load balancers result in packet loss reduction by 50%, lower response time and higher throughput.


Introduction
The Internet of Things (IoT) has emerged with features of ubiquitous connectivity (anything, anywhere, and anytime). Things include actuators, sensors, or objects (smart watch, smart car, smart pen, smart chair, smart bag, etc.). With IoT, it is possible to turn anything to smart called smart X. These smart Things communicate with each other to achieve a complex and difficult goals without human intervention. IoT becomes more and more an interesting concept almost in all domains of life because of its features and capabilities. Things are low power, low bandwidth, low cost and low memory, and identified by Internet Protocol (IP) to be linked to the world through the Internet [1][2]. Cloud computing (CC) is a suitable way for storing and managing web services by applying "pay-as-you-go" model. CC isolates resources and services from customers in order to facilitate the operation of applications. However, it has an impact on real time applications because of possible high delay. CC has three different types, namely: Software as a Service (SaaS), Infrastructure as a Service (IaaS), and Platform as a Services (PaaS) [3][4]. Fog computing (FC) has the same features of CC (virtualization, network, processing, and computing) and made by Cisco. These features are provided near to Things to solve the problem of high delay and is specialized for sensitive applications like healthcare [5]. Cisco said that there will be around 25 billion of Things, objects, and actuators by 2020. These Things generate an enormous number of messages. Governments and societies benefit from these messages to improve the quality-of-life of end users [6]. Messages from Things can be analyzed and processed using Data in Motion (DM) technique that is introduced by IBM to provide analyzing messages without to need storing it [7][8]. Virtualization is a technology used to reduce power, cost, space and complexity by making multiple virtual servers on a single physical server called Virtual Machines (VMs). It provides a middleware layer called Hypervisor to separate the hardware and software layers [9]. Due to high number of messages from Things, servers may crash when processing these requests. Thence, Load Balancing (LB) can solve this problem by distributing the messages across multiple servers [10]. The main contribution of this paper is to propose IoT based Fog-to-Cloud and Data-in-Motion with LB (IoT-F2CDM-LB) architectures using virtualization technology with Message Queue Telemetry Transport (MQTT) protocol to reduce the packet loss and achieve the highest throughput. The rest of this paper is organized as follows: section 2 explains the MQTT protocol. Section 3 introduces LB techniques and load balancers. Section 4 lists literature review. Section 5 describes IoT-F2CDM-LB architectures. Section 6 and 7 lists the hardware and software, and discusses the design and implementation of proposed architectures respectively. Section 8 and 9 show the results with discussion. Finally, section 10 concludes this paper.

MQTT Protocol
MQTT protocol is an open source, application layer based on Transmission Control Protocol (TCP). It operates as publish/subscribe model with asynchronous connections. It has a low overhead (2 bytes header), comparable to client/server model. It is simple to implement and is recommended for smart applications like smart hospital, smart home, smart school, etc. MQTT improves performance of high delay and low bandwidth. The Broker part of the MQTT is needed to enable connections between clients (publishers and subscribers) [11]. Publishers send messages within a specific topic, then the broker distributes these messages to the subscribed clients. The topics of MQTT consist of levels sectioned by slash in order to clarify a specific topic among of all topics in the same network. Clients use wildcards such as: + (Plus symbol) represents all values in single level and only one topic. # (Hash symbol): represents all values in multiple levels [12].
The MQTT protocol has a retained message which means that broker saves the last value on each topic and sends to new subscribed clients in order to save messages from failures. MQTT has three levels of Quality of Service (QoS) to guarantee the message delivery [13]: QoS level 0, message is received at most once, without an acknowledgment; QoS level 1, message is received at least once, with an acknowledgment. Finally, QoS level 2 in which message is received at exactly once, and this level requires four-way handshake. Last Will and Testament (LWT) message is sent by broker to advertise all subscribers that there is something wrong on connections [14].

LB
LB is the process of enhancing performance by distributing traffic among a number of server pools. There are a few load balancers that support MQTT protocol. The researchers used High Availability Proxy (HAProxy) open sourced [15], Nginx Plus [16], and Elastic Beam [17] where license requirement is needed. This section explains two main load balancers used in this paper: HAProxy and Server Load Balancing (SLB) router.
The HAProxy is an open source software used to distribute load among multiple servers based on TCP. HAProxy consists of two lists: Back-end list that contains servers that receive messages from front-end list. This list can be defined by: IP address, port, and LB technique used. The other is the Front-end list that represents messages from clients. This list can be defined by: IP address and port of clients, and Access Control List (ACL). ACL is used to provide some rules to permit or deny messages arrive from clients to servers [18]. HAProxy uses health check to monitor the availability of back-end servers. Health check works by establishing TCP connection between the HAProxy and the back-end servers. If one of servers fails to process messages from the front-end, then HAProxy removes the server from the back-end list [19]. HAProxy provides a redundancy by adding two HAProxys and act as an active or passive model using a Keepalived to prevent the load balancer gets overwhelmed with huge number of messages because of single point of failure. Keepalives is a software used for routing messages, it works by creating a shared virtual IP address between two HAProxys. For example, clients use 10.0.0.3 as the destination IP address as shown in Figure 1 [20]. The SLB router is an IOS image of Cisco router. Inside SLB router, a virtual server is defined to represent a list of real servers called server farm. SLB router redirects messages from clients to this virtual server, and then the virtual server redirects messages to one of the servers in the server farm. According to a specified LB technique [21], SLB can be in one of the two modes: Directed mode indicates that virtual server can be configured with any class of IP regardless of the class of the servers in the server farm. Therefore, this type needs to perform NAT in order to translate the IP address of virtual server to one of destination servers in server farm. For example, if the IP address of virtual server is 192.1.1.1, then the clients sends the messages to destination 192.1.1.1, and the SLB router receives these messages and redirects to one of IP address of servers using NAT protocol as shown in Figure 2 [22]. The second mode is dispatched. It indicates that IP address of virtual server is configured with servers in server farm. Each server has its own real IP address plus virtual address of virtual server. This type needs the servers to be connected with the same Local Area Network (LAN) of SLB router. For this reason, it could not be capable in network with multiple routers as shown in Figure 3 [23]. SLB router provides a redundancy using (Hot Standby Redundancy Protocol) HSRP protocol between two SLB routers by creating a shared virtual IP address as shown in Figure 4 [24].  These load balancers distribute messages from multiple clients to several servers simultaneously according to a specific technique to reduce the traffic on single server. LB techniques can be classified into several types [25]; the most common techniques are as follows: 1) Round Robin (RR) This technique distributes the traffic among the servers sequentially. For example, two servers receive messages from clients in sequence as shown in Figure 5. 2) Weighted Round Robin (WRR) is the same as RR; however, some servers have capabilities more than others. Therefore, weights can be added to the servers. For instance, if there are two servers: the first server receives 5 messages (because it is weighted to 5), the second server receives 1 message (because it is weighted to 1), and so on as shown in Figure 6. connections, and server 3 is not processing any connections. Some servers have more overloads than others because clients stay connecting to servers much longer than other servers. So, server 3 receives three messages. Both servers 1 and 3 have now the same number of active connections. Then, load balancer performs RR on server 1 and 3 until number of active connections in both servers reach 15. Thence, load balancer performs RR on three servers 1, 2, and 3 and so on.

Literature survey
There are few works have been proposed the integration between Fog and Cloud or simply (F2C) as in [26], the authors introduce F2C architecture and its advantages with main challenges and implement IoT based F2C and IoT based Cloud architectures using Tareador simulation tool to provide the performance comparison between them in terms of execution time and speedup. The results show IoT based F2C is better than IoT based Cloud because it is reduced the execution time. However, the authors have not considered IoT protocols in the architecture and not implemented the proposed architectures practically. But, the authors in [27] implement F2C computing on real application for chronic obstructive pulmonary disease (COPD) patients and the results show that the F2C improves the quality of life of patients. None of two previous papers have considered the delay on their proposed, while the authors in [28], propose distributed service allocation strategy for both resource offering and service requirements based on F2C computing in order to reduce delay of service allocation and decrease traffic load on Cloud. Then, the same authors in [29] minimize the delay of the services and provide the capacity requirement. In addition, the queuing theory are considered in F2C concept as in [30], the optimal workload allocation is proposed based F2C architecture to reduce power consumption and delay. The results show that F2C is better performance that Cloud. We notice that the previous researchers have not been tested and implemented F2C with IoT protocols. Some authors try to propose an opposite path from Cloud to Fog (C2F) as in [31], the authors propose C2F architecture for monitoring healthcare network and smart homes. The results show C2F provides the better service to Things. Finally, the authors in [32] propose IoT architecture based on Cloud to combine MQTT and Hypertext Transfer Protocol (HTTP) protocols and to distribute traffic among virtual servers using HAProxy. The performance evaluation of protocols is presented in terms of number of clients and Central Processing Unit (CPU) cores.
The results show the MQTT protocol has better performance than HTTP. These authors have considered protocols, however the architecture is based on Cloud and not based on F2C. This paper proposes a new IoT architectures based F2C with LB and virtualization using different load balancers, namely: SLB and HAProxy. Up to our knowledge, SLB has not been used previously by researchers in IoT applications. Also, previous researchers have been used HAProxy on Cloud layer not on Fog layer.

IoT-F2CDM architectures with LB
Messages are handled by a single Fog server in the traditional network, which might become loaded on heavy incoming messages. This condition will lead the Fog server to act slowly and results in packet loss. Consequently, packet loss may lead to wrong decision-making. To solve this situation, multiple server arrangements for LB can share processing power and prevent packet loss. For experimental purposes and due to limited number of available physical servers, virtualization technology is used to create the required number of servers for testing. The proposed IoT-F2CDM-LB architectures consider five layers: Things, gateway, Fog, Cloud, and monitoring with HAProxy and SLB. Messages from Things are distributed according to LeastConn technique across multiple servers to mitigate the packet loss. When a new publisher tries to connect, this technique translates message from publisher to server which has the least number of connections. This technique is used when servers have the same capabilities and is suitable for protocols of long sessions like MQTT protocol. Several servers are virtualized on single Fog server using type 2 to reduce the cost, power, and space. These virtual servers receive messages from load balancer and send to Cloud for permanent storage using DM technique.
The main hardware and software tools used in the proposal architectures as shown in Table 1 and Table 2 respectively. These software perform several tasks such as writing specific codes, simulate networks, emulates sensors, and monitoring messages.

Network design and implementation
This section discusses the design and implementation of IoT-F2CDM architectures over two sites: Site A (located at Al-Nahrain University, College of Information Engineering), and site B (located at the Ministry of Higher Education (MOHE), Department of Research and Development (RRD)). The layers of proposed architectures are as follows:

Things layer
In this paper, Things layer consists of one real heartbeat rate sensor and several virtual sensors from Tsung. Heartbeat rate sensor is programmed with C/C++ language. The real sensor is placed on patient's body to capture the heartbeat rate messages in real time. While, virtual sensors are emulated in Tsung tool to generate a high traffic using XML programming languages to match the real sensor. Tsung is used to reduce the cost, power, complexity and space of real sensors. Things publish messages using MQTT protocol with QoS level 0 and 1.

Gateway layer
Messages from Things layer are transferred to the upper layers through gateway layer located at site A. Mikrotik AP transmits messages from sensors to Fog layer. Cisco switch mediates the Things and Fog layer, and connects several hardware together. Cisco router is used to forward messages from Fog to Cloud layer through the Internet. Open Shortest Path (OSPF) protocol is configured on router to provide routing mechanisms.

Fog layer
This layer receives messages from Things layer to provide a temporal storage and distributes messages to four Fog servers using LB techniques, Then selects the important messages using a DM technique and sends to Cloud layer. VMs are created to act as virtual Fog servers in order to create the required number of servers and apply LB. Four Fog VM servers and two HAProxys are virtualized with MQTT QoS level 0 and 1 using VirtualBox as shown in Figure 7. All VMs are configured to operate Apache2, PHP5, MySQL, PHP-Mosquitto, DM technique, and network configuration.

Figure 7. VMs in VirtualBox
A. HAProxy It is used to distribute a huge number of messages based on TCP to four virtual Fog servers. These servers are setup to bridged setting in VirtualBox. The IP address of these servers are configured in HAProxy backend list. While, the IP address and ports of clients (Things) are configured in HAProxy frontedend list. Messages from clients are transferred to load balancer using MQTT with QoS of MQTT level 0 and 1. Then, these messages are separated to four virtual servers according to LeastConn fashion. Each virtual Fog server has a DM procedure to select the important messages, and then send to Cloud layer. Due to a single point of failure, the Fog layer provides availability in terms of servers. Two load balancers as an active/passive model is configured to the architecture using the Keepalived mechanism. Keepalive is used for monitoring the connections between the frontend and load balancer. Reroute the traffic to secondary load balancer if an interrupt is detected as shown in Figure 8. MQTT protocol is used for communication in this method. The previous research has been implemented HAProxy on Cloud layer, while this proposed architecture based F2C implements HAProxy on Fog layer and provides redundancy.
B. SLB router SLB router is used in this proposed architecture as shown in Figure 9 because it is easy to employ and easy to maintain   Figure 10, where two SLB routers are installed and configured. Two SLB routers are connected to Cisco switch. On the other terminal, two SLB routers are connected to real hardware of the IoT architecture (Gateway layer) through cloud. The cloud symbol inside GNS3 is not the intended CC but it represents the interface between the network inside GNS3 and the real world. The Cisco switch is connected to four Fog servers that are installed and setup to host settings in VirtualBox and is connected to Internet through cloud.
Two SLB routers are configured in directed mode in this proposed architecture. IP address of virtual Fog servers, port, and LB technique are defined in server farm of SLB routers. If one of servers is off or connection between SLB and server is interrupted, then the SLB waits for a settled time (set to 60 second -an adequate time for detecting failures) and the failed server is removed from the server farm. Due to SLB routers are in directed mode, NAT protocol is configured to map IP address of load balancer to one of virtual Fog servers according to LeastConn. In addition, OSPF routing protocol is configured in both SLB routers for forwarding messages to Cloud layer. The SLB routers act as active or passive using HSRP protocol to provide redundancy by creating a shared virtual IP address between them. The Things put this shared IP address in their destination field.

Software
Descriptions Graphical Network Simulator-3 (GNS3) version 2.0.2 [35] GNS3 is an emulation tool made by Cisco to visualize a virtual environment of networks. Also, it can connect GNS3 to the real world. GNS3 is based on daynamips that is a program to emulate an actual Cisco IOS routers series VirtualBox [36] and VMware workstation [37] VirtualBox and VMware are type 2 virtualization software used to create multiple VMs with different OSs in single physical PC or server as a Guest OS. VirtualBox has some differences than VMware. VirtualBox is more friendly, free, has low overhead, and consumes less power than VMware. However, VMware is a close source with licenses requirement, has the ability to drag and drop between the host OS and Guest OS PDE is a program used to process Java programming language and create applications with charts and sound 6 Istabraq M.  Messages from Things are transferred to load balancer (HAProxy or SLB) using MQTT with QoS of level 0 and 1. Then, these messages are sent to four virtual servers according to LeastConn technique. Each server selects important messages using DM technique and sends them to Cloud layer. DM technique is proposed to implement in Fog layer. The authors adopt the idea from DM in IOx Cisco router. This router consists of the traditional IOS image of Cisco router and Red-Hat OS for computing and storage. Cisco implemented DM with HTTP protocol. In this paper, proposed DM subscribe messages from Things using Python API with the help of PHP-Mosquitto broker. These messages are stored in MySQL database, and are filtered in real time to select important messages.
Three messages (maximum, average, and minimum) are selected every one hour from database using PHP5 and MySQLi within topic (/sensor/MAXIMUMvalue/), (within the topic /sensor/AVERAGEvalue/), and (within topic /sensor/MINIMUMvalue/) respectively. These three messages are sent to Cloud layer, while the rest are deleted as shown in Figure 11. This proposed DM technique are implemented with MQTT protocol with QoS level 0 and 1.

Cloud layer
This layer receives three messages (maximum, average, and minimum) using Node.JS with the help of Mosquito broker and located at site B. These messages are stored permanently in near real time in MongoDB. Physician or patient's family can monitor the history of the patient through Cloud layer.

Monitoring layer
Local and global monitoring tools are provided using smart phones, tablets, and PCs. PDE can monitor messages directly from sensors locally located at site A using TCP protocol. IoT MQTT Dashboard and MQTTool are installed and configured on Android and IPhone devices respectively to monitor messages using MQTT protocol form Things layer. MQTT_spy is installed and configured on VM using VMware to monitor messages from servers and debug brokers.

Practical testing and results
This section presents the results of proposed IoT-F2CDM-LB architectures with explanations. These results are extracted from transferred messages from Things layer to Fog layer located at site A using MQTT protocol with QoS level 0 and 1.

Average throughput
Throughput is the number of messages per second that the destination server can handle. Two methods are used to find the throughput in this paper: Tsung captures messages when it runs. Throughput is computed using the following formula in Wireshark  The average throughput is specified and calculated by averaging the collected values. Figure 12 show the average throughput in packets/s for both load balancers (HAProxy and SLB) with MQTT QoS level 0 and level 1. As the figure show, the use of LB increases Fog network throughput in both MQTT QoS level 0 and 1. Throughput is affected by the destination server capabilities to handle messages, and distance from destination server. Throughput decreases when average response time increases. The results reach the saturation level for all cases because of limited bandwidth.

Average packet loss
Packet loss is the number of packets fail to reach the destination server when they travel through network. Wireshark is used to calculate packet loss using the following equation [48]: (2) Then, the average packet loss is calculated by averaging the collected values. Figure 13 show the average throughput in packets/s for both load balancers (HAProxy and SLB) with MQTT QoS level 0 and level 1.
The results show that the use of LB reduces packet loss to its minimum value (half in QoS 0 and 1 times in QoS 1). This result comes from the fact that using LB will distribute the traffic over four virtual Fog servers, thus all messages arrive safe and sound.

Average delay
The delay is the time taken of message transferred from sender to destination server. It is captured using WireShark in timestamp. Figures 14 show the average delay in Fog layer with MQTT protocol with LB. This figure shows that messages is a little impacted in LB. and MQTT QoS level 1 is higher than MQTT QoS level 0 by factor 1 because the latter one has an ACK in application layer. Delay occurs because packet arrival rate to link exceeds output link capacity and packets queue in routers, messages must wait for turn. RTT can be calculated using the following formula [49]: (3) In (3), T_1 is the timestamp the sender imitates a request message targeted to the destination server, T_2 is the timestamp the server receives the request message, T_3 is the timestamp the server respond to the request message after processing delay, and finally T_4 represents the timestamp the sender receives the response message. Since the RTT is affected by the link status between the sender and receiver, the average RTT is specified and calculated by averaging the collected values. Figure 15 show that MQTT QoS level 1 is higher than QoS level 0 by factor 2 because of QoS 0 does not have an ACK, and LB in both levels of QoS is affected by factor 1.

Average response time
The Response time is the elapsed time taken by a webserver to respond to a request initiated from a web client. This response time is measured using two tools: Tsung and Wireshark. Response time of HTTP is computed using the following commands in Wireshark: HTTP.request.method=="GET" || HTTP.response.code = 200. Then, the response time can be found in "time since request" in HTTP header. While the response time of MQTT QoS level 0 can be found by applying "MQTT" in the filter of Wireshark, then from the PUB message the value of response time is extracted. Finally, the same step for MQTT QoS level 1, expect that the value of the response time is seen in PUBACK. The average response time is specified and calculated by averaging the collected values. Figure 16 show the average response time of QoS level 1 is higher than QoS level 0 by factor 2 because QoS 1 have an ACK in transport and application layers.

HAProxy monitoring
Health check is configured on HAProxy to monitor the front and backend lists. It works by establishing a TCP connection between HAProxy and the backend servers in order to know if the servers listen to IP address and port or not. Figure 17 presents the traffic during four hours when four messages arrive every minute to HAProxy, then these messages are distributed across servers according to the LeastConn technique.

SLB monitoring
Traffic from Things can be monitored in SLB router as shown in Figure 18. The IP address of virtual load balancers (the first one is active, and the second one is passive) and number of transactions in active SLB are shown in the figure. Active SLB receives twenty-one messages for 5 minutes and distributes them to four virtual Fog servers according to LeastConn technique.

DM technique
This subsection presents the three messages (maximum, average, and minimum) arrived at Cloud layer from each four virtualized Fog servers as shown in Figure 19.

Discussion
The IoT-F2CDM-LB architecture proposes the DM technique with QoS level 0. Also, the Things with QoS level 1 and level 2 can use DM technique without modification in the programming script. In this IoT-F2CDM-LB architecture, two different database structures: MySQL in Fog layer and MongoDB in Cloud layer. The reason for using MySQL in the Fog layer is to allow distributed design and handle complex transactions (multi-row transaction). On the other hand, MongoDB is more scalable to store messages and therefore installed in the Cloud layer. However, both databases can be used in both layers. Two brokers are used, namely: Mosquitto and PHP-Mosquitto. The main difference between them is that the PHP-Mosquitto is compatible with PHP5 scripts and MySQL database. This chapter uses two different methods for LB, namely: HAProxy and SLB router. SLB router reduces the number of physical hardware used since everything is installed inside the router itself. The number of virtual Fog servers could be extended to more than four when the network senses increment in packet loss. API in Python script is used to make python and PHP5 with MySQLi scripts work together in the Fog layer and provide interoperability between MySQL and NoSQL databases.

Conclusions
The work in this paper proposes an architecture based on Fog and Cloud Computing to solve the performance degradation in the Cloud response time when large volume of Things traffic towards that Cloud. Introducing Fog layer nearby the Things layer contributes to network performance enhancement in terms of response time and packet loss. Two IoT architectures are proposed with LB (formed by two types of load balancer, namely: HAProxy and SLB router) and proposed DM technique using MQTT protocol with QoS level 0 and 1 in the Fog layer. The LB distribute the received messages from Things across a specified number of Fog servers according to LeastConn technique and results in reduction of packet loss, delay, RTT, and response time to half that without LB in the carried tests and achieves the highest throughput. Up to the author's knowledge, both LBs have not been evaluated on Fog layer by researchers previously. Finally, the results obtained have been conducted through practical implementation and can be considered as a possible solution to IoT based Fog/Cloud performance enhancement.