perception actions QoS Constraints Critic Learning Element Problem Definitions Feedback Learning Goals Sensors Computation / QoS Element ActuatiorsExperiments Changes Knowledge AGENT perception actions QoS Constraints Critic Learning Element Problem Definitions Feedback Learning Goals Sensors Comput

Robotic automation is being increasingly proselytized in the industrial and manufacturing sectors to increase production efficiency. Typically, complex industrial tasks cannot be satisfied by individual robots, rather coordination and information sharing is required. Centralized robotic control and coordination is ill-advised in such settings, due to high failure probabilities, inefficient overheads and lack of scalability. In this paper, we model the interactions among robotic units using intelligent agent based interactions. As such agents behave autonomously, coordinating task/resource allocation is performed via distributed algorithms. We use the motivating example of warehouse inventory automation to optimally allocate and distribute delivery tasks among multiple robotic agents. The optimization is decomposed using primal and dual decomposition techniques to operate in minimal latency, minimal battery usage or maximal utilization scenarios. These techniques may be applied to a variety of deployments involving coordination and task allocation between autonomous agents. Received on 29 May 2018; accepted on 22 July 2018; published on 13 August 2018


Introduction
Integration of robotics, cyber-physical systems and the cloud has come to the forefront with Industry 4.0 [1] requirements. Industry 4.0 proposes the "smart factory" solution, wherein, modular cyber-physical systems coordinate to make decentralized decisions. Such coordination is required for high throughput, complex flows among multiple participants in industrial deployments. Some of the key requirements include [1]: 1. Interoperability: Machines, Internet of Things (IoT) [2] enabled devices and humans connected and coordinating with each other.
2. Information transparency: Physical systems enhanced with sensor data to create added value information systems.
3. Technical Assistance: This involves the use of intelligent devices to aid in informed decision making. Robotic automation may be identified to perform repetitive, unsafe or precise tasks.
agents to autonomously solve problem subsets, without relying on a failure prone central controller. When there are inter-dependencies among sub-problems, agent coordination should ensure solutions to the global problems.
Multi-agent systems [9] [10] are typically employed to model interactions between heterogeneous, distributed and mobile communicating devices. This serves as the right modeling approach when dealing with networked robotic devices that autonomously cooperate to complete missions. Multi-agent deployments are suitable for realtime distributed control in dynamic and distribution prone environments. Important requirements of Industry 4.0 including information transparency and decentralized decision making are satisfied by multi-agent systems.
Due to the decomposition of tasks to agents, the typical AGV dependent warehouses can be upgraded with robotic Autonomous Mobile Robot (AMR) deployments [5] [6]. AMRs have more powerful on-board computational power that can understand dynamically changing environments, produce a real-time map of surroundings and efficiently plan paths to destinations. In warehousing environments where there are dynamic variations in demand and supply of inventory, assigning tasks to robotic AMR agents can be modeled as optimization problems [4]. Typically, such optimizations are solved at a central entity such as Warehouse Management Systems [11], which are neither scalable nor can handle autonomous entities. In this paper, exploiting the agent based architecture, the centralized optimization problem may be distributed. We make use of primal and dual decomposition techniques to distribute the resource optimization problem among agents. As a realistic use case, we employ "picker-toparts" autonomous robots in a warehouse setting, similar to the Kiva robot delivery system employed by Amazon 1 . We demonstrate that, through application of this distributed optimization framework, problems such as minimizing overall procurement latency and maximizing utilization of robotic agents may be solved. We further demonstrate that such a distributed optimization is fault tolerant in nature, striving for completion of tasks, despite robotic failures. To the best of our knowledge, this is one of the few papers addressing the problem of warehouse automation with intelligent multi-agent systems, decentralized optimization.
The main contributions of this paper are: 1. Propose an agent based model for warehouse automation.
2. Identify the optimization parameters that constrain resources of agents.
3. Formulate primal and dual decomposition techniques that can be distributed among agents.  4. Demonstrate optimization of task allocation to maximize resource utilization or minimize latency/energy usage.

5.
Demonstrate fault-tolerance in the task allocation problem due to distributed implementation.
The rest of this paper is organized as follows: Section 2 provides a brief overview of multi-agent system modeling. Description of the warehousing scenario with agent based interactions are provided in Section 3. Section 4 provides analysis of distributed optimization involving primal and dual decomposition. Porting such distributed optimization to the warehouse automation scenario is done in Section 5. In Section 6, the results of applying such optimizations to the warehouse automation scenario are presented. Related work and conclusions follow in Sections 7 and 8.

Multi-agent Systems
Classical architectures for control of distributed systems involve a central co-ordinating node. An example of this might be on the warehouse floor, where all robots submit data to a central (cloud) repository for coordination. While a centralized solution is easy to manage (global picture) and might lead to full efficient deployments, there are cases where central deployments are not possible. Typical drawbacks of centralized decision and control include (i) Inefficiency: As all the decisions are processed through a central node, latency and networking overheads are substantial (ii) Constraints: Multiple safety-critical scenarios (manufacturing, transportation) require real-time decisions that may be repeatedly be violated by the remote central node (iii) Reconfiguration: Complex deployments typically involve reconfiguration and dynamic changes, which needs autonomous responses.
Intelligent Agents [10] are autonomous, problemsolving computational entities that can coordinate over a distributed, dynamic environment. Multi-agent systems are a group of agents deployed with specific architecture, 2 EAI Endorsed Transactions on Industrial Networks and Intelligent Systems 06 2018 -09 2018 | Volume 5 | Issue 15| e2 coordination and messaging protocols. Each agent senses/actuates based on data computed from its own field of view. Typically, the autonomous agents have incomplete information and limited viewpoints of a global problem. As there is no global control or central daemon available, coordination among agents are needed. Consensus on the global task may be provided by sharing knowledge among agents in a peer-to-peer or hierarchical fashion [12]. Data is also decentralized as agents may sense, store and perform decision updates periodically.
Agent-based solutions provide multiple benefits in complex deployment scenarios. Scalability is one aspect, wherein the distributed decision making among autonomous agents allows scaling out of agents. Every agent is provided just the right amount of information needed to complete a sub-task. Robustness and Faulttolerance is ensured as there is no central element for decision making. As agents are autonomous in nature, failure of a node assigned a sub-task can be replicated or check-pointed to another node. Reconfigurability is simple to handle in agent based deployments, where reorganization or reprogramming of a particular agent is possible (without dependencies).
When considering such agent based deployments for robotics, partitioning distributed device clusters as agents may be performed: • Physical Distinction: Each robot is an agent.
• Encapsulation: Every agent should be encapsulated with enough information to complete a sub-task.
• Ownership: Only a few agents have permissions to modify data.
• Resources: Allocating limited resources (computation time, carrying capacity, network bandwidth) to a few robots can correspond to agents.
We further enunciate on dividing complex tasks to agents in a warehousing environment in the next section.

Case Study: Warehousing Automation
In order to study a realistic scenario, wherein, multiple agents autonomously interact with each other, we use the warehouse procurement set-up motivated by the Industry 4.0 requirements. The typical "parts-to-picker" model (items on a conveyor belt) is replaced by the "picker-to-parts"(Autonomous Mobile Robot) model. The set-up is described below: Consider a large warehousing factory (e.g. Amazon, Dell) where inventory is processed and delivered to customers. As it is a large factory, human interaction is kept minimal with a fleet of hundreds of robots handling the order processing. A set of server robots receive the goods/tasks that need to be delivered. This information is sub-divided to a set of mobile delivery robots that move along the factory floor. Robotic picking arms are located at each shelf, that pick and drop goods on the delivery robots. Inventory management is performed by periodically polling the racks for RFID tagged products. As it is a large warehouse, no centralized control is possible and the robots must handle task sets in an autonomous, collaborative manner [14]. There is significant variation in the demand of procured items for collaborative task completion. Such a scenario has motivated by [14][4], where-in hundred of autonomous robots are deployed on the warehousing floor. The objective is to move from (timebound) batch processing, which leads to increasing procurement latency, to a real-time procurement system. The use of multiple decentralized robots also prevents a single point of failure. This can also be used to scale up the number of robots with varying demand. Inventory management is done in conjunction with Radio-frequency identification (RFID) RFID tagging [15]. Procurement and dispatch of items from the warehouse can be monitored with a common database with RFID tag readers located on procurement robots and entry/exit points.
As seen in Fig. 2, we demarcate the problem space using specialized robotic agents spawned within a warehouse to automate delivery of products. A set of server agents A S tabulate the task requirements (product list); a set of mobile delivery agents A D are employed to subdivide/allocate delivery tasks and a set of picking agents A P are located close to the shelves to pick and drop items. Inventory is monitored using RFID tags through periodically polling, such that replacement of inventory is done periodically (A S , A P crosschecks). Properties of physical distinction, encapsulation and resource allocation amongst these agents are set as described in Section 2.
Once a set of products are to be procured by the server agent A S , the products are subdivided based on dimensions (length, breadth, weight) and location coordinates (rack, row, column). These are then assigned to the delivery agents A D that have a 3 EAI Endorsed Transactions on Industrial Networks and Intelligent Systems 06 2018 -09 2018 | Volume 5 | Issue 15| e2 limited capacity of battery lifetime, load bearing capacity and load bearing dimensions. The delivery robots A D then approach the picker robots A P to procure specific items in the provided list. The picker agent A P detects the proximity of delivery robots and picks/drops the required products. After procuring the items, the server robot A S updates the inventory list about decrease in stocks of a particular inventory item. Note that this delivery process requires handling of dynamic variations in demand and location of delivery robots. While in the computational setting, resources may include CPU cycles, memory, network and battery capacities, in case of such cyber-physical interactions, allocating tasks with physical and environmental constraints is also needed. Rather than processing each order individually, the server agent may queue up a batch of tasks that need to be completed. These tasks may be sub-divided among multiple delivery and picking agents to optimize procurement of products across multiple warehouse racks. The process of division/allocation of delivery tasks can occur in multiple modes: • Utilization of Delivery Agents -wherein, the tasks as subdivided such that utilization of individual agents are maximized. Each agent makes a choice as to participate or abstain from the procurement process (based on constraints).
• Procurement Latency -wherein, the autonomous delivery robots must coordinate amongst themselves to procure the set of items within a given latency constraint.
• Energy Limitation -wherein, delivery robots accept tasks as long as overall battery degradation of each agent is minimized.
This formulations allows us to move from AGV → AMR, distributing intelligence to the robotic entities. Further elaboration of how optimization formulations may be ported to distributed settings are elaborated in the proceeding sections.

Optimization Techniques
In this section, we formulate optimization problems when applied to multi-agent systems for resource allocation. We further provide an overview of primal and dual decomposition techniques, that are used for distributed optimization among agents.

Multi-agent Optimization
For the multi-agent optimization formulation, we consider m agents, where agent a i , i ∈ [1, m] has a local convex objective function f i (x), with f i : R n → R, and a local convex constraint set X i (known only by the agent).
x ∈ R n represents a global decision vector that the agents are collectively trying to decide on. The goal of the agents is to cooperatively optimize a global objective function f (x), which is a function of local objectives: where T : R m → R is a monotonically increasing convex function. The decision vector x is constrained to lie in a set x ∈ C, which is a combination of local and global constraints: where C g represents the global constraints. This model leads to the following optimization problem: The decision vector x can either be a resource allocation vector or a global parameter vector to be estimated via agent coordination. We further develop this formulation for the warehousing scenario in Section 5.

Optimization Decomposition
Resource allocation among multiple entities can be formulated as a Knapsack optimization problem [16]. However, a scalable solution with multiple agents coordinating in real time, cannot handle data and decision coordination with a central entity [17]. Moreover, in industrial warehouses, where demand of products varies dynamically, a distributed optimization approach is needed. As seen in Fig. 3, the centralized control/optimization model which needs a central authority is moved to a decentralized model, with agents coordinating with each other to solve a global problem. Such a decomposition approach has been done in the case of optimization problems [18], which we will now describe in brief. Building from eq. 3, we consider a set of m agents a i , i ∈ [1, m] that coordinate in order to optimize resource/task allocation. The centralized version of this optimization problem is: with the two individual sub-problems coupled by the constraints  master algorithm and two linear programming problems solved (possibly in parallel) at each iteration. The assumption is that the constraints are separable and the local objective functions/constraints may be distributed over the agents. Two possible techniques decomposition exist: Primal and Dual decomposition [16].

Primal Decomposition (Allocation).
In this decomposition technique, the coupling constraint m i=1 F i · a i ≤ h is subdivided into two sets of individually solvable constraints dependent on a shared variable z. The individual optimizations that are solved at each agent are given below (agent a m is the co-ordinator): The master optimization that is updated and solved at the co-ordinating agent is min m i=1 P i . In order to solve this update equation, the sub-gradient is given as: where S i and S m are the solutions to the decomposed problems in eq. 5, with step size α k .
Dual Decomposition (Pricing). The dual decomposition problem makes use of Lagrange multipliers to eliminate the coupling constraint [19], using a decomposition variable λ: The master update performed at the coordinating agent is given by: where S i are the solutions to decomposed problems in eq. 7, (·) + denotes the non-negative part of a vector and α k is step size. At each step, the master algorithm sets the prices for the resources. The individual agents each optimize independently, taking into account the expense and income generated by using the resources. The master algorithm adjusts the prices such that the price for an over used resource is increased and the price for an under used resource is decreased (non-negative).
Both the primal and dual decomposition techniques make use of the sub-gradient method for distributed optimization. The sub-gradient method is a simple algorithm for minimizing a non-differentiable convex function [16]. The sub-gradient method uses the iteration: Here z k is the k th iterate, g(k) is any sub-gradient of function f at z k , and α k > 0 is the k th step size. Several step size rules may be applied including: • Constant step size α k = b is a constant, independent of k.
• Non-summable diminishing. The step sizes satisfy Step sizes that satisfy this condition are called diminishing step size rules. A typical example is At each step, the best point found since the first iteration is tracked: The sub-gradient algorithm is guaranteed to converge to within some range of the optimal value [16].

Optimization in Warehouse Automation
In this section, we formulate the optimization problem in a warehousing floor, where (varying) number of products need to be procured and delivered via robotic agents. We demonstrate that this can be formulated as maximizing utilization, minimizing procurement latency or minimizing energy consumption of resources. The optimization problems are further decomposed using primal/dual techniques to be solved by individual agents. The real deployments may be ported onto Robot Operating System (ROS) 2 .
When an order with a given number of products are to be delivered from the warehouse, the agents (A D , A S , A P ) coordinate such that the procurement is satisfied. The coordination may be restricted based on parameters relating to device utilization or procurement latency. Table 1 describes some of the important parameters to be used in our formulation. This can be viewed as extension to the contract net protocol [12] consisting of the following steps: 1. Recognition of goals and whether it can be completed within constraints by an individual or group of agents.
2. Broadcasting of the task to other agents including objectives and constraints.
3. Agents bid for the task taking into account tradeoffs for completing tasks and resources to be allocated.
4. Contracts are awarded to successful bidding agents with communication on job allocation.

5.
A contracted agent may further sub-contract parts of the job to complete the task.
The algorithmic overview of our methodology is presented in Algorithm 1, wherein the traditional tasks 2 − 4 in the contract net protocol are set into the distributed optimization formulation. Note that the agents we consider typically operate as delivery agents A D as specified in Section 3. Problems such as path planning, stocking and pick-place are out of scope of this formulation. We elaborate on specific situations where the optimization may be formulated based on utilization, latency or energy constraints.

Maximize utilization
When there are multiple robots on the factory floor ready for delivery, one strategy is to maximize the per-trip utilization of resources. By utilization, we refer to load bearing capacity of the robots rather than utilization of hardware (CPU, Memory, Network). The optimization may be formulated to maximize the utilization of each node, subject to maximum dimensions/weight that the node can bear. In order to deploy this optimization over agents, we perform primal decomposition as given below: where, D i and W i are the maximum carrying dimensions of node i having normalized utilization U i . In order to satisfy the primal decomposition constraints, one of the agents (agent with index i = m) performs the pivot optimization satisfying the global constraint. A master update equation is performed at the co-ordinating agent node (Fig. 3) to update the variables (X , Y) that are used in the primal problems:

Minimize Procurement Latency
As collecting products from picking stations and delivery of these products to delivery stations would involve 6 EAI Endorsed Transactions on Industrial Networks and Intelligent Systems 06 2018 -09 2018 | Volume 5 | Issue 15| e2 latency overheads, formulating cases where latency must be minimized is of interest. The following optimization is proposed where the latency parameter l i is dependent on a number of factors including distance between picking and delivery robot d i , average robot speed v i and picking efficiency µ. As this reflects a pricing problem, we perform dual decomposition as follows on all agents: where l i is the procurement latency associated with agent a i . The update equations for each of the variables λ 1 and λ 2 is based on the gradient descent method: where a i refers to solutions provided from the optimization in eq. 14.

Minimize Energy Utilization
Minimizing the battery depletion rates of robotic agents may also be a priority. Battery discharge lifetimes have been studied using circuit, electrochemical, stochastic or analytical models [21]. For constant loads, the battery lifetime L can be derived from the battery capacity C and the discharge current I as L = C/I. However, modeling the discharge capacity of lithiumion or lead acid batteries is complicated by the nonlinear discharge profiles. Voltage can vary based on the discharge currents, rendering the actual battery capacity lower for high discharge currents (rate capacity effect). Additionally, during periods of low discharge, the battery can recover some of its capacity (recovery effect). We make use of an improved version of Peukert's law [22] that takes into account current drawn and temperature variation during battery discharge: where ∆C is the battery capacity consumed, I ref and T ref are the reference current and temperature rating of the battery, I and T are the current and temperature observed during battery usage, parameters δ, η are exponents relating non-linear battery depletion rates and constant γ on A-h relates the capacity removed to the non-dimensional discharge current and temperature components. This is formulated as an optimization problem with the objective to reduce total battery consumption of participating agents using dual-decomposition: The update equations for each of the variables λ 1 and λ 2 : where a i refers to solutions provided from the optimization in eq. 17.
We demonstrate the efficacy of these optimization formulations on deployment simulations in the next section.

Results
In order to demonstrate the optimization output on realworld robotic agents, we make use of the KUKA mobile platform (KMP) 1500 3 with specifications in Table 2. The KUKA KMP 1500 robot is an autonomous platform that may be deployed on warehouse floors. By the use of safety laser scanners and wheel sensors, the robots are able to perform real time Simultaneous Localization and Mapping. This allows the omni-directional to move in any direction and achieve positioning accuracy of upto ±1 mm.
We now apply the optimization formulations developed in Section 5 to a set of KUKA robots simulated on the warehouse floor to collect and deliver products. For our simulations, we restrict the number of delivery, picking and server agents, in order to remain in realistic thresholds where a 3 − 4 agents coordinate over complex warehousing tasks. A further assumption is that the static locations of the server A S and picking agents A P are preset with the optimization performed over picking allocations to delivery agents A D . The optimizations are solved in Scilab using the Karmarkar optimization solver, setting the sub-gradient update from Section 4 as α k = 0.1/ √ k. The following cases are studied:  • Utilization: In order to maximize utilization of the robotic agents, the primal-decomposition techniques provided in eqs. 12, 13 are deployed. Fig. 4 provides an output of this optimization when applied to increasing demand in products (assumption is that a 50 × 50 cm product carton weighs 1 kg). As seen from Fig. 4, allocation to the three delivery agents are gradually increased proportionally such that utilization of each agent is maximized. The convergence analysis of primal decomposition (eqs. 12, 13) for one instance is seen in Fig. 5. This demonstrates that the optimization may be accurately decomposed among participating agents.
• Latency: In order to minimize procurement latency of the robotic agents, the dual-decomposition techniques provided in eqs. 14, 15 are deployed. Fig. 6 provides an output of this optimization when applied to increasing demand in products when the delivery agents are randomly placed on the warehouse floor, velocity as specified in Table 2. As seen from Fig. 6, allocation provided to the distant agent a 2 is minimized so as to maintain acceptable latency. Compared to the naive deployment (agents randomly allocated), the optimization in Fig. 6 produces a latency improvement of 25% for 2000 kg procurements. The convergence analysis of the dual decomposition variable λ (eqs. 15) is seen in Fig.  7, where convergence to a stable neighbourhood is observed within a few iterations.
• Energy: In order to minimize battery degradation of the robotic agents, the dual-decomposition techniques provided in eqs. 17, 18 are deployed. Fig. 8 provides an output of this optimization when applied to KUKA robots with battery  specifications in Table 2. We notice variations in allocation ratios such that the battery degradations of each agent is minimized.
• Fault-tolerance: One of the key advantages of moving to an agent based system is fault tolerance. As there is no central entity replicating/checkpointing, the coordinating agent in Fig. 3 is responsible for varying the primal/dual update factors to take care of faulty agents. As seen in Fig. 9, despite failure of agent a 1 , the optimization reconfigures so that higher ratio of products are given to other agents. Note that we do not perform any check-pointing (partially completed tasks are not reported); rather, the optimization ensures task re-distribution to available agents in the next iteration.
The message passing among agents has overheads, whose analysis has been earmarked for future work.     coordinating multiple robots deployed on the shop floor. In such deployments, typical path planning algorithms go hand in hand with resource optimization and coordinated goal completion among deployed robots [24]. Warehouse Management Systems (WMS) [11] typically are used to coordinate machines and robot interactions in enterprise warehouses. WMS systems may also be placed towards the edge of the network, as proposed by newer architectures such as Fog computing [13].

Related Work
Handling such intelligent, autonomous robots requires an appropriate modeling and analysis framework. Multiagent system modeling [9] has been used to encapsulate modular autonomous systems in multiple fields. In the case of robotic systems, multi-agent systems may be used to encapsulate autonomous robots that can coordinate without central control [10]. Coordinating such devices in a decentralized setting is an active area of research [25]. One such widely used protocol is the contract-net protocol [12], that makes use of a bidding mechanism to allocate tasks. In this work, we further append such a protocol with optimization models, which are able to allocate tasks among autonomous agents.
The use of resource optimal protocols in the robotic world has been studied in centralized deployments [20]. Decomposition of optimization algorithms across multiple nodes has been used extensively in the networking literature to design cross-layer resource allocation mechanisms [19]. In [18] [17], theoretical models for decomposition techniques in multi-agent systems are provided. An early industrial use case for intelligent agents in manufacturing was provided by the Holonic Manufacturing Systems [26]. The use of genetic algorithms for coordinating multiple agents has also been proposed in literature, as seen in [27]. Such algorithms are necessary in conjunction with traditional robotic path planning [28], in order to ensure efficient use of resources on the factory floor.
We apply the distributed optimization among a set of agents to warehouse automation [4]. Deploying autonomous robotic agents in the warehouse setting has received considerable industrial interest, ever since Amazon acquired Kiva robotics in 2012 [14]. Robot automation employed in Amazon's warehouses has drastically reduced procurement times from 60 minutes taken by human participants to around 15 minutes. Automation has increased the inventory capacity per square foot by 50% and reduced Amazon's operating cost by 20% ($225 million/Warehouse) [14].
In this work, we take the realistic case of autonomous mobile robots deployed in warehousing environments. The agents coordinate among themselves using distributed optimization techniques to allocate tasks within warehouses. We model division of tasks based on agent utilization, procurement latency, energy depletion and fault tolerance capabilities.

Conclusions
Multi-agent intelligent systems are ripe for application into real-world industrial problems. In this paper, we exploit multi-agent system interactions to solve some of 9 EAI Endorsed Transactions on Industrial Networks and Intelligent Systems 06 2018 -09 2018 | Volume 5 | Issue 15| e2 the Industry 4.0 automation requirements in warehouses. Through the use of decomposed optimization techniques, the intelligent agents are efficiently enabled utilization, latency and energy resources when traded off with task allocation. Through distributed optimization solvers, these situations are analyzed based on commercially available warehouse robots to demonstrate efficacy in delivery tasks. Such a distributed optimization framework may be extended to other deployments involving intelligent agents.
In future, we would like to deploy this framework on testbed deployments defined using the Robot Operating System (ROS). Incorporating our optimization framework within an auction bid mechanism along with human agents is also another aspect for future consideration.