Evolving Mixed Societies: A one-dimensional modelling approach

Natural self-organising collective systems like social insect societies are often used as a source of inspiration for robotic applications. In return, developing such self-organising robotic systems can lead to a better understanding of natural collective systems. By unifying the communication channels of the natural and artiﬁcial agents these two collective systems can be merged into one bio-hybrid society. In this work we demonstrate the feasibility of such a bio-hybid society by introducing a simple one-dimensional model. A set of patches forms a one-dimensional arena, each patch represents a stationary robot, which is controlled by an AHHS (Artiﬁcial Homeostatic Hormone System) control software. The stationary robots are able to produce diﬀerent types of environmental stimuli. Simulated bees react diversely to the diﬀerent stimuli types. An evolutionary computation algorithm changes the properties of the AHHS and deﬁnes the interactions between the robots and their properties of stimuli emission. The task is an aggregation of simulated bees at a predeﬁned aggregation spot. We demonstrate that an evolved AHHS is a very feasible tool for controlling these stationary robots. Furthermore we show that an AHHS even works robustly in diﬀerent setups and dynamic environments even though the controller was not specially evolved for these purposes.


INTRODUCTION
Natural self-organising collective systems like social insect societies are often used as a source of inspiration for robotic applications.In return, developing such self-organising robotic systems can lead to a better understanding of the natural collective systems.By unifying the communication channels of the natural and artificial agents these two collective systems can be merged into one bio-hybrid society.Biology knows many examples of collective behaviour on a wide range of organisational complexity, from organisms as simple as slime moulds (Nakagaki, 2001) or bacteria (Camazine et al., 2001) via insects and fish to highly sophisticated representatives of birds and mammals.The most intriguing example for the efficiency of collective behaviour is found in social insects.For example in ants, detailed research has been conducted on phenomena like chain formation (Lioni et al., 2001) and bridge formation (Deneubourg et al., 1990).
One of the most studied social insects is the European honeybee Apis mellifera.Honeybees are not only crucial for agricultural and economical purposes (e.g.pollination), but have also successfully found their way into engineering approaches.For example, honeybees inspired the development of bio-inspired algorithms for controlling autonomous swarm robots (Schmickl et al., 2008;Bodi et al., 2012Bodi et al., , 2011)).
Robotic devices also become more important in biological studies.The simplest form for such robotic applications would be a sensor network for monitoring animals (Zacepins et al., 2011).A more ambitious approach is the usage of robotic agents in animal societies for the purpose of influencing the animal behaviour (Halloy et al., 2007).The ongoing EU-project ASSISIbf presents a new approach for closing the loop of interaction between natural and artificial agents for the purpose of forming a bio-hybrid mixed society consisting of animals (honeybees Apis mellifera) and stationary autonomous robots called CASUs (Combined Actuator Sensor Units) (Schmickl et al., 2013b,a).CASUs are equipped with several sensors (e.g., temperature sensors, proximity sensors) (Salem and Schmickl, 2014) and actuators (e.g., heating devices, vibration devices).This system should be able to adapt without a priori knowledge of the open system as we are using evolutionary computation for adapting the interface between the natural and the artificial society with the goal to produce common collective behaviours.
For closing the loop of interaction between bees and CA-SUs, it is first necessary to find a set of different stimuli, to which bees show different behavioural responses.It is already known, that honeybees are collectively attracted to temperature (Szopek et al., 2013).Finding other appropriate stimuli types is part of the research in the ASSISIbf project.Figure 1 shows a preliminary experiment using real honeybees and CASU prototypes.In this preliminary experiment the CASUs use temperature to pull the bees from the left to the right side of an experimentation arena.The temperature settings were controlled solely by the experimenter with no autonomy of the CASUs or interaction between the CASUs and the honeybees.The approach of the ASSIS-Ibf project is to form such a bio-hybrid society of bees and CASUs by using evolutionary computation.These evolved controllers should learn to use the different stimuli to alter the behaviour of the bee collective (e.g., aggregation, separation, path following) Therefore we decided to test in simulation, whether or not, the reaction-diffusion controlled Artificial Homeostatic Hormone System (AHHS) controller is the right tool for this task.
The aim of this work is to evolve a controller for such a bio-hybrid system consisting of two types of autonomous agents: predictable but unprogrammable agents (simulated honeybees) and programmable and evolvable agents (simulated CASUs).We answer the following questions: (I) Is it possible to evolve a controller to redirect reactive agents by the use of different types of environmental stimuli?(II) Is an AHHS controller, evolved for a specific setup, able to perform tasks in a more general environment?To answer these questions, we developed a one-dimensional proof of concept model of a bio-hybrid system in Netlogo (Wilensky, 1999).

METHODS
A set of 101 patches, located in a straight line, represents a one-dimensional experimentation arena (figure .2).Each patch can be considered as one CASU (Combined Actua- After the bees aggregated there, the left CASU was cooled down and the right CASU was heated up to 36 • C. The majority of the group followed the optimum, but some bees remained at the left CASU (B).
tor Sensor Unit) which is able to generate three different stimuli types via implemented actuators.The CASUs are controlled by an AHHS control software which controls the 'behaviour' of the CASUs by perceiving concentrations of virtual hormones and reacting to them according to a set of rules.These rules can then be changed by an evolutionary computation algorithm.

The AHHS control software
AHHS (Artificial Homeostatic Hormone System) (Schmickl and Crailsheim, 2009;Schmickl et al., 2011) is a reactiondiffusion-based system inspired by the Turing process (Turing, 1952) which describes processes of natural pattern formation and growth.AHHS has already been successfully implemented in robotic applications (Stradner et al., 2009;Schmickl et al., 2010;Hamann et al., 2010) and has been investigated in terms of pattern formation and diversity generation capabilities (Zahadat et al., 2013).An AHHS is defined by a set of artificial hormones and a set of rules.The rules define how sensory inputs and hormone concentrations participate in making changes in hormone concentrations and outputs of the system (see figure 3).Both, hormones and rules, may be changed by an evolutionary process.
In our one-dimensional model we use 8 different hormones H which can have values between 0 and 255.H1 forms a prime gradient from left to right, meaning that on the left side H1 has a value of 255 and on the right side a value of 0. This gradient feeds spacial information into the system.H2 has a constant value of 127 across all of the arena with exception of patches -5 to 5. Within this area H2 generates 'white noise' (figure 4), meaning that every patch from -5  to 5 produces a random hormone value in every time step.This area of white noise marks the aggregation spot for the honeybees.The hormones H3 -H5 are used to control the actuators within the CASUs, H3 controls an attractive stimulus A, H4 controls a repelling stimulus B and H5 controls a stumulus C which servers a stopping signal.H6 -H8 serve as 'free hormones' and are used by the AHHS for computing.

Stimuli and Honeybee model
Each patch contains 3 actuators which act as the sources of 3 different types of stimuli.These stimuli types are further called type A, B, and C and are different in terms of their physical properties and the reaction of the simulated bees to them (see Table 1).Stimulus A serves as an attractive signal (e.g., temperature and chemicals) and stimulus B serves as a repulsive signal (e.g., light, airflow).This means that a bee tends to move to a neighbouring patch if the intensity of stimulus A is higher or the intensity of stimulus type B is lower in the neighbouring cell.Stimulus C serves as a stopping signal (e.g., vibration), meaning that if the intensity of the stimulus is above a certain threshold value the bee stops on the cell, ignoring the attractive or repulsive stimulus.It's noteworthy, that the assumed reactions of the simulated bees to the mentioned stimuli in this model are more or less arbitrary.These reactions do only have the purpose to test the suitability of the described algorithms.It is part of the ASSISIbf project to determine the behavioural response of the honeybees on different environmental stimuli in laboratory experiments.
As already mentioned, the CASUs are able to generate 3 different types of stimuli and the simulated bees react differently to each type of stimulus.A bee moves to a neighbouring patch, called newpatch, based on following equation: The values of all parameters used in this simulation can be found in table 1.In this model the bees do not have any physical properties, which means that the bees are able to move through each other and can stack up on a single patch.We made this decision because real honeybees can form very dense aggregations and sit on top of each other.It's also noteworthy, that the parameters of the stimuli do not precisely correspond to reality and the effects of these stimuli on the bees are assumed to be linear for simplification of the simulation, since the purpose of this simulation is to test the before mentioned algorithms.

Experiment I:
The goal of this experiment is to evolve an AHHS controller that leads the bees to form an aggregation at a predefined target area which is marked by white noise of hormone H2.At the beginning, 11 bees are placed equally distributed within the arena.These bees move around in the arena according to their reaction to the presented stimuli.The genetic start population used in these evolutionary runs consist of 30 random genomes.For evolving the AHHS controller, a wolfpack-inspired evolutionary algorithm is used (Zahadat and Schmickl, 2014).This algorithm uses overlapping generations and a fixed population size representing a limited resource (e.g.food).The offspring is evaluated and ranked hierarchically according to their fitness.An evaluated offspring removes another individual with equal or lower fitness from the population.This way the population stays dynamic.
The fitness function in the presented model was defined as where nt is the number of bees in the target area, dt is the distance of a bee to the centre of the target area and dmax represents the maximum possible distance to the centre of the target area.This means that bees which aggregate in the exact centre of the aggregation spot gain more fitness, than bees which locate themselves merely within the aggregation area.The evolutionary runs were repeated 7 times for 151 time steps each.

Experiment II:
After the evolutionary runs, we picked the genome with the highest fitness and tested it in terms of robustness and generality.In this experiment we varied the number of bees in the arena and observed the number of aggregated bees in the target zone.The tested groupsizes were 2, 5, 7, 11, 15 and 20 bees.The experiments were repeated 24 times.

Experiment III:
In this experiment we tested the ability of the system to redirect the bees in a dynamic environment.Therefore we started with 11 randomly distributed bees and the aggregation spot placed off-centre.After 150 time steps we changed the position of the aggregation spot.In one setup the aggregation spot jumped abruptly (figure 9a and 9b), in another setup the aggregation spot moved continuously to its new position (figure 9c and 9d).We monitored the number of aggregated bees after 350 time steps.The tested gap widths between the relocated aggregation spots were in both setups 11, 21, 27, 29 and 31 patches.The experiments were repeated 24 times.

Experiment I:
The evolutionary computation algorithm was tested with 11 equally distributed bees.Their task is to aggregate in the target zone located in centre of the arena, marked by white noise of hormone H2.The system started with random genomes.As it is shown in figure 6, the fitness rises quickly.After the first evaluation the median fitness is 64 with a maximum fitness of 124,5.After 500 evaluations the median fitness already reaches 357,75 and the maximum fitness reaches the possible maximum fitness of 396.After 1500 evaluations the highest median fitness is reached with 383,25 and all but one genomes are capable of positioning the bees in the target-zone.After 2000 evaluations all genomes manage to relocate all bees in the target zone.

Experiment II:
We tested the generality of the fittest genome by varying the number of bees.As we show in figure 7 the evolved genome can handle different groupsizes of bees very well.The evolved genome is able locate 100% of the bees in the target zone, independent of the tested groupsizes.

Experiment III:
We tested the fittest genome in a dynamic environment.After 150 time steps we switched the position of the target zone and after 350 time steps we observed the number of bees in the new positioned target zone.As we show in figure 8 the evolved system is flexible enough to move 100% of the bees to the new target area when the gap between  the targets is between 11 and 21 patches.Once the gap exceeds 27 patches, the number of aggregated bees drops.But still, looking at the 27 patch gap, an average of 96.7% (stdev ±6,4) of the bees are aggregated in the target area.Beyond that distance the aggregation drops rapidly.With gaps of 29 and 31 patches only 13.3% (stdev ±12) and 9,6% (stdev ±7,5) of the bees aggregate at the new target area, respectively.When the target is not moved abruptly but continuously, 100% of the bees could be pulled into the repositioned target area, independent of the gap width.

Experiment I:
As we have shown in figure 6 the median fitness is 357,75 after already 500 evaluations.At this point also the maximum fitness of 369 is reached by at least one genome.After 2000 evaluations all evolved genomes reach high fitness val-  ues (median fitness: 383,25) and are able to pull the bees into the target area.The steep increase of fitness and the good fitness performance strongly indicate that the combination of our evolutionary computation algorithm and the AHHS control software is a feasible tool for developing robot that can learn to manipulate animals.

Experiment II:
As we have shown in figure 7 an evolved AHHS can use the CASU's actuators to lure the bees to a desired location.Although the AHHS controller was evolved for groups of 11 bees it is able to fulfil this task also with other group sizes.We tested group sizes between two and 20 bees and in all cases 100% of the bees were directed to the target area.Since we did not model any interaction between the bees, they can be considered as an indirect probe of the CASU stimuli landcape.Therefore this result is not really surprising.Nevertheless this result serves us as a sanity check of our model.

Experiment III:
When the target area changes its position abruptly, an evolved AHHS is capable to redirect the bees into the new target unless the gap between the two targets gets too big.As we have shown in figure 8, a gap of 11 to 21 patches can be vanquished without any losses to the aggregation properties.All bees can be redirected into the new target area.When the gap surpasses 20 patches, the number of bees located in the new target decreases.Reaching a gap distance of 27 patches the number of redirected bees starts to decrease slightly, but still 97.7% of the bees could be redirected.At a gap distance of 29 patches and above the number of redirected bees decreases significantly.This can be explained by the properties of AHHS, as a hormone gradient triggers the actuators of the CASUs.Since the evolution of AHHS was performed with a target area located in the centre of the arena, the hormone gradient does not spread far enough to overcome the gap.But when the target area does not jump abruptly to a new position, but continuously moves there, all bees can be redirected to the new target, no matter how far the gap between the target is.This can be explained by the fact, that not only the target moves continuously to the new position but also the hormone gradient does.This way the bees can easily be 'pulled' to a new location.

CONCLUSION AND FUTURE WORK
In this work we answered the question if an evolved AHHS is suitable for controlling CASUs and luring simulated honeybees into a predefined target zone.Therefore we generated a model of a one-dimensional experimentation arena containing CASUs, controlled by an AHHS controller.We showed that the combination of the wolfpack inspired evolutionary algorithm and AHHS is very promising tool for this tasks.We also answered the question if an AHHS controller, evolved for a specific setup, is able to perform tasks in a more general environment.Once evolved, AHHS is capable of reacting to dynamical environmental changes.Our results strongly indicate that an evolved AHHS is a flexible and reliable tool and a promising approach for controlling stationary autonomous robots to influence the behaviour of simulated honeybees.
We are well aware that, because of the strong simplification of the bee model and the stimuli model, there's a reality gap from our modelling approach to the real world setup.Nevertheless, our results indicate that the observed algorithms are a feasible choice for the real world problem.
In the future we plan to extend our model by implementing the actual physical properties of the stimuli used.Once we have experimentation data of how honeybees react to the given stimuli types, this will also be implemented into our model.Furthermore we will transfer the gained knowledge to the real world and test our approach on real CASUs with real honeybees.

Figure 1 :
Figure 1: Snapshots of a preliminary experiment with real CASUs and a group of honeybees.(A) The left CASU was heated up to 36 • C (preferred temperature of young bees).After the bees aggregated there, the left CASU was cooled down and the right CASU was heated up to 36 • C. The majority of the group followed the optimum, but some bees remained at the left CASU (B).

Figure 2 :
Figure 2: Screenshots of the one-dimensional experimentation arena.Yellow dots represent honeybees.The green area represents the target for the bees.The purple area represents a mixture of two environmental stimuli: red = temperature, blue = light.This screenshot shows the different behaviours of bees and CASUs for an unevolved (random) (A) and an evolved (B) genome.

Figure 3 :
Figure 3: Graphical representation of the AHHS and the interactions between hormones inside an AHHS, the interaction between two AHHS controllers and the interactions between AHHS and CASUs.H1 -H4 represent different artificial hormones and their interactions.H4 controls the properties of the CASU (located above H4).

Figure 4 :
Figure 4: Hormone levels of the prime gradients used in our model.The x-axis represents the space of the onedimensional experimentation arena.Hormone H1 generates a gradient throughout the arena.Hormone H2 is used as a marker for the target area by producing white noise at this location.

Figure 6 :
Figure 6: Results of experiment I: This figure shows the median fitness (with min, max, q1, q3) of the best genome of each evolutionary run (N = 7 repetitions).fmax represents the maximum possible fitness when all bees aggregate at the exact centre of the target area.f all inn represents the fitness when all bees aggregate in the target area (For details on the fitness function see equation 3 and 4).

Figure 7 :
Figure 7: Results of experiment II: Mean fraction of bees (with stdev) aggregated in the target area.(N=24 repetitions/experiment)

Figure 8 :
Figure 8: Results of experiment III: Mean fraction of bees (with stdev) aggregated in the target area after it has been moved to a new position.The solid line represent the results for abrupt target changes, the dashed line represents the result for continuous target changes.(N=24 repetitions/experiment)

Figure 9 :
Figure 9: Example screenshots of experiment III: spontaneously and continuously moving targets with a gap of 11 and 21 patches.The yellow lines indicate the trajectories of the bees.The green areas represent the target areas.The stimuli are represented by a mixture of red (temperature) and blue (light).

Table 1 :
Characteristics of the stimuli used in the models.