Participating researchers:

• Professor Oliver Deussen
• Professor Bastian Goldlücke
• Professor Dietmar Saupe

To be able to study collective animal behaviour in as much detail and as close to reality as possible, animal groups must be embedded in realistic and reactive 3D environments.

Here, it is important to create the illusion of an unbounded natural habitat. This means that the artificial lab environment must be as realistic as possible and adapt to the animals’ behaviour in real time. The aim is to study the visual cues that different species react to and to determine how sophisticated a virtual environment must be to elicit natural responses - a locust’s perception of the world, for instance, differs radically from a bird’s view of its environment. To that end, we must generate realistic environments with natural structures, including simulated active organisms, for instance when studying how red-billed quelea respond to virtual birds of prey.

Research priority 1 will seek to resolve the following challenges: First, we must create the technical groundwork for the observation environment that will be used to expose animals to a realistic version of their three-dimensional habitat and to collect high-resolution data on all individuals in a large collective across temporal and spatial scales. The behavioural experiments with desert locusts, zebrafish and red-billed quelea carried out in research priority 2 enable us to interactively adjust our artificial environments to the animals' actual behaviour.

The technical groundwork for conducting experiments and collecting highly detailed data on swarm behaviour laid in research priority 1 enables us to address fundamental questions in biology, perception research and computer science as regards emergent behaviour and collective responses to various influences. At the same time, the actual technicalities involved in laying this groundwork are at the cutting edge of what’s possible today and require new insights and methodologies in the areas of computer graphics, image processing and computer vision as well as real-time signal processing. The Imaging Hangar and Fish Labs will be central to the proposed experiments.

We will use projection systems to generate realistic environments for all three model species. These environments are updated in real time depending on the animals' behaviour and field of vision. To that end, we also need to collect the required movement data of the swarm in real time, which can be done by recording the behaviour of each of the collectives using various imaging methods, including high-resolution cameras and light field technology. The aim is to capture the animals’ geometry and movements in their local contexts and then to adjust our detailed models for each individual in the collective in line with our observations. That way, we can obtain precise information about the position, orientation and kinematics of the animals as they move through space. If an organism's three-dimensional structure is sufficiently known, we are able to determine the position of the eyes, especially, which means that we can synthesize the individual organism’s visual perception and field of vision using methods borrowed from computer graphics. The data obtained from 3D representations of the animals' field of vision depend on the individual animal’s proper motion, the motion of others and the environment.

In research priority 3, we apply data analysis methods to examine which visual characteristics individuals in a swarm use to make movement decisions. Working closely with research priorities 2 and 3, research priority 1 will thus not only develop new methods for reconstructing three-dimensional movement data in real time as well as interactive representations of realistic three-dimensional worlds. It will also make a major contribution to expanding our knowledge about the visual perception of desert locusts, zebrafish and red-billed quelea.