The camera stays on automatically

The people on stage move quickly, jump, disappear and return. Their facial expressions and many of their gestures would remain hidden were it not for the moving close-ups of faces and bodies on the screen behind the stage. This allows even the audience in the very back row to see anger, resignation or hope. This is not uncommon for the Berlin-based company Nico and the Navigators. The award-winning troupe is known for integrating digital components into its performances. In order to further develop its artistic language, it asked Prof. Dr.-Ing. Steffen Borchers-Tigasson from the Electrical Engineering programme at HTW Berlin for support. The expert in automation and artificial intelligence (AI) is now further developing the camera technology based on AI.

‘Technologically feasible for only a few years’

When the request came, Prof. Dr.-Ing. Borchers-Tigasson did not hesitate for a second. He had already attended private performances by Nico and the Navigators, including some with video projections. ‘The company is incredibly tech-savvy,’ he says. He was immediately interested in the task of developing an automatic, AI-based camera tracking system. “Technologically, this has only been possible for a few years,” says the scientist. At the same time, he says he really enjoys giving artists a new tool and opening up new possibilities for them.

More freedom for dancers and actors

Giving the dancers and actors on stage more freedom was exactly what client Oliver Proske wanted. He founded Nico and the Navigators with Nicola Hümpel 25 years ago. As a producer and managing director, the trained industrial designer and stage designer is responsible for the digital stage, the concept and the appropriate technology in many productions. This technology has become too static for him. ‘It restricts the artists,’ he believes. This is because the cameras have to be preset to the second and all performers have to be at precisely defined points on stage in order for projection-ready close-ups to be possible. ‘To become more flexible, we experimented with a sensor system ourselves for a long time,’ says Proske. However, the aesthetic results were so unsatisfactory that the company sought expert advice.

An intelligent system is needed

The ideal solution is an intelligent camera system that automatically follows artists on stage with complete flexibility, regardless of how fast they move, and produces images that can be selected for projection by the camera director in real time. ‘The technical requirements are high,’ admits Prof. Dr.-Ing. Borchers-Tigasson.

Recognising and following people

Firstly, the camera has to be mobile and able to zoom in and out independently. Secondly, it must not only transmit image data in the form of pixels. Rather, it has to recognise people, follow them reliably and, in case of doubt, track them down again if they leave the stage briefly and return later, which happens regularly in dance and theatre. And if two cameras are pointed at the same scene, the system has to distinguish where the same person is acting. This is easy for the human eye and brain thanks to hair colour, body size and other characteristics. The camera has a much harder job. It has to perform a kind of facial recognition. ‘This is exactly where artificial intelligence comes into play,’ says Prof. Dr.-Ing. Borchers-Tigasson.

‘But nobody wants a robot camera’

This raises the question of whether such an intelligent system will make humans behind the camera redundant in the medium term. Quite the contrary, say all those involved. ‘Nobody wants a robot camera,’ says Oliver Proske. Instead, the technology will give rise to completely new forms of artistic expression. This is exactly what Nico and the Navigators are interested in.

Artistically sophisticated camera direction

The scenario in which the new technology is used can be imagined as follows: During the performance, the camerawoman receives real-time images from the cameras positioned in the room. She can change their settings live using a so-called ‘interface’. The camerawoman then decides which actors to focus on, whether the respective camera should zoom in, from which angle to film and which images to show behind the stage. ‘This is a creative and demanding process that places higher demands on the camerawoman while at the same time giving her more artistic freedom,’ says Prof. Dr.-Ing. Borchers-Tigasson, referring to ‘camera direction’.

Electrical engineering students were also involved

The scientist developed the intelligent camera system step by step in collaboration with the theatre company. Electrical engineering students were also involved in the initial phase with their final theses and project work. ‘It wasn't difficult at all to spark the students’ interest,’ says the university lecturer. In autumn, he accompanied the company to their rehearsal rooms and adapted the technology to the environment and the plot.

Practical test in December at Berlin's Radialsystem

In December 2023, the intelligent camera system will be tested in a real environment for the first time when the play ‘sweet surrogates’ is revived. If all goes well, it will also be used for the performance at Berlin's Radialsystem, where ‘Nico and the Navigators’ perform regularly. Prof. Dr.-Ing. Borchers-Tigasson will of course not be sitting in the audience, but will be backstage, stage fright included.