Encounter with the digital twin of our brain

At a research station in the Humboldt Lab, the ‘Adaptive Digital Twin’ project of the Cluster of Excellence ‘Matters of Activity’ and Charité Berlin will take a look inside the human brain. 3D models of this complex organ will be shown as holographic representations. In clinical practice, such visualisations support the preparation of operations, for example.

Holographic projections are considered an obligatory fixture of science fiction series. However, they have now arrived in reality – as holographic displays in the Humboldt Lab will show. Two such ‘Looking Glasses’ can be used to inspect 3D models of the human brain, virtual twins of our most complex organ.

Such visualisations are not only appealing for aesthetic reasons, they also serve medical purposes. ‘This is something we work with every day,’ reports Lucius Fekonja, research associate at the Department of Neurosurgery at Charité – Universitätsmedizin Berlin and the Cluster of Excellence ’MATTERS OF ACTIVITY. IMAGE SPACE MATERIAL’ (MoA). He is leading the development of the ‘Adaptive Digital Twin’, an ‘adaptable, digital twin’ of the human brain. This project is based at the Image Guidance Lab (IGL), an interdisciplinary working group of the neurosurgery clinic at Charité and the MoA Cluster of Excellence at Humboldt University in Berlin, headed by PD Dr Thomas Picht.

‘Adaptable digital twin’ of the human brain

Various visualisation strategies are being developed there that are used in neurosurgery. The ‘adaptable digital twin’ of the human brain is modelled using different data: Magnetic resonance imaging (MRI) images are fed in, as well as the results of neuropsychological tests – combined with statistical analysis to classify or correlate the data and the clinical picture. ‘The structural data is combined with functional data – for example, the anatomy of the individual brain with speech function,’ explains Fekonja. This means that the Adaptive Digital Twin is more than just a snapshot. It is in motion and reacts to changes. The aim is to simulate processes and interventions in this way, reports the project manager. ‘For example, we can see what happens when we stimulate certain parts of the brain.’

The Adaptive Digital Twin works in a similar way to an online map service. It offers orientation and instructions on the virtual map of the brain. How well this works depends on the quality of the information fed in. ‘The more precise the data is, the more precise the model is,’ says Fekonja. After all, accuracy is particularly important when it comes to brain surgery.

This three-dimensional map serves various purposes. For example, doctors can use it to visualise the location of a tumour in the brain. One aim is to make certain interventions superfluous by enabling virtual navigation in a patient’s head, explains the project manager. The Adaptive Digital Twin can also be consulted during an operation itself.

Visitors gain insights into an ongoing research project

The research station in the Humboldt Lab will consist of the holographic displays, two high-performance computers and a tablet on which, for example, films of operations can be shown. The plan is for a scientist from the Image Guidance Lab to move their workstation into the exhibition once a week and work directly on the aforementioned equipment, reports Friedrich von Bose, Deputy Head Curator at the Humboldt Lab. ‘This gives visitors an insight into the work on the three-dimensional visualisations and allows them to address any questions that arise directly.’

For the audience, this is an opportunity to gain an insight into an ongoing research project. As complex as the brain may be, there are plenty of points of reference. ‘Many people have come into contact with neurosurgery at some point in their lives, for example because they know someone who has had to undergo surgery for a brain tumour,’ says von Bose. Contact with visitors is also exciting for the Cluster of Excellence. What effect does her work have on people who otherwise have nothing to do with neurosurgery? Is it understandable what they do there? What are possible ethical concerns?

The researchers can visualise different parts of the brain on the holographic displays – for example the grey and white matter. ‘A tumour can also be highlighted as a three-dimensional object,’ explains Fekonja. As with a games console, visitors will be able to move the 3D models around the room to view them from all sides. The monitors also offer the opportunity to make comparisons between different brain states, including those caused by illness. Various international working groups are researching digital twins, reports the project manager. The special feature of the Image Guidance Lab is that the applications are being developed in the context of neurosurgery. ‘Advanced imaging methods are usually developed in basic research and take time to find their way into clinical practice,’ says the scientist.

The Adaptive Digital Twin, on the other hand, is developed and used directly in practice. Another unique selling point is that the design aspect is considered from the outset. ‘The design should be as user-friendly as possible,’ emphasises Fekonja, who studied scientific visualisation and knowledge visualisation in Lucerne and Zurich and wrote his dissertation in theoretical medicine at Charité. The Cluster of Excellence ‘Matters of Activity. Image Space Material’ has the vision of rediscovering the analogue in the activity of images, spaces and materials in the age of the digital. Forms, design and materiality play a central role.

Do we recognise ourselves?

The Adaptive Digital Twin therefore also demonstrates what interdisciplinarity means in modern research. ‘For us, this is a special object on several levels,’ emphasises Friedrich von Bose. For example, it opens up the discussion as to what the ‘scientific object’ in this installation is – is it the ‘Looking Glass’ with the holographic projection? Is it the imaging process on which it is based? It becomes clear here, especially in contrast to the objects from the history of science exhibited in the neighbourhood, that it can be the act of exhibiting itself that turns something into a scientific object. The Adaptive Digital Twin also once again poses the much-discussed question of the significance of images in science – and in medical practice. Visualisations play a central role in the relationship of trust between doctor and patient, says the curator. Under certain circumstances, a doctor who makes a drawing of the brain by hand during a consultation can build up greater trust than if he or she uses a high-resolution MRI scan.

How the holographic projections affect visitors to the exhibition will be revealed in the Humboldt Lab. What feelings creep up on us when we look at this digital twin? Do we recognise ourselves – or do we feel like we are on a visit to the Starship Enterprise? Are we thrilled, amazed or even petrified? The reactions to the modern visualisation possibilities are also interesting for the researchers.