Project report

Augmented Ubiquity – an immersive work of art

Augmented Ubiquity is an immersive multimedia artwork in virtual space that you can explore freely with your eyes and ears. Sound and sculptures form the framework for this unique dance and music performance. Lou Kilger's composition, Ria Rehfuß's dance performance, Manuel Steitz's VR space, and the immersive audio scene intertwine perfectly to create a completely new kind of artwork.

Project participants

Lou Kilger Sascha Etezazi Benedikt Samuel Jäger

Ria Rehfuß @riarehfuss
Manuel Steitz @wunderkammer_co
Hella Bachnetzer

Project status

finalization

The final result

Here you can experience a 2D version of the immersive scene: You can look over Sascha's shoulder as he explores the scene. You see his view enlarged, while at the same time you can see how he moves in the “real” world in the lower left corner. You hear the dynamically binauralized sound that Sascha heard. Of course, it's not the same as experiencing it yourself, but it's pretty close!

Quote You are in a surreal space. In the center stands a strange figure, surrounded by abstract sculptures. You hear a deep flute coming from the figure. From the left, you hear small rods and shells tinkling; the sculpture next to you must be a wind chime. You walk towards the figure, the flute grows louder. Suddenly, a drum sounds and the figure collapses.
Quote AN ENTIRELY NEW ART FORM

Augmented Ubiquity is a multimedia composition for the VR space that utilizes all the possibilities of the medium. Lou Kilger composed specifically for this space and developed a dance choreography with Ria Rehfuß. We captured both in full volumetric form using the latest technology, such as 4D Gaussian splats and dynamic binauralization. We achieve six degrees of freedom in audio and video, allowing you to freely explore the space and experience a new level of immersion; you alone decide your viewing and listening angles. This leaves 360° videos with fixed listening and viewing positions and 3DoF far behind.

The Visuals

We used our body scanner to record the dance performance. It consists of 112 precisely aligned cameras that can capture all objects that fit into the scanner in three dimensions. However, it is actually only designed for static objects, but we wanted to record a dance performance. To do this, we developed a new workflow and lured our body scanner out of its comfort zone. This enabled us to record the entire performance in high resolution and three dimensions. To relax, the body scanner was then allowed to capture the three sculptures statically again.

Manuel Steitz then used 4D Gaussian splat technology to transform the individual perspectives into a moving 3D model. This technology is based on neural radiance fields and machine learning. The highlight: 4D Gaussian splats are photorealistic and can capture transparencies and reflections in a particularly impressive way.

The Performance

Our body scanner is rather small for a dance stage... So it was clear: we needed a professional dancer who could deliver a captivating performance even in a confined space. Ria Rehfuß developed her performance in close collaboration with Lou Kilger.

The composition

From the outset, Lou Kilger designed the composition “Augmented Ubiquity” to be freely explored. This is important because you can choose your listening position in Augmented Ubiquity completely freely, but the composition has to work at every point in the room. To achieve this, Lou Kilger carefully considered where each sound should be positioned in the room and how they influence each other.

Imagine it the other way around: if you stand right next to the drums in a band, or next to the trombones in an orchestra, do you still hear the music the way it was intended?

Lou has really broken new artistic ground here!

The sound mix

Quote THE SOUND MIX

We have utilized innovative audio technology to create a completely new listening experience for you. With SPAT Revolution, we have added the appropriate acoustics to the visual VR space and precisely positioned and aligned the sounds. For dynamic binauralization, we have also elicited something special from the software; the constant changes in listening position are not actually intended there.

With “Augmented Ubiquity,” you experience the audio scene binaurally; it is dynamically adjusted to your position and head movements. This means that each of your ears receives exactly the audio signal it needs to hear the sound sources and the room from the correct direction and at the correct distance. This allows us to achieve a new level of immersion on the audio level as well.

Whats's next?

We believe Augmented Ubiquity is truly impressive. Currently, you can experience Augmented Ubiquity at our KIO. Our goal is to make the work available as a standalone app for VR headsets so that even more people can explore it.



BEHIND THE SCENES