Project report

Interface Modules for Musical XR

As part of this cooperation project with the Center for Music and Film Informatics (CEMFI) at the Detmold University of Music, a bridge module was developed to integrate the SPINE system for physical interfaces into the IVES 3D engine for MAX.

Project participants

Damian Dziwis
Aristotelis Hadjakos (CEMFI - Center of Music and Film Informatics)

Project status

Continued development

OVERVIEW

The possibilities and availability of immersive technologies in the field of extended reality (XR) open up new perspectives in the field of musical XR. These include new ideas for instruments that can be played in virtual reality (VRMIs) as well as new approaches to reception, composition and performance practice.

Current XR devices for virtual and augmented reality not only display virtual content, but can also combine real and virtual environments. This creates hybrid mixed realities. In addition to the standard interfaces such as controllers or hand tracking of XR systems for interacting with VRMIs, the development of custom musical interfaces can offer extended physical possibilities and experiences for interacting with VRMIs and other virtual, sound-generating systems.

In this research project, a modular open-source interface system was developed to realize VRMIs with physical musical interfaces or interactive installations.

To create the tool, the sensor-based SPINE music interface toolkit was combined with the modular 3D engine IVES for the MAX programming environment. The combination of these systems with their no-code/no-soldering approach makes them a customizable tool that can be used to quickly create VRMIs or extended virtual environments with corresponding physical interfaces.

IMPLEMENTATION

The tool developed in this project is designed to help create graphical virtual interfaces with suitable physical interfaces for virtual instruments and similar applications. It allows to quickly create 3D objects, environments and 3D sound, controlled by individual physical interfaces based on sensors. Resulting instruments can be created for conventional PCs as well as for XR devices.

The requirements of the tool were adapted to the target group, especially the field of computer music.

It enables users to quickly develop virtual instruments and physical interfaces. Modules are connected without the need for programming or soldering. Since not all requirements and use cases can be covered by prefabricated modules, more experienced users can change, expand and replace these modules as required by adapting the programming code and hardware.

To realize this, two existing toolkits, IVES and SPINE, were used and an interface module was developed to connect them.

Screenshots from the application

IVES and SPINE are ideally suited here, as they are modular open source systems that already meet the defined requirements and enable the development of visual 3D content and spatial audio (IVES) as well as physical, sensor-based interfaces (SPINE) - without programming or soldering. Experienced users can customize the toolkits to their specific needs.

With the SPINE toolkit, users can design a variety of haptic and non-tactile physical interfaces using multiple sensors. The IVES engine makes it possible to quickly design visual representations of virtual instruments and virtual environments and spatially locate sound. Integration into the MAX programming software opens up many possibilities for sound synthesis and algorithmic composition in a domain-specific language that is widely used in the computer music community and already familiar to many users.

The development of this interface module thus opens up many possibilities for virtual works and the realization of artistic ideas and concepts that would be difficult or impossible in physical reality.

Publication

The research project was presented with a paper publication at the renowned NIME (New Interfaces for Musical Expression) conference in Utrecht (Netherlands) in 2024: https://doi.org/10.5281/zenodo.13904903