“It Takes Two” - Shared and Collaborative Virtual Musical Instruments in the Musical Metaverse.

The relevance and technical possibilities of Shared Virtual Environments (SVEs) are constantly growing as part of what is known as the Metaverse. This includes software and web platforms for creating SVEs and the availability of hardware for experiencing these environments. SVEs offer unique capabilities that have yet to be explored, especially in music. In this paper we explore the concept of networked Virtual Musical Instruments (VMIs) for the Musical Metaverse, where virtual spaces are specifically designed for musical collaboration and social inter-actions. We describe three prototypes for shared, collaborative VMIs that incorporate specific features of SVEs, such as spatial audio, data sonification, and embodied avatar-based interactions. We conducted a user study to investigate how these instruments can support creativity and usability and to what extent they can deliver a sense of social presence and mutual engagement between users. Finally, we discuss how the three implementations of the proposed shared and collaborative instruments provide novel avenues for music-making in the Metaverse. Our results show that the three instruments exhibit varying degrees of creativity and usability. However, instruments that employ symmetrical and embodied interactions better support social presence and interdependence among users.

2024 IEEE 5th International Symposium on the Internet of Sounds (IS2), 1–10.

Boem, A., Dziwis, D., Tomasetti, M., Etezazi, S., & Turchet, L. (2024)

Interface Modules for Extended Reality in Music

The growing possibilities and availability of immersive technologies in the field of Extended Reality (XR) are leading to new potentials in the context of Musical XR, like the emergence of new concepts for virtual reality musical instruments (VRMIs), reception, and composition and performance practices. Recent XR devices for Virtual and Augmented Reality allow not only to present virtual content, but also to blend real and virtual environments to create hybrid Mixed Realities. Therefore, in addition to using standard interfaces such as controllers or hand tracking from XR systems for interacting with VRMIs, developing custom musical interfaces can provide enhanced physical possibilities and experiences for interacting with VRMIs and other virtual sound-generating systems.

Proceedings of the International Conference on New Interfaces for Musical Expression, 439–446.

Dziwis, D., & Hadjakos, A. (2024)

„Revisiting Reynolds - Autonomous Agents for Spatial Audiovisual Composition and Performances“, Conference on AI Music Creativity (AIMC), Brighton, Vereinigtes Königreich

„The Entanglement – Volumetric Music Performances in a Virtual Metaverse Environment", Journal of Network Music and Arts

TED-Talk „Being creative in the Art with Artificial Intelligence (AI)“, TEDxKIT, Karlsruhe

„Machine Learning-Based Room Classification for Selecting Binaural Room Impulse Responses in Augmented Reality Application“, International Conference on Immersive and 3D Audio (I3DA), Bologna, Italien

IVES - Interactive Virtual Environment System: a modular toolkit for 3D audiovisual composition in Max“, Sound and Music Computing Conference

„Body-controlled sound field manipulation as a performance practice“, AES Vienna, Vienna, Austria

„Development of a 7th Order Spherical Microphone Array for Spatial Audio Recording“, DAGA „Jahrestagung der Akustik“, Rostock