How I Turned My Music into 3D Visual Art: Melody Figments
What if every motion you made with your hands left a trail of colorful lights? Imagine if a pen, a pointer, or a percussionist’s mallets were paintbrushes!
In this project, I combined my passion for music, technology, and visual art to create exactly that: Melody Figments.
Melody Figments are pieces of digital art that visually represent the nuances of musical performance. By capturing the movements of a performer and translating them into dynamic visual forms, I wanted to create an immersive experience that combines sound and sight. Here’s an in-depth look at how this innovative concept came to life:
Conceptualization
With 30 motion-tracking nodes attached to my body, I embarked on a journey to map my movements and transform them into visual art. As a percussionist, composer, and public artist, my performances involve intentional, intricate movements. The idea was to capture these nuances and turn them into digital art, showing the unique landscape of each musical moment.
Collaboration with Mad Labs
To bring this vision to reality, I collaborated with Mark Roy and the Mad Labs team based in Dania Beach, Florida. I met them at a CODAworx summit, where they expressed interest in experimenting with motion capture technology. Soon after, I found myself in a motion capture suit, ready to translate my musical performance into data!
Recording and Initial Data Processing
In the studio, we recorded a piece of music specifically composed for this project. Each of my movements—whether slow and deliberate for soft sounds or quick and forceful for loud ones—was captured. Then, using Unreal Engine, we turned my full-body movements into an avatar, creating a digital map of my performance.
Inspiration from Nature
Now that we had this motion data, it was time to do something creative with it. However, at the time, we didn’t know exactly how to express that.
While hiking, I noticed intricate spider webs and saw a parallel between their patterns and the lines my hands created while playing music. This inspired me to capture the linear energy of my movements in a similar way. I snapped photos of the spider webs, creating mood boards and physical models using tulle and planting wire to represent these lines of musical energy.
Prototyping and Refinement
Back at Mad Labs, we used a particle system in Unreal Engine to transform my motion data into three-dimensional visualizations. We experimented extensively with colors, textures, thickness, and weight to accurately represent the musical dynamics. Loud sounds translated to brighter colors, while softer sounds became darker, creating a vivid, dynamic visual representation of my music.
Live Performance Integration
Taking Melody Figments further, we wanted to incorporate this visual art into live performances. Using an infrared camera, we developed a system to track my body in real time and translate it into visual art projected on surfaces behind me. This setup allowed audiences to see the music come to life, making the concert experience immersive and multi-sensory.
Final Presentation
The culmination of this project resulted in a series of Melody Figments that represented the recorded studio session. But we didn’t stop there; we translated this concept into live performances, where the visual art evolved in real time with the music. Audiences could see my body movements, hear the music, and witness the corresponding digital art, creating a deeply immersive experience.
Conclusion
Melody Figments showcases the power of merging music with technology to create unique, immersive art. By capturing and visually representing the essence of a musical performance, we can offer audiences an experience that engages multiple senses and brings them closer to the art.
Interested in exploring the Melody Figments project in more detail? Watch the full video here!