Future Music Hubs - DAE Research Internship
Seventeen weeks at DAE Research, building low cost, low complexity and efficient audio visual tools using gesture, sensor, and lcoal AI. Everything ran on a webcam, a consumer PC, and mostly open-source software: TouchDesigner, Ableton, Python.
MediaPipe Gesture Controller
The foundation the rest of this work was built on. A Python tool using MediaPipe hand and gesture tracking, sending data over OSC into Ableton via Max for Live and into TouchDesigner. Packaged into a standalone Windows executable so it could be handed to someone without a Python environment.
Made with Python, MediaPipe, OSC, PyInstaller.
Gesture Clip Launcher
Hands controlling Ableton's clip launcher directly. Hand position and gesture select the track and clip slot over OSC, with TDAbleton handling the heavy lifting on the Ableton side.
Made with TouchDesigner, TDAbleton, MediaPipe.
Conductor
Right hand velocity and vertical position mapped straight to Ableton's BPM. Left hand pinch and horizontal movement sweep a master filter. About as simple a mapping as it gets, and it landed harder than most of the more complicated systems.
Made with TouchDesigner, MediaPipe, Ableton.
VideoSynth: Object Movement to MIDI
Tracks how objects move in a camera feed, relative position and distance between them, and turns that into four-voice MIDI: scales, arpeggios, or chords. Slow objects near the center produce pad-like chords. Fast, chaotic movement produces fast note runs.
Made with TouchDesigner, MediaPipe.
Weather Instrument
Live weather data, temperature, wind speed and direction, humidity, precipitation, cloud cover, pulled from the Open-Meteo API and mapped to a generative Ableton project and a GLSL particle system that traces the silhouette of a masked webcam subject, colored by current conditions outside.
Made with TouchDesigner, Python, Ableton, Open-Meteo API.
Apparel Color to Chord
A color detection system that reads the dominant clothing color of people on camera using MediaPipe's segmentation, then maps that color onto a circle of fifths and a color wheel to generate a MIDI chord. Built toward tracking multiple people at once, one person per chord tone, tonic, third, fifth, based on color, distance between them, and brightness.
Made with TouchDesigner, MediaPipe, Python.
Drum Sequencer
A 16-step drum sequencer synced to Ableton's BPM, plus a stackable window system in TouchDesigner so multiple patches or controls can sit alongside a full screen Ableton set during a live performance. Also built a small browser-based tool to generate scale-correct MIDI note sets on the fly.
Made with TouchDesigner, Ableton, JavaScript.
lament_vision - DITHERMETHIS
A generative text and music installation. A vision model reads a live camera feed and describes what it sees, a language model rewrites that description as a fragmented, dark poem with an adjustable severity dial, and the resulting mood drives MIDI chord generation. Wrapped in a dithered, low-fi DOS visual aesthetic to match.
Made with Python, local vision and language models, TouchDesigner, MIDI.
MediaPipe Gesture Controller
The foundation the rest of this work was built on. A Python tool using MediaPipe hand and gesture tracking, sending data over OSC into Ableton via Max for Live and into TouchDesigner. Packaged into a standalone Windows executable so it could be handed to someone without a Python environment.
Made with Python, MediaPipe, OSC, PyInstaller.
Gesture Clip Launcher
Hands controlling Ableton's clip launcher directly. Hand position and gesture select the track and clip slot over OSC, with TDAbleton handling the heavy lifting on the Ableton side.
Made with TouchDesigner, TDAbleton, MediaPipe.
Conductor
Right hand velocity and vertical position mapped straight to Ableton's BPM. Left hand pinch and horizontal movement sweep a master filter. About as simple a mapping as it gets, and it landed harder than most of the more complicated systems.
Made with TouchDesigner, MediaPipe, Ableton.
VideoSynth: Object Movement to MIDI
Tracks how objects move in a camera feed, relative position and distance between them, and turns that into four-voice MIDI: scales, arpeggios, or chords. Slow objects near the center produce pad-like chords. Fast, chaotic movement produces fast note runs.
Made with TouchDesigner, MediaPipe.
Weather Instrument
Live weather data, temperature, wind speed and direction, humidity, precipitation, cloud cover, pulled from the Open-Meteo API and mapped to a generative Ableton project and a GLSL particle system that traces the silhouette of a masked webcam subject, colored by current conditions outside.
Made with TouchDesigner, Python, Ableton, Open-Meteo API.
Apparel Color to Chord
A color detection system that reads the dominant clothing color of people on camera using MediaPipe's segmentation, then maps that color onto a circle of fifths and a color wheel to generate a MIDI chord. Built toward tracking multiple people at once, one person per chord tone, tonic, third, fifth, based on color, distance between them, and brightness.
Made with TouchDesigner, MediaPipe, Python.
Drum Sequencer
A 16-step drum sequencer synced to Ableton's BPM, plus a stackable window system in TouchDesigner so multiple patches or controls can sit alongside a full screen Ableton set during a live performance. Also built a small browser-based tool to generate scale-correct MIDI note sets on the fly.
Made with TouchDesigner, Ableton, JavaScript.
lament_vision - DITHERMETHIS
A generative text and music installation. A vision model reads a live camera feed and describes what it sees, a language model rewrites that description as a fragmented, dark poem with an adjustable severity dial, and the resulting mood drives MIDI chord generation. Wrapped in a dithered, low-fi DOS visual aesthetic to match.
Made with Python, local vision and language models, TouchDesigner, MIDI.
Houdini Procedural Weapon tool
During my formal education as technical augio engineer I opted to try my hand at procedural 3D tool building using Houdini. The result was a powerful and relative hassle free tool that allows for granular shape sculpting and granular detail controlling.
Made in Houdini
Made in Houdini
Scifi Environment
The result of our 3D Environment exam using Maya and Unreal Engine. The task was to manifest our theoretical and practical knowledge gathered from the 3D courses at DAE in to a single cohesive environment. As a audio engineer I also opted to liven up the scene using basic blueprint driven audio delegation, along with a simple interaction event to add to the atmosphere.
Modeled in Maya, Environment and animation made in Unreal Engine 5
Modeled in Maya, Environment and animation made in Unreal Engine 5
Altreel
Scifi Environment
An alternative cinematic reel showcasing the previously mentioned 3D environment
