The visual engine draws direct lineage from the Rutt Etra Video Synthesizer (1971–73), the landmark analog instrument developed by Steve Rutt and Bill Etra that displaced the electron beam of a raster display using external voltage signals, bending the scan line itself into sculptural form. Here, that logic is ported into a software context: rather than voltage, hand data deflects and modulates the scan processing, warping, stretching, and rupturing the image geometry in direct response to performer movement. The result carries the same quality that made the Rutt Etra so distinctive: the image doesn't just react to the performer, it deforms with them, the body inscribed into the structure of the picture itself.
On the audio side, mapped gesture parameters feed into Ableton as continuous control signals, allowing the performer to shape synthesis, effects processing, sampling, and mixing with open-air movement. The tight coupling between the OSC data streams means a single gesture can bend a synth filter and distort the scan geometry at the same time, collapsing the boundary between sound and image into a single performative act.
The system is designed for two contexts:
Live A/V Performance — the performer works within the tracking field as a kind of living control surface, the hands conducting both the sonic and visual material in real time. The interface disappears; what remains is gesture, image, and sound in direct conversation.
Large Scale Interactive Installation — deployed without a dedicated performer, the system opens to audience participation. Visitors enter the tracking field and find their own movements reshaping the visual and sonic environment, the scan-processed image responding to bodies it was never specifically trained to expect.
Built entirely on open, extensible tools — Python, MediaPipe, OSC, Max/MSP, Ableton — the system is modular by design, with each layer independently mappable and reconfigurable for different venues, scales, and conceptual applications.