Do You Hear Me?: An Interactive Multimedia Composition Integrating Sign Language, Pure Data, and Real-Time Visual Synthesis for Hearing-Impaired Performers

This paper presents the creative and technical analysis of the multimedia composition, "Do You Hear Me?", an interactive work designed to empower hearing-impaired performers to create sonic art through sign language. Historically, individuals with hearing impairments have faced significant barriers in experiencing the nuances of music and participating in live musical performances alongside typical musicians. To address this gap, this composition utilizes a video score as a primary framework. As the video score is projected, a hearing-impaired performer signs the corresponding text. A camera system tracks the real-time movement and spatial coordinates-specifically the horizontal (x) and vertical (y) axes-of the performer’s left hand, right hand, and facial center. These spatial coordinates are converted into numerical data streams mapped to dual software environments for real-time synthesis. In Pure Data (PD), the tracking data dynamically modulates core audio synthesis parameters, including sound frequency, audio filter coefficients, and amplitude. Concurrently, the data is routed into VDMX to analyze and manipulate live cinematic visual effects. By transforming expressive sign language into a unified tactile, auditory, and visual synthesis engine, this project establishes an accessible, inclusive methodology for interactive performance, bridging the gap between deaf communities and electronic music composition. For hearing-impaired participants, the experience of the composition may vary according to the nature and degree of hearing ability. However, the work is designed so that musical agency does not depend solely on auditory perception. Participants can experience the piece through the visible rhythm of signing, the physical sensation of movement, the visual transformation of projected images, and the immediate cause-and-effect relationship between gesture, sound, and image. In this sense, the performer is not merely interpreting a pre-existing score but actively shaping an audio-visual environment through the body. By transforming expressive sign language into a real-time synthesis engine, Do You Hear Me? proposes an inclusive methodology for interactive performance. The project bridges sign language, electroacoustic composition, and multimedia technology, while expanding the understanding of musical participation beyond conventional listening and instrumental performance.