Hearing gestures: Vocalisations as embodied projections of intentionality in designing non-speech sounds for communicative functions

The development of intuitive user interfaces is a vast challenge in the years to come. Sound as a communicative medium has a lot of potential, that has not yet been fully utilised in the development of user interfaces. My dissertation aimed at tackling with both of these issues, by discovering new theoretical and methodological foundations for user interface sound design.

My focus was on questions of, firstly, how do we utilize our bodily knowledge when we communicate with sound and interpret the sounds we hear? And secondly, how this tacit, embodied type of knowledge can be utilized in sound design?

In particular, sound seems to convey bodily knowledge when it is perceptually conceived as bodily activity. A Finnish sound-artist Simo Alitalo has called this phenomenon "knowing through sounds of doing". In my approach to sound design, sound is fundamentally outlined and conceived as activity, which, in terms of embodied experience, is significant in itself.

The empirical part of my dissertation studied how speech melody may convey interpersonally meaningful gestures. These gestures of person-to-person interaction may indicate, for example, the communicative intent of a speaker.

The results demonstrate that gestural patterns in speech can be methodically utilised in designing non-speech user interface sounds, which communicate their intended meanings intuitively. Also, the outlined method can be used across sensory modalities, that is, in designing tactile user interface messages.

Despite the currently strong general emphasis on user experience (or UX, if you like), there is an ever-growing need for design tools through which experiential aspects of interaction could be accounted for. My thesis proposes a partial answer to such a need.

The PhD thesis is available on the university's web archive [follow this link]. You can also view the press release of my dissertation via [this link] (in Finnish only).