Multimodal Interaction utilizing Voice Command in Real-Time Strategy Game

Abstract

Today speech recognition systems are commonplace and find applications in automatic dictation, simple instrument and computer control, personal identification, and toys. Recent technological advances in connected-speech recognition and position sensing in space have encouraged the notion that voice and gesture inputs at the graphics interface can converge to provide a concerted, natural user modality. However, speech is slow for presenting information, is transient and therefore difficult to review or to edit, and interferes significantly with other cognitive tasks. Hence, even though speech recognition has made great advances over several decades, the pinnacle of speech recognition ability—mimicry of human performance—has yet to be reached. The work described in the paper involves the user commanding inside real time strategy game scenario. The main contribution of the paper is the presentation of an idea and experimentation result on how to design the voice commands in such a way that it is integrated into cognitive tasks and intuitive body gesture of the user of real time strategy game application. Because voice can be augmented with simultaneous pointing, the free usage of pronouns becomes possible, with a corresponding gain in naturalness and economy of expression. This makes voice commanding tasks invisible to the user. Conversely, gesture aided by voice gains precision in its power to reference.