January 14, 2001
Conference Paper

Creating Tangible Interfaces by Augmenting Physical Objects with Multimodal Language


Rasa is a tangible augmented reality environment that digitally enhances the existing paper-based command and control capability in a military command post. By observing and understanding the users' speech, pen, and touch-based multimodal language, Rasa computationally augments the physical objects on a command post map, linking these items to digital representations of the same; for example, linking a paper map to the world and Post-it notes to military units. Herein, we give a thorough account of Rasa's underlying multiagent framework, and its recognition, understanding, and multimodal integration components. Moreover, we examine five properties of language: generativity, comprehensibility, compositionality, referentiality, and, at times, persistence--that render it suitable as an augmentation approach, contrasting these properties to those of other augmentation methods. It is these properties of language that allow users of Rasa to augment physical objects, transforming them into tangible interfaces.

Revised: September 13, 2002 | Published: January 14, 2001


McGee D.R., and P.R. Cohen. 2001. Creating Tangible Interfaces by Augmenting Physical Objects with Multimodal Language. In IUI 2001: 2001 International Conference on Intelligent User Interfaces, Santa Fe, New Mexico, USA, January 14-17, 2001, 113-120. New York City, New York:ACM Press. PNNL-SA-34270.