Sensor Synaesthesia: Touch in Motion, and Motion in Touch

  • Ken Hinckley, Microsoft

We explore techniques for hand-held devices that leverage the multimodal combination of touch and motion. Hybrid touch + motion gestures exhibit interaction properties that combine the strengths of multi-touch with those of motion sensing. This affords touch-enhanced motion gestures, such as one-handed zooming by holding one’s thumb on the screen while tilting a device. We also consider the reverse perspective, that of motion-enhanced touch, which uses motion sensors to probe what happens underneath the surface of touch. Touching the screen induces secondary accelerations and angular velocities in the sensors. For example, our prototype uses motion sensors to distinguish gently swiping a finger on the screen from “drags with a hard onset” to enable more expressive touch interactions.

Watch Next