The Association for Computing Machinery has honored Information School Professor Jacob O. Wobbrock and colleagues with a 10-year lasting impact award for their groundbreaking work improving how computers recognize stroke gestures.

Wobbrock and co-authors Radu-Daniel Vatavu (University Stefan cel Mare of Suceava, Romania) and Lisa Anthony (University of Florida) were presented with the 2022 Ten Year Technical Impact Award in November at the ACM International Conference on Multimodal Interaction (ICMI). The award honors their 2012 paper titled Gestures as point clouds: A $P recognizer for user interface prototypes, which also won ICMI’s Outstanding Paper Award when it was published.

The $P point-cloud gesture recognizer was a key advancement in the way computers recognize stroke gestures, such as swipes, shapes, or drawings on a touchscreen. It provided a new way to quickly and accurately recognize what users’ fingers or styluses were telling their devices to do, and even could be used with whole-hand gestures to accomplish more complex tasks such as typing in the air or controlling a drone with finger movements.

The research built on Wobbrock’s 2007 invention of the $1 unistroke recognizer, which made it much easier for devices to recognize single-stroke gestures, such as a circle or a triangle. Wobbrock called it “$1” — 100 pennies — because it required only 100 lines of code, making it easy for user interface developers to incorporate gestures in their prototypes.

“In academic terms, $1 went viral,” he said. “It really met a need because within a year, there were commercial projects that used it." Those included “Mr. Spiff’s Revenge,” a game made by POW Studios. The game used the $1 recognizer for creating shields, launching fireballs, drinking potions and smashing enemies. It won first place in the 2008 Dr. Dobb’s Challenge.

The $1 recognizer inspired a wave of follow-up work, known as the “$-family” of gesture recognizers, from Wobbrock and other researchers in human-computer interaction. The innovations from the $-family made it possible for prototype designers to quickly enhance the interactivity of their projects without having to understand complex mathematical and statistical approaches to gesture recognition, which had been the state-of-the-art until then. 

The $P recognizer is not limited to single strokes, but can quickly recognize gestures with an arbitrary number of strokes, called multistroke gestures. Its breakthrough was to consider multistroke gestures as “point clouds,” looking at the entirety of what the user draws, whereas previous methods relied heavily on how each gesture was made, limiting their flexibility and reducing their recognition speed. The $P algorithm compares each completed gesture with a set of previously stored point-cloud templates to determine what gesture the user is trying to convey.

Because the code for the $P recognizer is open-source, it is difficult to measure the true reach of its adoption in everyday products such as smartphones and tablets. The $-family has given prototype designers a way to rapidly develop ideas that eventually make their way into people’s pockets and homes, Wobbrock said. For example, designers at Microsoft Research have used $-family recognizers in some of their prototypes, which one day might become products.

“With the advent of touchscreens and their widespread use in personal devices, tabletops and wall displays, gesture recognition is a fundamental form of input today, which the $-family recognizers have made much more accessible and widely used than before,” Wobbrock said.