You try to remember the name of that movie, but it just won’t come to you. So you tap your fingers on your thighs and your smart glasses know exactly what you’re seeking. Information School Ph.D. candidate Mingrui “Ray” Zhang imagines a future like that — where we can type anywhere as if there were a keyboard at our fingertips.

In fact, TypeAnywhere is the name of a text entry tool created by Zhang and collaborators. TypeAnywhere uses sensors on people’s fingers to simulate the presence of a keyboard wherever they tap. A paper on the project is one of four contributions from Zhang at ACM CHI 2022, the top international conference on human-computer interaction. He’s the lead author on two other papers accepted to the conference and second author on another.

Zhang’s papers are among 51 from University of Washington faculty and students accepted to this year’s conference, including 20 with authors from the Information School. The UW is annually among the leading contributors to CHI.

With TypeAnywhere, Zhang said, “Your fingers are the keyboard.” Computing is taking new forms such as smart clothing, television screens, smart assistants and VR/AR glasses. Carrying different keyboards for multiple devices would be clunky, and speech-to-text isn’t always possible, so Zhang sought a communication solution that would work across devices. With TypeAnywhere, the user wears sensors to track finger movements and can type on any surface.

“It basically enables you to communicate anytime, anywhere, because you are wearing the keyboard on your fingers,” Zhang said. His co-authors on the paper were Shumin Zhai, a principal scientist at Google; and Jacob O. Wobbrock, a professor who is Zhang’s advisor in the iSchool.

Zhang’s research focus is on ubiquitous computing — the concept that computing can be incorporated into people’s everyday lives and environments. He is particularly interested in text entry in all its forms and the role it plays in society, and he likes the challenge of developing technology.

“Some researchers like to understand a problem, so they do processing, analysis, interviews and qualitative work,” Zhang said. “My work is more focused on building stuff. After understanding the problem, we need to propose a solution but also build it.”

Building a better communication experience with computers is the focus of Zhang’s dissertation, which he is planning to defend in June. In addition to novel ways to input text, he is investigating ways to train keyboard algorithms to understand users’ behavior and adapt to serve them better. Some of his recent work has been focused on accessibility, including a second paper accepted to CHI on a GIF annotation system for vision-impaired users. In the paper, Zhang, Wobbrock and Mingyuan Zhong of the UW’s Allen School of Computer Science & Engineering create a crowdsourced database of descriptions to help users understand what is happening in animated GIFs.

“Doing accessibility work gives you a feeling of achieving something meaningful,” Zhang said. “You are actually helping others.”

Coming to the iSchool from China, Zhang saw the value of inclusion as a student who at first struggled with being immersed in a foreign country. His Ph.D. cohort gave him lots of opportunities to connect, and he made connections with other Chinese students in the program.

“I think those are the main reasons why I don’t feel isolated,” Zhang said. “Even though in my first two years, I might not have understood their language or the culture very well, the cohort was constantly having activities to connect to each other.”

“My work is more focused on building stuff. After understanding the problem, we need to propose a solution but also build it.”

Zhang has partnered with iSchool Assistant Professor Alexis Hiniker on some of his recent work, including a third paper accepted to CHI. Zhang and co-authors — including iSchool Informatics student Ravenna Rao and UW Ph.D. students Amanda Baughan and Kai Lukoff — designed and tested alternative Twitter interfaces to see if features built into the app could do more to promote users’ sense of control over their experience and screen time. They found that designs that support users’ agency may perform better than screen-time tools that restrict their use of the app.

Zhang and Hiniker also contributed another CHI paper in a similar vein, designing ways to give Twitter users a better experience and help them avoid mindless scrolling. 

“People are addicted to social media platforms and using those apps to spend their days and their time without noticing they are spending too much time. We wanted to find the design patterns that cause people to become addicted,” Zhang said.

Wobbrock said Zhang is one of the most productive Ph.D. students he has mentored, and he praised Zhang’s willingness to build his knowledge base to make himself a better researcher.

“He’s got a great attitude,” Wobbrock said. “He enjoys the work he does. He likes to think through hard problems, so it’s a delight to watch him work.”

With graduation looming, Zhang has been interviewing both with research universities and with companies in the private sector. Wobbrock expects him to have plenty of career options.

“Whether he ends up in academia or industry, I think Ray’s future impact is going to be quite strong because he’ll continue to be creative, to push his own learning boundaries, to take on new projects,” Wobbrock said. “On either side of that academia/industry fence, he has a chance to really push the state of the art in input and interaction technology forward quite a lot. I am thrilled to see where Ray’s work goes next.”