Informatics student Chris Oh discovered a fascination in what computers “see.”

The expanding field of computer vision (CV) teaches computers how to recognize patterns in digital images and videos. The technology enables self-driving cars, traffic monitoring, innovative video games and sorting mail by reading hand-written zip codes. It’s in factory robots monitoring for problems and in our cameras’ ability to recognize faces.

Oh pursued his interest in CV by auditing a computer science class and taking online classes to learn more. The next step was to volunteer for hands-on work through a research project called Viziometrics, directed by iSchool Assistant Professors Jevin West and Bill Howe at the DataLab.

Viziometrics tests the hypothesis that there is one graphically represented figure in a scientific paper that best summarizes the results that the researcher wants the reader to know. The figure could be a graph, chart, photograph, visualization or an equation, and the goal was to see if the computer could automatically identify this key figure.

Oh joined midway through this project when most of the infrastructure was already in place. He helped another Informatics student, Lia Kazakova, who graduated in spring ’17, to test the prediction with a survey and correlate the results.

“To do this, we first needed to assess whether a central figure existed,” says Oh. “To find out, we asked the authors of more than 480,000 research papers from PubMed whether they could identify a central figure in their paper. We only had about a 1 percent response rate, but it gave us a fascinating look at how scientists communicate with figures. We are using this labeled data set to see if we can develop methods for automatically identifying these graphical abstracts.”

His first research discovery was that sending out an email survey isn’t easy. He had to learn about spam rules, implementing an unsubscribe button, and IRB approvals for research done with human subjects – all the ground rules for conducting a good research survey.

“The University of Washington system only allows you to send 10,000 emails per day, so it took me 40 days to send out the survey,” says Oh.

It was worth it. The Viziometrics project, presented at the iSchool’s 2017 Capstone event, won the Informatics “Best in Class” award for Oh and Kazakova.

His other CV research project was with a team of Informatics students doing their Capstone on self-driving cars. The project, called Road IO, would help the development of these cars along by crowdsourcing dashcam footage from cooperating drivers. They then would package the crowdsourced data for sale to autonomous car companies.

Both the Viziometrics and Road IO projects were also presented at the UW Undergraduate Research Symposium in May 2017.

Research is one way to explore CV. Another is to work for a company that is using it.

Oh did a summer internship with New York fashion conglomerate HBC Digital. There, he worked with a CV scientist who analyzes dresses on HBC Digital’s websites for similar patterns and textures to recommend to online customers based on their searched interests and purchase patterns.

“What I got to work on was analyzing a customer’s previous purchase data to be able to recommend cross-categorical information. For example, if you buy a handbag, you might be interested in buying a shoe of this color and a heel of this height.”

He will be taking a full-time position there as a software engineer once he graduates. Before that, he is taking on one more CV research project in his third year with Assistant Professor West.

Artstor, a non-profit, has a digital library that includes “more than 2 million high-quality art images for education and research from a wide variety of contributors around the world.” The problem is that the images are hard to find, as they aren’t tagged in a meaningful way so they aren’t easily accessible.

“We are trying to see if we can extract meaningful metadata from those images and cluster them into groups of similar features,” says Oh. “For example if you just looked at Picasso, you might be interested in a related image that might not be from Picasso. We might show images that have the same texture, color scheme, or image definition  – something that the human eye might not be able to catch but the machines might be able to find in those paintings.”

This project is just beginning, so there is more learning for Oh to do this year.

Oh wasn’t always interested in research.

“When I took the research methods class, I didn’t really enjoy that class,” chuckled Oh. “I didn’t think I would really enjoy research. But then, after that quarter, I found out about Viziometrics and got interested doing research for the project. I went back to Professor Jacob Wobbrock, who was my instructor, and asked for the resources he taught in class that I hadn’t absorbed at the time. I was amazed by the different approaches to research and developed an appreciation for the field.”