If you were a travel vlogger who had just gotten back from, say, a trip to watch the U.S. men's World Cup soccer team trounce Australia, and you wanted to create a video about the experience, what would your first step be?

A new tool called Obby, created by a team of five Informatics students for their Capstone project, enables videomakers to use natural language to search for the specific clips they want to feature. For example, a user could search for “U.S. scores goal” and find that clip amidst all of the footage they had uploaded to Obby, saving valuable time.

Rising seniors Chelsea Li, Derek Liu, Cleo Reyes and Keira Wong and recent graduate Kathryn Rochleau-Rice worked together on projects for several years after meeting as first-year direct-to-major Informatics students. Their two-quarter Capstone project, a full-stack semantic video search application, represented the culminating experience of their three years working together. Faculty praised their professionalism and skill at working with their Capstone sponsor, Microsoft.

“They've scoped a tight end-to-end pipeline from upload to semantic search, and are already addressing real engineering challenges, such as processing performance,” said André Bearfield, Capstone instructor. “They have built a strong working relationship with their sponsor that keeps their MVP [minimum viable product] grounded in actual user needs.”

Rochleau-Rice, who served as the Team Obby’s project manager, said that the team originally considered a project with a different focus, but then they shifted to video search, inspired in part by friends who create videos. The team wanted users of their design to be able to identify specific clips easily and quickly within both long- and short-form content, no matter the background or context. “For vloggers, it’s very organic. They’re taking a lot of videos of a lot of different things, a lot of different objects, a lot of different settings, and so we knew that we wanted to focus on that type of content creation,” she said. 

The students built Obby on Azure, the Microsoft cloud computing platform. Wong, who is a Microsoft Learn Student Ambassador, helped forge the connection between Team Obby and Bella Chan, the project sponsor. Chan is a program/product leader working in Azure product engineering at Microsoft. She was impressed by how professional the students were about incorporating her feedback, she said, such as opportunities for benchmarking, UX/UI improvements, and supporting the ideas they want to test and pursue — always backed by data.

“They’re going for four phases [of software development], which is great to see,” Chan said. “They complement each other. Their documentation is flawless, too.” Chan connected the team with additional colleagues who provided valuable mentorship to the students on semantic search, which can deliver broader results than lexical search. 

By tagging content and querying their own video libraries, users can avoid scrubbing through hours of footage to find a specific moment, a pain point the team was familiar with from their sessions with user testing and feedback. Semantic searching can typically handle more nuance in meaning, and can be more user-friendly for people without technical backgrounds. Obby also incorporates video editing tools, allowing users to trim their video clips. 

“Everyone’s been so attentive and proactive, and everyone takes ownership of their work.”

Team Obby built the tool with a hybrid search pipeline that combines multiple methods of searching for the greatest likelihood of identifying the right clips. Users upload their videos, the videos are added to the Azure Video Indexer, and then "AI tools sample frames throughout the video and generate searchable metadata for each segment. The result is that all of this information is searchable on the front-end using everyday language.

“This project is very engineering-intensive,” said Liu, who focused on back-end software engineering. “Katie, as our product manager, has done a really good job of planning out meetings, delegating work, and defining what tasks should be done every week.” Rochleau-Rice is also a certified Scrum Master, and brought this experience to project management for Team Obby. 

“She also helped us communicate with our sponsors and helped write up a lot of the documentation as well, along with conducting user testing, feedback, documenting feedback, and then also working with access requirements,” Liu said of his teammate.

Security measures, Reyes said, included taking steps so that no user can access or edit another person’s videos without their permission and ensuring that sensitive credentials are never live in the front-end. 

Liu said that as part of their software development process, the team also reviewed ethical considerations including privacy leaking via AI, bias in semantic understanding, and unequal access for video creators. They explored ways to address these concerns, including by acknowledging and testing for bias, documenting user feedback and communicating limitations. 

Keeping in touch through Discord, working out bugs and planning next steps, the team met weekly over the two quarters, in the evenings after their other responsibilities. Each of the students juggled many competing demands on their time, including jobs, research, teaching assistantships and student leadership positions. 

Rochleau-Rice said, “I'm very grateful for our team, because if it was any other group of people, I know things would slip through the cracks. Everyone’s been so attentive and proactive, and everyone takes ownership of their work.” 

Designers, engineers from Microsoft, and content creators all tested the product and shared about their specific use cases. The students used what they learned to help reduce processing time for uploading videos and improve the user experience overall.

“We had a really ambitious vision for how we wanted to build this,” Reyes said. The team wanted to develop the strongest pipeline possible and to learn about how they can apply this process in other ways in the future, such as to their shared interest in health-care technology. 

Potential next steps include making a mobile application in addition to the current web application, further reducing processing time, enhancing the user interaction experience, and adding cloud drive integrations for easy access to stored footage, Reyes said.

Chan couldn’t attend the Information School’s Showcase to see the students demonstrate Obby, but as a proud project sponsor, she sent the students chocolate to mark the occasion.

“They knew it was a challenging project,” she said, “but the thing is, they liked those challenges.”

Pictured at top, from left: Derek Liu, Keira Wong, Kathryn Rochleau-Rice, Chelsea Li and Cleo Reyes.