ALS is a ravaging disease. People living with it slowly lose control of their legs, use of their hands, and the ability to speak and to breathe on their own. Many, however, can still control their eyes. And those functioning eyes and some inexpensive computer hardware are all they need to start using My Visual Piano, an exciting new software application developed by iSchool students for their 2019 Capstone project.

The application allows musicians and music lovers with physically debilitating conditions to literally play piano with their eyes, running notes, laying down chords, and feeling the thrill of making live music again. It just takes a little practice, says Informatics student Parker Ciambrone. “Eyes really weren’t evolved to be an output device. We are repurposing them here.”

Ciambrone, who handles coding and development for the project, is one of four senior Informatics majors on Team Dubstep. Others on the team: project manager Emily Ye, who has played classical piano for 19 years; designer/developer Wesley Wei, a violinist; and project researcher Cole French, who has played piano since age 5.

Sponsored by Microsoft’s pioneering Hands-Free Music project, the team combines a love of music and technology with a passion for breaking down barriers for the severely disabled. “There’s not a lot of technology supporting the ALS field,” says Wei. “We want to show that it is possible to dive into this field and make something really useful.”

Their application uses on-screen buttons to represent keys. Users holding their eyes on a single piano note can suspend that note until they break the gaze. They can also move their eyes to different on-screen targets to run through minor or major chords and change octaves. Unlike other musical applications that are loop-based – creating music by compiling clips from sound banks – theirs is a live-performance tool. Users play in real time.

The only operation requirements are an updated Windows 10 system and a Tobii eye tracker, available online for as low as $150. “Previously, eye-tracking devices could cost thousands of dollars. Now Microsoft has begun integrating technology directly into Windows,” says Ciambrone.

The piano project took root when Ann Paradiso, a principal design manager at Microsoft, came to the iSchool to talk about the Hands-Free Music project she directs at Microsoft. Working with patients with ALS – Amyotrophic Lateral Sclerosis, or Lou Gehrig’s disease – her team has developed technologies that support composing, jamming and other forms of musical expression. The aim is to “restore, redefine and reinvent the music-making experience” for people with impairments.

“iSchool students work in groups a lot. Having that experience helped us get together very quickly and work together to create something helpful for a big community.”

When Ye heard Paradiso speak at the iSchool, something clicked. She’d been wanting to do a project that involved music. “I also wanted to make something that helps others and brings joy into their lives,” she says.

She asked Paradiso to sponsor an iSchool team that could riff off work the Microsoft team was doing. Paradiso agreed and introduced Ye and colleagues to both the latest eye-tracking technologies in her lab and to a special research subject, Jeremy Best, a bright, creative musician with ALS who agreed to be a consultant on My Visual Piano.

Among challenges Team Dubstep faced was mastering an entirely new programming language, C#, in a short window of time. “C# is a language no one really knows at the iSchool. So we had a new interface we had to work with,” says French.

Microsoft wasn’t expecting much those first few weeks, but by Week 5, the team had a working prototype, built on top of Microsoft’s Windows Gaze Input APIs. “They were like, ‘Wow!’” says French.

The team’s mentors, Paradiso and Microsoft principal software developer Dwayne Lamb, were impressed by the students’ hard work. “Emily, Cole, Parker, and Wesley did a fantastic job, showing ingenuity, adaptability, creativity, and a lot of heart in their process,” Paradiso says. “They are all honorary members of the Microsoft Hands-Free Music Project for life.”

The students attribute much of their success to skills learned at the iSchool, especially the ability to work as a team, playing to each other’s strengths. “iSchool students work in groups a lot,” says Ye. “Having that experience helped us get together very quickly and work together to create something helpful for a big community.”  

Once their application is debugged, fully tested, and finalized, the team will hand My Visual Piano over to Microsoft to be part of its Hands-Free Music collection. The app will also be released as open-source coding or as a download for others to build on.

Hopefully, software developers will pick up where the team left off after their project wraps in June, say the students. “We can’t develop the whole range of depth for this project in five-ish months,” says French. “By letting it be open source, it allows other people to help develop our instrument to its full potential.”

The team is excited to see what – and who – comes next for My Visual Piano. Says Ciambrone: “It’s an incredibly rewarding space, being able to see people have some of their creative channels restored.”

Pictured at top, from left: Team Dubstep's Cole French, Parker Ciambrone, Wesley Wei and Emily Ye.