The words and images of famous American writers like James Baldwin or Mark Twain circulate widely online, and are often invoked in social and political conversations. These texts and images — frequently paired together — sometimes draw from their youth or old age, or from very different parts of their prolific careers. 

These pairings reflect different interpretations of the same writer and ultimately help shape how they are remembered. Text-only searches can miss the visual details that contribute to those interpretations. 

These cultural figures are not only cited by individuals online, but also by government officials and representatives. A new project led by University of Washington Information School Assistant Professor Benjamin Charles Germain Lee (pictured at top) aims to help scholars examine how nuances in image-caption documents shape representations of historical figures across millions of government documents. Supported by a $403,833 Schmidt Sciences research grant announced Oct. 14, the team plans to develop artificial intelligence tools to trace how historical and literary figures have been quoted, pictured and remembered across agencies and presidential administrations.

Lee, the principal investigator, will work with co-principal investigators Melanie Walsh, an Information School assistant professor; Kyle Deeds, an assistant professor of computer science at Boston University; and Trevor Owens, chief research officer at the American Institute of Physics. Ying-Hsiang Huang, an Information School doctoral student who worked on the team's GovScape search system, will also join the project.

The three-year project, “Toward a Multimodal Understanding of 250 Years of U.S. Government Documents,” brings together expertise in computer science, information science, literary studies and the history of science. The researchers plan to assemble more than 10 million image-caption pairs from over 100 million pages of government documents spanning 250 years, use the data to train and fine-tune AI models, and release the data, models and research tools for others to use.

The work builds in part on GovScape, a search system developed by Lee, Deeds and collaborators. Its published version covers about 10 million PDFs from the End of Term Web Archive’s 2020 collection and supports keyword, semantic and visual searches. The new project would expand that foundation, helping scholars trace recurring words and images and interpret their changing contexts.

Multimodal AI models process more than one type of information. Vision-language models, one type the team plans to use, process images and text together. This could help researchers find and study quotations embedded in images that text-only searches can miss, a limitation Walsh encountered in her previous studies. For example, when searching for Baldwin, a civil rights activist and famous author, Walsh could only enter queries using his full name, thereby excluding posts that use his likeness in images or illustrations. 

Walsh, whose earlier research examined Baldwin’s circulation on social media, will lead the project’s literary studies research. She pointed to scholarship examining how Martin Luther King Jr., once viewed as radical, came to be celebrated across the political spectrum, his legacy “whitewashed.” She is interested in whether Baldwin’s portrayal has changed in comparable ways and hopes to explore references to other literary figures, including Mark Twain.

“We might not know in advance who is going to be the most influential in this government data,” Walsh said. “And the answers might surprise us.”

For Lee and Walsh, the grant creates an opportunity to bring together interests they have discussed for years. Their connections already extend to shared student meetings, brunches and bubble tea outings with their lab groups. Lee has also collaborated with Owens on digital preservation and government archives for nearly a decade, and with Deeds on GovScape. “It's the perfect collection for us to come together around and to work on,” Lee said.

All four investigators have technical abilities that will help shape the research tools, with Lee and Deeds leading the development of the dataset and AI models, and Walsh and Owens drawing on their disciplinary expertise to guide what those tools need to address. 

Walsh said that explaining the assumptions and methods of one field to collaborators in another can deepen everyone's understanding of the work. “You have to make explicit things that are implicit in your community,” she said. “But actually, the process of making those things explicit helps me too,” referring to the new perspectives she draws from such conversations.

Throughout the project, the team plans to evaluate their models against existing ones and develop tools applicable to other digital collections.

For humanities scholars, access to those resources could make it possible to connect individual examples with broader patterns across an archive too large to examine one document at a time.

“We're all excited to satisfy our curiosity, work with the large PDF set, start to use these AI models to be able to identify visual content and textual mentions, and start digging into the data,” Lee said.