Conference Agenda

Building on our experience with materials from the British Library’s Endangered Archives Programme and 19th-century court records from the Circuit Court of Istmina, Chocó, Colombia, which are damaged and in different manuscript and typescript formats, we will teach participants how to utilize AI to generate research data from document images. Using vision-language models (VLMs), we will demonstrate how to extract text and other metadata from images and publish this data in accessible formats for research. This transformative process makes it possible to search and analyze collections of digitized documents in ways that facilitate exploratory analysis and computational research methods. Machine-readable text makes documents more accessible to researchers, students, and the public. Additionally, auto-cataloging of the collection provides standardized metadata, such as the occurrences of person names, place names, and case summaries, which facilitates research across the entire collection. VLMs are particularly useful for handwritten text recognition and the description of materials at the case, box, and collection levels.

Our workshop seeks to address three key outcomes:

• Provide a flexible and customizable workflow for researchers to transform document images into text and metadata.

• Demonstrate various research output formats with the extracted data, including a searchable static website, a spreadsheet, and a Word document with images and text side-by-side.

• Participants should end the session with a conceptual understanding of the methods and technologies involved, working code examples, and experience processing a sample collection. These materials can then be adapted for participants' projects.

We will address common problems facing researchers with optical text recognition (OCR) and handwritten text recognition (HTR). The most common problem is finding a model that is “good enough.” Most out-of-the-box general models do not provide sufficiently accurate results for researchers’ needs. These models work well for modern languages but do poorly with historical documents or languages with non-Latin scripts. Platforms like Transkribus and eScriptorium provide essential tools for annotating such text and creating models for specific scripts, periods, and languages.

However, our team’s experience with eScriptorium resulted in some frustration–too often, generated text was spotty or so inaccurate as to be useless. The segmentation is inconsistent across different types of documents in the collection. Any text missed by the segmenter is not processed in the recognition stage. In response, we shifted to transformer-based vision-language models (VLMs), which provide significantly improved handwritten text recognition (HTR) and extraction of complex metadata for our nineteenth-century typescript and manuscript legal court documents from Colombia. Rather than working with whole images, we split the document into chunks and use smaller, fine-tuned models for specific tasks. For example, we can generate legal case summaries and identify names and places in the context of the case. We identify not only a person’s name, but also that they were the defendant or judge. A place is not just a name, but the place where the crime occurred or the court's jurisdiction. The capabilities of VLMs allow us to create automatic workflows that not only extract text for research but also offer the metadata essential for creating archival finding aids.

Timeline (2 hours total)

First session (45 minutes)

• A brief introduction to the EAP and records of the Circuit Court of Istmina, Chocó (EAP1477).

• Discussion of archival collections as research data. We will detail how datafication workflows can build on existing digitization work to meet researchers’ needs.

• What are vision-language models? We will provide context on recent developments in HTR/ATR and share examples of applied machine learning tools for document processing.

• We will demonstrate the process using documents from the Istmina Circuit Court Archive, using our Fichero (catalog) software.

Break (10 minutes)

Second Session (45 minutes)

• A hands-on workshop to process a sample of materials from the Endangered Archives Programme. We encourage participants to evaluate how models perform on materials from different time periods, languages, and domains.

• Participants will compare the outputs of traditional OCR engines like Tesseract and Kraken against generative VLMs. We will discuss the problems and advantages of this new architecture for use in the computational humanities.

Resources

Workshop participants will need a laptop with an internet connection. All code notebooks will be available in a GitHub repository and can be run in the browser using Codespaces or Google Colab. These systems require a Google or Microsoft user account. Participants are welcome to download and run the code locally if they prefer.

We will use the Qwen2-VL-2B-Instruct model available on HuggingFace for the workshop. This relatively small model can be downloaded and run on consumer computers. We will also provide an API key to access Qwen models on Alibaba Cloud. Our process is model and provider-agnostic, so participants can use the models and resources they prefer going forward.

Preparation

Participants should know basic Python syntax and be familiar with Jupyter Notebooks, JSON, and YAML. We will provide links to instructional material on these topics for preparation.

Additional information about instructors

Andrew Janco has taught several previous workshop series, including “New Languages for NLP: Building Diversity in the Digital Humanities” at Princeton’s Center for Digital Humanities. He provides technical expertise to Professors Farnsworth-Alvear and Tubb on their projects related to gold and platinum mining in the Chocó region of Colombia. This work includes the fichero catalog system and software.

Ann Farnsworth-Alvear is a co-editor of The Colombia Reader and author of Dulcinea in the Factory. She teaches Latin American History at the University of Pennsylvania. Her current work explores the extraction of platinum and gold from the San Juan River and its tributaries in 1870-1970; it depends on collaborations with other researchers–especially Daniel Varela of the University of Michigan and the Semillero de Jóvenes del Centro de Memoria Muntú Bantú, in Quibdó, Colombia.

Daniel Tubb is an anthropologist working on gold mining, agrarian change, and rural life in Colombia. He is the author of Shifting Livelihoods: Gold Mining and Subsistence in the Chocó, Colombia, and he is working with Ann Farnsworth-Alvear and Andrew Janco on projects exploring extraction and memory in Colombia’s San Juan and Atrato River regions. Their work includes the development of fichero—a digital tool for cataloguing and transcribing historical documents related to mining and land struggles.

Kelly López-Roldán is a doctoral student at the University of Pennsylvania. She has participated in various digitization projects involving materials in Colombia, including EAP1477, EAP1531, and EAP1740. Her experience with public history projects focused on memory and engagement has included publishing collectively-authored books that showcase community-level research with primary sources.