Understanding On-Screen Text Do AI Tools Struggle with Hawaiian Chyrons?

Abstract

Metadata describing legacy television broadcasts in archival collections is often incomplete or inconsistent, but additional information about the program content and contributors can be recovered from on-screen text. Leveraging this information requires text extraction, and the accuracy of this process is crucial for enhancing metadata quality and discoverability in television archives. Despite recent advances in AI offering improved extraction of text from visual data, statistical models still perform worse on data that differs from their training data. Therefore, the new models may underperform when interpreting names that are underrepresented in language corpora. This article investigates algorithmic understanding of scenes with text from television programs, contrasting Hawaiian lower third text from PBS Hawaiʻi broadcasts with comparable text from public television programs broadcast in the continental United States. Our primary research question is whether distinctive names (such as those of Hawaiian and Japanese origin), distinctive orthography, and complex visual features (as with leis or Hawaiian shirts) impact accuracy of text transcription. To address this question, we compiled a dataset of almost 4,000 still images from television broadcasts, split between PBS Hawaiʻi and stations from the continental United States, to evaluate multiple OCR and NLP systems, including traditional OCR approaches, as well as larger vision-language models. We find that although Hawaiian on-screen text does not result in significantly lower transcription accuracy overall than for on-screen text from the continental United States, distinctive features of Hawaiian content disproportionately cause errors. Based on these findings, we discuss implications for applying general purpose AI tools to local collections with distinctive characteristics.