"Multimodal analysis using TV data: new tools for the study of language and gesture"
by Dr Daniel Alcaraz-Carrión, University of Wisconsin-Madison
Abstract
In this talk, I will describe some of the data and methods offered by the Red Hen Lab. The first section will be devoted to the NewsScape archive, a television repository with over 400,000 hours of TV news recorded from 2004 until the present. This dataset allows researchers to look up specific linguistic expressions and to obtain all the instances in which they were uttered on TV. The NewsScape library gives access to massive amounts of multimodal data useful for big-data approaches to linguistics, political science and computer science, amongst many other disciplines. To illustrate this, I will present some of my research on temporal and numerical cognition, mixing corpus-based linguistic methods and large-scale gesture analysis.
The second section will focus on new tools for the analysis of visual data and multimodal communication. I will present Open Pose, an open-source Python package that automatically detects body key points, shifting gesture recognition from a manual to a machine-assisted annotation. Following that, I will introduce the Red Hen Anonymizer, a software capable of substituting facial features by using computer-generated images while maintaining facial gestures (e.g., lips, eyebrows). I will finish by introducing some of the tools that are currently being developed in Red Hen, such as a visual lexicon for Aztec hieroglyphs and the integration of PRAAT for the analysis of acoustic features.
Speaker’s bio
Dr Daniel Alcaraz-Carrión is a postdoctoral researcher at the department of psychology at University of Wisconsin-Madison. He obtained his PhD in Linguistics and English Language at Lancaster University in 2018. His research uses large multimodal databases to examine different aspects of multimodal communication, including language, gesture and other visual representations. He is particularly interested in how people communicate about highly abstract concepts such as time and number, and how language and gesture can vary cross-linguistically. He is a member and collaborator of several international research groups, including the Red Hen Lab, the Daedalus Lab and the Cognitive Development and Communication Lab.