Lunch at 12:30pm, talk at 1pm, in 148 Fitzpatrick

Title: Item Response Theory (IRT)

Abstract: In many cases, improvements in NLP tasks are measured based on aggregate performance results on large-scale datasets. However, these metrics do not account for differences between examples such as difficulty and discriminative ability. As the field of NLP embraces leaderboards to track progress on challenge tasks, fine-grained evaluation of models and examples is needed. In this presentation, I will describe one type of fine-grained analysis which leverages Item Response Theory (IRT) for learning latent parameters of examples and models. I will present an overview of IRT and describe examples of its use in NLP. I will conclude with an introduction to the py-irt Python package for learning IRT models on NLP datasets.

Bio: Dr. Lalor is an Assistant Professor at the Mendoza College of Business at the University of Notre Dame. He completed his Ph.D. at UMass Amherst in the College of Information and Computer Science. At UMass he was a member of the Bio-NLP group, working with Dr. Hong Yu. His research interests are in machine learning and natural language processing. He is particularly interested in model evaluation and quantifying uncertainty, as well as applications in biomedical informatics.