Lunch at 12:30pm, talk at 1pm, in 148 Fitzpatrick

Title: Is the Retrieval-augmented Language Model still needed in the GPT-3 era?

Abstract: When interacting with each other and with the world, humans tap into many different forms of world knowledge (e.g., commonsense, world facts, trending news). To incorporate this capability into AI applications, information retrieval provides models with access to a large collections of documents that can contain such knowledge. Therefore, retrieval-augmented language models have long been considered as a key component in the development of general-purpose language models, with the advantages of being smaller, simpler, and more efficient. Recent, large language models such as GPT-3 have demonstrated impressive performance on knowledge-intensive tasks, mainly due to their powerful capabilities of memorizing and deducting world knowledge. Meanwhile, large language models can generate contexts that are more specific to the input query than retrieved information (since the documents for retrieval are always fixed). Therefore, the necessity of retrieval-augmented language models has become an important topic in the GPT-3 era. In this talk, I will first give a research overview by comparing retrieval-augmented language models with large language models. Then, I will share some of my recent work, including training general retrieval-augmented language model framework, improving factuality of retrieval-augmented language model with knowledge graph and replacing retrieval with large language model generators.

Bio: Wenhao Yu is a Ph.D. candidate in the Department of Computer Science and Engineering at the University of Notre Dame. His research lies in language model + knowledge for solving knowledge-intensive applications, such as open-domain question answering and commonsense reasoning. He has published over 15 conference papers and presented 3 tutorials in machine learning and natural language processing conferences, including ICLR, ICML, ACL and EMNLP. He was the recipient of Bloomberg Ph.D. Fellowship in 2022 and won the best paper award at SoCal NLP in 2022. He was a research intern in Microsoft Research and Allen Institute for AI.