What Came Before and What Comes After XLNet

Senior Capstone Experience by Michelle Ly ’21

Submitted to the Department of Computer Science

Advised by Dr. Kyle Wilson

Description: This paper presents an overview of XLNet’s state-of-the-art system in the field of Natural Language Processing and specifically text generation. I look at previous language models Transformer-XL and BERT in order to understand the foundations that XLNet is built upon. Context is vital for the advancement of Natural Language Processing, not only in terms of text generation but understanding how these machine learning systems can improve. Overall, XLNet generates text that sounds human-passable, and it’s also able to build around a new prompt based on its pretrained knowledge. This collaboration between machines and humans on improving text generation is what natural language processing focuses on. XLNet, while on the cutting edge of machine learning, is already being surpassed by bigger models. However, it still deserves recognition for what it does bring to the field of natural language processing.

Read Michelle’s SCE below:

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s