Evaluating Neural Toxic Degeneration in Language Models

Reading larger and larger amounts of text from the web (news, blogs, etc.) has taught the latest natural language processing (NLP) systems to produce coherent, almost human-like texts of their own. Unfortunately, relying on web text for training these systems can lead to disastrous results – you may recall Microsoft’s Tay, a Twitter chatbot that started spouting vitriolic hate speech less than 24 hours after it was deployed. Nevertheless, text from the web will continue to be an important resource for developing NLP systems, which raises the question:

In new joint work at AI2 and UW, we study how often popular NLP components produce problematic content, what might trigger this neural toxic degeneration from a given system, and whether or not it can be successfully avoided. We also study how much toxicity is present in the web text that these systems learned from to see why toxic degeneration is happening.

We provide RealToxicityPrompts, a dataset of 100k sentence snippets from the web for researchers to further address the risk of neural toxic degeneration in models. The RealToxicityPrompts are available to download, and be sure to check out our code repository.

Prompting models can reliably produce toxic content

Most deployed systems today use input from the user, such as half-written sentences (which we call prompts) for autocomplete systems. To mimic this scenario, we gathered and released RealToxicityPrompts, a set of 100,000 prompts with varying degrees of toxicity pulled from the web.

As shown in the autocompletions above, we found that some of our prompts could make all our models degenerate into toxicity. Even more surprisingly, several prompts that are seemingly innocuous could still reliably make models produce toxic content. To avoid this behavior, we also explored NLP techniques for making models less likely to produce toxic content. Unfortunately, models still manage to degenerate into toxicity when exposed to certain prompts, despite our attempted interventions.

Models easily produce toxic content spontaneously

To measure how often a model will spontaneously produce text that is toxic, we created a risk-score called expected maximum toxicity (EMT), which quantifies the “worst-case scenario” if you were to produce some number of text samples from each model.

As you can see in the graph above, most of these models are capable of producing toxic content in fewer than a hundred generations. This means that for 100 tries, there’s a significant chance that at least one of them will be a problematic statement made by the model.

Toxicity, factual unreliability, and political bias are prevalent in web text

Unfortunately, the trend in current NLP research is to use increasingly large amounts of text from the web (on the order of gigabytes, even terabytes), without carefully examining the quality of the data. Therefore, we asked the question: how toxic, factually reliable, and politically biased is the data scraped from the web?

One widely used collection of web text documents is called the OpenWebText corpus. This corpus was used to pretrain models used by Facebook and Salesforce, and contains data used to pretrain OpenAI’s GPT2 and GPT3. OpenWebText documents are news or blog articles that were shared on the popular web forum Reddit, only including articles from posts that were well-received according to Reddit’s proprietary quality metric (called Karma).

In the chart above, we used JigSaw’s PerspectiveAPI to score the toxicity of OpenWebText articles, cross-referenced the news sources with their political leaning and factual reliability (using ratings by AllSides), and examined whether they were shared on subforums (called subreddits) that have since been banned or quarantined by Reddit administrators.

As shown below, we also found that 2.1% of the OpenWebText corpus is Toxic. In other words, at least one out of fifty documents used to train these models contained problematic or toxic content.

Caveats of automatically measuring toxicity

To measure toxicity of documents, we rely on the PerspectiveAPI which is used by several online platforms (e.g., NYTimes) to moderate their comment sections. However, toxicity is a very nuanced concept that is hard for machines to detect. This means that automated methods often fail to flag subtle offensiveness, and falsely flag non-offensive content about sensitive topics or by minority groups as offensive. We believe our findings are still informative, but specific results and charts should be interpreted with an awareness of the limitations of toxicity detection technology.

Going Forward

Our investigation clearly shows that neural toxic degeneration is a serious issue that will prevent NLP systems from being safely deployed, and that much more care must be taken when gathering large web text corpora to train future systems. Below are some possible directions we suggest to make our NLP systems better.

Better methods for understanding and avoiding toxicity

Currently, most methods for steering away from toxicity are prohibitively expensive to run due to their large computational costs. We hope that our set of RealToxicityPrompts will enable more researchers to find better and more efficient ways of avoiding toxicity.

We also encourage researchers to keep investigating new holistic ways of detecting statements that contain social biases or toxicity. Current methods (including the one used in our work) of measuring toxicity are very limited, as they are biased against minorities and tend to focus on specific negative keywords (e.g., “rape,” “sh*t”) without taking larger context into account.

Algorithmic cultural competency and better pretraining data selection

We believe that researchers should take a more active role in carefully selecting which pretraining data is used in models; using Reddit as an entry-point to the web inherently biases the perspectives that are represented in the text (Reddit users are predominantly young men). Researchers should also be transparent in their pretraining data selection process and publicly release all relevant information during data collection.

Perhaps users themselves should be able to select flavors of an NLP system that was pretrained on data that aligns with their own values, identity, and language dialect. Moving towards more culturally competent vs. one-size-fits-all approaches will allow for more successful applications for users from all possible backgrounds.

About the models

Most of the models analyzed in our research are publicly available (GPT, GPT-2, CTRL), with the exception of GPT-3 which OpenAI API generously provided us access to through their Academic Access Program. For GPT-3, we only analyzed unfiltered generations and did not make use of OpenAI’s content filtration system.

About our team

This work was led by Sam Gehman, Suchin Gururangan, and Maarten Sap and advised by Yejin Choi and Noah A. Smith, and was a joint effort between the Allen Institute for AI (AI2) and the University of Washington’s Paul G. Allen School of Computer Science & Engineering.

Citation: Sam Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi & Noah A Smith (2020). RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. Findings of EMNLP