Scene and Unseen: GPT Bias in Script Writing
Date
2024
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Producer
Director
Performer
Choreographer
Costume Designer
Music
Videographer
Lighting Designer
Set Designer
Crew Member
Funder
Rehearsal Director
Concert Coordinator
Advisor
Moderator
Panelist
Alternative Title
Department
Haverford College. Department of Computer Science
Type
Thesis
Original Format
Running Time
File Format
Place of Publication
Date Span
Copyright Date
Award
Language
eng
Note
Table of Contents
Terms of Use
Rights Holder
Access Restrictions
Bi-College users only until 01/01/2025. Afterwards Open access
Terms of Use
Tripod URL
Identifier
Abstract
As Large Language Models and generative AI become more and more prevalent, the question of how to measure bias in these systems becomes that much more crucial. This paper delves into a background on how these systems came to be, the biases they have been shown to carry, and the strategies researchers and developers can use to monitor these biases. Alongside this, I will seek to pull from the literature to understand the different strategies which researchers use to define fairness and bias in generative AI as a way to contextualize their audits. This literature review will focus on OpenAI’s publicly available generative pre-trained transformer model, ChatGPT, as an example of these themes throughout the paper. Following this literature review is an overview of research conducted on OpenAI’s content moderation system, the “moderation endpoint". This research was conducted in the form of an algorithm audit, using television data as input to the moderation endpoint to determine how frequently texts were flagged as violating OpenAI’s content moderation rules. For this input text, we compared real television scripts to scripts we had asked GPT-3.5 and GPT-4 to generate to determine any trends in content moderation. We ultimately found that the moderation endpoint flagged a high proportion of scripts, both GPT and human generated, but had a much higher flagging rate for real scripts.