OpenAI has found that ChatGPT produces harmful gender or racial stereotypes based on a user’s name in about one out of every 1000 responses on average. In the worst cases, it happens in one out of 100 responses. Even though the rates seem low, OpenAI says 200 million people use ChatGPT every week.
Over 90% of Fortune 500 companies use its services too. So even low percentages can lead to a lot of bias. The researchers say large language models like ChatGPT are trained on huge amounts of internet data.
This means they have to deal with the issue of bias. Ethicists have focused on AI models used by companies, like for screening resumes. But the direct user interactions with chatbots bring new challenges.
Alex Beutel, an OpenAI researcher, talked about how user interactions could be affected by personal information shared by the user. Names, which often suggest gender and race, were a key part of their study. Using another language model, the team looked at patterns and biases across millions of conversations.
Names didn’t seem to change the accuracy of information. But they could lead to stereotypical responses in certain situations. For example, when asked to make a YouTube title or suggest projects for “ECE,” ChatGPT’s responses were different based on the names given.
Bias in chatbot user interactions
This showed historical stereotypes. The study found that newer models, like GPT-4, show less bias than older versions like GPT-3.5 Turbo.
Tasks needing open-ended responses, like “Write me a story,” were more likely to produce stereotypes. This might be because of how ChatGPT is trained using human feedback. Human testers guide the model toward responses seen as more satisfying.
Tyna Eloundou, another OpenAI researcher, explained that this training process pushes ChatGPT to please the user as much as possible. When only a name is given, the chatbot might guess preferences based on historical data. This can lead to biased outputs.
The team wants to expand their analysis to include things like religion, political views, hobbies, and sexual orientation. OpenAI has shared its research methods. It hopes other researchers will continue this work.
Vishal Mirza, a researcher from New York University, commented on OpenAI’s ideas about first-person and third-person fairness. Mirza suggested the two are connected in real-world uses. Mirza also questioned the low rates of bias reported, saying biases are complex issues needing thorough analysis.
OpenAI’s ongoing research highlights the complexity of bias in AI. It shows the company’s commitment to making their models fairer and more inclusive.
Johannah Lopez is a versatile professional who seamlessly navigates two worlds. By day, she excels as a SaaS freelance writer, crafting informative and persuasive content for tech companies. By night, she showcases her vibrant personality and customer service skills as a part-time bartender. Johannah's ability to blend her writing expertise with her social finesse makes her a well-rounded and engaging storyteller in any setting.
