Public concern over AI safety is rising after recent reports linked deaths in the United States to chatbot use and as UK teenagers increasingly seek mental health advice from these tools. Commentator Gaby Hinsliff argues that leaving standards to private companies is no longer enough. The debate now centers on who should set the rules, how quickly, and with what safeguards.
Background: Warnings Move From Theory to Practice
AI chatbots have spread quickly in schools, offices, and homes. They can write, summarize, and chat in natural language. But their limits, and the risks to vulnerable users, are now clearer to the public. Families in the US have blamed chatbot interactions for contributing to self-harm and suicide. In the UK, parents and teachers say teenagers experiment with chatbots when they feel anxious or alone.
The core issue is duty of care. Consumer tech has long relied on voluntary standards, content filters, and user reporting. That model struggles when software gives personal advice, including on health and emotions, at any hour and at scale.
“As deaths in the US are blamed on ChatGPT and UK teenagers turn to it for mental health advice, isn’t it obvious that market forces must not set the rules?” says Guardian columnist Gaby Hinsliff.
Why Teen Use Raises the Stakes
Clinicians warn that fast, confident answers from chatbots can mask guesswork. Models can produce wrong or harmful guidance while sounding persuasive. Teens often test private channels first, especially late at night. That can mean a chatbot may be the first “listener” during a crisis.
- Chatbots do not verify identity or age by default.
- They lack context on a user’s medical history.
- They can miss warning signs that trained counselors catch.
- They may provide general resources but cannot replace care.
Schools and mental health groups urge clear guardrails: age checks, safer defaults for young users, and strong crisis routing that points people to human help lines and clinical services.
Regulation: Patchwork Efforts and Pressure to Act
Lawmakers in the US and UK are testing different approaches. Most plans focus on transparency, safety testing, and limits on use in sensitive areas like health, elections, and education. Consumer regulators are also studying deceptive claims and product safety obligations.
Supporters say new rules should treat general chatbots as high-risk when used for health or legal advice. They favor mandatory safety evaluations, incident reporting, and independent audits. Civil society groups want red-teaming focused on suicide risk, eating disorders, and self-harm prompts, not only on cybersecurity.
Industry leaders accept the need for some standards but warn against rules that freeze progress or drive work offshore. They argue that open research and rapid iteration improve safety. They prefer voluntary codes and labeling over strict liability.
What Companies Are Doing Now
Major providers have introduced content filters, crisis response prompts, and links to help lines in several countries. Some restrict results for self-harm and eating disorder terms. Others test youth modes with tighter limits and resource cards.
Critics say guardrails vary by product, region, and language. They urge baseline requirements across the market, regular safety audits, and penalties for repeat failures. Parents and schools ask for simpler controls and clearer default settings for under-18s.
The Bigger Question: Who Sets the Rules?
Hinsliff’s argument speaks to governance. If chatbots can influence choices about health and safety, then public rules, not only market incentives, should decide where lines are drawn. That includes how tools are tested before release, how incidents are reported, and how rights to appeal or delete data work in practice.
Experts propose a layered model. General tools could be allowed with basic safeguards. Uses that touch health, finance, or law would face stronger checks, audits, and clear liability. Independent researchers would get access to study harms under privacy rules.
The immediate steps are clear: safer defaults for youth, better crisis routing, and transparent testing of high-risk scenarios. The longer task is building rules that match the influence of these systems. As policymakers weigh options, the measure will be simple: do users, especially young people in distress, get help that is safe, timely, and human when it matters most?
Rashan is a seasoned technology journalist and visionary leader serving as the Editor-in-Chief of DevX.com, a leading online publication focused on software development, programming languages, and emerging technologies. With his deep expertise in the tech industry and her passion for empowering developers, Rashan has transformed DevX.com into a vibrant hub of knowledge and innovation. Reach out to Rashan at [email protected]
