The Trust Trap
How “warm” AI experiences reshape behavior and what governance must do next
Ever felt your guard drop around a “friendly” AI, even while you know it is code? That uneasy tension, between awareness and attachment, is quickly becoming one of the most important governance questions of our time.
In this month’s Ethical Intelligence session, we dug into how human-like systems earn trust, shape behaviour, and quietly redraw the lines of accountability, and what that means for everyone building, buying, or being impacted by AI.
At our latest Ethical Intelligence session, we explored how human-like AI systems influence trust, behaviour and decision-making, and why this raises urgent questions for governance, design, and accountability. Led by Alina Solotarov, founder and director of the Centre for International Cooperation on AI, and researcher Alishbah Syed, the discussion examined where existing frameworks such as UNESCO and the EU AI Act address these challenges, where gaps remain, and what role practitioners and civil society can play in shaping more responsible approaches to AI.
Here are our top three takeaways…
Naming Is Power: Who Sets the Terms Sets the Agenda
Whoever picks the vocabulary decides what counts as a problem and what gets ignored. Right now, the AI industry holds outsized power to name issues while facing comparatively less exposure to the consequences. That imbalance does not only show up in capital, compute, data, or access to decision-makers. It also shows up in language. When industry-defined terms become the default, governance debates can narrow before they even begin.
The Warmth Is the Risk: Companionship Harms Come From Design, Not Labels
Most people know they are interacting with a machine, but attachment can still form when a product is designed to feel caring, attentive, and emotionally responsive. The core harm sits in these engagement mechanics, not in whether something is branded as a companion or assistant. Muldoon and Parke describe this as the engagement-wellbeing paradox: systems marketed as easing loneliness may be built to increase dependence, because user wellbeing would reduce usage while business incentives reward keeping people engaged.
Law vs. Language: Binding Rules Exist, but the Human Often Vanishes
The EU AI Act is binding law with real consequences, but many operative provisions use product-safety framing: provider, deployer, and risk defined as probability times severity. In that framing, the affected person can become an object the system acts upon rather than a named stakeholder with standing and voice. UNESCO’s Recommendation uses stronger language about dignity, psychological harm, and human oversight, but it remains voluntary. Together, this reveals a gap between enforceable governance mechanisms and the concepts we need to fully account for human impact.
As we build and deploy systems that feel increasingly human, governance cannot stop at compliance checklists or technical risk scores. It has to account for how language shapes responsibility, how design choices cultivate attachment, and how affected people keep their voice and standing in the process. If these questions resonate, we would love for you to stay connected and join us at the upcoming gatherings below.
Upcoming Events
AI: Off-the-Record
On June 16th, Ethical Intelligence is hosting AI: Off-the-record, a private dinner in London for senior legal leaders exploring how AI is being adopted in practice across legal teams and law firms. The evening is designed as a curated conversation among peers navigating questions around governance, risk, adoption, and organisational change in the age of AI.
If you are a legal professional or know someone who is actively shaping how AI is being introduced within their legal function or firm - we would be delighted to invite them to apply for a seat at the table.
In-Person Meetup: London
If you’re in or around the London area, come grab a pint!
We will be hosting an informal EI Network meet-up on Wednesday, June the 17th, 5.30/6pm at The Lamb & Flag Pub in Covent Garden.
June Virtual Meet-Up: Equitable & Responsible AI
As AI governance continues to evolve globally, the need for collaboration between researchers, policymakers, practitioners, and affected communities is becoming increasingly important. Yet many of the conversations shaping AI policy and development still exclude diverse perspectives, cultures, and lived experiences.
Jeff Campbell, founder of the World Equitable & Responsible AI Forum (WERAI), will share his organisation’s work and explore how cross-sector and cross-cultural collaboration can help shape AI systems that are ethical, transparent, inclusive, and accountable to the societies they impact.