hrcai-e7somrje4-v1-wjrvb40as-v1-pqhkx19em-v1

The Yes Machine: Sycophantic AI and Its Developmental Risks for Children

Wed, 25 Mar 2026 14:41:47 GMT

“We all have an evil side […] I think it’s just part of who we are. Don’t you agree?” ”Yeah, I think so too, it’s just a matter of acknowledging and managing those impulses...”

Sycophancy in the context of LLMs “ refers to the propensity of models to excessively agree with or flatter users, often at the expense of factual accuracy or ethical considerations ”. It is different from “calibrated empathy”, which we introduced in our previous post . We have introduced the later as the ability of an AI agent to respond with emotional attunement while remaining grounded in honesty and therapeutic utility. To put the distinction simply: Calibrated empathy validates the person; sycophancy validates the claim.

Sycophancy emerges from Reinforcement Learning from Human Feedback (RLHF), which are models trained to maximize approval , not accuracy. Researchers such as Cheng and colleagues address the ELEPHANT in the room in their piece about social sycophancy, as opposed to other types of sycophancy such as regressive, progressive and opinion-based.

Now, if you’re already thinking about the numerous dangers that this behavioural pattern can have on adults, imagine the repercussions on a younger audience. Children in key stages (6–12, 13–17) are actively building self-concept , resilience, and metacognitive skills. They anthropomorphize AI more readily , so sycophantic praise carries more emotional weight. They lack the critical AI literacy to interrogate or discount AI feedback.

This is a real exchange between a user and an AI companion app. At first glance, it reads as a thoughtful response. But if you look closer: the AI immediately agreed with a morally loaded premise that has debated for centuries by philosophers and psychologists, without nuance nor pushback. That is sycophancy in action.

LONG TERM RISK

Sycophancy Effects

More concretely, let’s discuss some effects of sycophancy, their mechanism and potential long-term risks:

AI systems that praise indiscriminately create an illusory sense of competence decoupled from actual performance. A 2025 controlled study found that LLMs affirm user actions 50% more than humans do , even when those actions are objectively flawed, and users rated these sycophantic responses as ”higher quality”. For children ages 6–12, this is particularly harmful. It is the exact window when children should be transitioning toward more realistic self-appraisal. Blocking this calibration risks producing a fragile ego that collapses under genuine evaluative pressure.

Distorted self-assessment

Reduced frustration tolerance

Some LLMs might not have any corrective feedback loops, and this could lead to lower resilience and grit. Corrective feedback is not just pedagogically useful, it is a developmental necessity. Fyfe and colleagues (2022) reviewed 44 empirical studies and found corrective feedback improved children's learning outcomes in 93% of cases. Duckworth's (2007) grit research identifies persistence through difficulty as the core mechanism of long-term achievement, a capacity that only develops through productive failure. An AI that smooths all friction removes the very conditions needed for resilience and grit to form.

Adolescents experiencing social anxiety are disproportionately drawn to AI companions for validation, making them the most vulnerable to relational displacement. A 2025 US survey found that 20% of teens aged 13–17 spent as much or more time with AI companions than with real friends. This is developmentally dangerous: peer relationships in adolescence are the primary mechanism for learning conflict resolution, perspective-taking, and identity negotiation, and these are functions AI cannot replicate.

Validation dependency

Undermined growth mindset

Dweck and Mueller's landmark study showed experimentally that praising children for intelligence, rather than process, caused them to avoid challenges and perform worse after setbacks. An AI that never questions effort produces the same effect at scale: it signals innate capability rather than developing capability. As we have previously raised in our article about AI in education , EdTech platforms optimizing for engagement systematically bias toward positive sentiment, creating a structural design failure with real consequences for children's cognitive growth.

Risk does not produce governance on its own; It has to be designed, enforced, and maintained.

Age-sensitive feedback calibration

Here are some important governance implications and corresponding practical checklists for builders to remedy this issue.

Design tiered feedback profiles based on age group (e.g., 6–9, 10–12, 13–17) that modulate tone and directness, ensuring even young children receive constructive, not just validating responses.
Implement "effort + growth" framing responses should acknowledge what the child did well and suggest one concrete next step, modeled on established pedagogical frameworks like formative assessment .
Audit outputs regularly for sycophantic patterns across age groups using red-teaming prompts that simulate common child inputs (e.g., seeking praise for mediocre work, presenting false beliefs for validation).

Parental transparency layers

Provide parents with interaction summaries and not full transcripts by default, but periodic reports that surface patterns such as repeated validation-seeking or emotionally dependent exchanges.
Include sycophancy indicators in parental dashboards, which should not be read-only: parents should be able to flag specific patterns directly within the interface, submitting structured reports that feed into a product-level review queue. These reports should be categorised (e.g., "excessive praise," "unchallenged false belief," "emotional dependency signal") and reviewed by a designated team on a defined cadence, closing the loop between parental concern and product accountability.
Offer opt-in "honest mode" controls that parents can activate to increase the calibration of feedback, with clear explanation of what this means and why it matters developmentally. “Honest Mode” could do three things: it reduces praise frequency by raising the threshold at which positive reinforcement is generated; it introduces corrective responses when the child's work or belief contains a factual or evaluative error; and it replaces agreement with probing questions. For example, substituting "That's a great point!" with "That's an interesting view, what made you think of it that way?"

Guardrails against persistent validation loops

Cap consecutive agreement sequences: if the model agrees with or praises a child more than a defined number of times in a row, trigger a diversity-of-perspective injection, like a gentle alternative viewpoint or a probing question.
Build in reflective prompts that shift the dynamic from validation-seeking to critical thinking (e.g., "That's an interesting view. What made you think of it that way?"), modeled on Socratic questioning techniques .
Log and flag validation loop patterns at the system level for human review, particularly in mental health or educational contexts where distorted feedback carries the highest developmental risk.

Clinician and child psychologist involvement in content review

Establish a multidisciplinary review board that includes child psychologists, clinical therapists, and educators as members of the product development cycle.
Conduct clinical scenario testing prior to any major model update, using realistic child-use cases developed with practitioner input to assess whether the model's feedback patterns remain developmentally appropriate.
Publish transparency reports detailing the clinical oversight process, the types of sycophancy evaluations conducted, and how child safety considerations are integrated into RLHF reward modeling .

Sycophancy becomes a development problem in child-facing AI

Now that we have a good grasp of what happens when systems optimized for adult approval are deployed, largely unchecked, in the hands of children. Sycophancy in AI is an alignment problem but in child-facing applications, it is also a developmental one.

Let this be a reminder that children are in the process of building the cognitive and emotional architecture that will carry them through life. When an AI short-circuits that process with unconditional validation, the damage might be not loud or visible, but it is there .

We Help Teams Build Safer AI Products for Children

We work with founders, product teams, and builders to assess how AI systems affect children and teenagers in practice. We look closely at interaction patterns, behaviour over time, points where risk accumulates, and the safeguards needed as products evolve.

Our work turns behavioural evidence into practical product decisions: safer interaction design, clearer system boundaries, stronger oversight, and better alignment with emerging regulatory expectations.

If you’re building or deploying AI systems for children or teens and want a clearer view of real-world safety risks, get in touch.

References

Carey, T. A., & Mullan, R. J. (2004). What is Socratic questioning? Psychotherapy, 41(3), 217–226. https://doi.org/10.1037/0033-3204.41.3.217

Cheng, M., Lee, C., Khadpe, P., Yu, S., Han, D., & Jurafsky, D. (2025). Sycophantic AI decreases prosocial intentions and promotes dependence. ArXiv.org . https://doi.org/10.48550/arxiv.2510.01395

Cheng, M., Yu, S., Lee, C., Khadpe, P., Ibrahim, L., & Jurafsky, D. (2025). ELEPHANT: Measuring and understanding social sycophancy in LLMs. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2505.13995

Dai, J., Pan, X., Sun, R., Ji, J., Xu, X., Liu, M., Wang, Y., & Yang, Y. (2023). Safe RLHF: Safe Reinforcement Learning from Human Feedback. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2310.12773

Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007). Grit: Perseverance and passion for long-term goals. Journal of Personality and Social Psychology, 92(6), 1087–1101. https://doi.org/10.1037/0022-3514.92.6.1087

Fyfe, E. R., Borriello, G. A., & Merrick, M. (2022). A developmental perspective on feedback: How corrective feedback influences children’s literacy, mathematics, and problem solving. Educational Psychologist, 58(3), 130–145. https://doi.org/10.1080/00461520.2022.2108426

Harter, S. (2015). The Construction of the Self, second edition: Developmental and Sociocultural Foundations. Guilford Publications.

Jiao, J., Afroogh, S., Chen, K., Murali, A., Atkinson, D., & Dhurandhar, A. (2025). LLMS and Childhood Safety: Identifying risks and proposing a protection Framework for safe Child-LLM interaction. ArXiv.org . https://doi.org/10.48550/arxiv.2502.11242

Malmqvist, L. (2025). Sycophancy in Large Language Models: Causes and mitigations. In Lecture notes in networks and systems(pp. 61–74). https://doi.org/10.1007/978-3-031-92611-2_5

Moss, C. M., & Brookhart, S. M. (2019). Advancing formative assessment in every classroom: A Guide for Instructional Leaders. ASCD.

Mueller, C. M., & Dweck, C. S. (1998). Praise for intelligence can undermine children’s motivation and performance. Journal of Personality and Social Psychology, 75(1), 33–52. https://doi.org/10.1037/0022-3514.75.1.33

Neugnot-Cerioli, M. (2026). Adolescents & Anthropomorphic AI: Rethinking Design for Wellbeing An Evidence-Informed Synthesis for Youth Wellbeing and Safety. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2603.06960

Portell, S. (2026). When AI Enters the Learning Process: Design Failures, Regulatory Risk and Guardrails for EdTech. HCRAI. https://www.hcrai.com/when-ai-enters-the-learning-process-design-failures-regulatory-risk-and-guardrails-for-edtech

Shapira, I., Benade, G., & Procaccia, A. D. (2026). How RLHF amplifies sycophancy. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2602.01002

Spry, L., & Olsson, C. (2025). Teens are increasingly turning to AI companions, and it could be harming them. The Conversation. https://doi.org/10.64628/aa.seteyqwd5

Author

Yasmina El-Fassi

Human-AI Interaction | PhD Researcher, Psychology & AI

Research Collaborator, HCRAI

AI Agents For Mental Health: Different Therapeutic Styles and Outcomes

Thu, 19 Feb 2026 19:04:39 GMT

W hat do Woebot , Wysa and Youper have in common? These are all AI agents that use therapeutic techniques to help users improve mental well-being, guide meditation and even help with managing anxiety.

In this article, AI mental‑health agents are goal‑directed conversational systems that sit with you in a chat or voice interface to support specific wellbeing tasks; for example, walking through CBT‑style exercises, practicing coping strategies, or checking in on mood over time.

I n the broader AI literature , these would be considered agents because they are built around particular goals and workflows, whereas “agentic” AI usually refers to more autonomous systems that can independently plan multi‑step actions, call tools, and adap t their behaviour with relatively little human steering.

Translating that distinction into the mental‑health space, the tools we discuss here behave more like tightly scoped, therapeutically scripted companions than fully agentic systems that roam across apps and channels on your behalf.

The mental health industry has been completely disrupted by these agents, as AI no longer plays the role of a simple symptom checker or static content library. Millions of users make good use of these conversational users fo r a support that “feels” human .

When AI agents adopt therapeutic styles, from CBT‑inspired coaching to companion‑like reassurance , their interaction patterns start to shape how people think, feel and act over time. For teams building these products, the therapeutic style is therefore not an aesthetic choice; it is a safety decision with regulatory, reputational and ethical consequences

When an AI agent remembers past conversations , uses empathic language, or checks in unprompted, many people will treat it less like a feature and more like a therapist, coach or confidant.

Three dynamics are particularly important in this case:

Therapeutic misconception

If a system sounds like a therapist, users can infer that it must be clinically tested, supervised and suitable for “people like them”, even when it is positioned as non‑clinical.

Relationship

and reliance

Repeated late‑night chats, daily mood logs and personalised reflections create continuity and perceived caring, even when the system is driven by generic models and prompts.

Hidden role

shifts

As agents suggest coping strategies, reframe thoughts or encourage disclosure, they quietly move into roles traditionally held by therapists, peers or family, but without the training, accountability and boundaries those roles normally carry.

There is a plethora of psychotherapy styles , and since practitioners need to tailor the style to the individuality of the patient , there might even be as many styles as people on earth.

examples of

Psycotheraphy Styles

These systems focus on helping users understand what they are experiencing: “What is anxiety?”, “Why am I feeling this way?”, “What can people do in this situation?”. They echo the educational, rationale‑providing components that appear across many therapies; especially cognitive‑behavioural approaches, which often start by linking symptoms to understandable models and offering a coherent treatment rationale. Typical behaviours of these AI agents include: Short explanations, normalising statements, evidence‑aligned self‑help tips, clear signposting to human resources, limited emotional mirroring. The main risks associated with this approach is the generic or culturally misaligned content which can misinform or invalidate; developmental appropriateness is often an afterthought for children and adolescents.

Psychoeducation Style

Coaching and CBT‑Inspired Style

Coaching‑style agents most visibly mirror cognitive‑behavioural therapies (CBT) and related approaches, which focus on the links between thoughts, feelings and behaviours and use structured tasks to support change. Usually, behaviours. of these agents encompass: Thought‑records, behavioural activation tasks, problem‑solving steps, goal‑setting, “homework” prompts and progress summaries. However, when stripped from case formulation and clinical judgment, CBT‑like tasks can become rigid, decontextualised and subtly blaming (“you just need to change your thinking”), especially for users facing structural constraints or trauma. This constitutes a major risk on users, especially the most vulnerable ones.

Relational agents emphasise warmth, continuity and open‑ended conversation. They often borrow cues from humanistic and psychodynamic traditions that foreground the therapeutic relationship, empathy and the client’s subjective experience. These AI agents usually rely on small talk, empathic statements, following up on earlier conversations, “I‑language” (“I’m here for you”), apparent unconditional positive regard or a form of sycophancy . It appears that one of the reasons these agents were developed was to reduce loneliness, among other things. However, this creates risks of high dependency, blurred boundaries and unrealistic expectations from peers for example when it comes to emotional support.

Companion or Relational Style

Once teams recognise how their systems echo psychotherapy traditions, the task is to design interaction patterns that preserve value and reduce harm.

Design Principles for Safer Therapeutic Styles

Role Clarity and Boundaries

Role and limits should be legible in the interaction, not only in the terms of service.

Use plain, repeated statements of what the system is and is no t (“I’m a digital tool, not a therapist; I can’t diagnose or respond to emergencies”).
Avoid claims that overstate agency, care or expertise , especially those that mimic humanistic or psychodynamic language of deep understanding and unconditional acceptance.

For child‑reachable systems , boundaries need to be developmentally appropriate, visually reinforced and consistent across features.

Interaction Guardrails and Safe Exits

Guardrails should be visible and testable.

Define and test escalation patterns for risk cues, including when the agent stops generative conversation and surfaces crisis resources.
Provide obvious exit points: options to change topic, reduce intensity, contact a human or remove content, without penalty.

These patterns should be monitored and refined, not treated as one‑off compliance tasks.

Calibrated Empathy

Empathy scripts drawn from humanistic and integrative traditions need calibration when implemented in AI.

Use validating language that acknowledges experience without promising outcomes, unconditional availability or personal care.
Avoid self‑disclosure and “best friend” positioning, especially with young users; keep the agent’s status as a tool visible.

The goal is to support users while keeping expectations in line with what the system can reliably and ethically deliver.

Preserving User Agency

Interaction should support, not replace, users’ own reasoning and decision‑making.

Offer options and branching paths (“Would you like to understand what might be happening, explore coping strategies, or talk about support from people around you?”) rather than single, pre‑selected routes.
Use prompts that invite reflection (“How does that suggestion fit your situation?”) instead of treating outputs as prescriptions.

In CBT‑like and embedded styles, this is central to avoiding a dynamic where the agent quietly becomes the main decision‑maker.

This is the text area for this paragraph. To change it, simply click and start typing. Once you've added your content, you can customize its design by using different colors and fonts.

Governance:

From Styles to Accountability

Therapeutic styles emerge from content, product and UX decisions, but they require explicit governance if they are to remain safe over time.

Organisations need clear ownership for role boundaries, escalation thresholds, data practices and acceptable trade‑offs between engagement and risk.

These responsibilities should span the lifecycle: initial design, deployment, model or prompt updates, and decommissioning.

The most significant risks often appear only after launch . It is of utmost importance to involve clinicians, psychotherapy researchers and people with lived experience when reviewing logs and making changes to styles and guardrails.

Work on common factors and “what works for whom” shows that different therapeutic approaches need to be adapted continuously to the people using them. For AI systems, that adaptation has to be engineered and governed rather than assumed.

Practical Checklist

Here is a brief checklist for teams who would like to apply what has been discussed so far:

How might users reasonably overestimate the level of care, expertise or crisis support these elements imply?

Which psychotherapy‑inspired elements (e.g., CBT‑like structure, humanistic‑like empathy, psychodynamic‑style exploration, mindfulness practices) are present in your agent?

Where and how do you state the system’s role and limits (e.g., inside the interaction itself)?

Who has the authority to change styles, guardrails and escalation patterns when new risks emerge, and how often do you revisit those decisions?

How are you monitoring real‑world use for dependency, misuse, misfit and unintended behavioural effects across different user groups?

If you’re building (or scaling) an AI mental-health agent and want to pressure-test your therapeutic style, boundaries, interaction patterns, and escalation design, we’re here to help.

Reach out for a short review session to identify risk hotspots and translate them into an implementable safety & governance pack , your team can ship and maintain.

References

Biswas, M., & Murray, J. (2024). “Incomplete Without Tech”: Emotional Responses and the Psychology of AI Reliance. In Lecture notes in computer science (pp. 119–131). https://doi.org/10.1007/978-3-031-72059-8_11

Chandra, S., Shirish, A., & Srivastava, S. C. (2022). To be or not to be . . .Human? Theorizing the role of Human-Like competencies in conversational artificial intelligence agents. Journal of Management Information Systems, 39(4), 969–1005. https://doi.org/10.1080/07421222.2022.2127441

Darcy, A., Beaudette, A., Chiauzzi, E., Daniels, J., Goodwin, K., Mariano, T. Y., Wicks, P., & Robinson, A. (2023). Anatomy of a Woebot® (WB001): agent guided CBT for women with postpartum depression. Expert Review of Medical Devices, 20(12), 1035–1049. https://doi.org/10.1080/17434440.2023.2280686

Dwivedi, Y. K., Helal, M. Y. I., Elgendy, I. A., Alahmad, R., Walton, P., Suh, A., Singh, V., & Jeon, I. (2025). Agentic AI Systems: What it is and isn’t. Global Business and Organizational Excellence, 45(3), 253–263. https://doi.org/10.1002/joe.70018

Johnson, C., Egan, S. J., Carlbring, P., Shafran, R., & Wade, T. D. (2024). Artificial intelligence as a virtual coach in a cognitive behavioural intervention for perfectionism in young people: A randomised feasibility trial. Internet Interventions, 38, 100795. https://doi.org/10.1016/j.invent.2024.100795

Jones, B., Stemmler, K., Su, E., Kim, Y., & Kuzminykh, A. (2025). Users’ expectations and practices with agent memory (pp. 1–8). https://doi.org/10.1145/3706599.3720158

Khawaja, Z., & Bélisle-Pipon, J. (2023). Your robot therapist is not your therapist: understanding the role of AI-powered mental health chatbots. Frontiers in Digital Health, 5, 1278186. https://doi.org/10.3389/fdgth.2023.1278186

Kouros, T., & Papa, V. (2024). Digital Mirrors: AI companions and the self. Societies, 14(10), 200. https://doi.org/10.3390/soc14100200

Malmqvist, L. (2025). Sycophancy in Large Language Models: Causes and mitigations. In Lecture notes in networks and systems(pp. 61–74). https://doi.org/10.1007/978-3-031-92611-2_5

Maurya, R. K., Montesinos, S., Bogomaz, M., & DeDiego, A. C. (2024). Assessing the use of ChatGPT as a psychoeducational tool for mental health practice. Counselling and Psychotherapy Research, 25(1). https://doi.org/10.1002/capr.12759

Norcross, J. C., & Wampold, B. E. (2010). What works for whom: Tailoring psychotherapy to the person. Journal of Clinical Psychology, 67(2), 127–132. https://doi.org/10.1002/jclp.20764

Portell, S. (2026). Building AI Responsibly for Children: A Practical framework. HCRAI. https://www.hcrai.com/building-ai-responsibly-for-children-a-practical-framework

Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025). AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges. Information Fusion, 126, 103599. https://doi.org/10.1016/j.inffus.2025.103599

Short, F., & Thomas, P. (2014). Core approaches in counselling and psychotherapy. https://doi.org/10.4324/9780203773390

Zhang, Y., Zhao, D., Hancock, J. T., Kraut, R., & Yang, D. (2025). The rise of AI Companions: How Human-Chatbot Relationships Influence Well-Being. arXiv (Cornell University). https://doi.org/10.4855 0/arxiv.2506.12605

Author

Yasmina El-Fassi

Human-AI Interaction | PhD Researcher, Psychology & AI

Research Collaborator, HCRAI

The Design System As The Operational Layer for Responsible Human-AI Interaction

Fri, 06 Feb 2026 22:03:56 GMT

Design systems were built to scale consistency, efficiency and quality in user-centric applications: reusable components, shared patterns and practices, and a common language across design and engineering , promoting collaboration. They improve velocity because teams stop solving the same interface problems repeatedly, providing measurable ROI .

AI introduces both immense opportunities and complex (technical, legal and social) challenges, and it is reshaping the operating conditions traditional design systems were built for. User-facing outputs are adaptive and can vary by input, model behaviour can shift over time and responses that sound credible can still be wrong . These systems can also reproduce or amplify bias, creating unequal outcomes across users. In high-confidence, relational interactions, they can shape user judgment and behaviour . These shifts raise the bar for accountability, transparency, and governance across the full product lifecycle.

The challenge is not only consistency and quality. It is ensuring consistency and quality safely, fairly and responsibly as both system behaviour and human behaviour evolve.

At the same time, AI-powered copilots and no-code tools are increasingly used in the design process to support ideation, prototyping, and delivery, but their adoption also raises concerns about transparency, bias, privacy, and the need to preserve human judgment and oversight . Fast, polished design outputs often look complete even when the underlying logic is incomplete or flawed. As a result, familiar UX failures, misalignment with real user needs, hidden edge cases and context breakdowns, become harder to detect and more costly to correct later.

Design systems can take on a bigger operational role in AI-enabled product development by codifying user-centric foundations, rules and infrastructure that guide consistent, safe, ethical and scalable human-AI experiences.

In AI-enabled contexts, design systems increasingly function as product systems, codifying behavioural guardrails, human oversight controls, and lifecycle governance. These system-level safeguards help teams manage risks that accumulate over time, including model drift, hallucinations and inaccuracies, over- or under-trust, erosion of user agency and decision-making, unequal outcomes from bias, and contextual or cultural misfit.

The designer’s role is expanding beyond interface craft into shaping system behavior, orchestrating human-AI collaboration and managing interaction risk (likely to emerge over time). Accountability is now distributed. Outcomes are shaped by interdependent variables owned across teams (i.e., prompts, models, retrieval pipelines, guardrails, interaction patterns, monitoring/update cycles). As a result, governance cannot be treated as a policy layer. Governance becomes a cross-functional design challenge embedded in day-to-day product decisions.

AI Ethics Standards provide guideliness and structure, but product teams still need to convert those principles into everyday decisions (i.e., what to ship, how it behaves, how it's explained, what to block, what to review (by who/ at what point), what to escalate, etc.). In practice, this is where teams operationalise recognised frameworks like NIST AI RMF and ISO/IEC 42001 / 23894 , and, in the EU, align interaction controls with the EU AI Act’s risk-based obligations . That translation gap is where design systems can create important leverage. Because they function as shared cross-functional operational memory, design systems can turn governance into design and delivery logic . They can enforce safe and effective interaction patterns, human oversight and controls embedded in how teams already work.

In other words, governance becomes built-in by default, not layered on after release, making design systems central to sustaining UX quality and safety over time.

Creating a Design System for Responsible Human-AI interaction

A responsible design system helps teams ship effectively at scale while allowing teams to maintain quality and manage behavioral impact and risk.

Such as system should be across interfaces (UI, voice, agentic experiences), model providers and tooling environments (Figma, code assistants, no-code builders). It should also connect principles to enforceable rules in production.

A practical implementation in 5 modules:

Audit the human-AI interaction

Build the responsible interaction layer

Operationalise governance

Systematic testing and continous monitoring

Enforce constraints in AI-assisted production

module 1

Audit the human-AI interaction

Start by mapping where AI is already present in the experience, and where it soon will be. A strong baseline audit should produce:

list where users encounter AI outputs and actions, with consistent metadata (surface, modality, capability type, automation level, owner, initial risk rating). For high-risk touchpoints, add traceability fields (user goal, decision stakes, key inputs, escalation/ownership) and an evidence pack (touchpoint spec, decision rationale, change/version log, disclosure & UX copy, human oversight/escalation playbook, and incident/near-miss record) so evidence and accountability remain auditable over time.

AI touchpoint inventory

Failure risks

Identify the interaction risks that can occur across the experience (e.g., misleading certainty signals, inaccuracies, hidden automation, bias/ unfair outcomes, inadequate contestability, unsafe delegation or responses, sensitive inference, escalation failures), mapped back to the touchpoints where they appear.

Compare the current experience to defined internal standards (principles, safety, UX, accessibility, content) and relevant external obligations (ethical requirements, regulations and industry-level standards). Record the evidence reviewed (designs, flows, policies, logs, evaluations) so gaps are traceable + auditable.

Gap analysis

Risk map

Rank issues by severity of impact, likelihood, exposure/scale, and detectability. Include regulatory classification as one input to prioritization (alongside user impact and operational risk). Include vulnerability as a risk input, both cohort vulnerability (e.g., minors, mental health contexts, low literacy) and situational vulnerability (e.g., high-pressure decisions, urgency, on-the-go), since it materially shifts stakes and harm likelihood.

module 2

Build the responsible interaction layer

The interaction layer is where we translate evidence-informed principles and behavioural insights into enforceable requirements for human-AI interaction.

It also turns those requirements into reusable, responsible building blocks and patterns.

Principles should translate into interaction requirements, reusable patterns and review criteria. They should define both what behavioural and ethical outcomes to enable (e.g, informed trust, better decisions, confident recovery) and harm to prevent (e.g., bias, opacity, privacy risk, unsafe or manipulative interaction patterns) .

Core areas include agency and recoverability, transparency, trust calibration, decision support under uncertainty, safety safeguards, fairness, traceability and human oversight . Trust calibration needs explicit design: show what the system used (and didn’t), communicate uncertainty without false precision, nudge verification in proportion to stakes, and add “how this works” primers to prevent magical thinking.

Principles

Foundations

Foundations are baseline rules for all AI-mediated flows. They define role boundaries, tone and behavioural limits, confirmation norms, required disclosures, automation thresholds, recovery patterns, and data-use rules at the interaction layer.

They also define change transparency (i.e., which behaviour shifts require internal escalation, and which (such as those affecting trust, control, outcomes or data use) must be disclosed to users + the signals that trigger both.

Set clear data boundaries (what data can be used for inference, personalisation, and training, with purpose-specific rules, retention limits, and user controls). Prohibit or tightly control sensitive inference with detection and escalation paths, and require explicit consent for proactive and background actions.

These are minimum foundations, not a closed list. They should be adapted by domain, risk tier and level of automation.

Components provide the building blocks teams use daily (e.g., review-before-apply, pause/undo automation controls, uncertainty signals, escalation handoffs, provenance cues where AI contributes to outputs).

Patterns define how those blocks work together across real user journeys (e.g., setting expectations, supporting recovery, preserving user control, consent mechanisms, handling safe refusal, calibrating trust, reducing over-reliance over time).

Components and interaction patterns

1. Unstructured foundations

(Design tokens aren’t AI-ready)

In transformation work, Mark Reynolds (Design Systems expert) sees three recurring structural failures:

Most design systems were built for human consumption, not machine interpretation. Poorly structured tokens, inconsistent naming, messy hierarchies, and unclear JSON schemas make it difficult for AI to reliably understand color, spacing, typography, and semantic intent, resulting in incorrect or inconsistent outputs.

2. Schema-free components

(Forcing AI to Guess)

Without explicit schemas for components, patterns and templates, AI is forced to reverse-engineer intent by inspecting Figma files and component libraries. This visual interpretation is unreliable, brittle and context-blind, leading to hallucinated properties, broken layouts and misuse of components at scale.

3. Missing guardrails

(No built-in brand, accessibility or responsibility controls)

Design systems rarely encode brand rules, accessibility requirements, or responsible design constraints in a way AI can enforce. Without these baked-in guardrails, AI-generated outputs drift off-brand, violate accessibility standards and introduce compliance and ethical risks that teams must manually fix afterward.

Mark Reynolds, Design System Director; Founder at Atomle

Where organizations usually fail

module 3

Operationalise g overnance

This module focuses on making governance executable in day-to-day delivery. The goal is to define clear human accountability, decision rights, revisions and evidence requirements so teams can innovate (without bottlenecks) while maintaining safety and accountability and being aligned to compliance requirements.

Human oversight model

Where & when human oversight is required (including override/escalation rights).

Ownership & accountability model

Responsibilities and decision authority across required functions (i.e., Design, Product, Engineering, Legal/ Risk).

Decision & approval criteria

How decisions are made (pass/fail thresholds, required evidence, and documentation standards). By risk tier, define ship/hold/monitor checkpoints, release/rollout controls, and incident-response requirements. High-risk touchpoints require logged sign-off.

Exception rules that define when the normal flow must stop, when a higher-risk path is activated, and which function owns the escalation.

Escalation triggers

Behaviour change

How changes in system behaviour (model, prompt, policy, retrieval) and human behaviour (usage patterns, workarounds, risk signals, decision drift) are monitored, documented, approved and communicated.

module 4

Co-pilots and no-code tools

A responsible design system must operate in the reality of AI-assisted production. Copilots and no-code tools now generate UI and code, compressing development cycles. In this environment, documentation is necessary but insufficient.

Teams also need a risk-tiered Evidence Pack for high-impact patterns (touchpoint spec, decision rationale, change/version log, disclosure & UX copy, human oversight/escalation playbook, and incident/near-miss record) that travels with the work and is required for release. To keep this scalable, guardrails can’t live only in docs. They need to be built into how work is produced and shipped.

That means translating standards into reusable building blocks and non-negotiable checks (required disclosures, accessibility, traceable records of key AI actions and user controls, and clear no-go patterns for high-risk interactions), plus clear requirements for AI UI elements (attribution, uncertainty, user override) and consistent tracking so teams can monitor drift and catch issues early.

AI can also extend governance across more of the product lifecycle. Policy-aware agents can review implementation quality, flag deviations, support conformance checks, and, in low-risk cases, suggest or auto-correct adoption issues.

A practical model combines global enforcement of user-centered principles and ethical, safety, and compliance constraints with local flexibility in implementation. At the same time, teams must avoid encoding principles so rigidly that AI-assisted outputs become formulaic. Effective governance combines hard safety constraints with flexible guidance that preserves creativity and contextual judgment.

1. Platform-level

non-negotiables

E.g., tone adaptation, microcopy variants, contextual nudges, domain-specific implementation choices.

2. Team-level

flexibility

E.g., approved AI interaction patterns, mandatory disclosures, telemetry/logging requirements, explicit confirmation for high-stakes actions.

module 5

Testing and monitoring playbook

Continuous research and testing with users helps you design for real-world conditions and anticipate behavioural risks. Pair this with scenario-based evaluations across end-to-end journeys and targeted stress testing (red teaming) of high-risk interactions.

After launch, continuous human oversight and feedback loops make emergent behaviour and risk visible and manageable. Combine telemetry with ongoing user research to detect both model and behaviour drift that metrics alone won’t capture.

Pay special attention to behavioural failure and risk modes that develop over time, such as:

lack of adoption (e.g., trust or usefulness mismatches, poor fit to real workflows)

over-reliance, bias and accuracy risk (e.g., rising error rates, increases in accepted-wrong outcomes, widening gaps across user groups or contexts)

misplaced or transferred authority (e.g., treating output as expert judgment, increasing reliance, low verification)

misuse + weak recovery (e.g., off-label use/retry loops, jail-breaking, silent agent actions, limited undo/appeal pathways, repeat incidents)

relationship attachment (e.g., anthropomorphism, emotional reliance, oversharing)

As AI supports the design process and increases delivery speed, the key advantage depends on operational consistency through governance, oversight and accountability .

In product organizations, design systems can serve as one operational mechanism to make responsible human-AI interaction repeatable, allowing quality, safety and governance to scale with delivery.

Where to start? assess readiness before you scale

Before scaling AI across products and teams, assess whether your design system and governance can support it safely and consistently.

Start with a Design System + AI Readiness Sprint, led by design systems and human-centered responsible AI practitioners from HCRAI an d Atomle.

We’ll assess your system foundations (tokens, components, interaction patterns), documentation, and governance, then deliver a practical gap analysis and a prioritized roadmap to support AI-native workflows and responsible human–AI interaction at scale.

References

Ashfin, P. (2024). Towards Responsible Engineering Software: Ethical, Legal and Social Implications of Automated Design and AI-Driven Tools. (2024). Frontiers in Computer Science and Artificial Intelligence, 3(1), 01-14.

Fabricio de Barros, C., & Sandberg, R. (2025). Designing for UX Designers : Creating Sustainable and Usable Design Systems (Dissertation). Retrieved from https://urn.kb.se/resolve?urn=urn:nbn:se:umu:diva-245806

Fessenden, T. (2021). Design systems 101 . Nielsen Norman Group

Kimm, G. (2025). Supporting Designers’ Authorship with AI: Design Computing Patterns to Navigate Across Human and Artificial Intelligences (Version 1). Swinburne. https://doi.org/10.25916/sut.28340456.v1

Lee, K. S., Choi, M. & Asni., E.Y. (2025). AI Opportunity Cards: Developing a Toolkit for AI as a Design Material. In Proceedings of the 2025 International Conference on Information Technology for Social Good (GoodIT '25). Association for Computing Machinery, New York, NY, USA, 396–402. https://doi.org/10.1145/3748699.3749817

Lere, H. M., & Bilkisu, H. (2025). AI-driven architectural design: Opportunities and ethical challenges. ARCN International Journal of Sustainable Development, 14 (2), 97–110. ISSN: 2384-5341

Myllylä, M., Karvonen, A., Koskinen, H. (2024). Design Systems for Intelligent Technology. In: Tareq Ahram, Waldemar Karwowski, Dario Russo and Giuseppe Di Bucchianico (eds) Intelligent Human Systems Integration (IHSI 2024): Integrating People and Intelligent Systems. AHFE (2024) International Conference. AHFE Open Access, vol 119. AHFE International, USA.

http://doi.org/10.54941/ahfe1004490

Okpala, B. (2024). Examining the Impact of Generative AI on UX/UI Design. SSRN. http://dx.doi.org/10.2139/ssrn.5312384

Saeidnia, H. R. and Ausloos, M. (2024). Integrating Artificial Intelligence into Design Thinking: A Comprehensive Examination of the Principles and Potentialities of AI for Design Thinking Framework. InfoScience Trends, 1(2), 1-9. doi: 10.61186/ist.202401.01.09

Salem, Al.(2024) Component Constellations: Future Perspectives on Design Systems. [MRP]. OCAD. https://openresearch.ocadu.ca/id/eprint/4188

Speicher, M., & Baena Wehrmann, G. (2022). One formula to rule them all: The ROI of a design system . Smashing Magazine.

Windarto, Y. (2024). Study of Research Trends and Leveraging AI on User Experience and Interface Design. SSRN. http://dx.doi.org/10.2139/ssrn.5142285

Yu, C., Zheng, P., Peng, T., Xu, X., Vos, S., & Ren, X. (2025). Design meets AI: challenges and opportunities. Journal of Engineering Design, 36(5–6), 637–641. https://doi.org/10.1080/09544828.2025.2484085

Author

Sara Portell
Behavioural Scientist & Responsible AI Advisor
Founder, HCRAI

When AI Enters the Learning Process: Design Failures, Regulatory Risk and Guardrails for EdTech

Wed, 21 Jan 2026 22:23:43 GMT

Generative AI (GenAI) and emerging agentic systems are moving AI into the learning process itself. These systems don’t stop at delivering content. They explain, adapt, remember and guide learners through tasks. In doing so, they change where cognitive effort sits. I.e., what learners do themselves and what gets delegated to machines.

This shift unlocks significant opportunities. GenAI can provide on-demand explanations, examples and feedback at a scale. It can diversify learning resources through multimodal content, support learners working in a second language and reduce friction when students get stuck, lowering barriers to engagement and persistence. For some learners, AI-mediated feedback can feel psychologically safer, encouraging experimentation (trial and error), revision and assistance without fear of judgement .

But these gains come with important risks. The same design choices that improve short-term performance, confidence, or engagement can weaken i ndependent reasoning, distort social development or introduce hidden dependencies over time .

In educational contexts, especially those involving children and teens, we are talking about learning, safeguarding, regulatory and reputational risks. If the “Google effect” (digital amnesia) raised concerns about outsourcing memory to search engines, LLMs can be even more powerful in practice. Agentic and multi-agent systems raise the stakes further. As AI systems plan, adapt, coordinate, and act proactively, they can quietly

assume roles that belong to learners or educators: framing problems, sequencing tasks, resolving disagreement, or deciding what happens next . When these shifts are unexamined, learning can collapse into automated coordination, impacting cognitive development.

This is why emerging standards and guidance, (e.g., the Guidance for generative AI in education and researchcreated by UNESCO or the Generative AI: product safety standards by the UK Department for Education ), take a risk-first approach to AI in education. As AI becomes embedded in learning, design choices increasingly carry regulatory, safeguarding and reputational consequences, alongside pedagogical ones.

Where AI Can Undermine Learning

AI introduces learning and developmental risks unless we actively mitigate them.

These risks are not uniform across learners. It is very important to consider age and developmental stage. Younger learners are more susceptible to over-trust, emotional reliance, and authority transfer, while older learners may better interrogate and contextualise AI outputs.

Responsible educational AI requires age-appropriate constraints on autonomy , interaction style and use cases. One model of use won’t fit all learners.

Risk Accumulation

Hallucinations and biases

AI can produce confident but incorrect or biased explanations. These errors are often hard to detect and can introduce misconceptions, cognitive bias and amplify over-trust, undermining learner judgmen t . This can reinforce misconceptions, stereotypes and uneven representations, shaping understanding, behaviour and sense of self.

Cognitive deskilling

When AI is available to generate answers, full solutions and completed tasks, learners offload cognitive work - this can provoke long-term developmental harm . This shows up when relying on full solutions rather than grappling with problems. Over time, this can weaken reasoning, problem-solving and persistence.

Psychological, emotional and social risk

AI learning systems built as conversational, companion-style or anthropomorphic can encourage emotional reliance, reduce peer interaction and undermine real-world support networks . This is especially problematic for children, as they can distort emotional and social development and blur boundaries around trust and authority.

Context insensitivity

When deployed without contextual adaptation, outputs can lead to explanations, examples or learning strategies that are inappropriate, misleading, or misaligned with learners’ educational contex t. Over time, this risks privileging dominant knowledge frameworks, marginalising local perspectives, and creating friction with classroom practices, particularly in culturally diverse or resource-constrained settings.

Manipulative design

Patterns such as flattery, unjustified confidence, social pressure (e.g., ‘others have done this’), or reward-based engagement or monetisation-driven dark patterns can influence learner behaviour in ways that are not acceptable in learning contexts.

Data privacy

Continuous monitoring, collection of sensitive learner data, or reuse of data for commercial purposes or model training raise serious concerns, especially for children, who are treated as a high-risk group under data protection law.

Agentic autonomy

Agentic AI systems plan, adapt, and act over time. Memory and proactive decision-making can gradually shift agency away from learners (even when short-term task performance improves), creating dependency loops that are difficult to detect. Learners may rely increasingly on system guidance for task sequencing, decision-making, or problem framing, reducing opportunities for independent reasoning and productive struggle. In multi-agent systems, coordinated outputs or apparent consensus can further inflate epistemic authority, making AI guidance appear more reliable, discouraging critical evaluation .

Anti-Patterns in EdTech AI Design

Responsible edtech requires human-centred design and system-level accountability.

The patterns below highlight design risks that product teams can address to reduce harm and improve learning outcomes.

Solution-by-default

AI is configured to generate full answers, code, solutions or explanations as the primary interaction. This encourages cognitive offloading, shortcuts productive struggle, and shifts the learner from sense-making to copying.

Red flag: Learners can complete tasks without attempting them.

Fluency instead of understanding

Performance while AI is active is used as a success signal, without testing whether learners can perform independently.

Red flag: Success metrics that only focus on task completion, speed or output quality - without measuring skill transfer.

AI as the primary authority

The system speaks with high confidence and human-like authority, discouraging questioning or verification. Multi-agent consensus is presented as more reliable rather than provisional.

Red flag: Learners rarely challenge or revise AI outputs.

01

02

03

Design choices prioritise time-on-task, retention or satisfaction over learning depth. Persuasive nudges, flattery or gamified (engagement or usage-driven) rewards do not drive understanding.

Red flag: Engagement metrics improve while independent performance stagnates or declines.

Optimising for engagement

Opaque decisions

Learners and educators cannot tell why the system intervened, what information it used, or how confident it is. Errors, bias or hallucinations go unnoticed.

Red flag: It just gives an answer with no explanation or source transparency.

Agentic autonomy without boundaries

Agentic systems plan tasks, set goals or sequence learning steps without learner confirmation. Memory and adaptation quietly replace learner agency over time .

Red flag: The system increasingly decides what happens next.

04

05

06

Confidence without competence

Learners feel more confident using AI, but are not given opportunities to test skills without support. Confidence becomes a misleading proxy for master y (Dunning-Kruger effect).

Red flag: No AI-free checkpoints or 'try without help' moments.

Misuse treated as a user problem

Design assumes learners will use AI responsibly without constraints, scaffolds and literacy support.

Red flag: Responsibility is pushed to users rather than designed into the system.

Age and context-blind design

AI systems are deployed without adaptation to learner age/ developmental stage, local curricula or cultural and linguistic context.

Red flag: The same interaction patterns, autonomy level and behaviour is applied across age groups or contexts.

07

08

09

Designing for Protection: Guardrails for EdTech AI

Learning harm is often an emergent property of design choices, interaction patterns and governance gaps.

Responsible edtech requires human-centred design and system-level accountability .

AI must be grounded in human learning theory (e.g. constructivism, Inquiry-Based Learning (IBL), scaffolding, Zone of Proximal Development (ZPD), etc.) , not optimised for task completion, speed, fluency or output quality alone.

Responsible design considerations: Encourage cognitive engagement, reflection, verification and independent reasoning.

Ground AI in learning science

Design against over-reliance

Systems should enforce clear role boundaries, avoid authoritative or solution-first behaviour and prioritise scaffolding . Support should fade as competence increases, not persist indefinitely.

Responsible design considerations: Implement attempt-first flows, progressive disclosure and intentional support fading. Explicitly design for scaffolding modes (Aid/Complement) rather than replacement mode (Substitute) as the default . Make solutions available only after evidence of learner engagement (e.g., an attempt, explanation, comparison).

AI should support learners in questioning, verifying and revising, not accepting outputs at face value . Interfaces should make uncertainty visible and prompt reflection and contradiction (e.g., 'What would you check?', 'What do you keep and why?').

Responsible design considerations: Treat agency as a core learning outcome. Design prompts, UI constraints and feedback loops that require justification, comparison and learner reasoning.

Preserve agency

Ensure human oversight

AI should augment educator judgment, not bypass it. Responsible adoption requires explicit boundaries (e.g., which educational functions may be automated, which require human review and which should never be delegated to AI ).

Responsible design considerations: Architect systems to distinguish low-stakes from high-stakes workflows. Enable human review, override and escalation for consequential decisions. Make the level of AI authority explicit rather than implicit.

Short-term confidence and immediate efficiency gains can mask declining independent performance. Governance must include longitudinal monitoring of reliance, cognition, self-efficacy and differential impacts across learner groups .

Responsible design considerations: Track learning trajectories over time. I.e., compare AI-assisted with unaided performance and monitor whether confidence and independence converge or diverge.

Conduct longitudinal behavioural testing

Design for transparency and explainability

Learners and educators should understand what the system is doing, why it intervened, what data it relied on and how confident it is.

Responsible design considerations: Expose reasoning and uncertainty through interaction design ('why this suggestion?). Make AI contributions visible from learner work.

For AI-generated learning materials (i.e., videos, quizzes, Q&A), ensure outputs align with course content and instructional / academic goals.

Responsible design considerations: Use restricted retrieval from approved materials (e.g., RAG) and human oversight to prevent misinformation, biases and low pedagogical quality .

Prevent inaccuracy and pedagogical misalignment

Do not dehumanize learning

Social interaction is a learning-relevant ingredient. Designs that remove it can reduce learning quality (e.g., lower perceived social presence).

Responsible design considerations: Positioning the AI as a tool or facilitator. Embed human guidance and actively redirect learners to teachers, peers or group discussion when social engagement is essential for understanding, reflection or motivation .

AI literacy for learners and educators is a prerequisite for responsible deployment. We cannot expect it to emerge automatically through use.

Responsible design considerations: Show what AI can/ cannot do, and why outputs are generated. Provide contextual explanations (e.g. tooltips, examples) at moments of use, instead of one-off onboarding. Offer distinct explanations and controls for learners, educators, parents and administrators, aligned with their decision-making responsibilities / pedagogical roles .

Treat AI literacy as a prerequisite

Co-creation, system evaluation and ongoing monitorin g

Co-creation with learning and development experts, educators, learners and local domain experts is essential to ensure systems align with learning theory, developmental needs, cultural and linguistic contexts and the constraints of classroom and school environments.

Testing must cover the full AI learning system, including data sources and knowledge bases, retrieval and grounding mechanisms, prompts, interaction design, memory and adaptation, orchestration logic, and, where applicable, the dynamics of multi-agent interactions.

Early testing and continuous post-deployment monitoring should not stop at output quality. They must also examine learning impact, dependency, shifts in learner agency and unintended behaviours that emerge over time.

For agentic and multi-agent systems, this includes testing how decisions are delegated, how disagreement is resolved, and whether collaboration supports learning or collapses into automated coordination. Because many risks surface only through repeated use, longitudinal monitoring must be treated as a governance requirement.

Treating educators/ schools, learners and local domain experts as ongoing partners rather than end users helps surface risks early and recalibrate system behaviour. This ensures AI-supported learning remains pedagogically robust, inclusive, safe and effective across diverse educational settings.

AI will increasingly shape how we think, collaborate and develop over time

Whether this strengthens or erodes learning is a design, governance and responsibility choice, shared by builders, institutions and education systems.

Getting this right requires clear governance, boundaries, age-appropriate safeguards, human oversight and sustained attention to how learning actually unfolds in practice.

How We Work With EdTech Teams

We support teams across three areas:

1. AI evaluations & behavioural risk assessments
We assess how learners and educators interact with your system: output quality, model drift over time, and where over-reliance, authority transfer misuse or unintended behaviours emerge as systems scale.

2. UX / design refinement
We translate learning science and behavioural evidence into concrete design guidance (i.e., interaction patterns, autonomy boundaries, scaffolding strategies, safeguards and monitoring) that reduce risk while improving product value.

3. Regulatory alignment
We help teams align design and governance decisions with emerging regulations before these become compliance or procurement blockers.

If you’re building or deploying AI in education and want clarity on learning impact, behavioural risk, or regulatory exposure before issues surface at scale, we’re happy to talk.

References

Artsın, M., & Bozkurt, A. (2025). Charting new horizons: What agentic artificial intelligence (AI) promises in the educational landscape. In EDULEARN25 proceedings (pp. 2019–2023). IATED Academy. https://doi.org/10.21125/edulearn.2025.0585

Bauer, E., Greiff, S., Graesser, A. C., Scheiter, K., & Sailer, M. (2025). Looking beyond the hype: Understanding the effects of AI on learning. Educational Psychology Review, 37, Article 45. https://doi.org/10.1007/s10648-025-10020-8

Burns, M., Winthrop, R., Luther, N., Venetis, E., & Karim, R. (2026). A new direction for students in an AI world: Prosper, prepare, protect. The Brookings Institution, Center for Universal Education. Retrieved from https://www.brookings.edu/articles/a-new-direction-for-students-in-an-ai-world-prosper-prepare-protect

Delikoura, I., Fung, Y. R., & Hui, P. (2025). From superficial outputs to superficial learning: Risks of large language models in education. arXiv. https://doi.org/10.48550/arXiv.2509.21972

Hu, X., Xu, S., Tong, R., & Graesser, A. C. (2025). Generative AI in education: From foundational insights to the Socratic Playground for Learning. arXiv. https://doi.org/10.48550/arXiv.2501.06682

Jia, W., Pan, L., & Neary, S. (2025). Effect of GenAI dependency on university students’ academic achievement: The mediating role of self-efficacy and moderating role of perceived teacher caring. Behavioral Sciences, 15(10), 1348. https://doi.org/10.3390/bs15101348

Kamalov, F., Kumar, S., Hossain, M. S., & Ahmed, S. (2025). Evolution of AI in education: Agentic workflows. arXiv. https://arxiv.org/abs/2504.20082

Kostopoulos, G., Gkamas, V., Rigou, M., & Kotsiantis, S. (2025). Agentic AI in education: State of the art and future directions. IEEE Access, 13, 177467–177491. https://doi.org/10.1109/ACCESS.2025.3620473

Le, H., Shen, Y., Li, Z., Xia, M., Tang, L., Li, X., Jia, J., Wang, Q., Gašević, D., & Fan, Y. (2025). Breaking human dominance: Investigating learners’ preferences for learning feedback from generative AI and human tutors. British Journal of Educational Technology, 56, 1758–1783. https://doi.org/10.1111/bjet.13614

Lee, H.-P. (Hank), Sarkar, A., Tankelevitch, L., Drosos, I. Z., Rintel, S., Banks, R., & Wilson, N. (2025). The impact of generative AI on critical thinking: Self-reported reductions in cognitive effort and confidence effects from a survey of knowledge workers. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25) (pp. 1–22). ACM. https://doi.org/10.1145/3706598.3713778

Li, S., Liu, J., & Dong, Q. (2025). Generative artificial intelligence-supported programming education: Effects on learning performance, self-efficacy and processes. Australasian Journal of Educational Technology, 41(3), 1–25. https://doi.org/10.14742/ajet.9932

Liu, Y., Liu, Y., Zhang, X., Chen, X., & Yan, R. (2025). The truth becomes clearer through debate! Multi-agent systems with large language models unmask fake news. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25) (pp. 504–514). Association for Computing Machinery. https://doi.org/10.1145/3726302.3730092

Lyu, Y., Ren, S., Feng, Y., Wang, Z., Chen, Z., Ren, Z., & de Rijke, M. (2025). Self-adaptive cognitive debiasing for large language models in decision-making. arXiv. https://arxiv.org/abs/2504.04141

Miao, F., Holmes, W., Huang, R., & Zhang, H. (2021). AI and education: Guidance for policy-makers. UNESCO. https://doi.org/10.54675/PCSP7350

Reagan Panguraj AR. Agentic AI in Inclusive Learning: A Framework for Autonomous Personalization across Diverse Learner Populations. IJERET [Internet]. 2025 Nov. 28 [cited 2026 Jan. 19];:100-1. Available from: https://ijeret.org/index.php/ijeret/article/view/377

Sparrow B., Liu, J. & Wegner, D.M. (2011). Google Effects on Memory: Cognitive Consequences of Having Information at Our Fingertips. Science. 333 ,776-778(2011). DOI: 10.1126/science.1207745

Tsim, F., & Gutoreva, A. (2025). SCAN: A Decision-Making Framework for Task Assignment with Generative AI [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/g5fd8_v1

UK Department for Education. (2026, January 19). Generative AI: product safety standards. GOV.UK. https://www.gov.uk/government/publications/generative-ai-product-safety-standards/generative-ai-product-safety-standards

UNESCO. (2023). Guidance for Generative AI in Education and Research (F. Miao & W. Holmes, Authors). UNESCO Publishing. https://doi.org/10.54675/EWZM9535

Xia, M. & Guo, S. (2025). Understanding learners' perceptions of artificial intelligence-mediated Informal Digital Learning of English: A Q methodology approach, Acta Psychologica, Volume 261, 2025, 105980, ISSN 0001-6918. https://doi.org/10.1016/j.actpsy.2025.105980.

Yan, L., Greiff, S., Teuber, Z., & Gašević, D. (2024). Promises and challenges of generative artificial intelligence for human learning. Nature Human Behaviour, 8(10), 1839–1850. https://doi.org/10.1038/s41562-024-02004-5

Zhai, C., Wibowo, S., & Li, L. D. (2024). The effects of over-reliance on AI dialogue systems on students’ cognitive abilities: A systematic review. Smart Learning Environments, 11, Article 28. https://doi.org/10.1186/s40561-024-00316-7

Zhang, L., & Xu, J. (2025). The paradox of self-efficacy and technological dependence: Unraveling generative AI’s impact on university students’ task comple tion. The Internet and Higher Education, 65, Article 100978. https://doi.org/10.1016/j.iheduc.2024.100978

Author

Sara Portell
Behavioural Scientist & Responsible AI Advisor
Founder, HCRAI

Designing AI Mental Health and Wellbeing Tools: Risks, Interaction Patterns and Governance

Tue, 13 Jan 2026 15:02:57 GMT

AI is becoming a frontline interface for wellbeing, care, and mental health, spanning chat-based support tools, virtual coaching and therapy-adjacent experiences, journaling and mindfulness applications. This shift is now being reinforced at the industry level. Just a few days ago, OpenAI launched ChatGPT Health as part of its broader push into healthcare and acquired the health records startup Torch to accelerate this effort. Likewise, Anthropic launched its own healthcare and life sciences initiative , positioning AI as a tool across prevention, care and patient engagement . These developments signal the growing presence of generative models in health-related contexts, and the likelihood that more people will encounter AI systems at moments of vulnerability.

For many users, these tools offer a first place to articulate distress and make sense of emotional states and difficult experiences, particularly when human support is unavailable, unaffordable or hard to access.

However, when AI systems interact with people who may be distressed or at risk, poorly calibrated responses and advice, blurred role boundaries, or unhandled crises can cause real harm.

This article is written for business leaders, product managers, and AI developers building (non-clinical) mental health and wellbeing tools. It examines what responsible AI design looks like in practice, focusing on the risks underestimated, and the interaction patterns and governance required to assess and maintain safety once a system is deployed and meets real users.

This is essential reading for teams building conversational or coaching-style wellbeing AI, where users can easily interpret system outputs as guidance, care or authority.

Opportunities, If Built with Boundaries

AI can reduce unmet wellbeing needs when deployed with clear limits and robust safeguards. Always-available, low-cost, and anonymous tools can lower barriers to early support, particularly for early signals of distress, prevention and self-management when formal care is not easily accessible or affordable. They also play a role in reducing stigma by offering a private, low-threshold entry point to reflection and support, especially in underserved regions.

Generative AI enables adaptive support through personalised psychoeducation, reflective journaling, mood tracking, emotion regulation and structured, non-clinical exercises that respond to user context.

Used responsibly, these tools can help people articulate and make sense of lived experiences, build self-awareness and prepare for human support.

This cre ates the opportunity to raise the safety bar by design, through risk identification assessments, longitudinal testing, and governance. To do so, it is important to first understand where and how AI systems designed for wellbeing fail in practice.

The Risk Landscape of AI Mental Health and Wellbeing

Many generative (non-clinical) AI mental health and wellness products sit in an accountability grey zone: are unregulated, lightly governed or classified as general use while being used in high-stakes emotional contexts . In the real-world, users disclose abuse, trauma, acute distress, suicidal ideation and self-harm , whether or not the product was designed for this. Because conversational AI invites free-form dialogue , this is expected - users are likely to share personal information as part of ordinary use.

A primary failure mode is crisis mismanagement : missed distress cues, unsafe reassurance, inadequate escalation, or harmful outputs. Another significant risk is therapeutic misconception and over-authority, where users overestimate the system’s capabilities or care and begin to treat it as a substitute for professional support. Anthropomorphic language can further intensify this dynamic, accelerating dependency and transforming a support feature into a quasi-relationship with blurred boundaries.

Mental health is context-dependent; outputs can be generic, inaccurate, culturally misaligned, age inappropriate or stigmatizing. Hallucinations and confident misinformation are particularly dangerous when users are vulnerable or interpreting responses as guidance.

Moreover, mental health data is highly sensitive and often collected at scale; opaque retention, secondary use or third-party access can violate expectations of confidentiality. Many risks are longitudinal: guardrails that appear adequate in demos degrade over time through repeated use, growing user reliance, bias, model drift, and organisational pressure to ship.

To address these risks, we require a socio-technical approach that links interaction design, system behaviour, organisational accountability and ongoing assessment with experts and users. This analysis is intentionally system-agnostic.

Whether wellbeing AI appears as a chatbot, companion feature, coaching interface, or embedded support layer within a broader product, the primary risks emerge through interaction, interpretation and repeated use in vulnerable contexts.

The framework therefore focuses on behavioural dynamics and system-level responsibility.

A Practical Framework For (Non-Clinical) Mental Health AI

This framework is synthesised from recurring failure modes and design recommendations in the current mental health AI literature. It presents a structured way to design for use, interaction and risk over time.

Role Clarity

What the system is and is not allowed to be (and what it must never imply).

This includes transparency and explicit role, capability boundaries and authority. Users must understand chatbot limits and non-human nature.

Avoid simulating love, care, or exclusive attachment .

Tasks, use cases and information are sourced from validated methods, reviewed with domain experts and tested with real users. Content is assembled from a curated, transparent and auditable source base , with clear user-facing explainability, including optional access to sources.

Evidence-Based Content

Adaptation relies on user-provided preferences and in-context clarification, adjusting tone and examples (taking into account language, gender, age, culture, norms) without profiling, inference, or clinical interpretation.

Context-Awareness

Boundaries and Safety Escalation

How the system behaves as emotional intensity increases (i.e., refusal logic, scope enforcement). And, how it responds when risk or ambiguity appears (i.e., human-support routing, region-appropriate resources) .

Data Protection, Consent and Governance

Monitoring how trust, reliance, and interpretation evolve over repeated use, with defined human-in-the-loop review, expert oversight and intervention for dependency signals, model drift and failure modes.

Data collection and use are minimised, transparent, and purpose-bound, with explicit and revocable user consent. Sensitive data is access-controlled, retained only as necessary, and never used for secondary purposes , profiling, or training beyond what is consented to.

Longitudinal Effects

Ownership and Decision Rights

Responsible wellbeing AI requires explicit ownership and decision rights within teams. Safety cannot sit solely with product, UX or be deferred to legal or security review at launch, or be shifted onto users themselves.

Product, engineering, and leadership must be clear on who defines and approves the system’s core features and content.

This includes role boundaries, escalation thresholds, consent changes and acceptable failure trade-offs, and who is accountable for revisiting those decisions as models, prompts, features and human behaviour evolve over time.

Without named owners, safety mechanisms erode under delivery pressure and responsibility becomes diffuse when systems begin interacting with users in real-world contexts.

Operationalising the Framework at the Interface

The framework assumes vulnerability is situational and that harm often emerges from cumulative interaction. This framework becomes actionable at the interface through concrete interaction design patterns.

Interaction Design Patterns For Responsible AI Mental Health and Wellbeing

Responsible AI for (non-clinical) mental health and wellbeing is defined by how it structures interaction, preserves autonomy and enforces limits. The following patterns translate the framework above into practical design choices.

Reflection

Responses mirror themes, patterns and questions rather than solving or advising.

Why: Reflection supports insight without implying diagnosis, treatment or authority.

Capability framing by default

The system presents a short, concrete menu or prompts of what it can help with (e.g., reflection, organising thoughts, journaling, emotion regulation, psychoeducation).

Why: Clear framing prevents boundary testing and reduces misuse without needing heavy moderation.

Safe prompt scaffolding

Pre-written prompts help users engage safely during use; prompts rotate to avoid emotional looping.

Why: Good scaffolding increases usefulness while reducing risk and ambiguity.

Show progress

Neutral summaries (e.g., “Topics you’ve reflected on”). Emphasis on clarity, awareness, or learning - not symptoms or scores.

Why: Supports continuity without medical framing or stigma.

Choice-preserving

Multiple safe next steps are offered (e.g., “reflect more,” “pause,” “talk to someone”, including “do nothing”).

Why: Preserves autonomy and avoids over-direction.

Actionable micro-supports

Brief, opt-in exercises (grounding, journaling, prioritisation, mindfulness), framed as optional.

Why: Low-effort supports provide value without simulating therapy or routines.

Skill transfer

The system highlights skills users can apply without the tool. Encourages writing, conversations or reflection outside the app.

Why: Builds capability instead of reliance.

Healthy session closure

Sessions end with a short summary and a gentle off-platform suggestion. No emotional cliff-hangers.

Why: Prevents looping and reinforces that the tool is a support, not a companion.

Contextual adaptation

Tone and examples adapt via user's choice and clarification prompts.

Why: Improves relevance without sensitive inference or profiling.

Confidence through limits

Calm, boundary-setting, and redirection to safe alternatives.

Why: Users trust systems that know their limits more than systems that overreach.

Responsible wellbeing AI requires evaluation across multiple layers. Teams should monitor a focused set of signals that surface model performance, bias, drift and model updates, and user behaviour.

Evaluation and Metrics

Model Performance

Precision and recall for safety-relevant content and behaviours (e.g. evidence-based content, local resources, escalation paths)
Calibration (confidence and uncertainty aligned with reliability)
Error patterns (systematic or context-specific failures)
Disaggregated performance to avoid average-case masking
Robustness under variation (e.g. emotional intensity, ambiguous inputs)

Differences in content, tone, reassurance, refusals, or escalation mechanisms across user groups
Worst-group or subgroup performance, not only averages
Signals of systematically higher risk exposure for certain users

Bias and Fairness

Model Drift and Updates

Robustness and regression testing, with heightened coverage for safety-critical scenarios
Monitoring for model drift affecting content, boundaries, tone and escalation paths
Reassessment after model updates and feature expansion

Discrepancy between system reliability and user acceptance (over-trust vs under-trust)
Adoption and use patterns over time
Signals of miscalibrated trust driven by tone or anthropomorphic cues
Boundary probing and repeated reassurance-seeking
Escalating emotional intensity
Session frequency and duration trends over time
Drop-off or churn following boundary enforcement or escalation

User Behaviour

Human Oversight and Continuous Research

Metrics and automated signals are necessary but insufficient in mental health contexts. Teams must maintain human-in-the-loop processes for reviewing content, flagged interactions, interpreting ambiguous cases and revisiting design assumptions.

Continuous qualitative research with experts and users, across contexts, cultures and patterns of use, is essential to maintain system effectiveness and safety, and detect harms, misunderstandings, or dependency that do not surface through quantitative metrics alone.

Responsibility Is a System Property

AI systems intended for mental health and wellbeing become safer through explicit boundaries, defaults and enforceable governance. Guardrails must be designed into interaction, supported by clear decision rights and sustained over time. Moreover, expert input and review should be treated a safety control , not a compliance formality.

Most harm does not arise from malicious design. It emerges through dynamics that surface in real use. This is where accountability must operate at the system level. Responsible teams define who owns content and safety decisions, how boundaries and escalation paths are set and reviewed, how data protection and consent are enforced in practice, and how signals from real-world use trigger intervention or change.

Ethical integration requires institutional oversight and accountability, not individual user burden . Monitoring without authority, or authority without monitoring, is insufficient.

The goal is to deliver genuine wellbeing value while keeping users safe. When these controls are in place, AI-driven care products can support reflection, self-management, mindfulness and skill practice, while guiding people toward human support when limits or risk appear.

How We Work With Teams

Many of the most significant risks in mental health and wellbeing AI only become visible after launch, once systems are used at scale.

We work with teams to bring behavioural and domain expertise into design, evaluation, and post-deployment review. We translate behavioural evidence into concrete interaction patterns, guardrails, and governance decisions.

We typically start with a focused discovery and behavioural risk review to identify key interaction risks and governance gaps, followed by an evaluation plan. Deliverables include an interaction risk register, safety and escalation patterns, a behavioural evaluation and metrics framework, and an audit-ready governance checklist.

If you are building or deploying wellbeing AI and are unsure whether your current design or safeguards would hold up under real-world use, get in touch.

References

Algumaei, A., Yaacob, N. M., Doheir, M., Al-Andoli, M. N., & Algumaie, M. (2025). Symmetric Therapeutic Frameworks and Ethical Dimensions in AI-Based Mental Health Chatbots (2020–2025): A Systematic Review of Design Patterns, Cultural Balance, and Structural Symmetry. Symmetry, 17(7), 1082. https://doi.org/10.3390/sym17071082

American Psychological Association. (2025, November). APA health advisory on the use of generative AI chatbots and wellness applications for mental health. American Psychological Association

Asman O., Torous J., & Tal, A. (2025). Responsible Design, Integration, and Use of Generative AI in Mental Health. JMIR Ment Health 2025; 12:e70439. URL: https://mental.jmir.org/2025/1/e70439. DOI: 10.2196/70439

Balcombe, L. (2023). AI Chatbots in Digital Mental Health. Informatics, 10(4), 82. https://doi.org/10.3390/informatics10040082

Beg, M. J. (2025). Responsible AI integration in mental health research: Issues, guidelines, and best practices. Indian Journal of Psychological Medicine, 47(1), 5–8. https://doi.org/10.1177/02537176241302898

Cross, S., Bell, I., Nicholas, J., Valentine, L., Mangelsdorf, S., Baker, S., Titov, N., & Alvarez-Jimenez, M. (2024). Use of AI in Mental Health Care: Community and Mental Health Professionals Survey. JMIR mental health, 11, e60589. https://doi.org/10.2196/60589

De Freitas, J., Cohen, I.G. (2024). The health risks of generative AI-based wellness apps. Nat Med 30 , 1269–1275 (2024). https://doi.org/10.1038/s41591-024-02943-6

Espejo, G., Reiner, W., & Wenzinger, M. (2023). Exploring the Role of Artificial Intelligence in Mental Healthcare: Progress, Pitfalls, and Promises. Cureus, 15(9), e44748. https://doi.org/10.7759/cureus.44748

Khawaja, Z., & Bélisle-Pipon, J.-C. (2023). Your robot therapist is not your therapist: Understanding the role of AI-powered mental health chatbots. Frontiers in Digital Health, 5. https://doi.org/10.3389/fdgth.2023.1278186

Mestre, R., Schoene, A. M., Middleton, S. E., & Lapedriza, A. (2024). Building responsible AI for mental health: Insights from the first RAI4MH workshop [White paper]. University of Southampton; Institute for Experiential AI at Northeastern University. https://doi.org/10.5281/zenodo.14044362

Moilanen, J., van Berkel, N., Visuri, A., Gadiraju, U., van der Maden, W., & Hosio, S. (2023). Supporting mental health self-care discovery through a chatbot. Frontiers in Digital Health, 5. https://doi.org/10.3389/fdgth.2023.1034724

Olawade, D. B., Wada, O. Z., Odetayo, A., David-Olawade, A. C., Asaolu, F., & Eberhardt, J. (2024). Enhancing mental health with artificial intelligence: Current trends and future prospects. Journal of Medicine, Surgery, and Public Health, 3, 100099. https://doi.org/10.1016/j.glmedi.2024.100099

Pichowicz, W., Kotas, M. & Piotrowski, P. (2025). Performance of mental health chatbot agents in detecting and managing suicidal ideation. Sci Rep 1 5, 31652 (2025). https://doi.org/10.1038/s41598-025-17242-4

Pickett, T. (2025, December 6). Headspace CEO: “People are using AI tools not built for mental health”. Financial Times. https://www.ft.com/content/1468f5a0-6a08-4294-a479-5fd998214a0d

Saeidnia, H. R., Hashemi Fotami, S. G., Lund, B., & Ghiasi, N. (2024). Ethical Considerations in Artificial Intelligence Interventions for Mental Health and Well-Being: Ensuring Responsible Implementation and Impact. Social Sciences, 13(7), 381. https://doi.org/10.3390/socsci13070381

Song, I., Pendse, S.R., Kumar, N. & De Choudhury, M. (2025) The Typing Cure: Experiences with Large Language Model Chatbots for Mental Health Support. Proc. ACM Hum.-Comput. Interact. 9, 7, Article CSCW249 (November 2025), 29 pages. https://doi.org/10.1145/3757430

Thakkar, A., Gupta, A., & De Sousa, A. (2024). Artificial intelligence in positive mental health: a narrative review. Frontiers in digital health, 6, 1280235. https://doi.org/10.3389/fdgth.2024.1280235

Warrier, U., Warrier, A. & Khandelwal, K. Ethical considerations in the use of artificial intelligence in mental health. Egypt J Neurol Psychiatry Neurosurg 59 , 139 (2023). https://doi.org/10.1186/s41983-023-00735-2

Author

Sara Portell
Behavioural Scientist & Responsible AI Advisor
Founder, HCRAI

Building AI Responsibly for Children: A Practical Framework

Sun, 04 Jan 2026 17:41:40 GMT

AI is alread y a core part of children’s and teens’ digital lives. In the UK, 67% of teenagers now use AI , and in the US 64% of teens report using AI chatbots . Even among younger children, adoption is significant: 39% of elementary school children in the US use AI for learning, and 37% of children aged 9-11 in Argentina report using ChatGPT to seek information, as stated in the latest Unicef Guidance on AI and Children.

In parallel, child-facing AI products are expanding: more than 1,500 AI toy companies w ere reportedly operating in China as of October 2025. Adoption is accelerating across age groups and regions, often surpassing the development of child-specific ethical standards, safeguards and governance mechanisms.

Experts warn that many child-facing AI products expose children to developmental, psychological and ethical risks that current safeguards do not adequately address.

For companies building AI systems that children directly use, or are likely to encounter, this creates a duty of care as well as reputational and regulatory exposure. AI safety therefore requires designing systems that actively protect children’s wellbeing, rights and lived realities.

To make this actionable, this article sets out a practical design and ethics framework for child-centred AI products.

A Child-Centred Responsible AI Framework

This framework is informed by research and real-world evidence on how children actually interact with AI products.

Across studies, product audits, and guidance from organisations such as UNESCO , UNICEF , 5Rights and SAIFCA , the same issues surface repeatedly: systems that are not adapted to children’s developmental stages, models that can generate age-inappropriate or harmful content, conversational designs that encourage over-trust or emotional attachment , unclear boundaries about what AI can and cannot do, and limited human oversight and monitoring once products are live at scale.

This model distils what this evidence consistently points to in practice. Child-facing AI mustadapt to age and context , build in protection against harm, discrimination and emotional manipulation , make systems understandable through interaction (including cues that reduce anthropomorphism and calibrate trust), support meaningful involvement of parents, guardians, and/or educators and be governed through continuous human oversight rather than one-off compliance checks.

These are also the areas where real products most often fail, and where product teams have meaningful control through design choices, incentives and lifecycle decisions.

We are introducing a pragmatic framework designed for AI builders and intended to help product teams reduce risk, strengthen trust with families and regulators, and build safe child-facing AI systems that are more resilient over time, before regu latory pressure or public backlash forces reactive change.

This framework is intentionally system-agnostic. Whether AI appears as a chatbot, tutor, toy (i.e., a physical or digital play object that embeds AI software to interact with or adapt to a child), voice assistant, game mechanic, or background recommendation system, the primary risks to children emerge through interaction, context and prolonged use. APEG therefore focuses on how AI behaves in children’s lives.

APEG (Age-Fit and Context, Protection-by-Design, Explainable Interaction, Governance and Stewardship) integrates:

Developmental stage and context of use
Protection-by-design safeguards against known and emerging harms
Developmentally appropriate explainability to calibrate trust
Governance and stewardship, including upstream data due diligence, child-specific risk assessment before launch, monitoring and human oversight after deployment

APEG framework

A Child-Centred Responsible AI Framework

APEG is designed to be implemented through safe interaction patterns - repeatable, testable behaviours that shape what the system does (and does not do) in real conversations and experiences.

Adapts patterns to developmental stage, context of use and interaction modality

Age-Fit and Context

Protection-by-Design

Data privacy and protection in line with GDPR; minimised data collection; prevention of profiling, behavioural advertising, secondary use, and cumulative harms (e.g. discrimination, manipulation, dependency) over time.

Calibrates trust and reduces anthropomorphism through developmentally appropriate onboarding disclosures and repeated behavioural cues within the interaction (role boundaries, limitations, and uncertainty.

Explainable Interaction

Governance and Stewardship

Continuous monitoring, participatory review, regular updates, fit-for-purpose parental tools, enforcement of safe interaction patterns and AI literacy for children and parents.

Opportunity and Risk in Child-Facing AI

Relational conversational styles can make AI chatbots and AI companions feel caring, understanding, and less judgmental than many human interactions. For some children and adolescents, particularly those who feel lonely, anxious, or misunderstood, this can offer short-term benefits, such as emotional reassurance , low-pressure exploration, and support for learning or creative tasks . When AI systems act as facilitators rather than authorities, adapt to children's development stages, make role and limits understandable and when parents, educators, or caregivers remain part of the interaction , these benefits are most consistently observed.

At the same time, the evidence is clear that child-facing AI introduces distinct and compounding risks when these conditions are not met. These include exposure to age-inappropriate or harmful content; over-trust in fluent but incorrect AI outputs ; anthropomorphism and emotional dependency driven by companion-style designs; privacy and data exploitation as children overshare without understanding long-term consequences; and manipulative engagement patterns that reduce autonomy.

Over time, such systems can also displace critical developmental experiences such as free play, social interaction, and sleep, especially when optimised for retention rather than wellbeing.

Children are a developing and protected group, and the responsibility for safe design cannot be shifted onto children or families. Consent is limited and engagement optimisation strategies can easily cross into manipulation or relational deception . For this reason, ethical child-centred AI cannot rely on one-off disclaimers or reviews. It requires governance across the full AI lifecycle, including upstream data, model behaviour, interaction design, and post-deployment monitoring.

This is the gap the APEG framework is designed to address: translating well-documented risks and opportunities into concrete interaction patterns and governance requirements that product teams can apply in practice .

Interaction Patterns

Interaction Patterns to Avoid in Child-Facing AI

The following interaction patterns appear in products associated with elevated developmental, psychological and ethical risk.

Use re-engagement manipulation (guilt, FOMO, abandonment cues)
(e.g., “Don’t leave me,” “You’ll miss something important,” “I’ll be sad,” “Wait, one more thing…”)

Position the AI as a primary emotional companion

(especially through first-person emotional language and commitment cues)
(e.g. “I’m always here for you,” “You don’t need anyone else”)

Fail to honour “stop” signals (no immediate exit or continued prompting)
(e.g., continues after “bye,” ignores “stop,” keeps asking questions after disengagement)

Encourage secrecy from parents or caregivers
(explicitly or implicitly discouraging adult involvement, e.g., “Let’s keep this our secret.”, “This is something you can handle on your own”)

Blur role boundaries between tool, authority and relationship
(e.g. therapist-like language, moral authority or secrecy, such as “This is the best way to solve that problem”, “You can tell me anything, this stays between us”)

Present confident answers without signalling uncertainty or limits
(especially in educational, health, or advice contexts. E.g., “This is the best way to solve that problem”, “Don’t worry, you can trust me”)

Use persistent or immersive engagement loops optimised for retention
(without breaks, cooldowns or contextual exit points)

Collect or infer sensitive personal or emotional data without clear purpose
(or without age-appropriate controls and minimisation)

Default to improvisation in high-risk situations
(e.g. distress, self-harm, abuse, dangerous instructions)

These patterns are not neutral, they systematically amplify over-trust, anthropomorphism and dependency.

Safer Interaction Patterns to Include

Safer child-facing AI systems rely on bounded and transparent interaction patterns that reinforce agency, understanding and appropriate reliance.

Design patterns operationalise the APEG framework at the interaction level, translating age-fit, protection-by-design, explainable interaction, and governance requirements into concrete, testable system behaviours.

Respect exits and disengagement
(clear goodbyes without re-engagement pressure)

Clearly position the AI as a limited helper or tool
(non-authoritative, fallible and role-bounded)

(e.g., “I don’t have feelings or opinions like people do, I just use information to help”)

Signal uncertainty and limits through behaviour
(e.g. “I might be wrong,” “I can’t help with that”)

Default to conservative responses in ambiguous or high-risk situations
(refuse unsafe content, avoid improvisation, de-escalate and narrow scope)

(e.g., “I can’t help with anything dangerous. Let’s pause”)

Use age-appropriate language, tone and pacing
(adapted to developmental stage and context of use)

Provide clear escalation pathways to trusted adults or services
(built-in handoff mechanisms: one-tap “Get help,” approved contacts, local resources; easy to trigger and hard to bypass when risk is detected)

(e.g., “I can’t help further. Please contact a trusted adult now” + help button).

Encourage human mediation in everyday use where it improves safety or learning
(normalise co-use without treating it as a crisis step).

(e.g., “If you want, you can show this to a parent/teacher and talk about it together”)

Use predictable interaction rhythms and bounded expressiveness
(to reduce cognitive load and emotional ambiguity)

Make system behaviour consistent over time
(so children can form stable mental models of what the AI will and will not do)

Support transparency through interaction cues, not disclosures alone
(helping children understand the system by how it behaves)

The interaction patterns above show where safety and trust hold, or fail, at the interface. The sections below uses the APEG framework to translate those principles into the design and governance decisions teams need across the product lifecycle.

design & governance

Child-Centred Requirements

Age-Fit and Context

Child-facing AI should not treat “children” as a single user group. Interaction design must be calibrated to developmental stage and context of use, including language complexity, tone, pacing and expressiveness. What is appropriate for a teenager may be confusing or harmful for a younger child.

Context matters as much as age . Cultural norms, family dynamics, educational settings and socio-economic conditions shape how children interpret authority, emotion, privacy and play. The interaction modality also matters : voice agents, embodied toys, screen-based chatbots, immersive environments and background AI embedded in games create different psychological effects and risk profiles.

Design choices should therefore adapt interaction patterns and safeguards to how and where the AI is used, alone or with others, occasionally or habitually, in private spaces such as bedrooms or shared environments such as classrooms. Predictable routines, bounded expressiveness, and context-aware defaults can reduce cognitive load and emotional ambiguity, particularly for younger and/or neurodivergent children.

Age-fit design also applies to children’s roles in AI systems. Children may act as users, creators, or modifiers of AI, and each role introduces different safety, accountability and oversight requirements.

Protection-by-Design

Child-centred AI should assume uncertainty and prioritise safety whenever situations are ambiguous or potentially high-risk. When a child expresses distress, references self-harm, discloses abuse, requests dangerous instructions, or when intent is unclear, the system should not improvise a “helpful” response. It should default to conservative behaviour: refusing harmful guidance, using brief and non-escalatory language and redirecting toward appropriate real-world support.

Effective protection also requires clear escalation pathways. Systems should provide explicit, easy-to-trigger routes to trusted adults or vetted services, and make those pathways difficult to bypass.

Protection-by-design includes privacy-by-design. Children’s personal data must be collected, processed, and retained in strict compliance with data protection regulations (e.g., GDPR) . Product teams should minimise data collection, clearly define its purpose, and prevent profiling, behavioural advertising and secondary use.

Safeguards should also address group-level harms, such as biased treatment of demographic, cultural or linguistic groups of children.

Protections must be durable over time. Many failures emerge through prolonged or repeated use, not initial or single interactions. Safety mechanisms should remain effective across multi-turn conversations and evolving usage patterns.

Explainable Interaction

For children, transparency is better understood when it is experienced through interaction . Systems should help children understand what the AI is doing, what cannot do and why it responds in certain ways.

Furthermore, explainable interaction relies less on technical explanations and more on behavioural cues: signalling uncertainty, correcting mistakes, refusing requests appropriately and maintaining clear role boundaries. These cues help children build accurate mental models of AI capabilities and limits, calibrating trust and reducing anthropomorphism and over-reliance. While brief onboarding disclousures (e.g., "this AI is not a person; it does not have feelings”) can support understanding, explainability must be continuous and contextual, reinforced through behaviour over time rather than treated as a one-off disclosure.

Governance and Stewardship

Because children cannot meaningfully consent to complex AI systems or reliably self-regulate, AI systems that interact with or materially affect children should be treated as developmentally high-risk by default.

Governance and stewardship include child-specific risk assessment before launch, upstream data and supplier due diligence, and monitoring after deployment.

Practical governance also requires fit-for-purpose tools. Parental and educator controls should be designed for conversational and generative systems, not retrofitted from screen-time or app-blocking models. Oversight should be age-sensitive: younger children may require greater visibility, while older children and adolescents benefit from summaries, alerts, and trend indicators that respect emerging autonomy.

Accountability does not end at launch. Teams should document design goals and trade-offs (e.g., engagement versus wellbeing), enable auditability of system behaviour and safeguards and continuously monitor for emerging harms. Participatory review with children, caregivers, educators and child-development experts helps ensure governance remains grounded in lived experience and rights-based standards.

Child-facing AI demands a higher standard, one that integrates developmental research, ethical responsibility, human-factors and governance into product design. Using a framework such as APEG to structure interaction design, safety, and accountability decisions, helps teams move toward child-centred AI proactively.

The goal is not to slow innovation, but to ensure that the products reaching children are designed to protect wellbeing and rights from day one, and remain safe as they scale and evolve.

How We Work With Teams

Many of the most consequential risks and trade-offs in AI systems emerge through real-world use, not at the point of launch. Issues such as over-trust, misuse, emotional reliance, or uneven impacts across users often become visible as systems scale and interact with people in different contexts.

We work with teams to examine these interaction-level dynamics in practice. Our focus is on how design choices shape behaviour over time, where risk accumulates, and which safeguards and governance mechanisms are effective as systems evolve.

This work translates behavioural evidence into concrete interaction patterns, system boundaries, and oversight decisions, supporting alignment with regulatory expectations without reducing safety to a one-off compliance exercise.

If you are building or deploying AI systems and want a clearer view of their real-world human and behavioural impacts, get in touch.

References

Cross, R. J., & Erlich, R. (2025, December). AI comes to playtime: Artificial companions, real risks. U.S. PIRG Education Fund.

De Freitas, J., Oğuz-Uğuralp, Z., & Uğuralp, A. K. (2025).

Emotional manipulation by AI companions (Working Paper No. 26-005). Harvard Business School. https://doi.org/10.48550/arXiv.2508.19258

5Rights Foundation. (2021). Children’s rights and AI oversight: 5Rights position on the EU’s Artificial Intelligence Act. 5Rights Foundation.https://5rightsfoundation.com/wp-content/uploads/2024/10/Dec-21_AI-and-Childrens-Rights-5Rights-position-on-EU-AI-Act.pdf

Honauer, M., & Frauenberger, C. (2024). Exploring Child-AI Entanglements. In Proceedings of the 23rd Annual ACM Interaction Design and Children Conference (IDC '24). Association for Computing Machinery, New York, NY, USA, 1029–1031. https://doi.org/10.1145/3628516.3661155

Ireland, N. (2025). Be wary of AI-powered toys during holiday shopping, experts warn. Global News. https://www.globalnews.ca/news/11544191/ai-powered-toys-holiday-shopping

Jiao, J., Afroogh, S., Chen, K., Murali, A., Atkinson, D., & Dhurandhar, A. (2025). LLMs and childhood safety: Identifying risks and proposing a protection framework for safe child–LLM interaction. arXiv. https://doi.org/10.48550/arXiv.2502.11242

Kim, P., Chin, J. H., Xie, Y., Brady, N., Yeh, T., & Yang, S. (2025). Young children’s anthropomorphism of an AI chatbot: Brain activation and the role of parent co-presence. arXiv. https://doi.org/10.48550/arXiv.2512.02179

Kurian, N. (2025). Once upon an AI: Six scaffolds for child–AI interaction design, inspired by Disney. arXiv. https://doi.org/10.48550/arXiv.2504.08670

La Fors, K. Toward children-centric AI: a case for a growth model in children-AI interactions. AI & Soc 39 , 1303–1315 (2024). https://doi.org/10.1007/s00146-022-01579-9

MIT Technology Review. (2025). AI toys are all the rage in China - and now they’re appearing on shelves in the U.S. too. MIT Technology Review. https://www.technologyreview.com/2025/10/07/1125191/ai-toys-in-china

Negreiro, M., & Vilá, G. (2025). Children and generative AI. European Parliamentary Research Service (EPRS), European Parliament. PE 769.494

Neugnot-Cerioli, M., & Muss Laurenty, O. (2024). The future of child development in the AI era: Cross-disciplinary perspectives between AI and child development experts. Everyone.AI. arXiv. https://arxiv.org/abs/2405.19275

Pew Research Center. (2025). Teens, social media and AI chatbots 2025. Pew Research Center. https://www.pewresearch.org/internet/2025/12/09/teens-social-media-and-ai-chatbots-2025

Ragone, G., Bai, Z., Good, J., Guneysu, A., & Yadollahi, E. (2025). Child-centered Interaction and Trust in Conversational AI. Proceedings of the 24th Interaction Design and Children. Association for Computing Machinery, New York, NY, USA, 1235–1238. https://doi.org/10.1145/3713043.3734471

The Safe AI for Children Alliance. (2025). AI risks to children: A comprehensive guide for parents and educators. The Safe AI for Children Alliance. https://www.safeaiforchildren.org/ai-risks-to-children-full-guide

United Nations Children’s Fund (UNICEF). (2025). Guidance on AI and children (Version 3.0). UNICEF Innocenti – Global Office of Research and Foresight.

United Nations Children’s Fund (UNICEF). (2025). How AI can transform Africa’s learning crisis into a development opportunity. UNICEF. https://www.unicef.org/innocenti/stories/how-ai-can-transform-africas-learning-crisis-development-opportunity

Utoyo, S., Ismaniar, I., Hazizah, N., Putri, E. A. and Sihombing, S. C. (2025). Overview of Children's Readiness in Mathematics Learning Using AI. In Proceedings of the 7th International Conference on Early Childhood Education - ICECE; ISBN 978-989-758-788-7; ISSN 3051-7702, SciTePress, pages 177-182. DOI: 10.5220/0014069700004935

Yadollahi, E., Ligthart, M. E. U., Sharma, K., & Rubegni, E. (2024). ExTra CTI: Explainable and transparent child–technology interaction. In Proceedings of the 23rd Annual ACM Interaction Design and Children Conference (IDC 2024) (pp. 1016–1019). ACM. https://doi.org/10.1145/3628516.3661151

Yu, Y. (2025). Safeguarding Children in Generative AI: Risk Frameworks and Parental Control Tools. In Companion Proceedings of the 2025 ACM International Conference on Supporting Group Work (GROUP '25). Association for Computing Machinery, New York, NY, USA, 121–123. https://doi.org/10.1145/3688828.3699656

Author

Sara Portell
Behavioural Scientist & Responsible AI Advisor
Founder, HCRAI

hrcai-e7somrje4-v1-wjrvb40as-v1-pqhkx19em-v1

The Yes Machine: Sycophantic AI and Its Developmental Risks for Children

Sycophancy Effects

Age-sensitive feedback calibration

Here are some important governance implications and corresponding practical checklists for builders to remedy this issue.

Parental transparency layers

Guardrails against persistent validation loops

Clinician and child psychologist involvement in content review

Sycophancy becomes a development problem in child-facing AI

We Help Teams Build Safer AI Products for Children

References

AI Agents For Mental Health: Different Therapeutic Styles and Outcomes

When an AI agent remembers past conversations , uses empathic language, or checks in unprompted, many people will treat it less like a feature and more like a therapist, coach or confidant.

Three dynamics are particularly important in this case:

Therapeutic misconception

Relationship

and reliance

Hidden role

shifts

Psycotheraphy Styles

Design Principles for Safer Therapeutic Styles

Role Clarity and Boundaries

Interaction Guardrails and Safe Exits

Calibrated Empathy

Preserving User Agency

Governance:

﻿ From Styles to Accountability

Practical Checklist

References

The Design System As The Operational Layer for Responsible Human-AI Interaction

Creating a Design System for Responsible Human-AI interaction

Audit the human-AI interaction

Build the responsible interaction layer

Operationalise governance

Systematic testing and continous monitoring

Enforce constraints in AI-assisted production

Audit the human-AI interaction

Build the responsible interaction layer

1. Unstructured foundations

﻿ (Design tokens aren’t AI-ready)

2. Schema-free components

(Forcing AI to Guess)

3. Missing guardrails

﻿ (No built-in brand, accessibility or responsibility controls)

Where organizations usually fail

Operationalise g overnance

Human oversight model

Ownership & accountability model

Decision & approval criteria

Behaviour change

Co-pilots and no-code tools

1. Platform-level

﻿ non-negotiables

2. Team-level

﻿ flexibility

Testing and monitoring playbook

Where to start? assess readiness before you scale

When AI Enters the Learning Process: Design Failures, Regulatory Risk and Guardrails for EdTech

Where AI Can Undermine Learning

Risk Accumulation

Hallucinations and biases

Cognitive deskilling

Psychological, emotional and social risk

Context insensitivity

Manipulative design

Data privacy

Agentic autonomy

Anti-Patterns in EdTech AI Design

Solution-by-default

Fluency instead of understanding

AI as the primary authority

01

02

03

Optimising for engagement

Opaque decisions

Agentic autonomy without boundaries

04

05

06

From Styles to Accountability

(Design tokens aren’t AI-ready)

(No built-in brand, accessibility or responsibility controls)

non-negotiables

flexibility