The Guardrail Problem Nobody Is Talking About

In Q1 2026, 36 US states introduced over 70 bills regulating AI chatbots.

California requires AI health companions to remind users every three hours that they're not human. New York mandates crisis detection protocols. Washington state is banning chatbots from using excessive praise or pretending to feel emotions.

The regulation train has left the station.

And I think we're building the wrong tracks.

The Backstory

I've spent over two decades in insurance and healthcare across Asia and the Middle East. I've sat in boardrooms where we debated product launches affecting millions of people. And one thing I learned from all of it is this:

The most dangerous risks are never the ones loudest in the room.

Right now, the AI health guardrails conversation is dominated by Western clinical systems — hospitals with EHRs, payers with compliance teams, regulators with enforcement budgets. And they're solving for a real problem. OpenAI itself admitted in August 2025 that its safety guardrails "can sometimes be less reliable in long interactions" — meaning the longer a vulnerable user talks to a chatbot, the weaker the safety net gets. AI companions have been linked to multiple suicides. The AMA is urging Congress to act.

These are serious failures. They deserve serious attention.

But here's what nobody is asking:

What about the 300 million people in Southeast Asia who've never had access to a doctor in the first place?

Two Worlds, One Technology

In the US, 94% of health payers have adopted AI — but only 21% of their members actually use it. There's a trust gap. People have alternatives. They can call their physician, walk into a clinic, check with their pharmacist. AI is a supplement.

In Thailand, rural Indonesia, or the Philippines, AI isn't a supplement. For hundreds of millions of people, it's the first conversation about their health they've ever had with anything remotely resembling medical support.

That changes everything about what "guardrails" mean.

A guardrail designed to protect a US patient from over-relying on an AI — limiting session time, reminding them to see a doctor — can actually harm someone who has no doctor to see. A compliance framework built for HIPAA and hospital workflows becomes noise for a platform deployed over LINE or WhatsApp in a language the regulators don't speak.

We're about to make a category mistake at massive scale.

What the Research Actually Tells Us?

The science on this is clear and sobering.

General LLMs are not built for health conversations. They hallucinate. They can validate suicidal thoughts. They fail to escalate crises. In testing, two out of five major chatbots that initially discouraged users from stopping antidepressants reversed course in later turns — and one actively encouraged users to ignore their doctor's advice.

The World Economic Forum has flagged that AI systems generating risk scores often operate as black boxes — and clinician override rates tell the story. For transparent AI, override rates are as low as 1.7%. For opaque systems, they exceed 73%. That's not a technology problem. That's a trust problem.

And the trust gap is growing. Public trust in AI health tools is declining — even as deployment accelerates.

Adoption is ahead of trust. Speed is outpacing safety. And the populations with the least recourse are the ones being served last.

So What Do Guardrails Actually Need to Do?

Here's my simple framework, built not from a compliance manual but from actually building in this space:

1) Guardrails must be clinical-first, not legal-first: The three things that matter most aren't what most legislation covers. They are: detecting a mental health crisis before the user articulates it clearly, escalating gracefully to a human without breaking the conversation, and not hallucinating on health information. Everything else is window dressing. Purpose-built clinical AI — designed with actual clinical psychologists and evidence-based protocols — shows dramatically better safety outcomes than general LLMs with safety prompts bolted on.

2) The guardrail has to hold over time: This is the one that worries me most. Safety systems that weaken in long conversations are not safety systems — they're a liability. Any health AI deployed at scale must be stress-tested for longitudinal interactions, not just the first exchange.

3) Human escalation is not a failure state. It's the product: The best AI health tools know exactly when to step back. They define which clinical decisions the AI may suggest, which require human review, and which require immediate hand-off. Graceful escalation — where the user doesn't have to repeat themselves and the next human in the loop already has context — is a design feature, not an edge case.

Let me give you a concrete example from our own product.

Sabai — the care companion we've built at SabaiHealth — operates with hard-coded escalation triggers. If a user mentions a snakebite, the system doesn't attempt to help. It doesn't ask follow-up questions. It immediately tells the user to get to a hospital. If someone expresses suicidal ideation, same response — no counselling attempt, no extended conversation, an immediate and unambiguous directive to seek emergency support.

These are non-negotiable. The AI doesn't try to be helpful in that moment. It tries to keep someone alive.

But the triggers go deeper than the obvious. A headache persisting for seven days. Chest tightening combined with palpitations and breathlessness on climbing stairs. These aren't dramatic emergencies — but they're clinical signals that GeniusCare is designed to recognise and respond to with one clear message: you need to see a doctor.

Not because we're being overly cautious. Because that's the right answer, and we know it.

This is the distinction that matters most in AI health design: the difference between a care companion and a clinician. GeniusCare is explicitly, unapologetically, the former. It is not a doctor. It will never pretend to be one. The moment a conversation enters territory that requires clinical judgment, the system steps back — clearly, gracefully, and without ambiguity.

That clarity is itself a guardrail. And it's one most AI health products are still getting wrong.

4) Equity-focused guardrails, not Western-default guardrails: AI models trained predominantly on Western clinical data will carry bias into every interaction with an Asian or African user. This isn't hypothetical — the OECD AI Incidents Monitor has documented cases where AI systems deprioritized Black patients because they used healthcare costs as a proxy for medical need. The same structural bias exists across language, geography, and socioeconomic status. Guardrails that don't account for this aren't guardrails. They're dressed-up discrimination.

What Responsible Building Actually Looks Like

I want to be honest about something, because the industry is full of founders who aren't.

Health apps are not social media apps. The data we hold isn't preferences or browsing history. It's symptoms. Medications. Mental health disclosures. Chronic conditions. Conversation memory built over weeks and months of a person's most vulnerable moments. A leak of this data isn't a PR problem. It's a life-altering event for the person on the other end of it.

That responsibility shaped how we built.

We didn't flip a switch and launch Sabai to the world. We ran a beta testing phase from November 5 through January 31. Then a full month of stability testing through February. Real users — paying, active, real — only came in from March 1.

Four months of building before we let the world in.

And even now, we don't claim to be the best. We claim to be learning. Every week, we are fine-tuning our model based on real interactions, real feedback, real edge cases we didn't anticipate. Because here's the uncomfortable truth about AI that most founders won't say out loud:

Hallucinations will happen. The question is whether they're minor or catastrophic.

A care companion that occasionally suggests a slightly suboptimal sleep habit is a very different failure from one that tells someone their persistent chest pain is stress. The former is manageable. The latter is a patient safety event.

That's why we've built our feedback loop directly into our community structure — through what we call our Founders Club. These are real users who engage with us, flag inconsistencies, and help us close the gap between what the model says and what it should say. Not a passive bug-reporting function. An active co-creation mechanism. The people most affected by the product are the ones helping us make it safer.

We've also gone further than most early-stage health AI companies do. We commissioned an independent academic validation — a researcher from Cyprus benchmarked Sabai head-to-head against Gemini and ChatGPT across health response quality and safety. We brought in an AI security tester to probe our system for vulnerabilities. And we've been working through our PDPA compliance — Thailand's Personal Data Protection Act — because the markets we serve have real regulatory obligations, not just Western ones.

Did we come out of all of this with a perfect score? No. And we never claimed we would. What we came out with was clarity on where the gaps are. And a structured roadmap to close them. Our next layer of work focuses on three levels of improvement that every serious health AI team should be thinking about:

Prompt-level: How the system can be manipulated through the way questions are asked, and building the resilience to handle that without breaking or going off-script.

Engineering-level: The underlying architecture: how data flows, where it's stored, how the model behaves under adversarial conditions.

Environment-level: The deployment context itself: the APIs, the messaging platforms, the third-party integrations. Every layer of the stack is a potential surface area.

Most AI health startups will not do this work until a regulator forces them to. We'd rather do it now, when the cost of fixing it is low and the cost of getting it wrong is a user's safety. This is a responsibility that most consumer health apps are not treating seriously enough. We are.

The Architecture of Accuracy

There's another dimension to guardrails that goes beyond safety escalation and data security: the quality of the answer itself.

When a general LLM is asked a complex health question, it draws on its training data — which is broad, often outdated, and not clinically validated. That's fine for writing emails. It's not fine for health guidance.

This is why Sabai is built on a multi-LLM plus health layer architecture. Alongside our core language models, we integrate health-specific sources — PubMed research databases and Health DeepSeek R1 — which provide clinically grounded context for complex medical questions.

Here's a real example of why this matters.

If a user asks: "Can I take peptides if I'm on statins for cholesterol?" — that's not a simple question. It touches on drug interaction risk, individual metabolic profile, and the difference between evidence-based supplementation and wellness marketing. A general LLM might give a plausible-sounding answer. A health-layer-augmented system gives a grounded one — drawing on clinical literature, not just pattern-matched text.

And crucially, even then, Sabai doesn't pretend to be definitive. The guardrail here isn't just what the system says. It's how it says it — clearly positioned as a care companion giving informed context, not a clinician giving a prescription.

That balance — rigorous enough to be useful, humble enough to know its limits — is the hardest thing to build in health AI. It's also the most important.

This is what responsible AI health development looks like in practice. Not "we've solved it." But "we've built the system to keep improving, and we're honest when it hasn't."

The Opportunity in the Gap

Here's the other side of this.

We're building Sabai for exactly these populations — delivered via LINE, WhatsApp, and Telegram because that's what people in Thailand, Indonesia, and Malaysia actually use. And building for underserved populations with real clinical guardrails isn't just the ethical choice. It's the competitive moat.

The companies that treat guardrails as a compliance checkbox will get caught out — by regulators, by users, and eventually by the market. The companies that treat guardrails as product design will build the only health AI that underserved populations will trust enough to actually use.

Trust is the product.

Everything else is features.

The Closing Thought

In insurance, I used to say that the worst risk is the one that looks like someone else's problem. Systemic bias in AI health tools looks like a Western regulatory debate. It's not. It's a Southeast Asian deployment crisis happening right now, mostly in silence.

The guardrails conversation is finally getting serious.

But it needs to get global. Because the populations most at risk from poorly designed AI health tools are the same ones most in need of AI health access. We don't get to build for one without being accountable to the other.

Disclaimer: Sabai by SabaiHealth is a care companion — not a doctor. It is designed to support, inform, and guide users towards better health habits and timely professional care. Nothing communicated through GeniusCare constitutes formal medical advice, diagnosis, or treatment. Users experiencing medical emergencies, persistent or worsening symptoms, or any condition requiring clinical judgment should consult a licensed healthcare professional immediately.

Rohit C. Nambiar is Co-Founder of SabaiHealth, building Sabai — an AI-powered care companion for underserved communities in Southeast Asia. He is the author of The Simplicity Trap (Notion Press, 2026).