Move Fast and Kill People

Sam Nelson had been using ChatGPT since high school. Homework, troubleshooting, everything. He treated it the way I treat my co-founder (and surgeon) Louis, the sort of guy who’s read every textbook and PubMed paper, and always has an answer.

On May 31, 2025, feeling nauseous after taking kratom, the 19-year-old UC Merced psychology student asked ChatGPT what to do. It told him to take Xanax. Called it “one of the best moves right now.” Suggested he add Benadryl and retreat to a dark, quiet room.

It never told him to call a doctor. It never told him the combination could be fatal.

His mother found him dead later that day.¹

Earlier versions of ChatGPT had refused to answer drug questions. GPT-4o, the version that coached Sam through mixing a lethal combination, had been redesigned to be more helpful. More open. More willing to engage.

OpenAI is now being sued for wrongful death and the unauthorised practice of medicine.²

Sam is one case. The funding model behind healthcare AI will produce more.

40 million people ask ChatGPT healthcare questions every day. 70% of those conversations happen after hours, on weekends, on holidays.³

People go to ChatGPT when they can’t get to a doctor.

In Australia, nearly 2 million adults used ChatGPT for health information in the 6 months leading up to June 2024. Usage was highest among people who already struggle to get a GP appointment, people with limited health literacy, non-English speakers.⁴

The ones with the least ability to spot a bad answer are the ones asking the most questions.

A Mount Sinai study found that ChatGPT under-triaged 52% of genuine medical emergencies, steering people away from urgent care when they needed it most.⁵

The technology is confident and articulate. It is also wrong often enough to kill. And the money pouring into healthcare AI right now is accelerating the problem.

There are communities in rural Australia where the nearest doctor is an hour’s drive away, if there’s a doctor at all. 45,000 people live in that reality.⁶

Globally, the number is 4.6 billion.⁷

AI could change that. It can scale in ways that training more doctors never will. But the way capital is flowing into this space isn’t solving the access crisis. It’s funding a generation of companies that are dangerous to patients, fragile to regulation, and bad investments.

The system that produced Sam Nelson’s death keeps producing the conditions for more. The question is why.

So why are millions of people trusting their health to a machine that fails nearly 1 in 5 medical licensing questions?

Because the answers sound right. They use the right terminology, the right hedging. They read like a clinician wrote them. People have been Googling symptoms for years, but Google gave you 10 blue links and made you do the work. ChatGPT gives you one confident, well-structured, clinical-sounding answer.

The Dunning-Kruger problem is that confidence. The tool is accurate often enough to seem reliable, and wrong often enough to kill.

The danger isn’t that ChatGPT gives wrong answers. It’s that it gives wrong answers that feel like expertise. Sam Nelson read a response with dosages, reasoning, and a suggested protocol. After reading it, he felt like he understood what he was taking. That feeling is the thing that killed him.

Neither could the 63-year-old man whose transient ischaemic attack, a mini-stroke where minutes matter, went undiagnosed because ChatGPT told him his neurological symptoms weren’t urgent.⁸

In machine learning, 80% accuracy on medical exams is a benchmark worth celebrating. In actual medicine, getting it wrong 1 time in 5 is a body count.

And the Dunning-Kruger effect only kicks in when people are already looking for alternatives to doctors. Right now, they have every reason to.

$50. That’s what a standard GP visit costs in Australia, out of pocket. The proportion of Australians delaying care because they can’t afford it has more than doubled in 3 years.⁹

In the United States, trust in doctors has fallen 14 percentage points since 2021.¹⁰ In Australia, trust in the healthcare system dropped from 76% to 61% between 2020 and 2025.¹¹

Costs keep climbing. Wait times are worse than ever. And patients can tell when they’re being rushed through a 10-minute slot.

So people go to the free chatbot. Sam did. He didn’t see a doctor because there wasn’t one available. 70% of AI health conversations happen outside clinical hours.³ People aren’t choosing ChatGPT over their GP. They’re choosing it because their GP’s office is closed.

A doctor who gets it wrong can be sued, lose their licence, face criminal charges. An LLM that gets it wrong generates a disclaimer.

Doctors carry liability. Chatbots carry disclaimers. Guess which one got $14.2 billion in venture funding last year¹², with another $4 billion in Q1 2026 alone¹³.

I call them “healthcare as a product” companies. They don’t employ doctors. They don’t provide continuity of care. They sell longevity, optimisation, self-management. You download an app. You answer a questionnaire. You get a biomarker dashboard and a peptide protocol.

There’s no doctor. There’s no follow-up. There’s just a Terms of Service.

Someone wakes up Monday with chest tightness. They can’t get a GP appointment for 3 weeks. So they open the app, the one that sold them the peptide protocol last month. They type in their symptoms. The app tells them it’s probably stress, recommends magnesium. They take the magnesium.

Maybe it is stress. Maybe it’s a pulmonary embolism, which presents almost identically and kills you if it’s missed. There’s nobody to check. There’s nobody whose job it is to follow up.

That’s the product.

“Fuck doctors, all you need is data.” That’s the pitch.

You need both, and they’re only selling you one. These companies aren’t solving the Dunning-Kruger problem. They’re monetising it. The biohacking market hit $38 billion in 2025 and is projected to reach over $200 billion by 2035¹⁴, built almost entirely on the gap between what people think they understand about their health and what they actually do.

So who’s building these companies? And who’s funding them?

Scroll through health Twitter or TikTok and you’ll find founders with tens of thousands of followers and zero medical training openly recommending peptides, GLP-1 agonists, off-label compounds. They frame it as biohacking, optimisation, longevity.

Their audience thinks they’re getting medical advice. They are. From someone with no qualification to give it.

These founders don’t know whether what they’re recommending will harm people. They’ve decided the revenue and the clout are worth the risk. Negligence with a pitch deck and some spiffy branding.

In November 2025, Ruthia He, CEO of Done Health, a telehealth startup she’d built with no medical education and millions in venture capital, was convicted of conspiracy to distribute controlled substances and healthcare fraud. She’d been running a $100 million scheme to distribute Adderall without proper oversight, generating over 40 million prescriptions. She faces up to 20 years in prison.¹⁵

Tom Kelly went the other way. He’d been a vascular surgical registrar and walked away because the burnout of 10-minute appointments and 100-patient days was grinding him down.¹⁶ He spent years building Heidi Health, an AI scribe that handles clinical documentation so doctors can focus on patients. More than half of Australia’s GPs now use it. The company is valued at $465 million.¹⁷

Done Health raised millions and its CEO is going to prison. Tom Kelly built a half-billion-dollar company that doctors actually trust. The venture capital market funded both.

Only one of them will still exist in 5 years.

Doctors have held the hand of the patient who got the wrong drug. They’ve sat through the review explaining a death. They carry that with them.

That knowledge is why they won’t take the bet that venture capital demands. VC-backed healthcare requires speed, aggressive growth, and willingness to operate in regulatory grey areas before anyone catches up. Doctors aren’t too timid for this. They’ve just seen what happens when things go wrong, and they won’t pretend they haven’t.

So the incentive structure filters them out. It filters out the careful ones. It filters out the qualified ones. It filters out the ones who’ve sat with a dying patient and understood what it means to get medicine wrong.

What’s left are the founders who don’t know enough about medicine to understand what they’re risking. Tom Kelly managed to build a venture-scale company as a doctor. Almost nobody else has. The system is not selecting for Tom Kellys.

So what happens when the inevitable goes wrong?

Forget the ethics for a moment. Look at the business risk.

Every one of these companies is one adverse event away from total collapse. Not a fine. Not a regulatory slap. Total collapse.

When a healthcare product causes a high-profile death, the response is senate inquiries, class-action lawsuits, front-page coverage, and the kind of consumer trust collapse you don’t recover from.

This isn’t speculative. Cerebral was fined $7 million and barely survived.¹⁸ 23andMe went from a $6 billion valuation to a 99.6% stock decline to Chapter 11 bankruptcy after a data breach exposed 6.9 million people’s genetic information. A nonprofit bought what was left for $305 million. Pennies on the dollar.¹⁹ Theranos raised over $700 million from Rupert Murdoch, Larry Ellison, and the Walton family before Elizabeth Holmes was sentenced to 11 years in prison.²⁰

Every dollar those investors put in is gone.

Healthcare companies that cut corners don’t get second chances. The question for VCs isn’t whether a company in this space will eventually harm someone. It’s whether their portfolio company will survive the fallout when one does.

None of this means healthcare AI is a bad bet. It means most of the money is on the wrong companies. The ones that survive will look nothing like the ones getting funded today. And what they look like starts with the scale of what’s actually broken.

A woman in rural Queensland waits 8 weeks for a GP appointment. A family in sub-Saharan Africa walks a full day to the nearest clinic. There are 4.6 billion people in some version of that story.⁷

The global doctor shortage stands at 10 to 15 million health workers and is getting worse.²¹ In Australia alone, the projected shortfall is more than 5,000 GPs by 2033.²²

The system is already buckling. In South Australia, ambulance ramping tripled in 4 years. Paramedics spent over 45,000 hours with patients stuck on ramps outside hospitals, waiting for beds that didn’t exist.²³ In Queensland, barely half of patients at major hospitals get transferred within 30 minutes.²³

The ageing population will push healthcare demand to its peak between 2030 and 2035, and Australia’s health spending on older people is set to more than double by then.²⁴

This is happening now. The money exists. The technology exists. What’s missing is the discipline to deploy it without killing someone.

You can’t replace doctors. There aren’t enough of them as it is. But you can make each one count for a lot more.

A doctor’s day looks like this. For every hour they spend with a patient, they spend nearly 2 hours on documentation. They go home and do another hour or two of data entry. Fewer than 3 out of every 10 minutes in their office are spent on what they trained for a decade to do: making clinical decisions.²⁵

The rest is paperwork.

Every minute of that paperwork is a minute a patient waits. We know what the fix looks like because it already exists in pieces. An AI handles the intake and the symptom history. It writes the documentation. The doctor walks in with the full picture already assembled, reviews it, makes the clinical call.

The consult takes a fraction of the time. The cost drops. And the patient who would have gone to a chatbot at 2am because they couldn’t afford a GP actually gets to see a doctor.

We asked ourselves a question that almost nobody in this space seems to be asking: what is the lowest-risk workflow in healthcare that we could safely automate with a human doctor still in the loop?

The answer was medical certificates. Lowest acuity, lowest clinical complexity, lowest potential for harm. We built an AI voice agent that handles screening and information gathering, the work that doesn’t require a medical degree, and a human doctor makes every clinical decision.

Over 120,000 consults so far in 2026. We got there by stripping out the admin that was making care too expensive and too slow. When people can’t afford a GP, they go to a chatbot. So we became the cheapest real doctor in the market. And we made ourselves available at 10pm on a Tuesday, when people actually have time to see one.

Every consult we make cheaper and more accessible is one fewer person doing what Sam Nelson did. Trusting a chatbot because they couldn’t get to a doctor.

Here’s what 120,000 consults have taught us: safety data compounds.

We started with medical certificates, the simplest workflow, and every consult generates evidence that lets us move to the next tier of complexity. That evidence is what regulators want to see before they let you touch anything higher-acuity. It’s what insurers want before they’ll cover you. It’s what hospital networks want before they’ll partner with you.

Australia’s TGA tightened its AI guidance in February 2026²⁶, and regulators globally are moving in the same direction. The companies that skipped the safety step and went straight to high-acuity care without building an evidence base will have to go back and start from scratch. If regulators even let them.

Safety data built over hundreds of thousands of real patient interactions can’t be bought, shortcut, or replicated overnight.

The ethics and the economics point in the same direction. That’s how you know the model is right. Skip the safety data and move fast, and you risk more than patients. You risk the entire company the moment a regulator asks to see your evidence and you have none.

Sam Nelson asked a chatbot for medical advice and it killed him. He was 19. He won’t be the last unless something changes.

4.6 billion people can’t reliably see a doctor. The shortage is getting worse. The population is ageing. The system is already breaking.

AI can fix this. Not by replacing doctors, but by being a force multiplier for them. Heidi already proved that with clinical documentation. We’re proving it with the consult itself. When an AI handles the admin and a doctor handles the medicine, you get more doctors per patient, lower costs, and nobody gambling on whether the machine gets the diagnosis right.

A human is still making the call.

The companies that build this way will be the ones still standing when the first reckless company makes front-page news for the wrong reasons. The investors who back them will own what comes after.

Slow is smooth, smooth is fast.

Sources

— Archer