Discussion about this post

User's avatar
Graham dePenros's avatar

Kevin,

I enjoyed that read. I was on an Advisor contract with a medical startup last year; for the sake of an NDA, I cannot be detailed. However, I have learned that when efficiency replaces judgment and trust, the cost is not some theoretical outcome; it is, as you say, real lives.

Efficiency is useful, but it’s no substitute for judgment, experience, or trust. Just look at the last person you hired who is great at the tasks and has all the skills but lacks judgment, experience, or trust. Fatal in a startup. Fatal in medical use cases: when a poorly architected AI agent or agentic AI system is trained on erroneous corpora, it starts issuing dangerous advice or diagnoses.

When systems optimise for speed and throughput over human context, it is incredibly dangerous. There are more than a few willing to set aside concerns about negative outcomes in favour of a payday.

This is a warning we should take seriously across all sectors

Best,

Graham.

John Groenveld's avatar

Did you make it to the MAHA moms conference this week? Does RFK Jr's fan club want their sons and daughters to grow up to be doctors and nurses? What recommendations did (would) you (Doctors Without Jobs) have for HHS.GOV?

TIA

John

groenveld@acm.org

6 more comments...

No posts

Ready for more?