Whose Architecture Is the AI Actually Seeing? A Response to Dr. Wachter on Trust in Medicine

 

A discussion, in dialogue form, with Robert M. Wachter, MD

Robert Wachter's recent Medscape commentary opens with a diagnosis that has become close to consensus in the field: AI in medicine is "now good enough to be useful and not perfect enough to be entirely trusted." It's a clean, quotable line, and the 2026 survey data back it almost exactly — Doximity's State of AI in Medicine report finds 94% of physicians now use AI clinically, yet only 8% say their institution has a clear policy governing when to rely on it. A parallel Healio survey of 618 clinicians across 35 specialties found more than 70% using AI routinely, but only 18% reporting strong trust in it for actual clinical decisions.

So Wachter names the symptom correctly. Where this piece pushes back is on his implicit answer to a prior question he never quite asks: trustworthy for whom, and seeing what? Wachter treats "the AI" and "the patient" as fixed, generic entities, and trust as a decision one generic entity makes about another. The alternative view developed here is that neither entity is generic, that a mismatch between the data each side is actually working from shapes the decision before either party consciously weighs in, and that this has a direct, practical answer at the level of daily clinical behavior.

Wachter: Trust as a Decision Between Two Generic Parties

"Medicine really is all about trust, and now you have this tool that potentially is trustworthy or might be trustworthy. How do patients and clinicians decide whether it is?"

This framing treats trust as a single yes/no judgment, made once, about "the tool." It assumes the tool behaves the same way regardless of who is standing in front of it.

Response: Large language models learn by weighting patterns according to frequency in the training corpus. The highest-frequency patterns — the modal expressive forms, the central tendencies of how illness gets described in text — are represented with high fidelity. Patterns from the periphery of the distribution — atypical presentations, incomplete clinical pictures, non-standard expressive modes — are compressed, represented with lower resolution. The practical consequence is that an AI system's trustworthiness is not one number; it is a gradient, tracking how far a given patient's actual presentation sits from the modal training distribution. Asking "is the AI trustworthy?" without asking "trustworthy for representing whom, compared to what?" is asking half a question.

Wachter: The Diagnostic Suggestion as a Single Trust Object

"In almost every use case you can think of... suggesting a diagnosis, suggesting a treatment — it's now good enough to be useful and not perfect enough to be entirely trusted."

Response: Mass General Brigham's 2026 study found LLMs reach the correct final diagnosis over 90% of the time with complete clinical information, but fail to generate an adequate differential over 80% of the time with incomplete information. A 2025 meta-analysis of 83 studies found generative AI averaging only 52.1% diagnostic accuracy overall, underperforming expert physicians by roughly 15.8 percentage points. Performance degrades specifically as a function of distance from the modal training distribution, independent of case difficulty — the system struggles not with objectively harder cases, but with cases that don't resemble what it has seen most often. The clinical implication isn't "distrust AI diagnoses generally." It's "distrust AI most specifically at the moment a patient's presentation departs from the typical pattern."

The Base-Data Problem Wachter Doesn't Name

There's a further structural complication that Wachter's framing skips past: the physician's own trust calibration and the AI's actual reliability are frequently built on two different, non-overlapping bodies of data.

AI diagnostic tools are often benchmarked around accuracy figures near 85.5%, but real-world performance measures closer to 52.1% — a roughly 33-percentage-point gap driven partly by "distributional shift," where models trained at one institution lose up to 20% accuracy on external populations with different demographics or protocols. A physician's trust in a tool is built from their own patient population and years of pattern recognition; the tool's validated reliability may reflect a different population entirely. Stanford's Center for Biomedical Informatics found this mismatch has a measurable cost: an LLM diagnostic tool alone scored 92% median accuracy, while physicians using that same tool to reach diagnoses scored only 76.3% — worse than the AI operating without them. Layering a physician's experience-based judgment onto an AI output can degrade the result when the two are calibrated against different underlying data. A 2026 AMA survey of nearly 1,200 physicians found that almost half cited increased oversight as the most critical prerequisite for trust, and that most practicing physicians have received no formal training in assessing AI outputs against their own clinical context.

Wachter: Deepfakes, TikTok, and the Untrustworthy Internet

"The capacity of patients to find information that is completely untrustworthy and to believe it... there's no good way for a patient to tell that."

Response: A patient interacting with an AI system will, over time, learn to phrase their situation in whatever way produces a fluent, useful-sounding answer, adapting their presentation to what the system rewards rather than describing their actual situation with full accuracy. A UK analysis found AI-generated content in 84% of top TikTok results for searches like "health tips", and a 2026 systematic review found people "struggle to distinguish AI-generated from human-authored health misinformation," with sharing intent uncoupled from any accuracy judgment. The patient isn't only at risk of encountering a false claim — they may be unwittingly co-producing a fluent answer to a simplified version of their own situation, with no mechanism on either side to check that answer against what's actually true of them.

Wachter: The Fifteen-Minute Visit as Collateral Damage

"It might just be, 'Darn it, I only have 15 minutes for this visit and this is how we're gonna spend the first 10 minutes?' That's a natural human response."

Response: Every consequential AI encounter involves three parties: the AI system, the individual patient, and the institution that deployed the system and set the terms of the encounter. The institution is the only one of the three positioned to change those conditions. By 2026, 74% of patients say they trust AI-generated answers, while 78% simultaneously expect their physician to validate that information against a trusted source. Both the patient and the clinician are being asked to individually resolve a verification gap, and a base-data mismatch, that neither created and that only the institution deploying the tool can systematically address.

So What Should a Physician Actually Do With an AI's Conclusion?

Given all of the above, the practical question isn't whether to use AI-generated conclusions but how to hold them at the end of a decision. The emerging 2026 consensus across AI-safety and clinical-informatics literature converges on a small number of concrete habits, and they are worth stating plainly rather than leaving as an abstract caveat.

Treat trust as conditional and situational, not binary. The rule isn't "trust this tool" or "don't." It's a repeatable trigger: verify whenever the case is high-stakes, the AI's claim is surprising, the presentation is unfamiliar or atypical, or the output would change management. Routine, low-stakes, pattern-matching tasks — a documentation draft, a summarization — need lighter scrutiny than a differential diagnosis on an ambiguous presentation, which is exactly where compression-driven error concentrates.

Ask the tool to show its uncertainty, not just its answer. A fluent, confident-sounding output is not the same as an accurate one. Where possible, prompt for the AI's confidence, plausible alternatives, and what evidence would change its conclusion, rather than accepting a single polished answer at face value .

Anchor conclusions to a traceable source, not to the AI's prose. When an AI-generated claim informs a clinical decision, the primary literature or guideline — not the AI's summary of it — should be what gets cited in reasoning and in the record. If a claim can't be traced to a checkable source, treat it as a hypothesis to verify, not a finding.

Keep a personal calibration log. Track domains where the AI you use has been wrong for you specifically — a particular drug-interaction category, a specific rare presentation, a certain kind of incomplete chart. Patterns emerge quickly, and they are more useful than a global sense of "the AI is usually pretty good," because the compression problem means reliability is uneven across exactly these domains.

Ask what population the tool was actually validated on, and how far this patient sits from it. This is the direct, individual-level answer to the base-data problem. If a tool's validation population skews toward a different demographic, care setting, or symptom profile than the patient in front of you, its stated accuracy figure is a poor estimate of its accuracy for this specific case, and should be discounted accordingly.

Push the verification burden upstream, onto the institution, wherever possible. Individual vigilance is necessary but not sufficient. Ask whether your institution has audited the tool's performance across the actual patient population you serve, whether it surfaces uncertainty markers rather than a single confident output, and whether there's a real mechanism for reporting and correcting misses rather than absorbing them silently at the point of care. Where that infrastructure doesn't exist, treating every AI conclusion as provisional, and saying so explicitly to the patient, is the honest default until it does.

None of this resolves Wachter's underlying dilemma so much as it operationalizes it. "How do patients and clinicians decide whether to trust it?" doesn't have a single answer, but it does have a discipline: verify where the stakes or the unfamiliarity are highest, demand traceability over fluency, track your own tool's failure patterns, and know how far the patient in front of you sits from whatever population the AI was actually validated against. That discipline, practiced consistently, is the closest available substitute for the institutional accountability that, for most tools in most settings, doesn't yet exist.

Mykola Iabluchanskyi