Stanford HAI recently published an important piece on why zero-shot clinical predictions are risky.
Their argument is about calibration: generative models of patient records behave more like simulators than predictors, and their outputs need rigorous evaluation before they are trusted as predictions. We’d take the point one step further. The deeper risk is irreversibility.
Same model, different standing. The output looks similar either way, which is exactly what makes the distinction easy to lose and expensive to recover.
When a model makes clinical predictions without task-specific grounding or calibration, its outputs may look plausible, but the assumptions that shaped them can’t be corrected after the fact. Once those assumptions are embedded, inference-time fixes don’t restore clinical defensibility.
That risk isn’t unique to zero-shot models.
It’s the same structural failure we see in post-hoc anonymization pipelines in healthcare data: once learning, linkage, or inference has already occurred, masking identifiers or adding controls downstream cannot undo the assumptions formed upstream.
In both cases, systems appear compliant or reasonable - until someone asks why a decision was made.
That’s the question modern clinical AI systems increasingly struggle to answer.
Stanford’s article is a valuable contribution to this conversation, and it reinforces a broader point the healthcare data community needs to grapple with: plausibility is not reversibility, and outputs are not provenance.
(For those interested, we’ve been exploring where these failures originate at the data and decision-infrastructure level in a short article series.)
Ray Diwakar writes on provenance, decision infrastructure and governed execution in regulated life sciences. StromaOS is the system these arguments led to.