There’s a version of audit defense that makes it sound almost like a search problem. A risk-adjusted code needs supporting documentation, so let’s say you point a capable AI model at the record and ask it to check and find the evidence behind the code. It returns with the relevant passage, say a note that pertains to a diabetes diagnosis. Problem solved, right?
It’s worth sitting with why the problem of audit defense is not solved here, a gap that could prove expensive to discover. The passage a retrieval system hands back may well be the one most relevant to the code, but as we know audit defense does not turn on relevance alone. It turns instead on sufficiency, in other words whether the documentation actually clears the MEAT criteria bar, the requirement that a condition be Monitored, Evaluated, Assessed, or Treated. An ICD code listed in the Assessment and Plan such as "E11.40 - Type 2 diabetes mellitus with diabetic neuropathy, unspecified" is obviously relevant yet entirely insufficient, and the difference between coding and the medical analysis that defends the coding is the whole problem. The AI model must perform an act of reasoning that aligns with that of a human auditor, and there’s the rub: The real question shifts from whether the system can find supporting text (it can) to how this system earns predictable confidence that the text it found is genuinely defensible. Approaching an answer to this question means breaking it down into three hard parts.
The first hard part is defining sufficiency at all, since MEAT is not simply an extractable data point. Consider a note that records a wound measuring three centimeters. The wound is a fact (a measurement capture) perhaps captured by a nurse, yet it is not, on its own, evidence that anything was Monitored, Evaluated, Assessed, or Treated. The MEAT lives one level up, in what the clinician did with the measurement. Perhaps the clinician judged it stable and chose to watch it, or they read it as deterioration and changed the plan. That reasoning may surface elsewhere, and the system must grade toward not a keyword or lab value but an act of clinical judgment.
The second hard part is that the AI doing the evaluation, as noted, is itself non-deterministic. Large language models do not return the same reading twice as a matter of course; give one the same patient and the same data, and its judgment can shift between passes. We test and measure this variability all the time in our reliability testing of our own AI engine. How do you control the error rate on an instrument that won’t hold still? Before confidence questions can even be asked, variance in the AI grader has to become something you yourself can reason about.
Thankfully, there are ways to do it. Rather than relying on the model alone, one must construct the analysis with several independent signals in such a way that there is more than one path to catching what a single pass would miss. For example, we can aggregate a median across repeated runs instead of trusting a standalone generation. We can write prompts that are clinically opinionated and validated by physicians rather than generic. We can reshape both prompts and outputs over successive design-partner cycles to better replicate real-world clinical workflows. This last step has proven to be critical to our overall performance, with double-digit swings in quality percentage improvement ordinarily occurring. Importantly, none of this makes the model deterministic, but it does make it steady enough to be regarded as accountable. And yet, essential is the fact that a human-in-the-loop always remains necessary.
The third hard part is the one that turns the modeling problem into a problem of epistemics: You cannot measure with perfect accuracy how often the system may be wrong because humans themselves can be non-deterministic. In other words, there is no perfect answer key against which to score a false-positive rate. For example, there is a dearth of objective, written consensus on exactly how auditors weigh what they see. Even in audit regulations where the rules are spelled out in explicit procedures, the posture of the individual reviewer measurably shapes the outcome, while clinical documentation itself may adhere to a softer standard. There is no divine oracle of reference to match on, meaning confidence remains always-constructed. One must evaluate the system's judgments against expert clinical review by tracking them against real audit determinations where they can be had and by treating agreement as something you build evidence for rather than a number read from a benchmark.
When these three hard parts come together, they explain why the two error types cannot be balanced the usual way, and this is where the key factor of design actually lives. A false positive ("Meets MEAT") on documentation that does not actually meet it is a code waved through into a population where an auditor may later find it and claw money back. A false negative ("Doesn’t meet MEAT) on documentation that actually does is perhaps less alarming but is also not free. A false negative could mean legitimate, defensible revenue left on the floor, and clinician time spent re-evaluating a code that was fine. The tempting move here is to borrow the clinic's own vocabulary and quote a sensitivity and a specificity. But those terms presuppose a gold standard (a disease definitively present or absent) and the third hard part mentioned above is precisely that no such oracle exists for auditor judgment. What can be measured honestly is agreement with expert clinical review, concordance with real audit outcomes, and the stability of the instrument itself.
That is what the categories are actually for: not grades on a curve, but routing decisions. At the poles, the machine's confidence is strong in both directions. Documentation with substantial evidence of clinical judgment stands as defensible, and a bare "E11.40 - Type 2 diabetes mellitus with diabetic neuropathy, unspecified" (the label with nothing behind it) stands just as confident in flagging an audit concern. In between are two different reasons to bring in a person, and they are worth keeping distinct. Sometimes the instrument is uncertain: repeated passes disagree, and the reading itself is shaky. And sometimes the chart is genuinely borderline, where no better model would eliminate the need for judgment. Calibration is the key: to make it steady enough to hold accountable, honest enough about the difference between relevant and sufficient, and disciplined enough to hand genuinely ambiguous cases to a person instead of guessing. The point of a human in this loop is not to rubber-stamp the model's output, but, importantly, to supply the one thing the whole apparatus is ultimately graded on—a point of view—at the exact moment the system is calibrated to know it needs one.
This level of design depth is unlikely to show up in a demo, where every retrieved passage looks like a match and every code looks supported. But in reality, where the rubber meets the road, it may show up months later, when a reviewer pulls two hundred charts out of thousands and reads them the way no relevance score ever can, asking whether the evidence was enough. The system built to look confident will do very well in the demo and very badly in the second room. The system built to know the difference between finding evidence and defending a claim, and between saying when it can’t and staying silent, is the one that holds up when an auditor is finally looking.