A New Way to Spot Missed Problems When Patients Return to the Emergency Department

Why return visits are hard to sort through

Patients who leave the emergency department often come back soon after. Hospitals watch these return visits closely. A quick return can signal a missed problem. It can also mean nothing more than a new complaint. Sorting the useful signals from routine noise is hard.

Most hospitals focus on returns within 48 to 72 hours. That window is common because it is easy to define. It also seems likely to catch serious failures. But real problems do not always appear inside that short span. A patient may return on day five or ten with a complication that began earlier. If the review stops at three days, those cases can slip away.

The workload makes the problem worse. Even in one high-volume emergency department, a very narrow rule still produced many records to check. The rule looked only at returns within 48 hours that ended in hospital admission. That should sound selective. It was not enough. Reviewers still faced a large pile of charts. Many did not contain an actionable quality problem.

This is why simple counting fails. A screening rule must be precise. In plain terms, it needs to flag few harmless cases. If too many flags are false alarms, reviewers lose time and trust. The goal is not to catch every return visit. It is to find the small group where a closer look may matter.

What clinicians notice when only diagnosis labels are visible

When clinicians saw only the main diagnosis from two emergency department visits, their judgments split quickly. One physician marked 21 of 99 pairs as worth a closer look. Another chose 11. A third urgent care clinician chose just 7. The same labels led to very different conclusions.

The biggest clue was how serious the return-visit diagnosis looked. Reviewers rated each label on a scale from mild to severe, using a typical case as the middle point. When that seriousness score rose, clinicians were more likely to say the pair needed follow-up. In one analysis, each small step up in seriousness raised flagging odds by about 7 percent. The number was an odds ratio. Each one-point rise multiplied the odds of a flag by about 1.07. That is not a 7 percent chance. It is a small multiplier that adds up. Over a wide range, it becomes a strong signal.

Clinicians also watched whether the second diagnosis could have been an alternative explanation for the first visit. If a chest pain return looked like it might have been heart disease all along, that raised concern. They likewise asked whether the later label was a complication of the earlier one. A fever after a procedure or a clot after a leg injury fits that pattern. These links mattered, but the clearest signal was medical gravity. How bad the return-visit problem seemed on its own drove decisions most consistently.

How an AI screen was built after chatbot answers proved too broad

When a chatbot was asked to judge emergency-department return visits, it said almost everything needed follow-up. It marked 94% of the pairs as concerning. Human reviewers were far more selective. One flagged about one in five cases. Another flagged about one in nine. The third flagged only about one in fourteen. That gap showed a general chatbot could not be left to make the call.

So the design moved from asking for an opinion to building a filter. Clinician judgments pointed to two useful signals. The first was how serious the return diagnosis looked on its own. The second was whether it fit as a possible alternative explanation or complication of the first visit’s diagnosis. These ideas were turned into a machine-readable map. The map is a knowledge graph. It links diagnoses that can plausibly point to a missed problem, a delayed diagnosis, or a worsening condition. An AI model helped fill in those links by repeatedly asking about diagnoses and their relationships.

For each pair, the screen checks whether the two diagnoses appear as a potentially concerning connection in that graph. It also looks at the seriousness of the return diagnosis using a gravity score built into the map. Only pairs that pass both tests are sent for human review. The cutoffs were set to make the list small and useful rather than exhaustive. In effect, the screen does not ask one open-ended question about a pair. It asks narrower questions. Is there a known risky relationship? Is the return diagnosis serious enough? If yes to both, it flags the case for closer look.

How well the new screen caught concerning return visits

The final screen was deliberately narrow. It did not try to catch every possible problem. It tried to flag only the pairs most likely to deserve a clinician’s attention. That choice shows up in its results. Among 99 return-visit pairs, clinicians judged 28 to be concerning enough for further review. The screen flagged far fewer than that. At one setting, it raised six flags; five were real concerns and one was not. At a stricter setting, it raised four flags, and all four were real concerns. In plain terms, when the screen said something looked wrong, it was almost always right. That is what makes it useful for busy reviewers: each alert carries weight.

The trade-off is that many concerning pairs slipped past. At the looser setting, it caught about one in six clinician-identified problems. At the stricter setting, it caught fewer than one in five. It missed most of them. But the comparison matters. Traditional review often looks only at returns within two or three days. The screen’s alerts included visits that happened later. At the looser setting, four of the five true flags were six- or seven-day returns. At the stricter setting, all four true flags fell in that later window. Those are exactly the cases a short-window review would never see.

So the screen did not replace human judgment. It acted as a high-confidence filter. It could add a small number of extra charts to review while keeping most alerts meaningful. Its weakness was coverage: it saw only a slice of possible problems. Its strength was reliability: when it spoke, clinicians had reason to listen.

A narrow net that may catch more

The central lesson is that precision can matter more than volume in quality review. Emergency departments already face too many return visits to examine closely. Most are harmless, but some hide a missed diagnosis or a complication. The old habit of checking only the first two days leaves later problems invisible. A screen built from diagnosis labels alone found a small set of pairs that looked genuinely concerning. It did not try to flag everything. That restraint is its strength.

When the screen flagged a pair, it was usually right. At one setting, five out of six flags matched clinician concern. At another, all four did. Several of those cases would have fallen outside the usual 48-72-hour window. That suggests a real opportunity: review more return visits without drowning reviewers in false alarms. The method also uses information already present in routine records. It can be explained in terms clinicians understand, such as how serious a diagnosis is and whether one condition plausibly follows another.

The obstacles are just as important. The test set was small, drawn from one health system. Clinicians did not always agree on what deserved follow-up. The screen missed many concerning pairs because it was deliberately conservative. It also saw only primary diagnoses, not the full story in a chart. Direct chatbot answers were too broad and needed to be replaced by a more structured approach. Before this could become routine care, larger tests are needed across different hospitals, specialties, and patient groups.

The promise is not that machines will replace clinical judgment. The promise is that they may point human eyes toward the few cases most likely to need them. If quality review can widen its window without widening its burden, what else might be found in the visits now left unexamined?