Electronic Vindication

Dilbert by Scott Adams, courtesy www.dilbert.com

My very first blog post as Doctor Dalai went online on 1/29/2005, beginning with a rant about ScImage lifted from one of my early AuntMinnie.com entries. In the subsequent fourteen years, my writing has become slightly more sophisticated (emphasize slightly), but my basic premise has not wavered: PACS interfaces by and large are not as user-friendly as they could be, in other words, they SUCK. Forgive the epithet, but that is a very accurate statement, and most would agree.

Most, but not the vendors. In browsing through my years of navel-gazing, I came across several entries wherein the makers of some of these gawd-awful products pushed back. A 2005 message from a long-defunct company called ITL stood out:

We appreciated the time, albeit brief, that Dr. Dalai afforded us, and respect his radiology expertise. While we are disappointed and disagree with his conclusions, it underscores the traditional distinction between image management products and information system solutions. Given his experience with, and knowledge of, particular image management/storage systems this is not surprising. While we don’t question the sincerity of his intentions, it does not provide license for him to make incorrect pronouncements as fact when commenting on RIS/PACS in general, and ITL in particular.

One could argue that PACS is in its own way a boutique product; after all, according to the JACR, there are something like 32,000 radiologists in the United States, barely a rounding-error on the 1,030,000 number of physicians here. Thus, my fourteen years of whining and moaning about PACS has had a little traction among radiologists and some of the vendors, but not much beyond that. PACS just doesn't rise to the threshold of attention of the wider audience.

Enter the Electronic Health (or Medical) Record, the EHR (or EMR). Consider this a superset of medical information, including not only the patient's imaging studies, but literally everything else: physician notes, lab results...anything and everything medical. Done correctly, the idea is grand, and I've supported it in the past. Quite vocally, in fact. I was even very much in favor of a National electronic health record, believe it or not. Think of it...your personal records are stored in the cloud, accessible from anywhere. What an incredible contribution to patient care, if done right. And therein lies the rub.

Remember the little 2005 rant from ITL quoted above? The response from the owner of the little imaging center that bought the damn thing was quite telling:

I've shared your thoughts with the ITL group that we met with on May 4, 2005. The enclosed attachment is their response to your concerns. Based on their response as well as the points I've listed below, I feel that I must go forward with the purchase of this product:
     * I've signed a contract
     * (Imaging Center) has the opportunity to save a significant amount of money vs. other PACS systems.
     * The product is on a single platform which has benefits that none of the others have.
     * Several of the Radiologists in (your) group that I've asked about time saved, said 20% to 40% (over reading from film). If you weigh that against 4 or 5 calls a day until they get used to it, your group comes out way ahead.

This scenario represents a microcosm of the big, bad EMR world out there. Those who don't understand what we do, and don't care, have inflicted much larger, more dysfunctional, and literally deadly software upon the entire medical field. And guess who suffers?

Let me give you the punchline in case you have something more important to do than read my musings (and I should hope you do!)...As with the majority of PACS systems that I have bemoaned over the past 14 years of blogging, the EMRs are generally not purchased by the end-users, and thus they are written for the CTO and IT department, and for other administrative types. They demonstrate limited, if any, real understanding of the physicians that use them for the life-and-death business of patient care. I've been accused of being a whiny, arrogant doctor for spouting that opinion, but please do show me where I'm wrong. You can't. But it makes perfect economic sense. Why bother writing software to make nasty doctors happy when they don't buy or even maintain it? All they do is use it.

The dysfunctional EMR's have been (rightly) blamed for everything from physician-burnout to diminished patient care, to actual deaths of patients. I've written about them a number of times (the link is to my 2016 post), and we are seeing article after article in the medical and general literature exposing the EMRs for the disappointments they really are.

You've probably heard of Dr. Atul Gawande, surgeon, author, Renaissance Man, and all-around brilliant guy. He recently wrote a nice piece on this topic in the New Yorker, "Why Physicians Hate Their Computers," and he pulled no punches:

But three years later I’ve come to feel that a system that promised to increase my mastery over my work has, instead, increased my work’s mastery over me. I’m not the only one. A 2016 study found that physicians spent about two hours doing computer work for every hour spent face to face with a patient—whatever the brand of medical software. In the examination room, physicians devoted half of their patient time facing the screen to do electronic tasks. And these tasks were spilling over after hours. The University of Wisconsin found that the average workday for its family physicians had grown to eleven and a half hours. The result has been epidemic levels of burnout among clinicians. Forty per cent screen positive for depression, and seven per cent report suicidal thinking—almost double the rate of the general working population.

Something’s gone terribly wrong. Doctors are among the most technology-avid people in society; computerization has simplified tasks in many industries. Yet somehow we’ve reached a point where people in the medical profession actively, viscerally, volubly hate their computers. 

Dr. Gawande picked up as well on the fact that doctors were being pushed aside in the decision process:

Many of the angriest complaints, however, were due to problems rooted in what Sumit Rana, a senior vice-president at Epic, called “the Revenge of the Ancillaries.” In building a given function—say, an order form for a brain MRI—the design choices were more political than technical: administrative staff and doctors had different views about what should be included. The doctors were used to having all the votes. But Epic had arranged meetings to try to adjudicate these differences. Now the staff had a say (and sometimes the doctors didn’t even show), and they added questions that made their jobs easier but other jobs more time-consuming. Questions that doctors had routinely skipped now stopped them short, with “field required” alerts. A simple request might now involve filling out a detailed form that took away precious minutes of time with patients.

He's nicer about the description than I was, but you get the idea. As I quoted in my own linked article, an insider from the BIG EMR vendor acknowledges that we docs were pretty much cut out of the loop. Dr. Gawande seems to have been luckier than most of us, though that is relative:

People wonder, "why aren't EHRs designed with providers in mind?" I've worked at Epic and can tell you why not:

• Physicians were on staff, but hard to reach. They were technophiles and barely practicing as others mentioned. 

• It really is a billing platform with some patient stuff tacked on. 

• Everything useful you see is probably a workaround and one level away from not working at all. 

• Quality Assurance (manual testers) are supposed to be a surrogate for users, as there is no beta testing. They are intentionally hired without CS background and maintained as laymen with a very lite, monkey-see-monkey-do training. However if they are not lickety-split quick to master the software, they are fired. Quality Assurance ends up being more like Self-Reassurance. 

• There is absolutely no testing of interoperability. There is however plenty of testing for the several convoluted ways of sharing data between Epic servers.

Should you doubt the potential for harm from these things, let me refer you to "Death By 1,000 Clicks: Where Electronic Health Records Went Wrong," by Fred Schulte of Kaiser Health News and Erika Fry from Fortune Magazine:

More gravely still, a months-long joint investigation by KHN and Fortune has found that instead of streamlining medicine, the government’s EHR initiative has created a host of largely unacknowledged patient safety risks. Our investigation found that alarming reports of patient deaths, serious injuries and near misses — thousands of them — tied to software glitches, user errors or other flaws have piled up, largely unseen, in various government-funded and private repositories.

In the case of a particularly bad product from a company called eCW.

The eCW spaghetti code was so buggy that when one glitch got fixed, another would develop, the government found. The user interface offered a few ways to order a lab test or diagnostic image, for example, but not all of them seemed to function. The software would detect and warn users of dangerous drug interactions, but unbeknownst to physicians, the alerts stopped if the drug order was customized. “It would be like if I was driving with the radio on and the windshield wipers going and when I hit the turn signal, the brakes suddenly didn’t work,” said Foster.

And by the way, you aren't even supposed to know about these potentially fatal glitches:

Compounding the problem are entrenched secrecy policies that continue to keep software failures out of public view. EHR vendors often impose contractual “gag clauses” that discourage buyers from speaking out about safety issues and disastrous software installations — though some customers have taken to the courts to air their grievances. Plaintiffs, moreover, say hospitals often fight to withhold records from injured patients or their families. Indeed, two doctors who spoke candidly about the problems they faced with EHRs later asked that their names not be used, adding that they were forbidden by their health care organizations to talk. Says Assistant U.S. Attorney Foster, the EHR vendors “are protected by a shield of silence.”

...KHN and Fortune examined more than two dozen medical negligence cases that have alleged that EHRs either contributed to injuries, had been improperly altered, or were withheld from patients to conceal substandard care. In such cases, the suits typically settle prior to trial with strict confidentiality pledges, so it’s often not possible to determine the merits of the allegations. EHR vendors also frequently have contract stipulations, known as “hold harmless clauses,” that protect them from liability if hospitals are later sued for medical errors — even if they relate to an issue with the technology.

So how does a poorly functioning piece of crap become entrusted with critical patient information? As usual, the government ruined what should have been a great and wonderful thing:

...10 years after President Barack Obama signed a law to accelerate the digitization of medical records — with the federal government, so far, sinking $36 billion into the effort — America has little to show for its investment. KHN and Fortune spoke with more than 100 physicians, patients, IT experts and administrators, health policy leaders, attorneys, top government officials and representatives at more than a half-dozen EHR vendors, including the CEOs of two of the companies. The interviews reveal a tragic missed opportunity: Rather than an electronic ecosystem of information, the nation’s thousands of EHRs largely remain a sprawling, disconnected patchwork. Moreover, the effort has handcuffed health providers to technology they mostly can’t stand and has enriched and empowered the $13-billion-a-year industry that sells it.

David Blumenthal, who, as Obama’s national coordinator for health information technology, was one of the architects of the EHR initiative, acknowledged to KHN and Fortune that electronic health records “have not fulfilled their potential. I think few would argue they have.”

...Seema Verma, the current chief of the Centers for Medicare & Medicaid Services (CMS), which oversees the EHR effort today, shudders at the billions of dollars spent building software that doesn’t share data — an electronic bridge to nowhere. “Providers developed their own systems that may or may not even have worked well for them,” she told KHN and Fortune in an interview last month, “but we didn’t think about how all these systems connect with one another. That was the real missing piece.”

This government-participation movie has played out in another country I've heard about (which shall remain nameless), where any public entity, like a medical school, is required to buy equipment and service from one particular company...which has absolutely no idea what it is doing, and is responsible for a good number of idle diagnostic machines. While the US government doesn't go this far, there is certainly enough money flowing to help us understand how we were delivered Pintos for the price of Mercedes.

And there was perhaps a more sinister thought behind the rapid governmental push for EMRs. Betsy McCaughey, former Lieutenant Governor of New York, writes:

Obama’s high-tech guru, Dr. David Blumenthal, wanted top-down control of the treatment decisions doctors make. Doctors have to follow computer prompts or be punished ­financially with lower Medicare payments. Blumenthal predicted doctors would resist the loss of ­autonomy, but he also promised it would result in better care and savings from fewer unnecessary tests and treatments. He was right about doctors resisting but wrong about everything else.

Stanford University researchers examining hospital records from six states, including New York and California, found that hospitals complying with the electronic records mandate don’t have better patient outcomes or survival rates.

As with PACS, the designers of the interfaces have absolutely no clue about physician workflow. Let's see...what would we like to do with and EMR? How about order a Tylenol for Mrs. Jones in Room 973? Good luck. Schulte and Fry continue:

In a study published last year in the journal Health Affairs, Ratwani and colleagues studied medication errors at three pediatric hospitals from 2012 to 2017. They discovered that 3,243 of them were owing in part to EHR “usability issues.” Roughly 1 in 5 of these could have resulted in patient harm, the researchers found. “Poor interface design and poor implementations can lead to errors and sometimes death, and that is just unbelievably bad as well as completely fixable,” he said. “We should not have patients harmed this way.”

Using eye-tracking technology, Ratwani has demonstrated on video just how easy it is to make mistakes when performing basic tasks on the nation’s two leading EHR systems. When emergency room doctors went to order Tylenol, for example, they saw a drop-down menu listing 86 options, many of which were irrelevant for the specified patient. They had to read the list carefully, so as not to click the wrong dosage or form — though many do that too: In roughly 1 out of 1,000 orders, physicians accidentally select the suppository (designated “PR”) rather than the tablet dose (“OR”), according to one estimate. That’s not an error that will harm a patient — though other medication mix-ups can and do.

Oops.  I personally think getting a suppository is traumatic and harmful, but that's just me. Atul Gawande asked a colleague about ordering things in her EMR:

“Ordering a mammogram used to be one click,” she said. “Now I spend three extra clicks to put in a diagnosis. When I do a Pap smear, I have eleven clicks. It’s ‘Oh, who did it?’ Why not, by default, think that I did it?” She was almost shouting now. “I’m the one putting the order in. Why is it asking me what date, if the patient is in the office today? When do you think this actually happened? It is incredible!” The Revenge of the Ancillaries, I thought.

Once again...those who built these monuments to bad technology did not ask the people who USE their little darlings how they should work...because the end-users do not buy these incredibly expensive systems. And our beloved United States Government was a BIG part of the problem:

The effort to digitize America’s health records got its biggest push in a very low moment: the financial crisis of 2008. In early December of that year, Obama, barely four weeks after his election, pitched an ambitious economic recovery plan. “We will make sure that every doctor’s office and hospital in this country is using cutting-edge technology and electronic medical records so that we can cut red tape, prevent medical mistakes and help save billions of dollars each year,” he said in a radio address...

In the depths of recession, the EHR conceit looked like a shovel-ready project that only the paper lobby could hate. In February 2009, legislators passed the HITECH Act, which carved out a hefty chunk of the massive stimulus package for health information technology. The goal was not just to get hospitals and doctors to buy EHRs, but rather to get them using them in a way that would drive better care. So lawmakers devised a carrot-and-stick approach: Physicians would qualify for federal subsidies (a sum of up to nearly $64,000 over a period of years) only if they were “meaningful users” of a government-certified system. Vendors, for their part, had to develop systems that met the government’s requirements.

They didn’t have much time, though. The need to stimulate the economy, which meant getting providers to adopt EHRs quickly, “presented a tremendous conundrum,” said Farzad Mostashari, who joined the ONC as deputy director in 2009 and became its leader in 2011: The ideal — creating a useful, interoperable, nationwide records system — was “utterly infeasible to get to in a short time frame.”

That didn’t stop the federal planners from pursuing their grand ambitions. Everyone had big ideas for the EHRs. The FDA wanted the systems to track unique device identifiers for medical implants, the Centers for Disease Control and Prevention wanted them to support disease surveillance, CMS wanted them to include quality metrics and so on. “We had all the right ideas that were discussed and hashed out by the committee,” said Mostashari, “but they were all of the right ideas.”

One major promise of the EMR is interconnectivity. I should be able to set foot in any hospital in the world, or at least in the United States, and have my data available instantly. But there was one small impediment to that ideal:

The notion that one EHR should talk to another was a key part of the original vision for the HITECH Act, with the government calling for systems to be eventually interoperable.

What the framers of that vision didn’t count on were the business incentives working against it. A free exchange of information means that patients can be treated anywhere. And though they may not admit it, many health providers are loath to lose their patients to a competing doctor’s office or hospital. There’s a term for that lost revenue: “leakage.” And keeping a tight hold on patients’ medical records is one way to prevent it.

As usual, those who think like the government, at least the government in place at the time, never, ever, EVER consider the unexpected consequences that human nature will inevitably provide. In this case, the possessiveness (dare I say greed?) of hospitals (and, yes, doctors) scuttled what should have been a given. And people have been harmed, even died, because one system would not talk to another. I guess the patient's home systems considers it a draw if the poor fellow dies or is treated by a competitor.

What is almost as sad as the loss of a patient is the loss of a soul. More and more physicians are burning out, losing their calling, losing their souls, precisely because of this disgusting situation. How did THAT happen? Gawande again:

Adaptation requires two things: mutation and selection. Mutation produces variety and deviation; selection kills off the least functional mutations. Our old, craft-based, pre-computer system of professional practice—in medicine and in other fields—was all mutation and no selection. There was plenty of room for individuals to do things differently from the norm; everyone could be an innovator. But there was no real mechanism for weeding out bad ideas or practices. Computerization, by contrast, is all selection and no mutation. Leaders install a monolith, and the smallest changes require a committee decision, plus weeks of testing and debugging to make sure that fixing the daylight-saving-time problem, say, doesn’t wreck some other, distant part of the system...

But those processes cannot handle more than a few change projects at a time. Artisanship has been throttled, and so has our professional capacity to identify and solve problems through ground-level experimentation. Why can’t our work systems be like our smartphones—flexible, easy, customizable? The answer is that the two systems have different purposes. Consumer technology is all about letting me be me. Technology for complex enterprises is about helping groups do what the members cannot easily do by themselves—work in coördination. Our individual activities have to mesh with everyone else’s. What we want and don’t have, however, is a system that accommodates both mutation and selection.

But selection is winning over mutation. Schulte and Fry:

The numbing repetition, the box-ticking and the endless searching on pulldown menus are all part of what Ratwani called the “cognitive burden” that’s wearing out today’s physicians and driving increasing numbers into early retirement.

In recent years, “physician burnout” has skyrocketed to the top of the agenda in medicine. A 2018 Merritt Hawkins survey found a staggering 78 percent of doctors suffered symptoms of burnout, and in January the Harvard School of Public Health and other institutions deemed it a “public health crisis.”

One of the co-authors of the Harvard study, Ashish Jha, pinned much of the blame on “the growth in poorly designed digital health records ... that [have] required that physicians spend more and more time on tasks that don’t directly benefit patients...”

“Everything is so cumbersome,” said Dr. Karla Dick, a family medicine physician in Arlington, Texas. “It’s slow compared to a paper chart. You’re having to click and zoom in and zoom out to look for stuff.” With all the zooming in and out, she explained, it’s easy to end up in the wrong record. “I can’t tell you how many times I’ve had to cancel an order because I was in the wrong chart.”

Neither an EMR or a PACS is a product of rocket science. They are basically large databases, with alphanumeric entries of notes and labs to be found in the "chart" side, and images to be found on the 'ology side, X-Rays, CT scans, even images of skin lesions (Dermatology) and histopath slides (Pathology). This shouldn't be hard. But it is because of the huge amounts of money involved, as well as the buffoonish, bulldozing intervention of a (supposedly) well-intentioned government.

This is a macrocosm of the PACS world, but even more intimately involved in the patient's care. And with even greater potential to harm the patient. Or kill him. Schulte and Fry quote Dr. Ratwani for the last word of this sad missive:

In a study published last year in the journal Health Affairs, Ratwani and colleagues studied medication errors at three pediatric hospitals from 2012 to 2017. They discovered that 3,243 of them were owing in part to EHR “usability issues.” Roughly 1 in 5 of these could have resulted in patient harm, the researchers found. “Poor interface design and poor implementations can lead to errors and sometimes death, and that is just unbelievably bad as well as completely fixable,” he said. “We should not have patients harmed this way.”

Let me repeat that for emphasis:

“Poor interface design and poor implementations can lead to errors and sometimes death, and that is just unbelievably bad as well as completely fixable. We should not have patients harmed this way.”

Which is what I've been saying about PACS for a very long time.

The "Heart" Of Darkness?

Black holes are the stuff of science, science fiction, and even fantasy, but they are real. Until now, however, no one has ever seen one. Until now:

Image courtesy PBS.com

From PBS.com:

On Wednesday, the Event Horizon Telescope released the first-ever image of a black hole — a historic moment shared by scientists spread across seven simultaneous news conferences around the world. As the PBS NewsHour reported Tuesday, the Event Horizon Telescope is a two-year-old, international collaboration bent on capturing direct pictures of black holes:

Two years ago, an international collective of scientists joined forces to take pictures of two black holes located at the centers of galaxies.

Scientists at eight radio telescopes observatories — stretching from Hawaii to Greenland to the French Alp to Antarctica — captured images of one black hole in our Milky Way — known as Sagittarius A* — and one in a nearby galaxy called M87, over the course of a week in April 2017. (The PBS NewsHour visited one in Chile when the project was still under discussion).

By linking together, the scientists created, in essence, a planet-sized telescope built to scan massive parts of the skies.

The concept of black holes has captivated scientists for two centuries. Despite decades of indirect evidence supporting their existence, black holes have never been captured by camera — until now. Scientists hope to use the image to probe the origins of our universe.

My friend Stacey saw this before I did and immediately made the comparison to a myocardial perfusion scan, in this case demonstrating an anterior wall defect:

Image courtesy JACC.com

Truly, the "Heart of Darkness"! Ah, if we could only look into the soul with as much insight...

The images of Pluto had already suggested to me that G-d plays games with us:

Image courtesy plutosafari.com

Now I know it's true!

Sudbury Is Still Waiting...

Bill Crumplin, with photo of his late wife Donna Williams, image courtesy Sudbury.com

You might recall my post of a few years ago about Sudbury, Ontario, Canada, a medium-sized town that turned to its citizens to fund a PET/CT facility when the much-touted Health Service would not provide it. The wait-time was not just an inconvenience; the health of Sudbury citizens was adversely affected by the lack of local scanning capability.

The wait goes on, it seems.

My friend Stacey discovered another such tragedy related to imaging, or rather, lack thereof. As reported on Sudbury.com:

Donna Williams’ dying wish was to raise money toward the purchase of a piece of medical equipment that would have helped doctors diagnose her illness sooner and perhaps changed the course of her treatment. And her life.

Williams, 54, died April 6, 2016, of cardiac amyloidosis, a disorder caused by deposits of an abnormal protein in heart tissue, making it difficult for the heart to function properly.

The Sudbury woman was scheduled to have a cardiac MRI in Ottawa, but wasn’t healthy enough to travel there and sadly died two weeks after being diagnosed.

Bill Crumplin said he and Williams, an exceptionally bright woman, did not make it to their fourth wedding anniversary before she succumbed “brutally fast” to the rare disease. Had she been able to have a cardiac magnetic resonance imaging (the full name of an MRI) scan in January 2016 rather than March, the outcome might have been different.

{snip}

SN is operating its one MRI machine 24 hours a day, 365 days a year, completing about 13,000 scans a year. Still patients must wait an average 52 days for a test that should be performed within the provincial target of 28 days.

{snip}

Health Sciences North Foundation executive director Mary Lou Hussak is leading the drive to raise more than $6 million for the scanners. The foundation is focusing all its efforts right now on raising the $3.5 million for the purchase and installation of a second MRI, then $2.6 million to replace the first.

So far, $1.1 million has been raised, without campaigning, thanks to a $500,000 donation from Carmen and Sandy Fielding and another so far anonymous $500,000 donation.

Hussak has no doubt residents of Sudbury and the Northeast will give generously to the MRI fund. While some have made huge donations, all amounts, large or small, will be gratefully accepted.

People can make individual or company donations, or hold small events such as birthday or tea parties or dinners in their home where friends can donate to the cause. Hussak encourages people thinking of holding such events to contact the foundation because it has resources that could make that easier.

{snip}

Meanwhile, construction continues at HSN’s Ramsey Lake Health Centre on the suite to accommodate a combined positron emission tomography/computed tomography scanner. There was a short delay in the early days, said Hartman, but efforts are being made to make up for lost time. The hospital estimates the suite will be finished in the spring and do the first PET scan this summer.

Oh, myyyyyyy....

So...it has taken three years to get the PET/CT scanner we heard about it 2016? Lovely. And the citizens of Sudbury have to take up collections and hold tea-parties to fund another MRI so critical patients (I have to say it) don't DIE waiting for their scans? The article neglects to mention the possibility of bake sales and selling Sally Foster gift-wrap, both big revenue-producers when my kids were in school. 

Houston, I mean Sudbury, we have a problem. 

The Canadian system is the quintessential  bureaucracy, and as such keeps meticulous records. In this case, wait-times are accessible via a Provincial website, http://www.hqontario.ca. Read 'em and weep:

I really don't need to add much of an editorial comment here, do I? But that never stops me, so I'll simply repeat more or less what I said about Sudbury's PET/CT in 2016, and it remains accurate in my humble opinion:

The healthcare here in the United States is good. Really, really good. People who can afford to do so come HERE to be diagnosed and treated. Those who can afford to come HERE do NOT go to Canada. Or Germany. Or Switzerland. Or Sweden. Or India. Or Anywhere Else. Even Mick Jagger, a British Citizen, is having a heart-valve replacement tomorrow in New York City, and not London. I'm not saying I contribute much, if anything, to it, but the best medicine in the world is practiced in the United States. I don't think our system is perfect, and I have called for massive reforms, dumping Obamacare, Medicare, Medicaid, and revamping the whole thing including a huge revision of the laws governing the Third-Party Payers, our beloved insurance companies. Ours is a dysfunctional system and it has to be revised. But even with its blemishes, it produces some of the best medicine in the world.

And yet, in our inexorable march toward Socialism, the latest incarnation of which is called "Medicare for All", led by demagogues who would have us trade everything that makes America America for "free stuff", American health care stands to be crippled in the process of making it "free". And by the way, nothing is "free". Someone has to pay for the "free stuff".

We need to look at the case of imaging in Sudbury and understand the implications. There isn't enough money in Ontario to provide enough "free" scanners for Sudbury, and in fact, funds for this life-saving technology are limited province-wide. Thus, the fine people of Sudbury will have to fund the purchase themselves. That they are willing to do so to help their fellow citizens is the heartwarming part of this story. The heartbreaking aspect is that they have to do so. I see this as a huge crack in the perfect facade of Canada's single payer system. It FAILED the people of Sudbury. And it fails the people of Ontario, and indeed all of Canada by limiting resources and thus rationing their care.

The profit motive has brought an overabundance of expensive CT, MRI, and PET/CT scanners to the United States, with the inherent likelihood of over utilization. My town of less than 200,000 may well have as many scanners as all of Ontario; certainly our small, relatively poor state has a significant multiple of the technology. Contrast this to the neighbourly situation in Ontario which requires the citizens themselves to raise money for life-saving technology that the government, even using the generously given tax-dollars (CAD) cannot provide.

This is not the answer to fixing healthcare in the United States. Sorry.

How IBM Watson Overpromised and Underdelivered on AI Health Care

My friend Phil Shaffer, a fellow retired Nuclear Radiologist, is an avid poster on Aunt Minnie. His AM post today about AI in general and Watson in particular is worthy of a wider audience, and here you are. It is based on an Engineering article in the IEEE Spectrum: How IBM Watson Overpromised and Underdelivered on AI Health Care . This is a cautionary tale for all who have anything to do with AI...If IBM stumbled in this venue, if IBM could fall victim to hype and hubris...

  
Well, we all knew that. Big hype, zero output.

I wouldn't bother to post this non-news, if it were not for the other questions it brings up.

IBM’s bold attempt to revolutionize health care began in 2011. The day after Watson thoroughly defeated two human champions in the game of Jeopardy!, IBM announced a new career path for its AI quiz-show winner: It would become an AI doctor. IBM would take the breakthrough technology it showed off on television—mainly, the ability to understand natural language—and apply it to medicine. Watson’s first commercial offerings for health care would be available in 18 to 24 months, the company promised.

In fact, the projects that IBM announced that first day did not yield commercial products. In the eight years since, IBM has trumpeted many more high-profile efforts to develop AI-powered medical technology—many of which have fizzled, and a few of which have failed spectacularly. The company spent billions on acquisitions to bolster its internal efforts, but insiders say the acquired companies haven’t yet contributed much. And the products that have emerged from IBM’s Watson Health division are nothing like the brilliant AI doctor that was once envisioned: They’re more like AI assistants that can perform certain routine tasks.

In part, he says, IBM is suffering from its ambition: It was the first company to make a major push to bring AI to the clinic. But it also earned ill will and skepticism by boasting of Watson’s abilities. “They came in with marketing first, product second, and got everybody excited,” he says. “Then the rubber hit the road. This is an incredibly hard set of problems, and IBM, by being first out, has demonstrated that for everyone else.”

The diagnostic tool, for example, wasn’t brought to market because the business case wasn’t there, says Ajay Royyuru, IBM’s vice president of health care and life sciences research. “Diagnosis is not the place to go,” he says. “That’s something the experts do pretty well. It’s a hard task, and no matter how well you do it with AI, it’s not going to displace the expert practitioner.” (Not everyone agrees with Royyuru: A 2015 report on diagnostic errors from the National Academies of Sciences, Engineering, and Medicine stated that improving diagnoses represents a “moral, professional, and public health imperative.”)

In many attempted applications, Watson’s NLP struggled to make sense of medical text—as have many other AI systems. “We’re doing incredibly better with NLP than we were five years ago, yet we’re still incredibly worse than humans,” says Yoshua Bengio, a professor of computer science at the University of Montreal and a leading AI researcher. In medical text documents, Bengio says, AI systems can’t understand ambiguity and don’t pick up on subtle clues that a human doctor would notice.

Both efforts have received strong criticism. One excoriating article about Watson for Oncology alleged that it provided useless and sometimes dangerous recommendations (IBM contests these allegations). More broadly, Kris says he has often heard the critique that the product isn’t “real AI.” And the MD Anderson project failed dramatically: A 2016 audit by the University of Texas found that the cancer center spent $62 million on the project before canceling it. A deeper look at these two projects reveals a fundamental mismatch between the promise of machine learning and the reality of medical care—between “real AI” and the requirements of a functional product for today’s doctors.

Watson learned fairly quickly how to scan articles about clinical studies and determine the basic outcomes. But it proved impossible to teach Watson to read the articles the way a doctor would. “The information that physicians extract from an article, that they use to change their care, may not be the major point of the study,” Kris says. Watson’s thinking is based on statistics, so all it can do is gather statistics about main outcomes, explains Kris. “But doctors don’t work that way.”

At MD Anderson, researchers put Watson to work on leukemia patients’ health records—and quickly discovered how tough those records were to work with. Yes, Watson had phenomenal NLP skills. But in these records, data might be missing, written down in an ambiguous way, or out of chronological order.

In a final blow to the dream of an AI superdoctor, researchers realized that Watson can’t compare a new patient with the universe of cancer patients who have come before to discover hidden patterns

If an AI system were to base its advice on patterns it discovered in medical records—for example, that a certain type of patient does better on a certain drug—its recommendations wouldn’t be considered evidence based, the gold standard in medicine. Without the strict controls of a scientific study, such a finding would be considered only correlation, not causation.

The question this raises in my mind is: Why?

It seemed so intuitive that this would work. Why doesn't it?

One thing that happens when you try to apply computers to any problem is that first you must break down the task and understand completely how humans do it. I think that what we are seeing is that there was a very incomplete understanding of how humans process information. Starting with a naive understanding of this, IBM brazenly predicted success. And failed. Miserably.

  

Another important point is that much of our scientific effort is reported as statistical differences, derived from controlled experiments. But this is NOT the way that medicine works. There is another level, as Luke Oakden-Rayner has pointed out.

He points out - convincingly - that experiments are NOT clinical performance.

Medical AI today is assessed with performance testing; controlled laboratory experiments that do not reflect real-world safety.

Performance is not outcomes! Good performance in laboratory experiments rarely translates into better clinical outcomes for patients, or even better financial outcomes for healthcare systems.
Humans are probably to blame. We act differently in experiments than we do in practice, because our brains treat these situations differently.

Even fully autonomous systems interact with humans, and are not protected from these problems. We know all of this because of one of the most expensive, unintentional experiments ever undertaken. At a cost of hundreds of millions of dollars per year, the US government paid people to use previous generation AI in radiology. It failed, and possibly resulted in thousands of missed cancer diagnoses compared to best practice, because we had assumed that laboratory testing was enough.

The unintentional experiment he references is Breast CAD.

He recounts how the initial studies suggested that there would be 20% more cancers found using CAD, however subsequent VERY LARGE studies showed (in one case) a 20% increase in biopsies for an increase in cancers found from 4.15 per 1000 to 4.20 per thousand (p = NS).

His diagnosis:

People are weird. It turns out that if you run an experiment with doctors being asked to review cases with CAD, they get more vigilant. If you give them CAD and make them use it clinically, they get less vigilant than if you never gave it to them in the first place.
There are a range of things going on here, but the most important is probably the laboratory effect. As several studies have shown [5, 6], when people are doing laboratory studies (i.e., controlled experiments) they behave differently than when they are treating real patients. The latter study concluded:

“Retrospective laboratory experiments may not represent either expected performance levels or inter-reader variability during clinical interpretations of the same set of mammograms”

Which really says it all.

He goes on to say that when people use computers they over value what computer input and under value the other evidence:

This effect has been implicated in several recent deaths in partially self-driving cars – it has been shown that even trained safety drivers are unable to remain vigilant in autonomous cars that work most of the time.

This effect has also been directly cited as a possible reason for the failure of mammography CAD. One particularly interesting study showed that using CAD resulted in worse sensitivity (less cancers picked up) when the CAD feedback contained more inaccuracies [8] (pdf link). On the surface this didn’t make a lot of sense, since CAD was never meant to be used to exclude cases; it was approved to highlight additional areas of concern, and the radiologists were supposed to use their own judgement for the remainder of the image. Instead, we find that radiologists are reassured by a lack of highlighted regions (or by dismissing incorrectly highlighted regions) and become less vigilant.

I’ve heard many supporters of CAD claim that the reason for the negative results in clinical studies is that “people just aren’t using the CAD as it was intended,” which is both accurate and absurdly naive as far as defenses go. Yes, radiologists become less vigilant when they use CAD. It is not surprising, and it is not unexpected. It is inevitable and unavoidable, simply the cost that comes with working alongside humans. 

There you go. Some food for thought.  

Rad-PAss: The Robot Radiology Physician Assistant

For the past many years, radiologists have lived in fear of AI. There are those who complain as well about radiological mid-levels, nurse-practitioners, physician assistants, etc., thinking that they, too, will encroach upon our territory. As you know from my previous entries on the subject, I personally don't buy into those fears. Artificial Intelligence and human helpers represent nothing more than assistants for us, ways to do our jobs better and faster. They will not replace us, I promise you.

As with most things in this business, technology marches on, and we see progress of a sort we could not even imagine a few years ago. I am proud to announce today that thanks to a consortium of industry and intellectual powerhouses, including IBM, Facebook, the University of Maryland, the ACR, Johns Hopkins University, US Robotics, Harvard University, and the Sirius Cybernetics Corporation, we now have the next step in the evolution of our profession, The Robot Radiology Physician Assistant:

It was only a matter of time until technology caught up to our needs, and here you see it in the flesh, er, silicon. We needed a non-threatening piece of machinery that could at once process images and help with those mundane radiological chores such as slinging barium, sticking needles, and shoving catheters. The Robot Radiology Physician Assistant, affectionately known as Rad-PAss, is the embodiment of these needs, a jack-of-all trades, of whom WE are the master. 

Rad-PAss's training is second to none, having been fed the entire body of radiological literature, textbooks, journals, videos of CME meetings in Hawaii, and even the dozen or so remaining pieces of film that have not yet been rendered down for their silver content. He went through "residency" with famed University of Maryland radiologist and AI guru Dr. Eliot Siegel, which took Rad-PAss about an hour. After this, Dr. Siegel certified his performance, also noting that he was much more personable than many human residents.

Rad-PAss can be deployed in multiple configurations. Here, we see him set to interface with the latest edition of GE's Universal Viewer PACS. With some luck, it will remain operational for the 3 milliseconds Rad-PAss requires to analyze today's workload. He will then assign preliminary reports to be reviewed and corrected if necessary by the radiologist. (So far, Rad-PAss has made no errors in interpretation, but we never say "never" in this business.) The robot can be loaded with any and all of the latest AI algorithms, which he will self-evaluate delete those he finds unsuitable or beneath his dignity.  

If you prefer, Rad-PAss can sit beside you in the reading room and look over your shoulder, just like the lawyers and some of your partners do, but he is on YOUR side!!

Of course, the most exciting possibilities presented by the Rad-PAss technology are to be found in the realms of fluoroscopy. Imagine turning over the drudgery of barium studies to an assistant that does not have to worry about radiation exposure! The modified delivery apparatus includes pressure sensors to optimize patient comfort, ummmmm, well, minimize discomfort.

And for the IR guys, how about a catheter-jockey assistant that can pounce on a bleeder faster than you can? Here ya go:

Suffice it to say, a brave new world dawns TODAY in Radiology. Do mark this date down in history!



Legal disclaimer: Robot images courtesy of ESET.com. Neither Dr. Siegel nor any of the institutions mentioned above have anything to do with this, or me, and are included here solely in a fictional manner. And if you didn't already realize that, you haven't checked the date of this entry.