Wednesday, May 12, 2010

Blunder Down Under

Believe it or not, I usually don't like to get into the middle of a problem, especially someone else's problem.  However, my very limited fame in the PACS realm seems to have created a lightning-rod effect of sorts.  I've become the go-to guy for those who would need someone to look at their new product or idea, and I keep getting asked to join various boards and committees.  As long as they don't ask for a membership fee, I'll help where I can.  I'm reminded of this stanza from Fiddler on the Roof's "If I Were A Rich Man":

The most important men in town would come to fawn on me!
They would ask me to advise them,
Like a Solomon the Wise.
"If you please, Reb Tevye..."
"Pardon me, Reb Tevye..."
Posing problems that would cross a rabbi's eyes!
And it won't make one bit of difference if I answer right or wrong.
When you're rich, they think you really know!
When you declare yourself the Dalai Lama of PACS, I guess they think you really know, too. . .

One of the few things I do know is a bad situation when I hear about it.

Last October, I was invited to Australia to give a talk to some customers of healthinc, which distributes AMICAS in those parts.  I was fortunate enough to meet some great people, radiologists and otherwise.  Other folks who were not able to get to RANZCR started communicating with me via phone and email.  (As an aside, I'll never get used to the time-zone difference and how it impairs telephone communication, and I'll never quite get over the fact that it's usually "tomorrow" for my friends on the other side of the world when I speak with them.)

The story I'm about to tell is true, and you may cringe when you hear the details I'm about to discuss.  Those who related this tale of woe asked to remain anonymous, and so they shall.  I've been in touch with quite a few of them, and my Skype bill reflects that, even at only a few cents per minute.  These are physicians trying to care for their patients, and they should be lauded for stepping up and speaking out about what could be a really bad should it continue unabated.

I will endeavor to avoid pointing fingers at low level bureaucrats and IT types... Well, I said I would endeavor to avoid; I didn't say I wouldn't. . .

Australia is made up of a number of states, some quite large in territory, but small in population.  New South Wales, which is where Sydney is to be found, has over 7 million citizens, and an area of 309,432 square miles.  Texas, by comparison, has a population of about 25 million people, and an area of 262,000 square miles.  Western Australia, wherein we find Perth, has a population of only 1.85 million souls spread over an area of 975,920 square miles.  Population density is thus quite similar to Alaska which has 700,000 people spread over 572,000 square miles. (Figures from Wolfram-Alpha, if you were wondering.)

Australia's healthcare system is an interesting blend of public and private, partially socialized, if you will, and the people I met over there by and large are happy with their care.  The public hospitals are run by the government, as you would expect, although there is a relatively larger number of outpatient imaging centers than seen here in the States, owned and run by radiologists or by private or corporate entities.  I had the opportunity to visit a few of these in Sydney, and I found them quite efficiently run with equipment that would rival many hospital departments over here.

When the government runs things, the government RUNS things. We in the U.S. need to heed this simple fact.  IT (being IT, and even more so when backed up by the government), tends to dig in its heels on certain things.  When it was time for PACS selection around 2003, about the time I was getting into the business myself, the WA Department of Health selected Agfa as their vendor.  As our hospital also found, Agfa at the time was the lesser of the evil "monolithic" PACS companies, and WA's IT types, just like ours, couldn't fathom going with a small company, as they might get bought out (rather prescient of them given the joys I'm now anticipating with AMICAS).  In a familar scenario, the WA PACS control group made the selection without much input from the radiologists who would use the darn thing.  The rads almost immediately took a dislike to the earlier iterations of Agfa, but they were told they would have to live with it, as a forklift upgrade to something else would be prohibitively expensive, and supposedly would take two years to accomplish even if it was approved, and there was no chance of that happening anyway.  Requests to allow access to Intelerad installations, which most had in their private offices, were often rebuffed with the statement that "you already have PACS here, and that should suffice" even though the most widespread access within the hospitals utilized the elderly, slow, cantankerous, and basically unusable Agfa Web 1000 (may it never darken my screen again as long as I live!!!)

So, the radiologists lived with Agfa in the hospitals, but the vast majority went with Intelerad for their own private offices.  Often, they loaded the InteleViewer client onto the Agfa workstations to provide access to their patients' data, and the programs coexisted peacefully.  However, the IT folks were not amused with this cohabitation, and at some hospitals ultimately wiped the program off of the Agfa workstations, and blocked its reinstallation.  Not to worry, the resourceful radiologists started carrying the software in on USB keys.  So, naturally, Agfa support blocked the loading of the foreign PACS clients (sounds scary) from USB drives.  Clever. The reason given, of course, was that the Intelerad software was crashing the Agfa computers (after a clean uneventful run of several years), and of course, the old tried-and-true excuse that loading extra software voids the warranty.  I've looked into the latter, and it simply isn't true.  The Agfa warranty (found here) says absolutely nothing about loading other software; you just aren't supposed to fool around with Agfa's programs. That's all. However, this is the U.S. version, and I'm told the Australian version might be more onerous.  The rads at one hospital were threatened with disciplinary action should they be so foolish as to try loading their software again.  What would IT do, take away Solitaire and Free Cell?  Taking away their access to PACS would be a little over the top.  But still, I must encourage everyone involved to follow the rules.  Hospitals ultimately had to buy extra PC's to accomodate Intelerad and other clients for the private imaging providers.

As noted, the Western Australia PACS contract was awarded to Agfa 7 years ago, and the contract is up for renewal (or retendering as it is called down there) in 2013.  In the meantime, the WA Department of Health's Health Information Network or HIN (which used to be called InfoHealth) recently awarded a contract to Agfa to create a statewide PACS and RIS, via restricted tender.  There was apparently no competitive bidding, although I'm told some other vendors made the trip thinking they had a chance at this large project.  The total value of the contract was $8,481,044 Australian (fixed price) (that’s $7,613,000 USD for those challenged in currency conversion).

And now, dear readers, comes the punchline to this walkabout.  IMPAX 6.4 was installed a few weeks ago throughout Western Australia, in the "largest international rollout" of the product.  (I wonder if the US is considered international to Agfa, a Belgian company.)  The problem?  Quite simply,…didn’t quite work as expected...

I've received a litany of complaints the likes of which I wouldn't have believed had I not heard them from multiple sources. Perhaps the paramount complication is that the darn workstations crash.  A lot.   The entire system has had significant down time, often for several hours at a stretch.  On occasion, the rads have had to read CT's directly off the console.  One site reported a seven-day backlog in CT reporting.

Even when it works, (granted, the majority of the time), it is problematic.  Some of the snags were rather severe with respect to patient care, such as images that don’t match patient names and/or records, and the occasional digital mammogram that was displayed with right and left reversed. More often than not, study lists don’t load previous cases in chronological order.

IMPAX 6 has a worklist that should automatically refresh, but sadly, it seems to crash a lot when refreshing, so the automatic function had to be disabled.  This has resulted in a number of duplicate dictations as cases don't automatically disappear from the list after being read. The speech recognition system (I could have told them not to go there!!) has a bad habit of randomly rewinding to the beginning of a dictation, and then overwriting the report that was already 10 minutes in the making. Reports are lagging for days, even ED reporting which impacts patient care. The modality worklist is failing intermittently, forcing the technologists to manually enter patient demographics, one of the error-prone procedures PACS was supposed to eliminate.  Training was abysmal to non-existent and on-site support equally lacking.

I was provided with a list of over 40 documented problems, and this number grows daily.  All of the complaints describe problems I would consider quite serious including downtimes of four hours or longer. Keep Dalai's Laws of PACS in mind:  PACS IS the radiology department, and when PACS is dead, so is the hospital.

The WA Health System is already under significant pressure. The PACS system over the last 2-3 years has become part of the problem, and certainly not part of the solution, and the recent troubles only make this worse. The workflow is incredibly clumsy compared with private practice PACS systems in WA.  The Senior Consultant Radiologists, many of whom have worked with multiple different PACS systems in private settings, have found the system frustrating, and the majority of Consultant Surgeons and Physicians are equally underwhelmed at the efficiency of PACS. What once was a tool to improve efficiency throughout WA Health, has now become a stumbling block to most clinical pathways.

Out of all this comes one positive:  The Impax 6.4 interface is a major improvement for remote viewing over groady old Web 1000.

One positive, and dozens of problems.  As the kids say these days, OMG!  This is unbelievable.  Patient care is being negatively affected in a manner that is totally beyond belief.   By the way, at least one hospital has had to declare a "Code Yellow" because of the PACS slowdowns; this means that the hospital has experienced a disaster and has had to stop accepting patients.  Ouch.  Keep this all in perspective.  This isn't just one clinic or hospital with a cranky PACS.   This is an entire state, and the problem affects multiple sites.

How did all this happen?  I'm not sure I can answer that with certainty, and pointing the finger of blame isn't going to be useful at this late date.  Suffice it to say that there might not have been enough planning, and that the vendor does not seem to have devoted adequate personnel to the job.  Whatever the reason, the system is in place, and even a tsunami wouldn't budge it.  So, it needs to be brought up to working status.  Somehow.

Despite all the grief I give Agfa, or maybe because of it, our system is functional and is operational the vast majority of the time.  I don't generally like the way it functions, but we do pump a tremendous number of studies through the darn thing every day.  In other words, my Agfa hospitals are not having difficulties to the severity described above. Thus, we know the system can be made to work.

As the Western Australia upgrade to IMPAX 6.4 is brand new (they had IMPAX 5 before this that at least stayed up most of the time), there is only one entity that has the ability, and the obligation, to fix it, and that is Agfa.  So, guys, fix it. Seriously.  This is a very bad, very serious, and very dangerous situation.  I'm putting out the pubic call to Agfa:  Send whatever resources you must, send whomever you must (you can't have my Agfa PACS Goddess, but you can have me if I can help, and not somehoe end up mysteriously lost in the Outback), and send them quickly, please. Western Australia needs you now.  Get this repaired before your reputation, and possibly patients, are further harmed.  Do what you need to do.

I'll stay in touch with my friends Down Under, and I'll report back as soon as something changes.  In the meantime, if you are in the market for a PACS, this is probably another vendor that you should watch.  And by that I mean wait and see how this is handled before deciding to go forward with this vendor, if they should happen to be on your short list.  Funny, I said the same thing just a week ago concerning AMICAS as well and I’m only at the beginning of the alphabet. First AMICAS, now Agfa.

Caveat emptor, eh?  But that, of course, as well as everything else I write, is only my humble opinion.

Fair Dinkum.


It seems the games have begun. From one of my friends Down Under comes this note:

As a strange “coincidence”, look what now comes up when you try to access your blog site from inside the public hospital system. This was not there last week.

Typically, it does not work properly, blocking some of the graphics, search tools, header, blog list, etc, but the article text still renders.

Access to the web site you requested has been restricted

The site you have attempted to access has been identified as inappropriate or unrelated to Department of Health business. All Internet access is recorded and regularly reviewed by Department of Health accountability personnel.

If you require access to this site for Department of Health business, please contact your line manager to request access. All requests for access to this site will be reviewed by the State Health Executive Forum.

For a copy of the 'Acceptable use of Computing Facilities' policy please click here.


stacey said...

Gosh Dalai,
Bet you didn't know you were running a porn site!!!
Censorship is very evil.....

PACSFerret said...

As you pointed out re the Merge debacle - it is people that form the key differentiator between vendors. Impax certainly can work and on occasion works very well indeed. But the venor hastohave the right people behind, under, AND in front of it. I've long been of the opinion that a tendering process should include an assessment of key staff as well as technical and financial issues.

Who they really are... said...

When the Dalai speaks, people listen...and then block your site. Cobardes!

Anonymous said...

Dear Dr Dalai
Don’t take it too personally, it’s just that the limits placed on the Agfa supplied and supported workstations now match the rest of the HIN workstations which include restrictions to EBay ,face book and YouTube. we can still view your webpage .

Anonymous said...

Ouch!!! I wonder what this portends for government-run healthcare in the US.....

Anonymous said...

Dr Dalai, unfortunately your friends down under have somewhat misinformed you. IT (HIN) does not Really have anything to do with PACS in Western Australia. PACS is owned and managed by Radiology, including all contact with the vendor. This is a large part of the problem. PACS support should fall under the application portfolio managers within HIN who are used to dealing with large vendors and then PACS be guided by the user group. Don't blame IT, blame Agfa and Radiology for not relinquishing control of PACS to those who know better how to manage applications and difficult vendors.

PACSFerret said...

Anon..... To a large extent you are right- lack of IT involvement is to some extent why PACS specifically and clinical system generally have progressed so slowly over the last 20 years. However:

1. Whether we like it or not the schism between radiology and IT is bigger than others in the healthcare arena. There are historical reasons for that but it must be factored into any project.

2. The schism is only driven deeper by IT department that insist on being more of a problem than a solution. I'm certainly not saying that is the case in WA but I see it often.

It is it seems to me a great example of what happens in an insulated environment. PACS vendors work hard to insulate radiology from IT be ause it is very much in their interests to do so (IT departments would NEVER have condoned the ridiculous money being charged by PACS vendors) and insulated environments are the very ground where Chinese whispers do the most damage.

Anonymous said...

Your web page certainly isn't blocked from my computer within the WA Health Department - at the moment at least....

Much of what you say is perhaps true, but I also think there are some that have over dramatised it all locally.

There are some very tired people in many fields who have been endevouring to get things right, and I take off my hat to them and thank them for their tireless efforts.

Believe me, I am hoping for a rapid solution to our woes though!


Anonymous said...

Excellent post. As an orthopaedic registrar (or 'resident' under the US system) I am regularly taking calls from all over WA - a massive distance, as you alluded to. So, when the system doesn't work and I can't see the images to provide advice to the GPs 2500km away, it may mean a unnecessary travel just for a simple fracture that the GP is unsure about, at considerable cost to the government and patient.

Rolling out the new system, they not only failed to provide any information or education to our department, they failed to inform us that for a 1-2 week period none of the old images would be available. Which is, as you can imagine, extremely challenging when you need to compare fractures, tumours, hip dysplasia...

Anonymous said...

So let me try to understand one thing...
After making enough and proper test cycles on the New software installed in the hospital, No problem appeared during the testing phase, but it appeared ONLY after deploying this on Production..!!

Or SIMPLY, NO testing took place at ALL...!!!!

According to my humble IT knowledge being working in Oracle for 10 years now, FULL and complete cycle testing should be done on a identical H/W structure and executed by the End User for long enough (Usually one business Cycle in low risk environments and 3 business cycle in high risk environments) BEFORE deploying the same software as a production.

So, My question is:

did the IT in the hospital conduct the proper testing before moving to the new software solution..???

Make sense right?

Anonymous said...

Anonymous has left a new comment on your post "Blunder Down Under":

A large part of the problem is that AGFA are running a "Sunset" business in Australia. That is, the traditional film business is declining much faster than their Digital technology revenues can replace it. -10% a year. As a result AGFA are required by Belgium to reduce their headcounts on an ongoing basis. In tha last 12 months significant long term technical talent has left the Australian business and AGFA are now struggling to maintain service to their shrinking customer base. The global focus for AGFA is now the growth markets of China and India. Anon

Anonymous said...

So what is the current status of IMPAX worldwide? Are there any serious issues remaining ?
Also - are there any IPad or IPhone apps out there for logging in remotely to view Impax studies ?