Wednesday, July 29, 2015

Time For Hope And Change

From one of my former partners, now bosses, who is taking back the night this week:
The situation with PACS is completely unacceptable. As you can see it is 5:55 AM and the system has been "down for three hours. My phone will not stop ringing with upset doctors that they have non reads and can't get images. ER can't hear my voice clips. I am rendering interpretations on monitors in the modalities on studies such as deer vs moped and attempted murder buckshot to the face cases. The problem is more than just my obvious legal exposure, but there is a notion somehow this is the radiologist fault, like we exert some control over this.

There was an ugly incident a couple of days ago with me and a PACS administrator in the reading room when I was told the problem was fixed and I had an issue. The next confrontation I fear will result in me saying something I will truly regret and this is my last warning email. Its time to go up the ladder and make changes.
And from the guy on the early evening shift:

PACS started the inexorable slow decline at about 9pm last night, with the usual >30 sec between studies, lack of initial response to mouse input resulting in cursor catastrophe when the machine finally caught up, and...a new one for me...the mouse cursor got stuck at the bottom of far right screen number took me forever just to find the cursor.

Is it unreasonable to institute a "no further imaging" point where, at our discretion, all imaging is halted until PACS resumes normal function? Otherwise, as stated above, we are liable for studies we cannot interpret per ACR guidelines, as the images are locked in a radiology purgatory. Continuing to scan patients and send them to this purgatory does absolutely no one any good at all. It only leads to the clinician unrest and anger our partner fought for us all last night/this morning.
This sounds eerily familiar...

Remember the Blunder Down Under?

To this day, five years later, my friends in Perth and elsewhere in Western Australia tell me their Agfa system still doesn't work properly. Now you see why Agfa has been very hesitant to share information with me!

It is quite possible this cannot be fixed. Agfa's PACS architecture is extremely complex, to the point that their own experts may not know what's going wrong with the system. Add to that a very clear reluctance on the part of our IT department to consider a network problem even in the face of clear evidence.

We are now backed in a corner. We cannot allow this to degenerate into another Western Australia debacle. For what it's worth, here are my suggestions:

First, someone needs to take a laptop to the data center and plug it straight into the server. If the PACS client works properly, we will know something about how the network is or isn't affecting performance. (Which we already know since different sites have different speed issues, and we get the best performance when connecting over the internet, but there are those who need proof...)

Secondly, given the crashes, it is clear that the problems go beyond the network, although I still think the network plays a significant part. Even if it isn't included in the contract (which I would like to peruse), the vendor needs to perform the next major update AT NO CHARGE given the current impairment to patient care we are experiencing.

Third, it is time to strongly consider moving to another vendor, this time using a Vendor Neutral Archive (VNA) which allows for easier migration in the future. I don't know the exact figures of how many exams and how much data we have stored over 20 years of PACS experience, but it would be a VERY major undertaking. Still, the switch to a VNA is something I strongly recommend even if we stay with Agfa. Keep in mind, migration from the old database could literally take years.

I haven't had hands-on experience with all of the vendors out there, but of course that doesn't stop me from having an opinion. While not everyone likes Merge/AMICAS, and they have had their problems at the hospital using it (although our group's system has had very, very few over the years), it has a much simpler architecture, built around a regular old web-server (Windows Server if you're interested), and as such it can handle a tremendous amount of traffic. I personally like the client (which I had a small hand in designing). The newer versions use a VNA database.

McKesson gets good reviews from the rads that use it, and in fact one of my bosses/former partners has had a very positive experience with it. McKesson was excluded from the 2003 PACS upgrade search because at the time, IT was phasing out other McKesson products for reasons known only to IT and would not consider any new McKesson product. Sectra out of Sweden has its fans and a fairly large US presence. Intelerad has a product similar to Merge/AMICAS that is certainly worth considering.

TeraRecon and a smaller advanced visualization company called Visage have something they call "deconstructed PACS" which overlays the existing database and provides another client interface. You still need your own VNA and other supporting components.

My short list ends there.

Based on our experience with GE's Universal Disappointment, and some insider knowledge, I would not even bother with them. Fuji continues to have many weird client problems, and locked-down software for which changes take years. Philips rebrands the web-based system once called Stentor. It is the last of the major programs that can't burn a DICOM CD readable by another PACS. People either love it or hate it. Siemens' latest PACS offering, syngoPlaza, hasn't taken off to any significant degree. There are a dozen more small PACS offerings out there that I would never recommend at all, let alone for an enterprise the size of ours.

For the very short term, IF our system cannot be brought under control, it will be necessary to do some form of overlay. From limited exposure at the last RSNA, the deconstructed PACS concept would work, BUT there are missing pieces such as the inability to generate a worklist, which requires another product. Years ago, an older version of AMICAS was used at Mass General as an overlay for their older (version 4.x) IMPAX to provide web-based access. With the proper interface engines, I think Merge could create an overlay to our database with full functionality. I think so, anyway.

It is validating, though sad, to have Dalai's First Law proven correct again and again:

PACS IS the radiology department.


Unknown said...

If it were my Impax (or any other vendor's) system with this long running a problem, I would have a GSC service case open as a down system along with an FDA customer complaint.

Dalai said...

From a comment placed on

Sorry to hear about you're trouble with the Agfa Pacs.

I'm a Belgian Pacs Admin (former trauma/radiology nurse with a very large affinity for IT).

We also run a complete Agfa installation (Impax 6.5.3 with Qdoc, integrated speechmagic recognition & Clinapps)at the radiology department and have similar performance problems from time to time.

Because we run everything within the radiology departement (complete seperate IT setup)
we can minimize the downtime very quickly.

We pride ourselves to have the fastest network/servers/workstations in the hospital &
keeping 30+ radiologists happy is a fulltime task (the radiology IT is just 2 persons).

Because I do all the installation work of the network/servers/workstations/software,
we have a much better control over the complete picture.

I did all the OS installs & network tuning of the Agfa Servers to be sure that everything was running smoothly before Agfa did the Impax install. Our Agfa environnement is completely Windows based (except the Oracle DB, that's running Solaris)

We still see drops in performance from time to time, corrupt userprofiles, SCP & SCU behaving strangely, Oracle DB that is acting slow.

Agfa is slow to fix certain issues, some issues don't get any fixes at all.
And their famous quote "We never ever have these problems on our other sites" is always a hoot, when reporting problems.

When contacting the helpdesk it depends on the person you get on the phone.
If it is a senior tech you will be lucky, if it is a junior sometimes things sometimes gets worse.

It is also very hard to get inside tech information from Agfa. We do a lot of our own interventions on the Agfa system. The knownledge is collected over the years when willing Agfa techs come by and share some inside information during upgrades.

It seems to help to do an iisreset on the application servers every evening & reboot every week (during maintenance).

We balance the load between 2 application servers (not clustered, standalone) and use 2 workflowmanagers to spread the load (also standalone)

We also run Philips Intellispace (CT & MRI), Siemens Syngo.via (Nucleair medicine), GE AW (CT, MRI),
Carestream Vue Pacs (CT), Philips Isite (student training),Sectra Pacs (MG second reading) as secondary PACS/3D Workstations

All the reading is done in the Impax system, with interactive speech.
Sometimes Qdoc is used when images are not directly available (John Doe on ER)

Dalai said...

From the same Belgian commentator:

We all could wear thinfoil hats, just to be prepared.

The new Agfa Agility is still at least year off when you want to migrate from a previous install.
The only installs at the moment are new installs, migration paths are still being investigated.

Agfa hopes that with Agility a lot of the performance/unexplained problems will be a thing of the past.
Everyone will run the same code at the same time (updates will be pushed a lot faster).

Don't know if it will work, I have my doubts as every hospital has it unique challenges.

Most people don't realise what a difficult taks it is to get everything up & running (and keep it running).

We have seen GE CT's killing the Agfa SCP with massive overload, Philips CT's that loop when sending dicomimages.
The Agfa Impax Oracle DB with indexing problems slowing things to crawl. Application servers that borked in the middle of the day.
Sometimes it is network related (to few queues in the masterswitch -> network overload), sometimes the underlying hardware is having problems (firmware problems with large diskloads) and sometimes the vendor software is going beserk (for no apparent reason).

It is true that Agfa has customized every installation and that results in problems that sometimes are difficult to diagnose.
It is becoming an art in itself to diagnose & isolate the problems and reach out to the right people (IT of Agfa).

And when the blame game begins, nerves get fried and arguments start.

I am a nurse by trade, I've worked in the trenches (ICU, ER, CT, CR/DR...), so I know how annoying it can be
if the images/RIS/Worklist are not available.

The pressure on the radiologists is rising as everything is evolving in "just in time" imaging (in Belgium)
Volume of patients are rising, the volume of images/exams are steadily rising,
so every time the system falters you have a backlog of hours.

Is it a sign of the times that problems don't solved adequately?
I don't know.

Unknown said...

Interesting. We are running 6.5.5 with pretty much the same architecture as yours (core servers are Solaris/Sun). Early in the upgrade process we did note intermittent (love that word) 'jerkiness' when scrolling large CT/MR series. That has gotten better with time and maybe Barco driver updates. We now have an interesting problem on the Barco monitors where the cursor will intermittently disappear or begin misbehaving. Never a dull minute. Over the years we have come to know the GSC support guys who are pretty knowledgeable and like you have acquired a large number of tips, tricks, and scripts. I will say that 6.5.5 has run far more smoothly than did its predecessor versions.

JeanetteC said...

Does anyone know how to resolve this error in the AGFA IMPAX Client when exporting/burning an image? Failed to create volume directory. Our hard drives are not full and this is happening on the clients. THX!

JeanetteC said...

We are running AGFA Impax Client and our users are getting an error "Failed to create volume directory", when exporting an image. All clients are getting it. We have plenty of hard drive space and we do not carry support on our software. Any assistance would greatly be appreciated.

Dalai said...

Hi, JeanetteC! I don't have an answer, but hopefully my readers can come up with something!