Time For Hope And Change

From one of my former partners, now bosses, who is taking back the night this week:

The situation with PACS is completely unacceptable. As you can see it is 5:55 AM and the system has been "down for three hours. My phone will not stop ringing with upset doctors that they have non reads and can't get images. ER can't hear my voice clips. I am rendering interpretations on monitors in the modalities on studies such as deer vs moped and attempted murder buckshot to the face cases. The problem is more than just my obvious legal exposure, but there is a notion somehow this is the radiologist fault, like we exert some control over this.

There was an ugly incident a couple of days ago with me and a PACS administrator in the reading room when I was told the problem was fixed and I had an issue. The next confrontation I fear will result in me saying something I will truly regret and this is my last warning email. Its time to go up the ladder and make changes.

And from the guy on the early evening shift:

PACS started the inexorable slow decline at about 9pm last night, with the usual >30 sec between studies, lack of initial response to mouse input resulting in cursor catastrophe when the machine finally caught up, and...a new one for me...the mouse cursor got stuck at the bottom of far right screen number 5...it took me forever just to find the cursor.

Is it unreasonable to institute a "no further imaging" point where, at our discretion, all imaging is halted until PACS resumes normal function? Otherwise, as stated above, we are liable for studies we cannot interpret per ACR guidelines, as the images are locked in a radiology purgatory. Continuing to scan patients and send them to this purgatory does absolutely no one any good at all. It only leads to the clinician unrest and anger our partner fought for us all last night/this morning.

This sounds eerily familiar...

Remember the Blunder Down Under?

To this day, five years later, my friends in Perth and elsewhere in Western Australia tell me their Agfa system still doesn't work properly. Now you see why Agfa has been very hesitant to share information with me!

It is quite possible this cannot be fixed. Agfa's PACS architecture is extremely complex, to the point that their own experts may not know what's going wrong with the system. Add to that a very clear reluctance on the part of our IT department to consider a network problem even in the face of clear evidence.

We are now backed in a corner. We cannot allow this to degenerate into another Western Australia debacle. For what it's worth, here are my suggestions:

First, someone needs to take a laptop to the data center and plug it straight into the server. If the PACS client works properly, we will know something about how the network is or isn't affecting performance. (Which we already know since different sites have different speed issues, and we get the best performance when connecting over the internet, but there are those who need proof...)

Secondly, given the crashes, it is clear that the problems go beyond the network, although I still think the network plays a significant part. Even if it isn't included in the contract (which I would like to peruse), the vendor needs to perform the next major update AT NO CHARGE given the current impairment to patient care we are experiencing.

Third, it is time to strongly consider moving to another vendor, this time using a Vendor Neutral Archive (VNA) which allows for easier migration in the future. I don't know the exact figures of how many exams and how much data we have stored over 20 years of PACS experience, but it would be a VERY major undertaking. Still, the switch to a VNA is something I strongly recommend even if we stay with Agfa. Keep in mind, migration from the old database could literally take years.

I haven't had hands-on experience with all of the vendors out there, but of course that doesn't stop me from having an opinion. While not everyone likes Merge/AMICAS, and they have had their problems at the hospital using it (although our group's system has had very, very few over the years), it has a much simpler architecture, built around a regular old web-server (Windows Server if you're interested), and as such it can handle a tremendous amount of traffic. I personally like the client (which I had a small hand in designing). The newer versions use a VNA database.

McKesson gets good reviews from the rads that use it, and in fact one of my bosses/former partners has had a very positive experience with it. McKesson was excluded from the 2003 PACS upgrade search because at the time, IT was phasing out other McKesson products for reasons known only to IT and would not consider any new McKesson product. Sectra out of Sweden has its fans and a fairly large US presence. Intelerad has a product similar to Merge/AMICAS that is certainly worth considering.

TeraRecon and a smaller advanced visualization company called Visage have something they call "deconstructed PACS" which overlays the existing database and provides another client interface. You still need your own VNA and other supporting components.

My short list ends there.

Based on our experience with GE's Universal Disappointment, and some insider knowledge, I would not even bother with them. Fuji continues to have many weird client problems, and locked-down software for which changes take years. Philips rebrands the web-based system once called Stentor. It is the last of the major programs that can't burn a DICOM CD readable by another PACS. People either love it or hate it. Siemens' latest PACS offering, syngoPlaza, hasn't taken off to any significant degree. There are a dozen more small PACS offerings out there that I would never recommend at all, let alone for an enterprise the size of ours.

For the very short term, IF our system cannot be brought under control, it will be necessary to do some form of overlay. From limited exposure at the last RSNA, the deconstructed PACS concept would work, BUT there are missing pieces such as the inability to generate a worklist, which requires another product. Years ago, an older version of AMICAS was used at Mass General as an overlay for their older (version 4.x) IMPAX to provide web-based access. With the proper interface engines, I think Merge could create an overlay to our database with full functionality. I think so, anyway.

It is validating, though sad, to have Dalai's First Law proven correct again and again:

PACS IS the radiology department.

Nothin' New

I wish I had something new to report on our current PACS issues. Sadly, I do not.

The Team of Experts from the Vendor, including the fellow who probably knows the software best of anyone in the entire world and beyond has been working feverishly. Things have been tweaked, caches have been cleaned, cookies have been baked, red heifers have been sacrificed, and the moon has been howled at.

Nothin' new. Bupkis. We don't seem to be getting anywhere. What hasn't been done to my knowledge is taking a workstation (or laptop) straight to the data center and plugging in right into the server. That might provide a hint as to what part the network and hardware might play in this little fiasco. Of course, the fact that the darn thing works near-perfectly (relatively speaking) when accessed from a regular old home broadband connection might be another clue. Yes, Colonel Mustard did it with a candlestick in the parlor. But hey, a clue is a clue!

When there's some news, I'll let you all know. But don't hold your collective breath.

Pointing Fingers At Meetings

PACS is complicated. Really complicated. Really, REALLY complicated.

So when something goes wrong, there is a lot of investigation that needs to be done, and occasionally, there will be a lot of 'splainin' to do. And here and there, people point fingers at the perceived source of the trouble. Sometimes they are right, sometimes they aren't.

As you might guess, one of our systems is having a problem, and we aren't getting to the bottom of it. And there is some finger pointing. As we are in the midst of working it through, but we do not yet know what's wrong, I won't name the name of any entity, person, nation, President, planet, or vendor involved. You may feel free to guess, but I'm not telling. Wild horses couldn't drag the information out of me. And I hate horses. Well, I hate riding horses, which is close enough.

So. Just the facts, Ma'am.

The PACS in question has never operated quite perfectly. You might say that none of them do, and I would have to agree. But this one has had a few operational issues, including occasional complete outages, which fortunately are few and far between. And you might say all of them do that, and I would have to agree.

But the system in question has always had some little glitches I don't see from our other sites. In particular, we have noticed over the years that scrolling of large data-sets can be painfully slow. Surprisingly, turning off annotations will speed things up considerably. No one has been able to explain this adequately, although some have suggested that it is due to this particular architecture having to communicate back to the mother ship server for every command and mouse stroke and click. Could be. Also, SOME of us, and not all of us, experience very slow searches, worklist refreshes, and so on. This bad behavior has been attributed to bad individual user profiles, and rebuilding them may or may not cure the problem. And it has also been said that because some of us keep multiple worklists active, and all of those worklists have to have a little chat with the server upon each refresh, we are the ones slowing down our own workstations. Blame the victim. Love it. But it might still be true.

We've been putting along like this for several years. I've complained, others have complained, things are looked into, things are tweaked, and sometimes we see some improvement. I do have to say the system has been usable, even with the glitches, except on those occasions when it dies completely, and then it isn't very usable at all. In the meantime, I'm informed that we are about the last site in this quadrant of the galaxy on our particular version of the product. Fits like an old shoe, I guess. We need an update, although no one is certain that will solve any of our problems.

Unfortunately, over the past month or so we have been seeing a definite deterioration in function. Things load slightly more slowly, then a bit slower still, and in some areas, including an ED and a remote site on the network, the speed drop has reached the point that the stations are inoperable. But here's a little clue for you electronic Sherlocks out there: Stations accessing PACS via the Internet, outside the Enterprise, demonstrate adequate speed. We get better service from a home station on a 50M Time-Warner home connection that the gigabit Ethernet inside the hospital. Hmmmmm. Aside from the network connection, the only other difference between a station within the network and outside is that our main reading stations have digital voice (NOT speech-recognition) software integrated, although the ED stations do NOT.

To me, this all points to a network problem, possibly/probably compounded by the way this particular PACS architecture works with the network topology.

Now here's where we get into trouble. It seems that IT and the vendor have been working on the problem for a month or more. And they have come up with nothing. Well, I've been talking with folks on all sides of this, and that's not quite true.

The vendor has run tests from sample workstations at multiple sites, and lo and behold, the site with the most trouble has significantly slower transmission speeds back to the gateway than the site with less trouble. I'm leaving out a lot of detail, but that's the gist of it. Sounds like the answer, yes? No. IT has run tests on the network, and the report is that everything is perfect, nothing wrong, nothing to see here, move along. And so the finger points to the vendor.

The vendor, for its part, is bringing in everyone who knows anything about how the thing works, and promises to do everything possible to get to the bottom of the bottoming out.

And that, dear readers, is where we sit today. All sides are supposedly working furiously on the problem. I think (I hope) they all realize the mission-criticality of what it is they are fixing. Remember Dalai's First Law:  PACS IS the Radiology Department. Right now, our beloved department is impaired. I personally don't care whether this is the fault of the software vendor, the hardware vendor, IT, or if the janitor slopped a wet mop on a server. Our system is absolutely vital for patient care, and we cannot begin to tolerate anything less than 100% function. And we cannot tolerate anything less than 100% cooperation to get us back to 100% function.

I have had the good fortune of creating an international speaking career based on some of the foolishness I've seen over the years on every side of the PACS equation, and that includes ridiculous behavior on the part of vendors, IT types, and yes, even radiologists. As a crotchety, cantankerous, semi-retired curmudgeon, I can say with confidence that with most PACS problems, all parties have some degree of guilt. It is really bad when one side digs in its heels and declares: "it's not my problem!" But there is something even worse: MEETINGS!  Many folks out there, often but not exclusively IT types who have come up through the IT bureaucracy and not via Radiology, know of one and only one way to handle a situation: We need to have a MEETING! Thus time and resources are wasted trying to get the Important People in the same room at the same time so they can all explain why something isn't their fault, and agree on the time of the next meeting. Yes, I know that's how things are done, but in my experience, it's more how things don't get done. My very favorite example comes from my early days of dabbling in PACS, now over 20 years ago. I was sitting beside the one and only PACS administrator, a good friend of mine, while an IT person was droning on and on about how thus and so wouldn't work, how it couldn't possibly work, and how they should all meet again in a month to discuss why it was a bad idea. My friend whispered in my ear, "I did it and it works just fine!"

No doubt I've stepped on a few toes with this little rant. I mean no disrespect, and I dearly value the friendships I have made throughout the years among IT folks and vendors alike. But I have very little tolerance for things that get in the way of patient care. A failing PACS is right up there. Let's not add squabbling and finger-pointing to my sh*t list.

Besides, if I should disappear tomorrow, you all might have to deal with some of my former partners (now my bosses) instead, such as the fellow for whom Dalai's Twelfth Law was written:

At least I speak the IT language. Which might not be all that appreciated in the end.

Dalai's note:  What happens in Vegas stays on the Internet it seems. I wrote about Bill Clinton's speech at RSNA, 2010 back in (surprise) 2010, and it was discovered only recently by Fox News. The anonymous donor who paid for Bill's rambling wreck remains anonymous, and in fact, there seems to be no record of the donation at all. Hmmmmmm....

Medical Bill: Mystery donor picked up $150G tab for 2010 Clinton speech

By Malia Zimmerman

Published July 10, 2015

| FoxNews.com

It was a big coup when a nonprofit medical trade group landed Bill Clinton as a speaker at its 2010 annual conference in Chicago -- so big that some members wondered how the former president was being paid.

Not to worry, members of the Radiological Society of North America were told: An anonymous donor footed his bill.

The $150,000 fee was a mere fraction of the $48 million Clinton took in from 215 speeches between 2009 and 2013, while his wife was secretary of state. Who paid Clinton and why they thought it was a fair bargain may never be known — but government watchdogs say it is a prime example of how elusive accounting can be for the ex-president's eye-popping earnings.

It was clear, however, that the husband of America's top diplomat was not chosen for his medical expertise.

“I think this is interesting that you would ask me to come and speak today to a group of people from all over the world, and everyone of you knows more about the subject than I do,” Clinton said at the beginning of his 45-minute address to an audience of 4,250.

Dr. Sam Friedman, a radiologist from Columbia, SC., said at first he was “peeved” when he heard Clinton was paid $150,000 for the “rambling” speech, during which Clinton took several “gratuitous shots” at Republicans and blamed U.S. doctors for many of the healthcare problems in third world countries. When he and like-minded members made their objections known to the organization, they were told the fee was paid by an “anonymous” donor. 

Radiological Society of North America spokesman Marijo Millette told FoxNews.com the group “strives to provide compelling speakers that will satisfy the educational needs and special interests of a diverse audience.”

Millette would not comment on Friedman's claim, which was also reported by trade media, but said Clinton's fee and travel expenses were paid to the Harry Walker Agency, which represents Clinton. The organization’s 990 forms, filed with the Internal Revenue Service and required to maintain its 501(c)3 status, do not list any payment to Clinton or his representative. Neither the executive director nor three executive board members contacted by FoxNews.com would divulge who paid Clinton's fee.

Matthew Whitaker, executive director of the Foundation for Accountability & Civic Trust, a Washington-based, non-partisan campaign and ethics watchdog group, said the anonymous donation “opens up a Pandora’s box of questions including who funded this speech and what their motivations were.”

“This issue has to be resolved," Whitaker told FoxNews.com. "There has to be an answer as to who gave the money. “It has the smell of someone trying to move money through an organization to curry favor with the former president. It also calls into question almost every speech Bill Clinton has made and who the ultimate funder is.”

Neither Clinton's representatives at Harry Walker nor at the Clinton Foundation responded to a request for the name of the mystery sponsor. It was not clear if other speeches by Clinton were similarly funded by anonymous third parties.

Tom Fitton, president of Judicial Watch, a Washington D.C.-based government watchdog foundation, said much of the $48 million Bill Clinton made from 215 speeches during the time Hillary Clinton was Secretary of State went to the Clintons' personal coffers, not to the foundation. Federal disclosure forms filed by Hillary Clinton for 2010 record her husband’s compensation as $150,000 from the Oak Brook, Ill.-based group, but critics say the lack of transparency about where the money really came from raises serious questions.  

“Bill and Hillary Clinton are married, so under the law, paying him for a speech is like giving money directly to her – to the Secretary of State,” Fitton said. “I cannot think of a comparable ‘pay to play’ scandal.”

Clinton gave 542 speeches around the world between 2001 and 2013, earning $104.9 million, and delivered another 53 speeches between January 2014 and May 2015, earning an additional $13.5 million, according to reports by Fox News and the Washington Post. The former president's speaking fees have ranged from $28,100 for a 2001 talk at the London School of Economics to $750,000 for a 2011 appearance at an event for Swedish communications company Ericsson.

While Clinton's knowledge of world events and charm as a raconteur is well-documented, critics doubt the sky-high fees are doled out by anonymous parties for sheer entertainment value.

"These donors don't cut checks because they want to hear a brief speech," said Sean Davis, co- founder of The Federalist, a conservative online magazine. "They do it to gain access or favors from the Clintons. The Clintons owe voters a clear explanation of who is funneling them this money and why.”

Belgian Rhapsody

Dalai's Note...Since IMPAX has decided to quit working this afternoon, I'm left with time on my hands, and we all know what the Devil does with idle hands...So please enjoy this repost from 2009, with a few minor enhancements . I wish I could say 2009 was the last time we had a PACS outage, but my creativity can only be stretched so far.... 

Is PACS online today-
Is that just fantasy?
Still caught in downtime
With no functionality-
Open your files
Look up from the dials and see
I’m just a poor rad, trying to get through the day
‘Cause PACS is easy come easy go
Why it glitches, I don’t know
Any way the circuits blow doesn’t really matter to me
To I.T…..

Mama, just read a scan
Had a CT of the head
Clicked it off, now PACS is dead
Mama, it had just come up
But now I’ve gone and crashed it all again
Mama, Oooo, ooo ee ooo
Didn’t mean to make it die
If it’s not online again this time tomorrow
Back to film, Back to film, as if PACS doesn’t matter

Too late, my PACS won’t scroll
Couldn’t label any spine
Lovely software past its prime
Goodbye to my worklist, it won’t update now
Gotta reboot yet again and hope it works
Mama, Ooo Ooo Ooo Ooo,
I can’t have it die
I wish sometimes I wasn’t a rad at all

I see a little problem getting back online
Waterloo, Waterloo, Can you make the damn PACS work?
Shorting out and sparking, keeping me remarking: OY!
Mike Cannavo, Mike Cannavo
Mike Cannavo, Mike Cannavo
Mike Cannavo make it work, oh make it work Ooo, Oooo,Ooo
But I’m just a poor rad and my PACS hates me-
He’s just a poor rad from a bad residency
Spare him the joy from this monstrosity
Easy on easy off, will you let me work
Just fix it! No we will not make it work-Make it work!
Just fix it! We will not let you work-let it work!
Just fix it! We will not let you work-let me work!
Will not let you work-let me work
Will not let you work-let me work
No,no,no,no,no,no,no,no
Mama mia, mama mia, mama mia let it work
The Psycho ward has a nice cell set aside for me, for me, for ME!
So you think you can patch me and tell me I’m fine
So you think that an upgrade will keep me online
Oh, baby, can’t do this to me baby
Just gotta hang on, just gotta stay online right here
Nothing ever matters
Anyone can see
Nothing ever matters-nothing really matters for I.T....
Any way the circuits blow….