A Post For Larry

My friend and colleague Larry tells me he has become an avid reader of the blog, so this one's for you!

For the record, I'm discussing our situation in this public manner in hopes of eliciting help from those who are smarter than I am, with more experience than I have, and who might actually have a clue about what is going on with our PACS and how to fix it. I believe this if anything REQUIRES discussion about a situation that is IMPAIRING PATIENT CARE. I have tremendous respect for privacy and security, and I will do nothing to compromise it. But as Mr. Spock might have put it, the needs of the many outweigh the needs of the few, and the need of the patients, and those of the physicians to care for their patients, outweigh everything. That is why the hospital exists, it is why the Radiology department exists, it is why IT and the vendors exist. Period. 

But back to Larry. He and other colleagues have been gathering interesting images from our PACS, some of which are reproduced below:

If we cannot show the patient, I guess we can all go have a three-martini lunch, since there's nothing else for us to do. Ha ha. 

This would all be funny if PACS was the conduit for Netflix or something equally frivolous. As our episodes of outage and electronic impairment (no, we didn't have a three-martini lunch!) demonstrate, there is a severe and immediate NEGATIVE impact on patient care, and we have to get a handle on it NOW. The good news is that we are really starting to see improvement. It's sad that we had to have a total meltdown of personnel and machinery for that to happen, but that's the way it is.

From the outside, and from rather minimal information often fed to me third-hand, I am not really seeing a concentrated, orchestrated effort to get to the bottom of this, but rather a lot of shot-gunning that has had little impact overall. Yes, we've seen some improvements, but clearly, we are far from fixed. (There are those out there who would like to get ME fixed, but we won't talk about that.)

I'm seeing three distinct but likely related problems.

1. General slowness of the system, which varies by site. Transmission tests I have managed to see show this variation. This must have something to do with the network. It was suggested early on to see if the client behaves properly on a station connected directly to the server. Has this been done? Connecting via the Internet outside of the enterprise yields good service. That's a big clue!

2. Client software and hardware crashes. Is this a software problem? If so, is it a problem with the local client, the server, both? Could it relate to the network? Is it a Windows 7 problem? The clues are all there.

3. Slow searches. This seems to relate to corrupt profiles. I discussed this with PACS people, and I'm told that the user profile is simply an XML file, just a list of parameters that tells the client how the particular user wants things arranged, etc. Supposedly, logging on to two or more workstations simultaneously corrupts the profile. So, rather than spend hours creating a new but still fragile profile, how about someone figure out why IMPAX is even capable of corrupting the old one? We simply don't have time to rebuild profiles during a hectic workday, and I'm not going to waste everyone's time doing so when we don't even know why it helps or whether the change will stick. And if logging on twice can create a patient-care-impacting situation, perhaps a hot-fix needs to be implemented to prevent this practice, and the FDA notified of such.

There is an interesting, illustrative little side show to tell you about. I heard third-hand that some were questioning if my little AutoHotKey script for keeping the RIS window active might be the source of some or all of our problems. The turn-around-time (TAT) of the final report is a very important metric for patient care, and of course quicker delivery of the report is obviously helpful. Our performance in this regard is tracked, and all of these factors led to my creation of the little macro that could...keep the RIS window open and keep reminding us to sign our reports. As it turns out, though, my little bitty AutoHotKey macro apparently became quite the topic of discussion. IT security got very upset at its very presence, calling it all sorts of nasty names, and apparently a few were directed at me personally. The punch-line, though, is that the use of the macro was APPROVED by that department in 2010, with the only reservation being that someone could conceivably use AutoHotKeys to create a script which would bypass the need for a password. Given that I'm the only one who knows how to do it, and I'm not about to create something like that, we should be OK. Fortunately, most of my colleagues have made it past the days of inscribing the password on the bezel of the monitor with a wax crayon.

Being a cantankerous, semi-retired, cranky old lout, I will simply note that this episode demonstrates how things don't get accomplished.  Accusations fly, fingers are pointed, and some of the real issues don't get the attention they deserve. It would have taken literally 2 minutes to go to a station where this complied script is running, and activate Task-Manager to see exactly how many resources are being utilized. But I saved everyone the trouble. I've run task-manager (on those machines which don't have it and right-click disabled) and found minimal impact. (In addition, it shows that IMPAX is in a non-responsive state during some of those hour-glass generating waits. There's another clue for you. Anyone notice this before?)

To prove my point on the machine I was using at the time, I had to use Microsoft's Process Explorer, as it was felt too dangerous to allow radiologists access to Task Manager on some machines. Here's the result for my little script NewSigner3.exe:

The compiled script NewSigner3.exe utilizes 0.04% of CPU time on the IMPAX client workstation

Yup. Wasting 0.04% of CPU time could have devastating effects.

Once I made these results available, I was told: "Oh, no one was thinking that the script itself was the problem; but it keeps the  RIS window open, and that's what's wrong!" But like a bunch of Greek philosophers of old, sitting around the Parthenon contemplating how many teeth a horse should have, rather than actually finding a horse and counting its teeth, no one checked this. So I did:

Citrix window executables for RIS window use 0.15% of CPU time

Oh, but it turns out that wasn't what was being considered either. No, it seems that the fact that the RIS window was being clicked automatically every 20 minutes or so to keep it active and to remind us to sign our reports "creates downstream problems in that it allows for people to be logged in on multiple stations." (See discussion about profiles, multiple logons, and XML files above.) Someone please help me understand how keeping the RIS window open has any effect on the PACS. It seems pretty clear that some folks in the loop haven't tried to understand just what this little script accomplishes, and maybe somehow think it tweaks the PACS client. But it doesn't, as would have been discovered with 30 seconds of effort. Or a 30-second phone call to me.

You would think I had attempted to unleash the worst virus in existence upon the network.

Anyway. I did say above that some progress had been made, which is true, and some of the problems have been identified. Eventually, some of those will be fixed, too. One server had not been updated in a long time and it was delaying arrival of prior studies. No, I don't know why it wasn't updated. Very large individual disks had been used for storage which might have been beyond the recommended size. I don't know how that happened either.

You might recall me saying something about image skipping or rotten scrolling performance if the annotations were on. Apparently, this relates to the five-year-old version of software we are using, and should resolve once we upgrade.

Progress is progress, however slowly it might, well, progress. Perhaps we can Git 'R Fixed sometime soon. Larry?