Monday, September 28, 2015

Making Mars Great Again!

Monday, September 14, 2015



Universal Improvements

I have referenced some of our trials and tribulations with the new GE Universal Viewer, which is technically version 5.x of Centricity.

We've found that some of our problems were of our own making, with the help of some unusual programming on GE's part. The disappearing measurements, for example, didn't really disappear. Rather, my hanging protocol created a clone of the main window, and if things are measured there, they won't show up, although they do live on as the Saved Presentation. Lesson learned. Various other "glitches" were due to our lack of understanding of something esoteric. Well, if they all worked the same, no one would buy the expensive PACS, now would they?

However, one problem has not yet been fixed, that of skipped images. We thought at first this was also due to the clone window handling, but apparently it is a coding error. Which is to be fixed. IN THE RELEASE AFTER NEXT!!  Um.....GE....a glitch that hides patients' image data is a VERY SERIOUS PROBLEM! This is the sort of thing that calls for emergency hot-fixes and mea culpas to the FDA. I do have to admit that the problem is not as bad as what we saw with Centricity 2.x, wherein scrolling would simply jump over images; UV 5.x tells us when it's skipping images by flashing "0% Loaded" in the missing frame. This is a definite improvement, can do better!

Anyway, a team from GE came by the other day to show us how things will improve.

The two apps people who came to visit were well-known to me, one having demoed Centricity to us about 12 years ago, and the other having shown me the Universal Viewer for the first time at RSNA 2012. They brought a laptop-based workstation and server, and a cargo-load of monitors. Given the fact that we just redid all the workstations, couldn't we have simply, temporarily downloaded UV 6.x on our station, saving a LOT of time and effort? Oh well, I guess that's why I'm not in sales.

The team showed us a few things we already had that we didn't know about. For example, two CT series can be synchronized by "Image Registration" as well as position or slice number. The modifier was there all along under the Sync menu, but I never thought to look there. Perhaps theres also a "Sync by Phase of the Moon when Study Acquired" or "Sync by State of Consciousness" in there, too. I'm sure there are other little Easter Eggs hidden in the somewhat scattered GUI I haven't found as yet.

UV 6.x apparently plays nicer with Windows than 5.x. We had complained about the Navigator pane disappearing behind the Patient Folder; resizing the latter will solve this temporarily, but the windows all revert back to where they were upon opening the next study. Ah, Windows. Some problems scrolling through old reports after clicking somewhere else supposedly will be improved in the next release as well.

One of the main deficiencies in UV to date has been the lack of the promised ability to read PET/CT without adding the Advantage Workstation Server. We were shown the AW functionality, which is FINALLY tightly integrated into UV about 12 years after it was supposed to happen. It does appear to work as I would expect it to, with all windows, planes, 3D renderings, cartoons, and other stuff linked. Adjust one, the others all follow, whether on UV or AW. Very nice. This gives us the full PET/CT reading capability, matching that of the old AW upon which we now depend. Works for me. There's also a nice OncoQuant module which we may or may not get that will allow serial lesion measurements. That is, after all, what we do in oncologic imaging. Having a table of what was there before could certainly help us to be certain we measure all the lesions seen last time. Each. And. Every. Single. One.

I took a few moments to rant about the control layout of UV, which is not terribly logical, and can get worse depending on what buttons are pressed, and what the state of the viewer might be. For example, when using measurements (obviously the bane of our existence), the buttons for linear measurement, ROI, etc., are placed to the right of the "Study Dictated" button, which is otherwise nicely placed at the end of the line. Yes, said GE, you are not the first to complain about that. Well, gee, GE, how about trying these things out with real users before deploying them? Little problems could be caught before they become big problems. That would be nice, and few companies do it. My services are available for a very reasonable fee!

Having heard of our difficulties with another PACS, the reps approached me after the demo and wondered if this might be the time to present their wares to the troubled site. Since the problems there might not be the fault of the PACS itself, I urged patience. A LOT of patience. Like wait until I've fully retired patience... Some of my former partners have told me quite clearly that they would really rather not have to use UV anywhere else. To them, I urge patience as well; UV represents tremendous progress over its predecessors, and you never know how much better it might get...

Thursday, September 03, 2015

Another Question

Hypothetical question:

Which vendor/product out there is capable of (rapidly) overlaying an existing PACS?

So far, Merge, Intelerad, and Visage seem to be able to do so. Any others? Any experience in doing this?

And how well will they work if your networks aren't in good shape?

Thank you for sharing your thoughts.

Wednesday, September 02, 2015

A Question For You...

A very simple question...

Has this blog been helpful to you? Has the description of our trials, tribulations, and occasional successes with PACS led you astray or in the right direction?

Just wondering...

Tuesday, September 01, 2015

Connectivity Deja Vu

Tonight, there was semi-emergent IMPAX downtime to allow for the updating of the Connectivity Manager. The 6-8 hour anticipated impact on PACS was planned about 8 hours before it happened, and is necessitated by our imminent move to IMPAX 6.6.1. The system is due back up in less than one hour, and we'll see if we fare any better than last time. You see, this is all quite familiar. Here is a regurgitation of my post, "Connectivity Banisher" from January, 2009:


In the world of Agfa Impax, a "Connectivity Manager" is:
. . . a middleware component in the integration between HIS, RIS, modalities, and PACS systems, linking patient and study data with images. To display the information available from a non-Agfa RIS in the Text area of IMPAX, connect to the Connectivity Manager. . .

The main purpose of the Connectivity Manager is to take data from one system, such as a HIS, and translate it into a format that another system, such as a modality, can understand. Connectivity Manager accomplishes this translation with mappings. The mappings tell Connectivity Manager how to translate incoming and outgoing messages to external systems. The following mappings must be configured so that Connectivity Manager knows which report source to go to for each study, and how to translate messages sent from IMPAX. . .

Map a reporting name into the Data Store by identifying the sending facility in the Connectivity Manager database. Identifying this value means it will work regardless of whether the sending facility sends their name along with the message or not. Also, if the sending facility changes their name at some point, mapping or configuration changes will not be necessary. The Default Assigning Authority identified in this mapping is the name of the report source entered in the Business Services Configuration Tool.

The sending facility is required to view reports in IMPAX. Connectivity Manager uses the Entering_organization and Requesting_Service mappings to populate the sending facility field. These mappings should include the Default Assigning Authority so that every report contains a sending facility.
Our Connectivity Manager was upgraded last night. And again this morning. From the rad's point of view, this means:
During this time Modality Worklists will not be available and Technologists will have to manually input ALL Patient Information. Studies sent to IMPAX will Fail Verification, and will not update with Reports until the downtime is ended.
I drew the short straw, and experienced the joys of the upgrade. Fortunately, the downtime lasted less than one hour, and not two. Of course, only a few of the techs got around to manually inputting ANY Patient Information. Still, we were OK. Until this morning, when this information (including the accession number by which we dictate) was suddenly absent once again. The culprit was, of course, the Connectivity Manager, which seemed to be confused by "multiple" inputs for the same patient. Now that's a problem, which we hope will be fixed by the experts before too much longer.

As usual, Dalai's Laws of PACS apply. In particular, the First and Third Laws are applicable:
I. PACS IS the Radiology Department.
III. Once PACS, never back.
When PACS malfunctions, the department malfunctions, and don't even consider asking anyone to go back to a manual process. It ain't gonna happen.

So, in the ideal land of Dalai-wood (Hooray for Dalai-wood!), PACS should never break. Since that isn't achievable, these thing need to be created with an eye toward simplicity and functionality. Based on what the "Connectivity Manager" is supposed to accomplish, I'm not really certain why it has to be a separate program or computer or whatever it is. Shouldn't the simple, basic, core PACS be able to talk to others? OK, provide a look-up table (user-configurable, of course), but do we have to have a big, separate, grandiose module that manages to bollux up the works when we upgrade it?

Yes, I know..."simple" and "basic" aren't in the vocabulary of a lot of PACS vendors. Neither is "easy". And "uptime" can be defined to the preferences of those making the definition. But as far as I'm concerned, if it isn't totally "up," then it's "down." (Which happens to be true for many things in life.) So, today, we were "down," courtesy of our dear Connectivity Manager.


A party close to the 2009 update had this closing comment:

Please understand this is not normal hospital type workflow, and with the hundreds of other mappings to get right is an easy oversight, the missing "Cerner ID" should have been noticed during your sites testing and identified prior to the go live.

But once again an easy oversight by all involved.

This also was not a standard upgrade, Yes your CM software was upgraded to the latest version, But behind the scenes each and every interface was recreated from scratch, including the HL7 Feeds, all the modality’s (100’s) and each and every Impax device and client, There was months of work evolved with this upgrade.

On a different note, I truly enjoyed reading your blog, And can fully understand the frustration these upgrades cause the people working with the software. I do try my best to ensure as little impact to your job as possible.

I have yet to understand what, beyond me, makes our system any different than any other. And I'm not sure how we were able to squeeze "months" of work into 14 hours. Perhaps the process has been perfected in the last 6.5 years. Right.



Initially, the update seemed to have gone well, and I even sent an email congratulating those involved. WRONG.

It apparently did NOT go well. In the middle of it things went completely down for about 30 minutes. No PACS, no nuthin'. Did we have adequate communications on this? Did we share the responsibility properly?

And our 6.6.x upgrade has been inexplicably delayed.

Not good. Not good at all.

Wednesday, August 26, 2015

Turning Point?

Image courtesy

There was a meeting involving everyone with a stake in our PACS problem. This included radiologists, IT folks, administrators, and representatives from the vendor. The purpose of the meeting was to outline where we were, where we are, and where we are going. I left the meeting feeling cautiously optimistic for the first time in quite a while.

I'll not delve into specifics much, as they probably won't help you with your PACS problems, nor will they help you help me with mine. The generalities however, should prove more valuable.

The bottom line is quite simple, really. Our problems stem in greatest part from lack of communication. This gap occurred primarily between IT and the vendor, but there was also a rather large chasm between the radiologist end-users and the other two entities. Let's talk about that one first.

As an electrical engineer, I'm well-versed in the concept of the feedback loop, particularly the need in a circuit for negative feedback:

You've all heard the screeching "feedback" from a public-address system going out of control. Most amplifiers have a negative feedback loop, illustrated by the wire above using some of the output of the amplifier to "tone down" the input. So it is with life in general. If you think you are doing everything properly, and you receive no negative modulation, you will keep doing the same thing whether or not your actions are correct.

We rads had (or more honestly, exercised) only limited feedback options when it came to PACS. We the rads never really put our heads together to create a grievance list. Yes, one rad here or there would get upset with slow speeds, workstation crashes, slow scrolling, etc. Sometimes these could be fixed by the PACS administrators, sometimes only marginally improved. Someone would get a personal profile remade, although we never learned why this would help, someone would get his worklists redone. And so it went. In this way, little problems propagated into big problems.

I'll take some level of personal responsibility. I was so, well, jaded may not be the word...disheartened? Complacent? Resigned? Well, I had more or less decided that nothing would change until we had a major upgrade, whenever that might be, and I found ways to tolerate the glitches. It took two of my former partners, now bosses, who recently started working nights, to say "ENOUGH!" The minor slow-downs become quite major when you're trying to pump out dozens and dozens of studies through the wee hours of the morning. To be fair, there was further deterioration of the system during their initiation into the dark side (I mean dark hours), to the point that the workstations crashed, and ultimately there were a number of system outages, which certainly brought the whole situation to a head. The newly-minted night-stalkers began the campaign that has brought us to the brink of...a solution.

Communication is paramount as always, and we've made some great strides in that realm. First, we insisted on having a call-team from both IT and the vendor. We were briefly relegated to the "Help" desk; when they actually answered, they were little more than a rather slow answering service for IT. Second, we streamlined the process for rads to report problems directly. This was prompted by a response to a complaint implying that no one had ever mentioned the problem before. When Donald Trump's cell phone number was published, instead of having a little hissy-fit as we saw with a certain Senator, Trump simply put a campaign ad on his line. This inspired us to create an email thread wherein any and all PACS complaints could be reported directly to the right people. And report we did. Initially, there were tens of emails per day; this has tapered off to only one or two. We have definitely made progress.

While we physicians can be quite problematic, the deeper institutional snafus lie with IT, the vendor, and their somewhat dysfunctional relationship. There will be yet another meeting to more precisely define just how that relationship will progress, and while I detest meetings, that is one that should have been held years ago. You see, there really wasn't a single point of failure, but there were quite a few, shall we say, lost opportunities for improvement.

It turns out that one of our major outages was a network problem, caused by an update push that got out of hand. Another slow-down was the result of the EMR grabbing too much bandwidth. There was a bug in a NetApp image server that took us down. OK, we assume these things happen and can be fixed.

But it took a village full of angry radiologists to bring to light that yearly service on some of the servers might not do the trick, particularly when a couple of the critical servers running SunOS/Solaris weren't touched at all. The latter had been running on an elderly version of the OS, and had a bug that was fixed umpteen versions ago. Update the OS, kill the bug. And here is where we had trouble. Putting it simply, everyone assumed the other entity was going to take care of stuff like this, if they assumed anything at all about it. And so nothing happened until the recent unpleasantness.

We found that hardware was sometimes purchased without consulting the vendor, and then retrofitted with the vendor's help when it wasn't quite right. Perhaps both parties could be a little more proactive here, so we can all ask for permission instead of forgiveness.

Computers are made by imperfect human beings, and are thus imperfect themselves. To assume otherwise is naive at best. And so in a mission-critical area such as PACS, one must be ready for the inevitable glitch. There has to be a downtime plan, and an out-and-out disaster recovery solution. Guess what? We have neither. To my knowledge, the downtime plan hasn't been changed since I spoke at RANZCR in Perth in 2010: after four hours of outage, we start printing to film. Unfortunately, we no longer have any film printers. The next best thing, which we have been having to do, is to read directly from the modality's monitor. It isn't optimal, but it works, sort of. As for a full-fledged disaster, data is stored offsite as required. But it's on tape and recovering might take a very long time. If we could muster the resources. Let's hope it doesn't happen.

PACS, as it turns out, is the only Tier One service that does NOT have a proper downtime solution. Why did we get left out? Money. It was hard to justify a complete, mirrored, automatic fail-over that would only be used a small fraction of the time. Unless you happen to be a trauma patient in the Emergency Department, where life-and-death decisions are put on hold while someone fiddles with the server. Then it seems perfectly justified.

In the end, we all serve one customer, and that is YOU, the patient. Everything we do in this business, every decision we make, every scrap of hardware and line of code we purchase and use is meant to promote your health and well-being. It was said by some that we radiologists were "paying the price" for the various challenges I've outlined. That's true to some extent, but the real victims, at least potentially, are the patients, and that CANNOT be allowed to happen.

I've been blogging about PACS for almost 11 years, and my basic message hasn't really changed much. PACS IS the Radiology Department, and the hospital cannot function without it. Making this all work, and work properly, is in huge part a matter of communication. You have seen what happens when the discourse fails or doesn't happen at all. The downtime plan, or lack thereof, illustrates what happens when one of the groups involved in PACS, the rads, becomes disenfranchised with respect to the decision process. One of us could have very easily convinced the powers that be that we cannot tolerate a four-hour gap in service. We weren't asked to do so; we didn't even know the question had been posed. Now we do. And I am cautiously optimistic that this will improve, as will the rest of our experience.

I would be remiss if I didn't take the opportunity to excoriate the majority of PACS and EMR vendors while I'm on this particular rant. You are still not making user-friendly software. We all know it. PACS is bad enough, but our EMR and its CPOE (Computerized Physician Order Entry) is so very poorly written and implemented as to drive a good number of physicians into early retirement. Seriously. This garbage is served up as caviar to, and purchased by, those who DON'T HAVE TO USE IT, and again the physicians are disenfranchised. This too will negatively impact patient care, and it CANNOT, well, it SHOULD NOT be allowed to happen. But it is.

Let's have a meeting about THAT, shall we?

And Another New Pres For TR

Looks like TeraRecon has a new CEO:

TeraRecon, a global medical image management company, announced that Jeff Sorenson is the company’s new president.
Sorenson has been with TeraRecon since 2004. His most recent position with the company was senior vice president of sales and marketing.
“TeraRecon is unique because it offers a complete and truly vendor-neutral 3D advanced visualization suite which can be extended to serve the medical image viewing needs of the entire health enterprise," Sorenson said in a statement. “I am excited to serve our valued customers and to drive innovation in this new capacity as president.”
Meanwhile, Venkatraman Lakshminarayan has stepped down as CEO and CFO. Lakshminarayan had been TeraRecon’s CFO since 2005 and its CEO since 2014.
TeraRecon also reported that a successful August has led to an increase in iNteract+ interoperability and enterprise image-enablement projects.
As a (hopefully) valued customer, I'm looking forward to working with Mr. Sorenson as well as Fred and the rest of the gang. Congrats!