Wednesday, November 07, 2007

Dalai's Laws In Action...

My friend and colleague Henry (that is his name, but he doesn't go by it) called me a few hours ago. "Dalai," he said, "I've got bad news and worse news for you. The Bad news is that the downtown campus of one of our hospitals just got cut off from the Internet. Some ditch-digger severed a main AT&T cable, and we're completely dead. The Worse news is that YOU are on call! See ya!" Actually, he didn't just leave at that point, but he did his best to inform all possible clinical services, and especially the ER, that PACS would be dead. Fortunately, the hospital's suburban campus is still online, and we can divert the patients over there. For the moment. So far, I have only had to be rude to the downtown Nucs tech, who didn't get the word and did a V/Q anyway. I was able to send our guy who gets done at 8PM over there on his way home, so all is not lost, and the normal V/Q will be read in a timely fashion.

This is a wonderfully unfortunate demonstration of Dalai's Laws of PACS, in particular, Laws I and IV. Law I, "PACS is the radiology department" is quite obvious when PACS is dead. All we can do is look at the CT console for those direst of emergency cases that can't be shipped out. I suppose we could film the hundreds of CT slices....nah. Because Dalai's Fourth Law says, "Once PACS, never back." We are not going back to film. Ain't gonna happen.

We can do little but wait for AT&T to fix the communication problem. We were told that could take from 2 to 8 hours. Right now, we're at 9 hours and counting. What should we have done here? This is a situation that is foreseeable, but not easily solvable. We have backups upon backups, but we don't have a backup for the most critical component of all in a web-based system: the Network! While it is the Internet that failed here, we would be just as dead if our LAN failed, so I'll group them together. In today's case, however, it was obviously the WAN that croaked. At this point in time, there aren't too many alternatives for broadband access. In my town, we have DSL, run mainly by AT&T, and cable, run mainly by Time Warner. There are various permutations with MetroE's and Frame Relays and such, but ultimately, all of these require physical lines, wires, fiber optic cables, or whatever, and those can be cut. The only other possibilities would be satellite internet, which is relatively slow and expensive, or a direct microwave connection to an ISP, which would be fast but still expensive. (No, I don't have time to pull the numbers, being on call and all, but trust me on this.)

I'm thinking WIMAX, the promised nationwide WiFi system, might solve this problem. Put an antenna on the roof, and you're connected. It would certainly have avoided an entire hospital being shut down by a ditch digger. Although I suppose a lightning strike or other such event could knock out WIMAX as well. Maybe we need to go back to dial-up?

The only good to come of today's experience is that we are able to tell the ER docs to think long and hard before they order useless studies. OK, we didn't use the word "useless" but the message got across. Maybe the mentality will sink in. Probably not.


dj said...

We just created our network for an Amicas system based upon WIMAX and are installing wired backups for our sites. Works great until the server room floods!

Martin P aka PACSFerret II said...

Hey Dalai... If you are truly in an 'average' town, there's a good chance all the ISPs (including WIMAX when it arrives) contract to a single Tier-1 provider who only gives 99.9% uptime guarantees.
Such things can't happen you say? Hell if it can happen to Silicon Valley it can happen anywhere. You don't need to revert to film , though - a stash of CD-Rs and a Segway would suffice.