Monday, June 25, 2007

A Survivor Of Voice/Speech Recognition Speaks Out

(Dalai's note: I was approached by a colleague, and asked if I would post his VR/SR experiences online, but anonymously so he would not have the vendor in question breathing down his neck. I am more than happy to do so, and here it is....)

I will start by telling of our voice recognition (VR) experience.

About one and a half years ago we launched VR to our hospital system. It was an integrated “front end” VR product that resided on our PACS workstations. You may or may not have preconceived ideas about what quality we might have purchased, suffice it to say we purchased what for all intents and purposes, seemed to be the best VR product on the market. No we didn’t buy some outdated piece of garbage.

It was a planned 6-week launch with training provided primarily by the VR company. Initially, we were expecting things to be a little stormy; how wrong we were. First, we had scheduling problems with training, there were rads that were unavailable for training and the amount of time dedicated to each rad was short, only a few hours. In addition, the rads were learning while still having to attend to some clinical duties, which was a BIG MISTAKE.

So after a few weeks, the trainers left, there were still rads that had almost zero actual training. Our IT/IS support staff tried to help out as much as possible. It was just over a month when things in the department finally caved in. We were chronically short staffed, radiologist wise, and work wasn’t being finished. Those of us who took to VR somewhat well were being completely overrun by those who did not, both in having to field multiple near hysterical calls and a markedly increased workload. There were multiple unexplained system crashes (denied initially but fixed somewhat in a later patch) and a greatly degraded workstation performance.

We had numerous direct meeting between the administration, radiologists, and VR company. Here is what we got, repeated requests for the radiologists to document every occurrence of problems (yeah, right, like we weren’t totally swamped as it was what’s one more thing to do), repeated denials of their software being to blame for workstation slowness and crashes (of course later we would learn that yes it was their software causing crashes and system performance issues), repeated promises to fix the problems.

After almost 2 months of this I, being the “radiologist champion” for VR and about 2 seconds from a complete nervous breakdown, told myself “Either I stop using VR or I hand in my 90 day notice and look for a new job”. I turned off the VR product and told my fellow radiologists that I would no longer be fielding their questions about “how to make VR work”. The rest of the group followed suit immediately. It imploded over night. Problems that still existed at this time were:

1) Random crashed requiring a system reboot
2) Unexplained, or rather un-admitted, slowness of the workstation
3) Consistent patterns of errors in VR, it did not appear to be learning no matter how many times corrected
4) Almost no one in the department (IT/IS included) really knew how to use the advanced functions, macro generation was “trial and error”
5) Lack of advanced support, many question to our IS/IT people generated a response “I’ll send and email to VR company and see what they say”
6) Many errors sent for “transcriptionist editing” came back uncorrected, the error rate of “transcritpionist editing” was far greater than the old system. Many report errors escaped the radiologist review as well and went out finalized. Many, many months later I would learn we had some actual “transcriptionist sabotage” going on and this was NEVER disclosed to the radiologists.
7) A horrific loss of productivity in an already short staffed group.
It died for a long time; of course the hospital had a vested interest in making this work.

We recently had a visit to a “model group” for VR and got to see them in action. The model group watched us and decided to purchase the product anyway. They are a large group; they owned the transcription, PACS, and other things and “contracted” these things out to hospitals. This was a business decision by a group that owned the entire infrastructure. They did things very differently and claim a 90+% “sign off” rate. Here is what we observed and were told:

They began by sending their IT folks to the VR company for intensive training. They refused the VR company's offer to do radiologist training; they were going to handle that all in house. Prior to launch their entire IT/IS support staff had been using the product for “months” and were fluent in its use. This was before a radiologist touched it. After launch they had their trainers go back for additional training to learn to “hack” the word lexicon to overcome pattern of repeated errors. This has worked very well for them and is a process discouraged by the VR company. What we learned, to our dismay, was that VR was not just “appearing” to not learn certain repeated phrases or words, it REALLY WASN’T learning them at all, no matter how many times we made corrections manually. The "model group" placed skilled VR support staff available 24/7, to immediately respond and deal with radiologist problems with VR.

They began by doing a “staged” launch of VR, starting with sections and working 1 section at a time. Each radiologists was initially afforded a full half day of one-on-one training with no clinical duties (no phone calls, no tech interruptions, no exams needing to be read, NOTHING); after that they each received 3 full days of one-on-one training while working (they estimated a total of 24-28 training work hours for each rad). By one on one they mean the trainer was physically in the same room or sitting outside the radiologists door. They had section meetings every morning to discuss VR’s use and share tips; they relied on a lot of “peer pressure” to make things go smoothly. The entire launch was slated to take about 1 year.

The made sure their workstation greatly exceeded the VR companies recommendations. They made sure their transcriptionists were folded into other duties as they lost their need for so many of them. I would add here that prior to VR this groups report turn around was poor, over 2 days for routine stuff.

We were also told the group was now considering “financially penalizing” rads that are using too much transcriptionist editing, this was not implemented yet and the penalties they were considering was not shared with us.

We then got to observe some rads in action. It really wasn’t magic. First, they have employed radiology “assistants”. These folks do a lot of the “scut work”, phone calls and so on; they basically make sure the radiologists are not disturbed and the hospital pays for them. One rad was a “macro master” although he appeared to spend much more time looking at the VR than the exam. One rad absolutely hated the VR system, so all is not roses at this model group. The next few rads showed all the problems we have seen all along, lots of little annoying errors. Every rad admitted it slowed them down but they had “made up” for it through intense training, heavy assistant use, awesome IS support staff and excellent equipment. They did feel, well, most of them, that it was “worth it” for various reasons.

They did confirm many things we radiologists already knew. Things like the software’s inability to learn certain works on its own, or rather the pattern of making repeated patterns of errors. The VR company’s gross underestimation of the training needed and IS support staff needs. The performance issues on the VR companies “recommended” systems. Basically this is something the radiologists need to “buy into”. They did admit to the limitations of VR but that it is improving and “As long as the radiologists see continual improvement they will stick with it”. They also, on their own, identified and developed work-arounds for many conflicts the software was having with other office software products.

Where do we go from here? A few in my group had prior experiences with VR, all universally negative. I had a hope, as did the administration, that “This is such a superior product, it will be different this time”. It was very disappointing to see not only every fear confirmed, but also some new ones learned. Any trust between the VR company and the radiologists has been completely destroyed beyond repair. This experience has certainly made us much more cautious in trusting the word of any vendor, especially software vendors as I feel they may not fully grasp that they are selling a medical product and not a video game (i.e. doctors have zero tolerance for “buggy” software and more than they would tolerate a “buggy” CT scanner). So it stands with my group, many of whom now have the well justified attitude of “Hell will freeze over before I turn on VR again”. I am not sure this can ever be turned around. I hope, if nothing else, this review serves as a warning, to radiologists, administrators and most importantly VR vendors that VR is not something to be taken on lightly.

In a similar vein to Dalai Lama here are my 10 commandments for VR:

1) Voice recognition (VR) exists as only one of multiple possible solutions to address the problem of radiology report turn around time. The underlying problem being the near instantaneous distribution of images through PACS unaccompanied by the radiologist report.
2) VR will only see its maximum benefit (i.e. near instantaneous report generation) IF the radiologists self edit and sign off the majority of reports.
3) VR with or without self editing slows radiologists down. It costs productivity as radiologists are now forced to spend additional time either self editing or reviewing the VR generated reports for errors. Do not assume your radiologists are going to quietly eat this productivity loss.
4) The loss of radiologist productivity can be compensated, to some degree, by the use of radiologist assistants to do some tasks formerly done by the radiologist. Also there is a gain by the lessened need to re-review cases that have been dictated but not transcribed. Despite this there may still be a long term net productivity loss. Again, do not assume your radiologists are going to quietly eat this productivity loss.
5) Poor system performance is a guaranteed PACS/VR killer.
6) VR and PACS are by definition “beta” software. There is no such thing as a perfect PACS or VR software; there are only “acceptable” performance parameters that will improve over time. VR may never be perfect.
7) The hospital shall accept it’s responsibility for supporting VR by ensuring it’s IT/IS support staff are well versed in PACS and VR and available as long as the radiologist are working. The IS/IT support staff shall have a full working knowledge of VR and be capable of training radiologists before a radiologist ever touches the software.
8) VR requires far more IT/IS/training support that the VR companies will admit.
9) VR is not a big money saver if properly implemented. Any savings from transcriptionists will be quickly eaten up by the need for hardware upgrades, software upgrades, training and the needs for additional support staff.
10) Before implementing VR, ask yourself “Am I doing this for the right reasons and am I willing to commit additional resources, possible indefinitely, to the proper launching and support of VR?”


Anonymous said...

I wonder...where is your RIS or PACS administrator in all of this mess? It seems from your rant that the support of your VR system is only handled by your IT team. That is a problem in itself. The PACS administrator should be responsible for this and should be an expert. VR is never a turnkey solution, and is only as good as the people that are providing support. If that is going to be IT, then they better know it really well and be available 24/7 on the spot!

You also had transcriptionists that were not helping you out. They have to be held accountable for their role in this too. If they are to be just proof-readers then they should do it well. They can't just do a crappy job just because they feel slighted that they have been replaced with software.

And time is saved in the big picture because the report gets done quicker. If it means the radiologist spends more time than usual, then that is OK, because the transcriptionist is out of the equation and the report doesn't sit in some preliminary queue for hours or days waiting to be signed off by the radiologist. The report is available immediately. And if the radiologist doesn't want to deal with VR, most systems have the ability to just let the radiologist just dictate like normal and it gets sent to transcription...where they do THEIR JOB in making sure it is typed accurately.

I hear so many bad reports about VR, and they all have the same common thread which is, little or no specialized, qualified, in-house support. And transcriptionists that won't play proofreader. I don't think that it is something that can't work if it is established correctly from within first.

Anonymous said...

Gosh, why didn't I think of that! Thank you for, I guess, basically restating my entire point. That was my whole point, we lacked the support, the training was pathetic, our hospital dropped the ball big time, and the "model group" was all over it. Did you even read my entire post or just the first half?

And I can just say that many radiologists, including one at the "model group", do not share your "opinion" that radiologists having to spend more time generating reports is "OK" as long as the reports get out faster. Guess what, hiring more transcriptionists also has the same effect, reports get out faster. What is an hours worth of radiologist time worth compared to a days salary for a trascriptionist? If you don't know (and I do) then you should check and you'll see why it doesn't make sense. A small productivity loss results in a huge income loss (comparable to what a transcriptionist costs). So your argument is incomplete in that, like all VR advocates, you refuse to acknowledge that there are other solutions to poor report turn around time than VR.

Our transcriptionist was held accountable, since they do not work for us rads I have no idea what they did.

Just out of curiosity, since you say it is "OK" to lose radiologist productivity so the hospital can save a dime on transcriptionists, are you a radiologist, a vendor or someone else? Because seriously, if you are not a radiologist then you should not be saying that it is "OK" for radiologists to lose productivity so VR can work. Radiologist time is VERY valuable and should not be squandered.

Of course your last paragraph is EXACTLY correct. So you cannot be a vendor as the vendors I have spoken with are all too clueless to realize this and share this knowledge with their potential clients. But of course they have a motive not to share this don't they, if they told their potential clients how much support and training VR truely needed they wouldn't sell as many producuts.

Anonymous said...

And not wanting to run on to much about anon1. The arrogant attitide that "radiologist productivity is less important than turn around time" is probably the core of what leads to disasterous VR launches. Our "model group" the radiologists owned everything so they made absolutely sure to mitigate, as much as possible and perhaps completely, the productivity loss experienced through using VR. While our hospital, not having any financial interest in radiologist producitvity, did a really half-assed job and made no attempt whatsoever to mitigate the producitivity loss. Like I have maintained, make maintaining radiologist productivity your #1 priority for launching VR and all good decisions will follow, make saving transcriptionists salaries and turn around time your #1 priority and disaster will follow, it's that simple.

We do have a PACS administrator, he was involved but IMHO he is one man running an entire regional hospital systems PACS and, not to put him down because he is awesome, simply was not up to making sure VR ran perfectly IN ADDITION to making sure the rest of the system functioned perfectly. He is, sadly, another example of our hospital totally underestimating the amount of support needed to run PACS and VR.

Anonymous said...

To answer your question...I am a PACS/RIS Administrator. I realize that any concept that includes added workflow to radiologists is not going to be swallowed easy. But keep in mind the goal here. The radiologists job is not the central focus of radiology...the patient is. That report is the number one most important thing that radiology produces. It is the "product" so to speak. At the end of the day, what matters is that the report is accurate and gets into the physicians' hands matter how.

Now whether or not that means radiologists spending more time on each report, hiring extra transcriptionists, or whatever other options is ultimately up to the management who has to weigh each of these things out.

Obviously if radiologist productivity is negatively impacted then management has to consider adding more radiologists, changing dictation software, adding transcriptionists. Like I said, the final product is what counts.

My intent is not to piss off all the radiologists. I work with radiologists that are in both camps. I just think that there seems to be many people that have bad VR experiences out there because of poor planning or because of poor adoption by stubborn radiologists who don't like change. This in turn gives all VR products a bad name. We are actually trying to get VR at our facilities.

When we get VR, I plan to be a freakin' guru!

Anonymous said...

Would you be able to tell me who the vendor was? I am researching VR systems now and knowing if the vendor is my top candidate for this service, will be extremely useful. You can e-mail me if you do not want to post this information.