A Survivor Of Voice/Speech Recognition Speaks Out

(Dalai's note: I was approached by a colleague, and asked if I would post his VR/SR experiences online, but anonymously so he would not have the vendor in question breathing down his neck. I am more than happy to do so, and here it is....)


I will start by telling of our voice recognition (VR) experience.

About one and a half years ago we launched VR to our hospital system. It was an integrated “front end” VR product that resided on our PACS workstations. You may or may not have preconceived ideas about what quality we might have purchased, suffice it to say we purchased what for all intents and purposes, seemed to be the best VR product on the market. No we didn’t buy some outdated piece of garbage.

It was a planned 6-week launch with training provided primarily by the VR company. Initially, we were expecting things to be a little stormy; how wrong we were. First, we had scheduling problems with training, there were rads that were unavailable for training and the amount of time dedicated to each rad was short, only a few hours. In addition, the rads were learning while still having to attend to some clinical duties, which was a BIG MISTAKE.

So after a few weeks, the trainers left, there were still rads that had almost zero actual training. Our IT/IS support staff tried to help out as much as possible. It was just over a month when things in the department finally caved in. We were chronically short staffed, radiologist wise, and work wasn’t being finished. Those of us who took to VR somewhat well were being completely overrun by those who did not, both in having to field multiple near hysterical calls and a markedly increased workload. There were multiple unexplained system crashes (denied initially but fixed somewhat in a later patch) and a greatly degraded workstation performance.

We had numerous direct meeting between the administration, radiologists, and VR company. Here is what we got, repeated requests for the radiologists to document every occurrence of problems (yeah, right, like we weren’t totally swamped as it was what’s one more thing to do), repeated denials of their software being to blame for workstation slowness and crashes (of course later we would learn that yes it was their software causing crashes and system performance issues), repeated promises to fix the problems.

After almost 2 months of this I, being the “radiologist champion” for VR and about 2 seconds from a complete nervous breakdown, told myself “Either I stop using VR or I hand in my 90 day notice and look for a new job”. I turned off the VR product and told my fellow radiologists that I would no longer be fielding their questions about “how to make VR work”. The rest of the group followed suit immediately. It imploded over night. Problems that still existed at this time were:

1) Random crashed requiring a system reboot
2) Unexplained, or rather un-admitted, slowness of the workstation
3) Consistent patterns of errors in VR, it did not appear to be learning no matter how many times corrected
4) Almost no one in the department (IT/IS included) really knew how to use the advanced functions, macro generation was “trial and error”
5) Lack of advanced support, many question to our IS/IT people generated a response “I’ll send and email to VR company and see what they say”
6) Many errors sent for “transcriptionist editing” came back uncorrected, the error rate of “transcritpionist editing” was far greater than the old system. Many report errors escaped the radiologist review as well and went out finalized. Many, many months later I would learn we had some actual “transcriptionist sabotage” going on and this was NEVER disclosed to the radiologists.
7) A horrific loss of productivity in an already short staffed group.
It died for a long time; of course the hospital had a vested interest in making this work.

We recently had a visit to a “model group” for VR and got to see them in action. The model group watched us and decided to purchase the product anyway. They are a large group; they owned the transcription, PACS, and other things and “contracted” these things out to hospitals. This was a business decision by a group that owned the entire infrastructure. They did things very differently and claim a 90+% “sign off” rate. Here is what we observed and were told:

They began by sending their IT folks to the VR company for intensive training. They refused the VR company's offer to do radiologist training; they were going to handle that all in house. Prior to launch their entire IT/IS support staff had been using the product for “months” and were fluent in its use. This was before a radiologist touched it. After launch they had their trainers go back for additional training to learn to “hack” the word lexicon to overcome pattern of repeated errors. This has worked very well for them and is a process discouraged by the VR company. What we learned, to our dismay, was that VR was not just “appearing” to not learn certain repeated phrases or words, it REALLY WASN’T learning them at all, no matter how many times we made corrections manually. The "model group" placed skilled VR support staff available 24/7, to immediately respond and deal with radiologist problems with VR.

They began by doing a “staged” launch of VR, starting with sections and working 1 section at a time. Each radiologists was initially afforded a full half day of one-on-one training with no clinical duties (no phone calls, no tech interruptions, no exams needing to be read, NOTHING); after that they each received 3 full days of one-on-one training while working (they estimated a total of 24-28 training work hours for each rad). By one on one they mean the trainer was physically in the same room or sitting outside the radiologists door. They had section meetings every morning to discuss VR’s use and share tips; they relied on a lot of “peer pressure” to make things go smoothly. The entire launch was slated to take about 1 year.

The made sure their workstation greatly exceeded the VR companies recommendations. They made sure their transcriptionists were folded into other duties as they lost their need for so many of them. I would add here that prior to VR this groups report turn around was poor, over 2 days for routine stuff.

We were also told the group was now considering “financially penalizing” rads that are using too much transcriptionist editing, this was not implemented yet and the penalties they were considering was not shared with us.

We then got to observe some rads in action. It really wasn’t magic. First, they have employed radiology “assistants”. These folks do a lot of the “scut work”, phone calls and so on; they basically make sure the radiologists are not disturbed and the hospital pays for them. One rad was a “macro master” although he appeared to spend much more time looking at the VR than the exam. One rad absolutely hated the VR system, so all is not roses at this model group. The next few rads showed all the problems we have seen all along, lots of little annoying errors. Every rad admitted it slowed them down but they had “made up” for it through intense training, heavy assistant use, awesome IS support staff and excellent equipment. They did feel, well, most of them, that it was “worth it” for various reasons.

They did confirm many things we radiologists already knew. Things like the software’s inability to learn certain works on its own, or rather the pattern of making repeated patterns of errors. The VR company’s gross underestimation of the training needed and IS support staff needs. The performance issues on the VR companies “recommended” systems. Basically this is something the radiologists need to “buy into”. They did admit to the limitations of VR but that it is improving and “As long as the radiologists see continual improvement they will stick with it”. They also, on their own, identified and developed work-arounds for many conflicts the software was having with other office software products.

Where do we go from here? A few in my group had prior experiences with VR, all universally negative. I had a hope, as did the administration, that “This is such a superior product, it will be different this time”. It was very disappointing to see not only every fear confirmed, but also some new ones learned. Any trust between the VR company and the radiologists has been completely destroyed beyond repair. This experience has certainly made us much more cautious in trusting the word of any vendor, especially software vendors as I feel they may not fully grasp that they are selling a medical product and not a video game (i.e. doctors have zero tolerance for “buggy” software and more than they would tolerate a “buggy” CT scanner). So it stands with my group, many of whom now have the well justified attitude of “Hell will freeze over before I turn on VR again”. I am not sure this can ever be turned around. I hope, if nothing else, this review serves as a warning, to radiologists, administrators and most importantly VR vendors that VR is not something to be taken on lightly.

In a similar vein to Dalai Lama here are my 10 commandments for VR:

1) Voice recognition (VR) exists as only one of multiple possible solutions to address the problem of radiology report turn around time. The underlying problem being the near instantaneous distribution of images through PACS unaccompanied by the radiologist report.
2) VR will only see its maximum benefit (i.e. near instantaneous report generation) IF the radiologists self edit and sign off the majority of reports.
3) VR with or without self editing slows radiologists down. It costs productivity as radiologists are now forced to spend additional time either self editing or reviewing the VR generated reports for errors. Do not assume your radiologists are going to quietly eat this productivity loss.
4) The loss of radiologist productivity can be compensated, to some degree, by the use of radiologist assistants to do some tasks formerly done by the radiologist. Also there is a gain by the lessened need to re-review cases that have been dictated but not transcribed. Despite this there may still be a long term net productivity loss. Again, do not assume your radiologists are going to quietly eat this productivity loss.
5) Poor system performance is a guaranteed PACS/VR killer.
6) VR and PACS are by definition “beta” software. There is no such thing as a perfect PACS or VR software; there are only “acceptable” performance parameters that will improve over time. VR may never be perfect.
7) The hospital shall accept it’s responsibility for supporting VR by ensuring it’s IT/IS support staff are well versed in PACS and VR and available as long as the radiologist are working. The IS/IT support staff shall have a full working knowledge of VR and be capable of training radiologists before a radiologist ever touches the software.
8) VR requires far more IT/IS/training support that the VR companies will admit.
9) VR is not a big money saver if properly implemented. Any savings from transcriptionists will be quickly eaten up by the need for hardware upgrades, software upgrades, training and the needs for additional support staff.
10) Before implementing VR, ask yourself “Am I doing this for the right reasons and am I willing to commit additional resources, possible indefinitely, to the proper launching and support of VR?”