Wednesday, December 17, 2008

Yes, SR Sucks

Speech recognition (SR, also incorrectly called Voice Recognition) is considered a boon to some (mainly administrators that don't have to use it) and a bane to others (most of the radiologists that DO have to use it).

There is a paucity of information out there on the topic, with most of the reports on and elsewhere being anecdotal, or written by those with an agenda on one side or the other.

From Ann Arbor, Michigan comes an article in the JACR that appears to be both objective and rather derogatory toward SR, and maybe a little bit toward docs as well. Leslie E. Quint, M.D., Douglas J. Quint, M.D., and James D. Myles, Ph.D wanted to

. . .determine the frequency and spectrum of significant dictation errors in finalized radiology reports generated with speech recognition technology.
To this end, they reviewed 265 reports, and found that 206 (78%) were accurate, containing no "significant" errors, but that means that 59 (22%) DID have "significant" errors. Those errors were:

Type of Error

Wrong word(s)-------------31
Nonsense phrase-----------19
Missing word(s)------------16
Wrong image number------9
Wrong date-------------------7
Wrong measurement-------4
Right/left substitution-----4
Missing or added ‘no’-------3
Nonsense date---------------2
Mm/cm substitution-------0


Some might not be impressed, but this really represents an unacceptable number of problem reports. Interestingly enough, the radiology faculty grossly underestimated the report error rate:

with only 8 of 88 rads guessing the error rate between 20 and 30%.

It is apparently a given that SR systems will bollux up a report, and that those poor docs forced to use SR should expect such. So:
The errors reflect a breakdown in the proofreading process, and there are multiple possible causes for this breakdown. Carelessness by the report signer is one possibility.
Personally, I chafe a little at this statement, but in the end, it is the radiologist's responsibility to send out an accurate report. Whining about the SR system (or the transcriptionist, for that matter) doesn't play well in a court of law. But still, if SR is doubling, tripling, quadrupling the number of errors in your reports, it becomes harder and harder to catch them all.

Quint, et. al. conclude:
Carelessness on the part of the editing physician or ill-conceived radiologist incentives (eg, rewarding speed over accuracy) may be partly to blame. Generally, the radiologists surveyed believed that the departmental error rate was much lower than it actually was, and most of the radiologists believed that they personally had lower than average error rates. This lack of awareness of the problem probably contributes to the high error rate. By delineating the frequency and spectrum of errors, we hope to raise awareness of this issue, thus facilitating methods for improvement.
So their point was to make everyone aware of the fact that SR creates lots of errors, and that those who use it need to be doubly vigilant.

I've got a better solution: Yes, I'll proofread carefully, but I'm keeping my human transcriptionists. Which is what I was going to do anyway.

1 comment :

alan said...

One of the problems with this, is something that I recently read on the AuntMinnie's website, which is true according to my experience. Humans have a difficult time seeing errors when they know the intent of what they said vs. what they see.

Add to that, the SR software inserts CORRECTLY SPELLED WORDS which are harder to see as errors, and that contributes to the high rate. If the errors were misspelled or whatever, it would be much easier, but when you know what you said, it almost reads what you think, even though this sounds illogical.