Having problems with message search?
Your arguments make little sense. For the record, I know exactly how SLMs operate, having worked side-by-side with some of the best speech scientists in the world to design, build, and tune some of the most advanced very-large grammars that take calls every day. I know how to test these systems. I know how to explain them to customers. I know how to make them easy for callers to interact with. I know how to deal with illogical behavior. Your points to the contrary are laughable and would be offensive if I didn't recognize them for what they are. I do wish you would know when to stop, though.
The article never said it was a QA or laboratory-type test. It was, though, highly representative of real-world opinions of our systems and those, whether you like it or not, or agree or not, matter far more than any testing protocol. What I want to see is our systems making real people happy, whether they "cooperate" with usage views such as yours or whether they read passages from Ulysses. Pointing at users and screaming "That's not fair! It's not a fair test!" will not make them happy. It will convince them further that the people producing technology don't give a crap about real people. I am not saying you are an evil people hater. As I've said before, I don't know you well enough to say such a thing. What I do know is that you frequently discuss and argue from a very system-centered view. Understanding and controlling the system is important, but will only be most beneficial when done so with a laser-focus on designing for the flawed, illogical, idiosyncratic, whimsical, wonderful people who will use it.
Can we be done now? Or perhaps, as a last word, you would like to educate us all on how "real testing" will save the IVR industry.
ph
On Thu, Jul 2, 2009 at 11:27 PM, Peter Nann <peter.nann@...> wrote:
I know! Thus my surprise.I'm not "pro bad systems", but I am very much against "bad testing".I know you love to turn people's arguments back on themselves, but I am not the evil people hater you are trying to make out I am.I am the one talking about making the system work for real users.You are wanting to see it work for fake, unrealistic test data.That will not help real users. Quite the contrary.If we were back at the farm, with the naive crowd laughing at the car stuck in the mud and taking their concerns seriously, we would all still be either a) riding horses or b) driving tractors, because the original users, who heard the car is proposed to replace the horse, all agree a car must be able to cut it in the field to do that.If you produced your real voice-mail box, with your weird and whacky voice-mails, and it didn't do an OK job on most of them, then we'd have something serious to talk about.Until then these sort of tests are pointless tyre-kicking.
Sent: Friday, 3 July 2009 11:42 AMThat is one of the funniest things anyone has declared to/about me in a long time.
Never mind, Peter. You go on standing up for those systems. They need all the help they can get.
Meanwhile, some of will design and build for people.
ph
On Thu, Jul 2, 2009 at 6:54 PM, Peter Nann <peter.nann@...> wrote:
"made complete sense" is completely irrelevant.The "design parameters" are simple - It's designed to work on real voice-mails.These tests were not real voice-mails.Phil, I am surprised at your apparent lack of understanding of how the SLM technology behind this works, and the concept of real test data.
Sent: Friday, 3 July 2009 1:23 AMPN,
You're right that I should have included the "owner" of the voice mailbox. So, my question corrected: "How would someone receiving or leaving a voice-mail have this kind of knowledge of the "design parameters" of the system?" Which is rhetorical, btw. Google markets it as a voicemail system that delivers transcriptions in email.
Those excerpts were not random. Though perhaps not typical, the audio content made complete sense to me, and I bet to all who sampled the article. As a husband/father/son/brother/friend/manager/employee/co-worker/acquaintance/business owner/... I have received voice mails of such varied content that I can assert that nearly anything goes in voice mail content, not to mention delivery aspects such as audio quality, speaker characteristics, etc. Either that long tail needs to be covered fairly well, or the system is nearly unusable.
If you want to stand by your system focus, so be it. Meanwhile, real people will look at the results in that article and say with conviction and validity, "That's why I hate that automated voice recognition crap. It can't even recognize the President's name!".
ph
On Thu, Jul 2, 2009 at 6:31 AM, Peter Nann <peter.nann@...> wrote:
The "leaver" of the message is not the user here. I place no assumption of anything on them.The person who signed up to get their received voice messages translated to text is the user.If that user expects this is going to work on any random excerpt of words thrown at it, then their expectations have not been properly managed.Do some REALISTIC voice-mail messages, from first word to last word, complete with a couple of appropriately contextualised Mills and Boon features if desired, then we can pull it apart.This was a bunch of "Top Gear challenge" style tests - Not indicative of real world requirements, and designed to have catastrophic failures.Amusing - yes, but the annoying thing is most seemed to hope for something better.I completely stand by my family car plus plough analogy.(No, that wasn't a Top Gear challenge that I know of...)
Sent: Thursday, 2 July 2009 1:17 PMCouldn't disagree with you more. Your analogy is completely off. How would someone leaving a voice-mail have the remotest idea of the "design parameters" of a system they don't even know about? That's old system-centered thinking. If a lover decided to read one of Shakespeare's sonnets to a sweetheart, why is the expectation of correct delivery by either party any less valid than someone leaving business meeting details?
ph
On Wed, Jul 1, 2009 at 9:36 PM, Peter Nann <peter.nann@...> wrote:
Guilty. I didn't take much notice of that part. My bad.I still get the feeling the article, and most people who responded, expected it to work on that stuff.And/or that one-day it should work well on it.In my opinion none of that showed the system was "broken".It was being used outside of its design parameters.It's kind of like taking the family car, finding that it doesn't pull a plough in the field very well, so saying "Geeze, they still have some development to do on that car idea..."My apologies again, I am stuck on a rant this week about bad test data...This touched a nerve...
PN,
Maybe there was a text reco problem on your end. ;-) The second paragraph pretty much puts the entire article into the proper light.
"The aim here was not to see how well the service, called Google Voice, handled transcriptions of the day-to-day mundanities of voice mail. Google itself acknowleges that the feature... is still sort of experimental. The company is pushing the limits of technology with this, so the goal was to see how far they could be pushed before they broke."
ph
--
www.design-outloud.com
www.twitter.com/designoutloud
vzn 469-853-9016
--
www.design-outloud.com
www.twitter.com/designoutloud
vzn 469-853-9016
--
www.design-outloud.com
www.twitter.com/designoutloud
vzn 469-853-9016
Offline 
Send Email