At 02:58 PM 1/9/2006, Desilets, Alain wrote:
>I don't have a good sense of the relative efficiency of upfront inspection
>(ex: Design Document Inspection) vs upfront testing (ex: Testing Paper
>Prototype) techniques for finding usability issues. But I suspect that
>both techniques are good, necessary and within the same range as far as
>efficiency goes.

In my opinion, the reason this is unclear is that it is *very* dependent on
the practitioners involved. Previous experience, exposure to research data,
innate skill, and a variety of other factors will affect both the results
of an inspection and the analysis of testing.

Something the usability/user experience community isn't addressing is how
much this is a craft, where individual prowess trumps collective wisdom and
process. We keep wanting to pretend what we do is an engineering
discipline, but unfortunately, it's a craft where the skill of the
craftsperson dictates success. This is the big elephant in the room that
no-one is talking about.

In research we conducted a while back, we traced where in the development
process usability problems were introduced into the design. Our major
finding was that they almost always were introduced at the same point:
someone on the design team needed to make a decision for which they didn't
have all the information necessary.

From this finding, I took away that the more important end result from any
type of usability activity is an informed decision.

Design Document Inspection can work if the inspection team arrives at
informed decisions in the process. That's going to depend on the existing
knowledge, skills, and deductive insights of the team. Paper Prototype
Testing arrives at informed decisions in a different manner. It's going to
depend on the test team's skills at creating tests, recruiting users, and
making proper inferences from the observations. Both techniques are as
likely to steer a team wrong as they are to steer a team right.

In other words, a poorly-executed usability activity will result in poor
decisions. There are no guarantees.

I think if we want to make usability into more of an engineering practice,
we need to control some of these variables a lot better.

Just my thoughts late this evening...

Jared


Jared M. Spool, Founding Principal, User Interface Engineering
4 Lookout Lane, Unit 4d, Middleton, MA 01949
978 777-9123 jspool@... http://www.uie.com
Blog: http://www.uie.com/brainsparks