there's a whole chapter on the spec workshops, and the idea is mentioned
in most other chapters as well.

--
gojko adzic
http://gojko.net


Jeremy wrote:
>
>
> Gojko,
>
> Is the specification workshop covered more in your book with examples?
> The ideas still seems pretty vague on how really get things the ideas
> and people together. I haven't ever tried one and wondered on the
> specifics. Deliverables and other techniques?
>
> Jeremy
>
> Gojko Adzic wrote:
>  >
>  >
>  > > Is there a good method to bring the team around to quality being their
>  > > responsibility? A smack in the face with the quality stick? Or make
> they
>  > > eat their own dogfood until they realize how important it is?
>  >
>  > specification workshops are a great way to do this, as they give testers
>  > a platform to explain problems and get everyone else on board:
>  >
>  > http://www.acceptancetesting.info/key-ideas/specification-workshop/
> <http://www.acceptancetesting.info/key-ideas/specification-workshop/>
>  > <http://www.acceptancetesting.info/key-ideas/specification-workshop/
> <http://www.acceptancetesting.info/key-ideas/specification-workshop/>>
>  >
> http://www.acceptancetesting.info/key-ideas/collaborative-specifications/
> <http://www.acceptancetesting.info/key-ideas/collaborative-specifications/>
>  >
> <http://www.acceptancetesting.info/key-ideas/collaborative-specifications/
> <http://www.acceptancetesting.info/key-ideas/collaborative-specifications/>>
>  >
>  > --
>  > gojko adzic
>  > http://gojko.net <http://gojko.net> <http://gojko.net <http://gojko.net>>
>  >
>  >
>
>

> If what the outcome are acceptance tests in a human read and
> executable form, then we agree. If the outcome are just minutes of
> the meeting and/or a picture of the white board (even with agreement
> to eventually create the acceptance tests), then we still disagree.
>

there are two outcomes.

one is tangible - photos of the whiteboard where the examples were,
ideally in a form that will be straightforward to clean up and convert
to acceptance tests.

one is a lot less tangible but in my mind more important - shared
understanding of what needs to be done. the discussion during the
workshop facilitates a transfer of knowledge and building a shared
understanding of what the piece of software should do when it is done,
why it needs to do it and what it doesn't need to do.

a workshop is not a meeting, it's not a design session and it is not a
presentation. it is a collaborative domain exploration exercise when you
have the scope and aims to nail down the specification.

--
gojko adzic
http://gojko.net

On Sun, Feb 1, 2009 at 5:10 AM, Gojko Adzic <gojko-yahoolist@...> wrote:
>
>> If what the outcome are acceptance tests in a human read and
>> executable form, then we agree. If the outcome are just minutes of
>> the meeting and/or a picture of the white board (even with agreement
>> to eventually create the acceptance tests), then we still disagree.
>>
>
> there are two outcomes.
>
> one is tangible - photos of the whiteboard where the examples were,
> ideally in a form that will be straightforward to clean up and convert
> to acceptance tests.
>
> one is a lot less tangible but in my mind more important - shared
> understanding of what needs to be done. the discussion during the
> workshop facilitates a transfer of knowledge and building a shared
> understanding of what the piece of software should do when it is done,
> why it needs to do it and what it doesn't need to do.
>
> a workshop is not a meeting, it's not a design session and it is not a
> presentation. it is a collaborative domain exploration exercise when you
> have the scope and aims to nail down the specification.
>

I fully agree with the primary objective of creating a shared vision.

I totally disagree with delaying the conversion into acceptance tests
more than a day, especially if it ends up getting handed off to
somebody who was not party to creating the shared vision.   Too many
times, I have seen a troublesome disconnect between the customer's
vision and what the acceptance tests end up requiring.  Any
intermediate representation of the requirements is wasteful and a
source of miscommunication.

Anyway, the kind of quality that the original questioner has in mind
seems to have very little to do with a shared vision of what the
software should do, the existence of acceptance tests, or satisfying
acceptance tests.  The original questioner is concerned with a shared
vision of what the quality of the code should be.

Would you recommend a workshop to come to a consensus on a shared
vision for code quality?  Would you expect the original questioner to
share that consensus vision if everyone else believed in a lesser need
for code quality?

> --
> gojko adzic
> http://gojko.net
>

> Would you recommend a workshop to come to a consensus on a shared
> vision for code quality? Would you expect the original questioner to
> share that consensus vision if everyone else believed in a lesser need
> for code quality?

that depends on the definition of code quality. if by that you mean
"solves the business problem for the clients in a way that they want",
then it falls very much under the category of acceptance tests, building
a shared understanding of the problem etc. if code quality is technical
code quality, eg following best practices when you are developing
software such as having good unit test coverage, avoiding duplication,
applying SOLID design principles etc, then this is a purely technical
discussion which can be solved by organising training sessions, and
mentoring.

--
gojko adzic
http://gojko.net

--- In agile-testing@yahoogroups.com, John Overbaugh
<john.overbaugh@...> wrote:

> There's a lot to be learned about engineering process and
discipline. I only
> sort of wish that we could get there with software engineering. I
say sort
> of because 1) it'd be pretty gratifying to reach the point where we
could
> build predictable quality every time, but 2) it would be so very
boring to
> do the same thing, over and over.

Sorry I snipped so much of this excellent post. When something is
"sort of" correct, isn't that a smell?

Suppose you just abandon the metaphor altogether.  Instead of
considering delivering software as a physical product that we
manufacture over and over, consider it as delivering a performance to
an audience.  We can give the same high-quality performance over and
over, even though we know that each performance and each audience is
unique and will never be repeated.  Doesn't that make more sense?
Doesn't that speak to an entirely different set of skills than the
skills required by the factory floor or the construction site?

Hello, Steven.  On Sunday, February 1, 2009, at 1:53:49 AM, you
wrote:

> The reality is that meeting the actual needs of the market beats
> quality (which is why we observe so many low quality systems surviving
> in the wild).  Get over it.  Just focus on how to attain the most
> quality we can while still delivering fast enough to discover, evolve
> and implement the right requirements faster than our competitors.

I disagree strongly with this and have blogged on the subject:

Quality-Speed Tradeoff — You’re kidding yourself.
http://xprogramming.com/blog/2009/02/01/quality-speed-tradeoff-youre-kidding-you\
rself/

Ron Jeffries
www.XProgramming.com
www.xprogramming.com/blog
I cannot find my duck.

2009/2/1 Ron Jeffries <ronjeffries@...>:
> Hello, Steven.  On Sunday, February 1, 2009, at 1:53:49 AM, you
> wrote:
>
>> The reality is that meeting the actual needs of the market beats
>> quality (which is why we observe so many low quality systems surviving
>> in the wild).  Get over it.  Just focus on how to attain the most
>> quality we can while still delivering fast enough to discover, evolve
>> and implement the right requirements faster than our competitors.
>
> I disagree strongly with this and have blogged on the subject:

Ron,

I think we disagree much less than you might think.  I think we both
agree that delivering fast and consistently enough to discover, evolve
and implement the right requirements faster than our competitors
requires a whole lot of consistent quality, but does not require the
same level of perfection that a mission critical system would.  Honing
the code any more than is required to facilitate future delivery is
likely to be a premature commitment to design patterns that might be
more complex that required, and is quite possibly wasted effort if
feedback tells us that the requirements we just implemented turned out
to be wrong.

I just believe we should let the objective of continuous delivery
drive the quality objectives instead of the developer or tester who
complains the loudest (and accuses everyone on the team who disagrees
of being uncaring or unprofessional).  If there is not at least one
concrete case where the code written in iteration N had to be reworked
(rather than refactored) in iteration N+K in order to deliver, then
maybe the team is creating sufficient quality despite the opinion of
the "holiest" team member.

If there is such an example brought up in the retrospective, then the
need for quality improvement can be driven by a concrete problem
instead of a rant.

Steve

>
> Quality-Speed Tradeoff — You're kidding yourself.
>
http://xprogramming.com/blog/2009/02/01/quality-speed-tradeoff-youre-kidding-you\
rself/
>
> Ron Jeffries
> www.XProgramming.com
> www.xprogramming.com/blog
> I cannot find my duck.
>
>
> ------------------------------------
>
> Yahoo! Groups Links
>
>
>
>

Hello, Steven.  On Sunday, February 1, 2009, at 6:29:24 PM, you
wrote:

> I just believe we should let the objective of continuous delivery
> drive the quality objectives instead of the developer or tester who
> complains the loudest (and accuses everyone on the team who disagrees
> of being uncaring or unprofessional).

You seem to have rants on your mind just now. I do not favor rants
as far as I am aware. The above behavior may or may not be
indicative of something wrong with the code, as your later(?)
posting points out. It /is/ indicative of something wrong with the
team's communications, and quite likely indicates that they are not
working together (or at least not well) and do not have a common
understanding of their coding standards. (See the XP Practice,
"Coding Standard")

> If there is not at least one
> concrete case where the code written in iteration N had to be reworked
> (rather than refactored) in iteration N+K in order to deliver, then
> maybe the team is creating sufficient quality despite the opinion of
> the "holiest" team member.

See also the XP Value "Respect". It seems to me to be obviously too
late to wait until the accident to put up the signal. The team
should be monitoring its code quality long before rework starts to
show up.

> If there is such an example brought up in the retrospective, then the
> need for quality improvement can be driven by a concrete problem
> instead of a rant.

Concrete examples are always good. I would take a rant as evidence
of one or more of the following:

   - Code in the system is not up to snuff;
   - Management is putting on pressure to go faster;
   - Some people are taking such pressure on themselves;
   - Some people are not as good at coding as they may think;
   - The team does not have a shared definition of quality;
   - The team does not work as a team.

I suspect that the first of these is almost always in operation, not
to the exclusion of others. How sure am I? I will visit any team
anywhere and inspect the code that has been called out. If I cannot
describe immediately why it is not up to snuff, I'll waive my fee.

Yes, sure, sometimes people call out issues when there aren't any.
But even if they anger us, it's likely that they see something
important.

Ron Jeffries
www.XProgramming.com
www.xprogramming.com/blog
Each of you is perfect the way you are ...
and you can use a little improvement. -- Suzuki Roshi

Caveat 1: I work for Atlassian
Caveat 2: Code coverage is only as meaningful as the tests being measured

John Overbaugh wrote:
> Magellan can be instrumented such that it literally associates lines of code
with test
> cases--automated or manual. Then as code changes, (the theory is) all you
> need to do is run the affected test cases.

Atlassian's Clover code coverage tool provides a similar function via its test
optimisation
feature for automated tests. It instruments your code, notes which lines of code
are
executed by your tests and runs only those tests which execute modified code
(plus any
new tests) during subsequent builds.
More information can be found here:
http://www.atlassian.com/software/clover/features/optimization.jsp

Note this will only be of use if;
1. Your source code is java
2. You can run your tests in the same JVM as the app your testing (test
optimisation does
not yet support distributed tests, but we're looking into it.)

If you still need to tie tests to use cases then there's a plethora of test
management tools
that can readily do this (including the humble spreadsheet). If you store your
tests in a
source code management tool like svn then commit messages should do the trick.

Regards,
Andrew

On Mon, Feb 2, 2009 at 7:49 AM, Ron Jeffries
<ronjeffries@...> wrote:
> I would take a rant as evidence
> of one or more of the following:
>
> - Code in the system is not up to snuff;
> - Management is putting on pressure to go faster;
> - Some people are taking such pressure on themselves;
> - Some people are not as good at coding as they may think;
> - The team does not have a shared definition of quality;
> - The team does not work as a team.
>
> I suspect that the first of these is almost always in operation, not
> to the exclusion of others. How sure am I? I will visit any team
> anywhere and inspect the code that has been called out. If I cannot
> describe immediately why it is not up to snuff, I'll waive my fee.
>

If by "why" you mean exactly how you can tell the code is not up to
snuff, I believe you.  If by "why", you mean the root causes of the
code not being up to snuff, I somehow doubt you can determine that
immediately.

Isn't crappy code is a symptom of deeper problems.  Have you ever
visited a team where crappy code was the only problem they had?

>
> Ron Jeffries
> www.XProgramming.com
> www.xprogramming.com/blog
> Each of you is perfect the way you are ...
> and you can use a little improvement. -- Suzuki Roshi
>

On Sun, Feb 01, 2009 at 10:33:33PM -0000, andrew_prentice_os wrote:
-> Caveat 1: I work for Atlassian
-> Caveat 2: Code coverage is only as meaningful as the tests being measured
->
-> John Overbaugh wrote:
-> > Magellan can be instrumented such that it literally associates lines of
code with test
-> > cases--automated or manual. Then as code changes, (the theory is) all you
-> > need to do is run the affected test cases.
->
-> Atlassian's Clover code coverage tool provides a similar function via its
test optimisation
-> feature for automated tests. It instruments your code, notes which lines of
code are
-> executed by your tests and runs only those tests which execute modified code
(plus any
-> new tests) during subsequent builds.
-> More information can be found here:
-> http://www.atlassian.com/software/clover/features/optimization.jsp

I feel compelled to say: if you're coding in Python, I added an
experimental feature to figleaf that lets you do things like this:

http://ivory.idyll.org/blog/feb-07/figleaf-goodness.html

It doesn't (yet) let you run only those tests which execute modified
code, although that's a good idea :)

cheers,
-titus
--
C. Titus Brown, ctb@...

First off, there was a related discussion in this group about 7months
ago on "Traceability Matrix in an Agile Project" that was nicely
summarized at
* http://www.infoq.com/news/2008/06/agile-traceability-matrix

I also wrote a Sept 2007 article entitled "Lean Traceability: a
smattering of strategies and solutions" that started with something like
"Trustworthy Transparency over Tiresome Traceability" over at:
* http://www.cmcrossroads.com/content/view/9089/264/

The article is basically for when you are in the situation of being
required to do traceability and the appropriate understanding and
mind-set to take, plus several better and simpler alternatives to manual
maintained matrices. (Also see 'eXtreme Traceability" at
http://blog.bradapp.net/2006/04/extreme-traceability.html)

More interestingly, about two years ago I was asked to sit on a panel
entitled "Just Enough Requirements Traceability" in the "Requirements
Engineering Track" of the IEEE COMPSAC conference
(http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4020054)
My assignment was to represent the "Agile" perspective on the issue
(which was, first and foremost to question why and if it really and
truly is necessary).

I was expecting to be just about the only person on the panel taking
that minimalist perspective and questioning the value of it. I was
surprised at how wrong I was about that. While there were 1-2 panelists
on the strongly pro-traceability side (or who simply didnt question the
need for it, because in their line of work it has always been
contractually or legally mandated), most of the rest of the folks on the
panel were plain about traceability being such a high-effort and
high-cost activity as to not be cost-effective to do when it is not
legally or contractually mandated.

One person even suggested that, even if you project decided it "needed
to do traceability" that you should still question "why", and perhaps
your need is only a "subset" of the shopping list of reasons often
supplied for "wanting" traceability. In which case, he suggested doing
traceability ONLY for those things where it was needed (perhaps only on
certain kinds of requirements that impacted that critical "quality
attribute", or perhaps only between requirements and tests, and not
necessarily from requirements to code).

A year after that, I ran into someone from the SEI that is part of their
Software Architecture group and teaches some of their courses on
software architecture. She mentioned to me that some of her Ph.D work
was on the subject of traceability. I mentioned that I'd be interested
in hearing more about her work in that area (seeing as it's an area I'm
often consulted on from both a CM perspective and an Agile one). She
immediately responded saying something like: "You mean aside from the
fact that it's a HUGE waste of time & money for most organzations that
provides little value (if any) and never as much as presumed?"

--
Brad Appleton <brad {AT} bradapp.net>
    Agile CM Environments (http://blog.bradapp.net/)
    & Software CM Patterns (www.scmpatterns.com)
"And miles to go before I sleep" -- Robert Frost

I agree for most projects its a complete waste of effort and money.
However, working in the Pharmaceutical industry, we are FDA Regulated
and mandated to 'provide documentary evidence that software works as
specified'.  So some form of traceability between requirement and
tests is needed.

I do it, by naming/grouping test suites (fitnesse, Selenium, etc) to a
Story or functional area of the system.  Either way, each story has a
clear set of automated tests.   The problem I'm stuck with, is that
the FDA Auditors don't get this and so we still have to produce a word
doc detailing it.  Keep in mind, if an FDA Auditor doesn't pass your
software audit, ANY drug made using that software can be rejected for
use in humans at worse or a LARGE penalty fine at best.


A


2009/2/2 Brad Appleton <Brad.Appleton@...>:
> First off, there was a related discussion in this group about 7months
> ago on "Traceability Matrix in an Agile Project" that was nicely
> summarized at
> * http://www.infoq.com/news/2008/06/agile-traceability-matrix
>
> I also wrote a Sept 2007 article entitled "Lean Traceability: a
> smattering of strategies and solutions" that started with something like
> "Trustworthy Transparency over Tiresome Traceability" over at:
> * http://www.cmcrossroads.com/content/view/9089/264/
>
> The article is basically for when you are in the situation of being
> required to do traceability and the appropriate understanding and
> mind-set to take, plus several better and simpler alternatives to manual
> maintained matrices. (Also see 'eXtreme Traceability" at
> http://blog.bradapp.net/2006/04/extreme-traceability.html)
>
> More interestingly, about two years ago I was asked to sit on a panel
> entitled "Just Enough Requirements Traceability" in the "Requirements
> Engineering Track" of the IEEE COMPSAC conference
> (http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4020054)
> My assignment was to represent the "Agile" perspective on the issue
> (which was, first and foremost to question why and if it really and
> truly is necessary).
>
> I was expecting to be just about the only person on the panel taking
> that minimalist perspective and questioning the value of it. I was
> surprised at how wrong I was about that. While there were 1-2 panelists
> on the strongly pro-traceability side (or who simply didnt question the
> need for it, because in their line of work it has always been
> contractually or legally mandated), most of the rest of the folks on the
> panel were plain about traceability being such a high-effort and
> high-cost activity as to not be cost-effective to do when it is not
> legally or contractually mandated.
>
> One person even suggested that, even if you project decided it "needed
> to do traceability" that you should still question "why", and perhaps
> your need is only a "subset" of the shopping list of reasons often
> supplied for "wanting" traceability. In which case, he suggested doing
> traceability ONLY for those things where it was needed (perhaps only on
> certain kinds of requirements that impacted that critical "quality
> attribute", or perhaps only between requirements and tests, and not
> necessarily from requirements to code).
>
> A year after that, I ran into someone from the SEI that is part of their
> Software Architecture group and teaches some of their courses on
> software architecture. She mentioned to me that some of her Ph.D work
> was on the subject of traceability. I mentioned that I'd be interested
> in hearing more about her work in that area (seeing as it's an area I'm
> often consulted on from both a CM perspective and an Agile one). She
> immediately responded saying something like: "You mean aside from the
> fact that it's a HUGE waste of time & money for most organzations that
> provides little value (if any) and never as much as presumed?"
>
> --
> Brad Appleton <brad {AT} bradapp.net>
> Agile CM Environments (http://blog.bradapp.net/)
> & Software CM Patterns (www.scmpatterns.com)
> "And miles to go before I sleep" -- Robert Frost
>
>

On Mon, Feb 2, 2009 at 10:38 AM, Andrew McDonagh
<andrewmcdonagh@...> wrote:
> I agree for most projects its a complete waste of effort and money.
> However, working in the Pharmaceutical industry, we are FDA Regulated
> and mandated to 'provide documentary evidence that software works as
> specified'. So some form of traceability between requirement and
> tests is needed.
>
> I do it, by naming/grouping test suites (fitnesse, Selenium, etc) to a
> Story or functional area of the system. Either way, each story has a
> clear set of automated tests. The problem I'm stuck with, is that
> the FDA Auditors don't get this and so we still have to produce a word
> doc detailing it.

It should not be hard to write a script that extracts the names of the
tests that are run for each story, so you can just run it upon demand
instead of maintaining the document.  Placing that information in a
word doc format would be only a little more difficult.

> Keep in mind, if an FDA Auditor doesn't pass your
> software audit, ANY drug made using that software can be rejected for
> use in humans at worse or a LARGE penalty fine at best.
>
> A
>

That's right, it isn't.  But then there's all sorts of other bumph
that the corporations Computer System Validation people want/need us
to add to it.   Keep in mind, the corporation is very Waterfall
oriented.  We have 37 documents that can be used on a single project,
regardless of size.  These documents have appeared over time with the
best of intentions....  My favourite useless ones is the TIP
(Technical Installation Plan) and TIR (Technical Installation Report).
  They are useless, because they are usually written by the people
installing the software, for themselves....because they are told to
write them, not because they need them.

2009/2/2 Steven Gordon <sgordonphd@...>:
> On Mon, Feb 2, 2009 at 10:38 AM, Andrew McDonagh
> <andrewmcdonagh@...> wrote:
>> I agree for most projects its a complete waste of effort and money.
>> However, working in the Pharmaceutical industry, we are FDA Regulated
>> and mandated to 'provide documentary evidence that software works as
>> specified'. So some form of traceability between requirement and
>> tests is needed.
>>
>> I do it, by naming/grouping test suites (fitnesse, Selenium, etc) to a
>> Story or functional area of the system. Either way, each story has a
>> clear set of automated tests. The problem I'm stuck with, is that
>> the FDA Auditors don't get this and so we still have to produce a word
>> doc detailing it.
>
> It should not be hard to write a script that extracts the names of the
> tests that are run for each story, so you can just run it upon demand
> instead of maintaining the document. Placing that information in a
> word doc format would be only a little more difficult.
>
>> Keep in mind, if an FDA Auditor doesn't pass your
>> software audit, ANY drug made using that software can be rejected for
>> use in humans at worse or a LARGE penalty fine at best.
>>
>> A
>>
>

>That's right, it isn't. But then there's all sorts of other bumph
that the corporations Computer System Validation people want/need us
to add to it. Keep in mind, the corporation is very Waterfall
oriented. We have 37 documents that can be used on a single project,
regardless of size. These documents have appeared over time with the
best of intentions.... My favourite useless ones is the TIP
(Technical Installation Plan) and TIR (Technical Installation Report).
They are useless, because they are usually written by the people
installing the software, for themselves....because they are told to
write them, not because they need them.

I'm curious:  what would the FDA auditor say if you brought to his/her
attention the risk associated with opportunity cost--that is, we know less
about the product /specifically because/ we're doing wasteful work that
takes time and resources away from useful work?

---Michael B.

>>Good point, I have wondered that myself before - is manufacturing QC
similar to software QC? One of my coworkers is an industrial engineer, I
will ask him. It seems like quality values would translate from one industry
to another. Maybe the feedback loop in a machine shop is quicker? If you
don't size the part exactly right, it gums up the works right away?

STRG:>In my opinion, if at all we have to compare software engineering with
manufacturing, the right analogy will be comparing SE with design activity
in manufacturing and not the manufacturing itself. Design activity is a
mental process and mistakes in assumptions, choosing right standards,
tolerances, loads, environmental conditions, etc will affect the final
product. and therefore, design activity is more comparable with software
development and testing process.

+1.  To me, software engineering is less like mass production--producing
zillions of uniform widgets--and more like design--making a newly-designed
widget each time out.  Similarly, I think less of a software development
industry and more of a variety of software development crafts.  Very
broadly, the factory model takes a good deal of responsibility for the
quality of the work out of the hands of the workers.  Craftspeople make
decisions about what fits and what works for them, and remain in close
contact with their clients.

---Michael B.

They'd fail the audit, because the documentation is required but if we
can't produce good quality evidence and have faith in it....and then
admit its failing.....     game over...  Its not the document or
process's fault...its ours...    (in their view)

2009/2/2 Michael Bolton <mb@...>:
>>That's right, it isn't. But then there's all sorts of other bumph
> that the corporations Computer System Validation people want/need us
> to add to it. Keep in mind, the corporation is very Waterfall
> oriented. We have 37 documents that can be used on a single project,
> regardless of size. These documents have appeared over time with the
> best of intentions.... My favourite useless ones is the TIP
> (Technical Installation Plan) and TIR (Technical Installation Report).
> They are useless, because they are usually written by the people
> installing the software, for themselves....because they are told to
> write them, not because they need them.
>
> I'm curious: what would the FDA auditor say if you brought to his/her
> attention the risk associated with opportunity cost--that is, we know less
> about the product /specifically because/ we're doing wasteful work that
> takes time and resources away from useful work?
>
> ---Michael B.
>
>

>> I'm curious: what would the FDA auditor say if you brought to his/her
>> attention the risk associated with opportunity cost--that is, we know
less
>> about the product /specifically because/ we're doing wasteful work that
>> takes time and resources away from useful work?

>They'd fail the audit, because the documentation is required but if we
can't produce good quality evidence and have faith in it....and then admit
its failing..... game over... Its not the document or process's fault...its
ours... (in their view)

I don't understand this part of the sentence:  "because the documentation is
required but if we can't produce good quality evidence and have faith in
it....and then admit its failing..."  Could you help me out here?

First, I didn't suggest /not/ producing the documentation (not yet, at
least).  I asked what would happen if you brought the problem to their
attention.

Second, what if the document or evidence were presented, but were of good
quality AND compact enough so as not to be wasteful?

Third, what happens when you discuss with your auditor ideas about waste and
associated risk, in light of specific FDA policies that address them, like
the Principle of the Least Burdensome Approach?

http://www.fda.gov/cdrh/modact/Leastburdensome.html

---Michael B.

Andrew McDonagh wrote:
> I agree for most projects its a complete waste of effort and money.
> However, working in the Pharmaceutical industry, we are FDA Regulated
> and mandated to 'provide documentary evidence that software works as
> specified'.  So some form of traceability between requirement and
> tests is needed.

Andrew, traceability between requirements and tests seems very
reasonable, and quite easy to do.  It traceability between requirements
and lines of code that seems a wasted effort to me.

   - George

--
   ----------------------------------------------------------------------
    * George Dinwiddie *                      http://blog.gdinwiddie.com
    Software Development                    http://www.idiacomputing.com
    Consultant and Coach                    http://www.agilemaryland.org
   ----------------------------------------------------------------------

Andrew McDonagh wrote:
> I do it, by naming/grouping test suites (fitnesse, Selenium, etc) to a
> Story or functional area of the system.  Either way, each story has a
> clear set of automated tests.   The problem I'm stuck with, is that
> the FDA Auditors don't get this and so we still have to produce a word
> doc detailing it.

You might look at Cucumber, which allows you to create very readable
tests.  I think there are similar efforts in Java, but the Ruby makes it
quite pleasant.  Perhaps these could just be sucked into the word doc.

   - George

--
   ----------------------------------------------------------------------
    * George Dinwiddie *                      http://blog.gdinwiddie.com
    Software Development                    http://www.idiacomputing.com
    Consultant and Coach                    http://www.agilemaryland.org
   ----------------------------------------------------------------------

I'm posting this for a friend.  If it sounds interesting, let me know
and I'll send you the email contact info:

---------------

Responsibilities:
Ensure a high level of quality for the Rupture client middleware:
Responsible for day-to-day quality assurance on a client component
that handles interaction between various GUIs and the backend.
Responsible for verifying that builds are ready to be released.
Develop test plans and test cases ensure coverage on middleware
features for each sprint.
Manage bug priority and severity levels and coordinate with developers
to ensure that high priority bugs are caught and resolved quickly.
Define and develop an automated test suite.
Verify and validate functional specs that are being delivered to
ensure clarity and testability.

Experience Required:
Experience working with and developing automation test suites.
4+ years experience developing and/or testing windows applications.
2+ years building test automation.
Erlang and C++ experience are preferred, but not required.


Skills:
Ability to multi-task and manage multiple aspects of a release cycle
as well as managing feedback from users and other members of the team
and coordinating bug fixes to ensure that releases are smooth.
Communication, project management, and prioritization are all
essential to this role.
Experience working in an agile environment preferred.
Able to work highly independently.  Ideal candidate would have
experience working as a sole QA person or in a similar role where
they’re directly responsible for the quality of releases.

--- In agile-testing@yahoogroups.com, Kim Gräsman <kim.grasman@...> wrote:
> Jack Reeves ... argues that software builds are like
> manufacturing and software development is more like
> design work.

That's been my experience too (when working as a release engineer
rather than as a tester).  I wrote an article on by blog suggesting
that agile software construction teams consider taking what they learn
as a result of agile improvements to their process and translate these
lessons learned into "standard work" where tasks are repeatable.  The
example of standard work that I gave was building/releasing software.
  This process needs to be repeatable (with no variance) from a given
set of inputs.

I'd argue that while the manufacturing procedure needs to be
standardized, I've found that I can be more effective using an
exploratory process to look for defects in the software itself during
manufacture.  As a release engineer (on teams where the QA was more at
the end), I used things like diffs between the last build and new
checkins in the current build to figure out which areas of the
software to explore.  After several years of using this process, I
found that I had filed over half of the bugs in the bug tracking
system.  Most of the remaining bugs were filed by the actual QA team
with a small percentage being filed by either end customers or the
software development team.  This team was not agile, but my work
within the team was very similar to what agile teams do today.

Since then, my experiences have largely followed what Lisa and Janet
have documented in their book with the exception that I still relied
more heavily on exploratory testing to account for lack of good
specification of stories at the beginning of iterations.  Based on my
own retrospective thought, as well as Lisa/Janet's and Gojko's
acceptance testing books, I've driven much harder to getting
actionable testcases in to the user stories.  This part is harder than
it sounds as you have to move the whole team to want to do this.
Where I couldn't move the team, I've compensated effectively with my
exploratory testing approach.

Please read my blog article and tell me what you think.

--
Bob Clancy
http://AgileTesterDotNet.wordpress.com/