On Sat, Sep 27, 2003 at 05:47:50PM -0400, Ron Jeffries wrote:

[snip]

> I believe that the curve effect is more or less present in any
> iterative method. Details matter: refactoring in particular.

Makes sense.

> It's not a simple question with a simple answer.

Understood; thanks for giving me your thoughts on the subject.

I'll clear my head for a day or so, then re-read the chapter in XP
Explained, Alistair's article, and this thread, and hopefully come
out understanding the nuances a bit more.

- Stephen

On Sat, Sep 27, 2003 at 05:47:50PM -0400, Ron Jeffries wrote:

[snip]

> I believe that the curve effect is more or less present in any
> iterative method. Details matter: refactoring in particular.

[snip]

I believe this discussion is confounding two related, but distinct phenomena:
1. The cost of mistakes
2. The cost of change

Cost of Mistakes

The curve effect of multiple shorter exponential curves is the cost of mistakes.
In a phasist process, some mistakes are likely to not be caught until the
customer has the chance to experience the implementation.  The long time delay
between when the mistake was made (e.g., during big up front analysis or design)
makes the exponential cost huge.  In an iterative process where individual
features are developed from the customer requirement to implementation in a
short time period, these mistakes are caught more quickly, making the
exponential cost much much less.

I do not believe refactoring or even expert OO skills has a great effect on this
cost.  The effect is almost entirely due to the significantly shorter time
between starting the work and getting customer feedback.

Cost of Change

On the other hand, cost of change is the cost of reworking the code to handle a
new  requirement that is in conflict with the work already done on the system
(even if the work already done was totally correct given the previously know
requirements).

BDUF proponents may argue that it is less costly to apply such changes if the
implementation has not yet started - just trace the impact of the change via the
mapping from requirements to design artifacts and make the necessary changes. 
XP argues that its cost of change is close to linear due to the application of
simple OO design and refactoring to keep that design simple.  I do not believe
that this issue can be understood as the decomposition of a single long
exponential curve into a series of short exponential curves.

Steven Gordon





The cost of change

The point that I got from Alastair's paper, whether or not he intended it,
is that the familiar curve from "Explained" is perhaps not the absolute cost
over time, but instead a "phase space" diagram - the cost of a given change
in XP vs the same curve in "conventional" approaches. To paraphrase his
summary, as the cost of a given change gets higher in higher in a
"bureaucratic specification" approach, the cost of the same change is much
lower in XP. I like his conclusion that the curve more accurately describes
the *phase space* comparison of the two approaches.

I also agree with Alastair's speculation that the shape of the curve is
qualitative, rather than quantitative.

Thanks,
Tom

Ron wrote:
> Good links, Mike, thanks!

Ron:

Here are a few more "primordial Scrum soup" references:


5) [NonakeTakeuchi95] Nonaka I., Takeuchi H.,The Knowledge Creating
Company, Oxford University Press, Oxford 1995.

Commentary:
Extends the concept of Scrum to define the Knowledge Creation cycle:

	 tacit knowledge --> externalized knowledge --> (back to tacit)

that has led to the concept of "Ba", or "shared knowledge spaces".

Again, deep down is simply the exploration on how effective
human communications, collaboration, and techniques, such as
"structured meetings" lead to knowledge creation, (in our
industry codified knowledge == software).


6) [Senge] Senge, P. M.,The fifth discipline : the art and practice
of the learning organization. New York,Doubleday/Currency, 1990.

Commentary:
	 Provides for archetypes that solve and explain systemic problems
	 such as the dynamics of communications, delays, interaction,
	 feedback loops, hand-offs, etc.

	 The Scrum patterns in many cases are solutions to these
	 archetypes instances.


7) [Tunde] Babatunde A Ogunnaike and W. Harmon Ray, Process
Dynamics, Modeling and Control, Oxford University Press, 1994.

Commentary:
	 Provides the arguments for the understanding of what is
	 an "empirical process" and how to manage it through
	 rapid feedback and empirical measurements.


8) [Wegner] Wegner, P.,Why Interaction Is More Powerful Than Algorithms,

Communications of the ACM 40(5): 80-91, 1997.

Commentary:
	 Provides strong arguments for the need of rapid feedback loops
	 relying on human interaction in software development.


9) [Goldratt] Goldratt E., The Goal, North River Press Inc., Great
Barrington, MA, 1992.

Commentary:
	 Provides additional insights in solving "systemic problems"
	 such as faster feedback, fewer hand-offs, smaller orders,
	 small inventories, etc.

- Mike

On Sunday, September 28, 2003, at 3:08:59 AM, Mike Beedle wrote:

> Here are a few more "primordial Scrum soup" references:

Thanks!

Ron Jeffries
www.XProgramming.com
The practices are not XP. They are a path to XP.

.... this simple thing for us was extremely difficult to achieve:
we needed weeks to get all people in the team into the habit of doing
the daily meeting.

SCRUM - as all agile process are based on self discipline.

--- Boris

On Wednesday, September 24, 2003, at 11:12  AM, Neil Padgen wrote:

> Finally, I'd suggest that you try and use *all* the Scrum practices,
> rather than "a few Scrum-like practices" that you mention.
> Especially, stick *rigidly* to the daily Scrum meeting, and ensure
> the whole team answers the Three Questions.  (We've been slack about
> this in the past, and as a result people haven't been aware of what's
> going on, and so we've missed Sprint Goals.  Now we have a laminated
> card with the Three Questions in big, bold letters, which sits in the
> middle of our meeting table - as a result, we all have much more of a
> handle on what's going on.)

Hi ---,

we do have a similar situation - we are a team, that develops and
maintain a system.
My current approach is to combine some tactical solutions:

1) We do have major bugs - that means the operation of the system is
affected. I mean our site is not running or something similar.

2) We do have bugs in our functionality that we find now - after month
of running the applications.

3) We do have development bugs (no we bring not always absolute bug
free code into production) right after development.

Bug 1 and 3 are urgent (in case 3 is a 1 bug) - we solve them
immediately. Regardless what our Sprint Backlog want from us to do.

Bugs from quality 2 we try to put into the Product Backlog. But this is
always a deal between us and the product owner.

When we do our Sprint Planning, we do only schedule 50% to 60% of our
development time for Backlog tasks. Sometimes this is not enough buffer
for fixing tasks and then we do not release all the functionality we
wanted to release.

---- Boris






On Friday, September 26, 2003, at 09:59  AM, neil_e_martin2003 wrote:

> Hi Everyone,
>
> We are grappling with this problem right now.  We have a small team,
> doing both development and operational support.  Some ops issues
> require instant attention.  The people who have the knowledge and
> experience are also those working on new development, so we'd rather
> not have a support team per se.
>
> I read the previous posts with interest and I've been considering our
> options.
>
> At the same time as adopting Scrum as an agile management method, we
> are also adopting some agile development practices, such as
> estimating effort on tasks using story points based on the "ideal
> developer day" concept.
>
> One thought that I had was simply to re-calibrate our idea of
> an "ideal developer day" by introducing support issues as just
> another thing which eats into development time.  We could quantify
> this by measuring how much time each person has spent on support over
> the last couple of weeks.  We might even be able to dynamically
> change our story point on a per-sprint basis, or is that just asking
> for trouble?!?!
>
> Would anyone care to comment?
>
> Thanks
> Neil
>
>
> ------------------------ Yahoo! Groups Sponsor
> ---------------------~-->
> KnowledgeStorm has over 22,000 B2B technology solutions. The most
> comprehensive IT buyers' information available. Research, compare,
> decide. E-Commerce | Application Dev | Accounting-Finance | Healthcare
> | Project Mgt | Sales-Marketing | More
> http://us.click.yahoo.com/IMai8D/UYQGAA/cIoLAA/9EfwlB/TM
> ---------------------------------------------------------------------
> ~->
>
> To Post a message, send it to:   scrumdevelopment@eGroups.com
> To Unsubscribe, send a blank message to:
> scrumdevelopment-unsubscribe@eGroups.com
>
> Your use of Yahoo! Groups is subject to
> http://docs.yahoo.com/info/terms/
>
>
>
>

(third time I'm attempting to post this)
Stephen:
> Alistair thought the curve is still exponential, XP just handles
> change better (by fixing code better and faster) so it has smaller
> constants and grows at a slower pace when compared to the
> high-bureaucracy expotential curve.

Ron:
Honestly I can't figure out what Alistair's point was. He
was mostly trying to show that the curve was still exponential,
and whatever the larger point was, I missed it.

Stephen:
> You're saying by XP handling change better, it does in fact change
> the curve to be logarithmic, as (perhaps due to the good engineering
> practices) the next change gets to start the curve all over again,
> there by keeping the curve from ever reaching a rapidly growing
> rate.

Ron:
No, I'm just saying that it remains exponential in T, but that
changes tend
to be limited to low values of T.
--->
Hi, Ron,
I'm really glad that you got the point of the article, however
much you deny it.
My point was to demonstrate that the curve is still exponential.
There was no larger point -- that was exactly the point that
was under controversy at the time.
The point of the article was to argue that the curve is
still exponential.
From your last posting, it seems you think so to.
Congrats
(I hope that now you can indeed figure out what Alistair's point was.)

Alistair

==============================================
Alistair Cockburn
President, Humans and Technology
Program Director, Agile Development Conference
(http://AgileDevelopmentConference.com)

http://alistair.cockburn.us alistair.cockburn@...
1814 E. Fort Douglas Circle, Salt Lake City, UT 84103
Phone: 801.582-3162            Fax: 775.416.6457

Author of
"Surviving Object-Oriented Projects" (1998)
"Writing Effective Use Cases" (Jolt Productivity Award 2001)
"Agile Software Development" (Jolt Productivity Award 2002)

"La perfection est atteinte non quand il ne reste rien a ajouter,
mais quand il ne reste rien a enlever." (Saint-Exupery)
==============================================

Stephen:
> Beck thought XP, with its good engineering practices, changes the
> curve to be logarithmic.

Jeffries:
No, I don't think he thought that. I think he thought that XP
flattens the
curve,

--->
The white book says: (p.22)
"The problem is that the curve is no longer valid. Or rather, with a
combination of technology and programming practices, it is possible
to experience a curve that is really quite the opposite."

Kent's writing supports Stephen's view

==============================================
Alistair Cockburn
President, Humans and Technology
Program Director, Agile Development Conference
(http://AgileDevelopmentConference.com)

http://alistair.cockburn.us alistair.cockburn@...
1814 E. Fort Douglas Circle, Salt Lake City, UT 84103
Phone: 801.582-3162            Fax: 775.416.6457

Author of
"Surviving Object-Oriented Projects" (1998)
"Writing Effective Use Cases" (Jolt Productivity Award 2001)
"Agile Software Development" (Jolt Productivity Award 2002)

"La perfection est atteinte non quand il ne reste rien a ajouter,
mais quand il ne reste rien a enlever." (Saint-Exupery)
==============================================

In a message dated 9/28/2003 8:28:07 AM Mountain Daylight Time, Ron
Jeffries writes:

> Beck thought XP, with its good engineering practices, changes the
> curve to be logarithmic.

No, I don't think he thought that. I think he thought that XP
flattens the
curve,
--->
Not only that, but it is very interesting to look at his white book
again in the context of modern XP ...
p.22:
"  'The cost to fix a problem in a piece of software rises
exponentially over time.' ... I resolved then and there that I would
never let a problem get through to production. ..."

It seems that Kent has come full circle.  One of the big ads for XP
these days is exactly that it keeps problems from getting through to
production !  Why? Well, because it is so much more costly to fix if
it gets through.  Therefore, please do pair programming, automated
testing, tiny changes, etc etc.

I thank Tom Stanbaugh for finding the relevant quote in my article
from some years ago (anyone know which year that got written?) which
is now in play:

"The exponential cost curve is mostly in detecting and
communicating the Mistake and naming the change that is
to be made. XP cannot change that curve, and indeed,
XP takes that increasing cost curve neatly into account.
So the first lesson I get is that one should not base a
defense of XP on the ABSENCE of the curve, but rather on
the PRESENCE of the exponential cost curve."

At this point, the exponential cost of change is still in full play
and is part of XP's standard sales pitch.

==============================================
Alistair Cockburn
President, Humans and Technology
Program Director, Agile Development Conference
(http://AgileDevelopmentConference.com)

http://alistair.cockburn.us alistair.cockburn@...
1814 E. Fort Douglas Circle, Salt Lake City, UT 84103
Phone: 801.582-3162            Fax: 775.416.6457

Author of
"Surviving Object-Oriented Projects" (1998)
"Writing Effective Use Cases" (Jolt Productivity Award 2001)
"Agile Software Development" (Jolt Productivity Award 2002)

"La perfection est atteinte non quand il ne reste rien a ajouter,
mais quand il ne reste rien a enlever." (Saint-Exupery)
==============================================

Alistair accurately stated:
"The exponential cost curve is mostly in detecting and
communicating the Mistake and naming the change that is
to be made. XP cannot change that curve, and indeed,
XP takes that increasing cost curve neatly into account.
So the first lesson I get is that one should not base a
defense of XP on the ABSENCE of the curve, but rather on
the PRESENCE of the exponential cost curve."


Alistair,

Well said.

If bugs are our software's enemies then ...

     ... the highest form of generalship is to
     balk the enemy's plans (in UNIT TESTING and ACCEPTANCE
     TESTING) ; the next best is to prevent
     the junction of the enemy's forces (in SYSTEM TESTING);
     the next in order is to attack the enemy's army in the field
     (in early PRODUCTION); and the worst policy of all is to besiege
     walled cities  (Here the bugs have taken over
     our software and have made cities within it, where they
     reside comfortably, while we struggle and fight them
     in MAINTENANCE .... )

	 from
	 THE ART OF (SOFTWARE) WAR
	 III. ATTACK BY STRATAGEM
	 SUN TZU

- Mike

On Monday, September 29, 2003, at 10:52:25 PM, Mike Beedle wrote:

> If bugs are our software's enemies then ...

>     ... the highest form of generalship is to
>     balk the enemy's plans (in UNIT TESTING and ACCEPTANCE
>     TESTING) ; the next best is to prevent
>     the junction of the enemy's forces (in SYSTEM TESTING);
>     the next in order is to attack the enemy's army in the field
>     (in early PRODUCTION); and the worst policy of all is to besiege
>     walled cities  (Here the bugs have taken over
>     our software and have made cities within it, where they
>     reside comfortably, while we struggle and fight them
>     in MAINTENANCE .... )

>         from
>         THE ART OF (SOFTWARE) WAR
>         III. ATTACK BY STRATAGEM
>         SUN TZU

Nice ...

Ron Jeffries
www.XProgramming.com
No one expects the Spanish Inquisition ...

Ron wrote:
> No one expects the Spanish Inquisition ...


Hmm, I would expect XPers to say :-)

	 I prefer a new edition of the Spanish Inquisition
	 than to ever let  .... untested bugs in my App!!!!

		 paraphrasing Professor Higgins, in My Fair "App" Lady

Of course, testing would be minimized to verify "what"
the software did if we only wrote "provable" software,

- Mike

-- quicksort in Curry
filter_ :: (a -> Bool) -> [a] -> [a];
filter_ _ []     = []
filter_ p (x:xs) = if p x then x : filter_ p xs
                           else filter_ p xs

qsort :: [Int] -> [Int]
qsort []     = []
qsort (x:l)  = qsort (filter (< x) l)    x : qsort (filter (>= x) l)

aacockburn wrote:

> At this point, the exponential cost of change is still in full play
> and is part of XP's standard sales pitch.

In fact, I don't think it's in full play!

You are right that XP takes care of the fact that a bug introduced in
implementation is much harder to fix the later it is found. Therefore
the early and excessive testing.

*But* - the thing XP calls into question, as I understand it, is that
"bugs" introduced in the *requirements* are much easier to fix (when
working XP-style). That's why we don't try to get them all "correct"
before starting with implementation.

What do you think?

Curious, Ilja

--
++ Business Intelligence: disy Cadenza Web - drei Klicks zum Ergebnis
++ http://www.disy.net/cadenza
++ disy Conference: Telefonkonferenz mit
++ integrierter Datenkonferenz http://www.disy.net/conference ++

      Ilja Preuß                                  preuss@...
      disy Informationssysteme GmbH               http://www.disy.net
      Stephanienstr. 30                           Tel: +49 721 1 600 624
      D-76133 Karlsruhe, Germany                  Fax: +49 721 1 600 605

++ disy: Kosteneinsparungen durch Optimierung Ihrer IuK-Prozesse ++

> *But* - the thing XP calls into question, as I understand it, is that
> "bugs" introduced in the *requirements* are much easier to fix (when
> working XP-style). That's why we don't try to get them all "correct"
> before starting with implementation.

I think XP challenges all assertions predicated upon a
presumption of a waterfall-lifecycle as XP challenges the
waterfall-lifecycle itself and the very notion of separable
"phases" of isolated activity-flow that focusing on producing
intermediate artifacts, as opposed to a single inseparable
flow of collaborative activities that focuses on producing
working software in "small enough" increments with high frequency.

XP says "what happens to all these waterfall-based presumptions
about cost-of-change of I go to the extreme of taking the
"limit" as the size of the delta (change) approaches zero?" Then
it is still true that the longer the period of time in between
fault injection and fault detection, the exponentially higher
the cost of fault-correction, because the more work that was
done that must be reworked.

But it is no longer true to make the phase-based assumption
that says 'the the lifecycle phase in which it was found, the
exponentially higher the cost to fix it. Because the delta is
so small and the phases are so much smaller and statistically
insignificant that it becomes a greater interruption of flow
to focus on intermediate artifacts and formal reviews for the
amount of work done in a single episode or feedback-cycle than
to rely upon the "in flow" feedback-loops in XP. And the amount
of work done between feedback cycles is whittled down so small,
that if you find it even only after you integrate your changes
and test them, it is still such a small unit of work that the
amount of rework is comparable rather than exponential.

I think that's what is meant by "flattening" the change curve
by taking the limit as delta-size approaches zero to make the
feedback loops be collaborative and/or in-flow activities rather
than separate across phase boundaries in distinct activity flows.
And that focusing on the whole via piecemeal growth works better
than the waterfall-based "magnum opus" divide-and-conquer approach
that deconstructs and separates to focus more on the parts and
their sum than on the synergistic whole.

Just my opinion ;-)

--
Brad Appleton <brad@...> www.bradapp.net
   Software CM Patterns (www.scmpatterns.com)
    Effective Teamwork, Practical Integration
"And miles to go before I sleep." -- Robert Frost

On Tuesday, September 30, 2003, at 10:51:05 AM, Brad Appleton wrote:

> I think that's what is meant by "flattening" the change curve
> by taking the limit as delta-size approaches zero to make the
> feedback loops be collaborative and/or in-flow activities rather
> than separate across phase boundaries in distinct activity flows.
> And that focusing on the whole via piecemeal growth works better
> than the waterfall-based "magnum opus" divide-and-conquer approach
> that deconstructs and separates to focus more on the parts and
> their sum than on the synergistic whole.

Yes. The biggest question may be whether one seeks to understand what is
meant by the notion of flattening the change curve, or merely to refute it.

Ron Jeffries
www.XProgramming.com
The opinions expressed here /are/ necessarily those of XProgramming.com.
But I might change my mind.

I've had great success with Agile, and wish to continue developing in
this manner, however the 6 Sigma wave is upon us. My concerns on how
well these 2 methodologies can integrate are increasing.  Has anyone
found a way to merge the 2 or experimented on weather the 2 can
coexist.

Thanks in advance for your replies,

MCCLIMG

On Tuesday, September 30, 2003, at 11:30:27 AM, acockburn@... wrote:

> I find it non-stop fascinating that people are so willing to say something
> that is exactly wrong, and cover it over by saying, "but let's not get into
fine
> details about it." I find this happens in discussions of Liskov Substitution
> Principle and the exponential cost of change curve. Both are mathematical, and
> therefore are Right or Wrong.

Paraphrasing from a Wendy's commercial that I like:

   Your concept of Right or Wrong in human discourse: very narrow.

Ron Jeffries
www.XProgramming.com
XP says: Don't just sit on your DUF, do something. Get some feedback.

Alistair,

This still does not address my earlier observation:

1. This analysis addresses mistaken or misunderstood or misimplemented
requirements -the much shorter feedback mechanisms of XP and other agile
software development methodologies catch these kinds of problems much sooner,
vastly shortening the time over which the exponential costs accelerate.

2. This analysis does not address the cost of changes not due to mistakes, but
due to evolving requirements, new requirements, market changes, etc.

Refactoring and simple design are a very minor contributor in the first case
compared to time lag, but anecdotally they give XP a huge advantage over phasist
approaches in dealing with the second kinds of changes.  If the change curve is
also exponential for this second type of changes, then how does refactoring and
simple design lessen their costs?  Or do they really?

Steven Gordon

-----Original Message-----
From: acockburn@... [mailto:acockburn@...]
Sent: Tue 9/30/2003 8:30 AM
To: scrumdevelopment@yahoogroups.com
Cc:
Subject: Re: [scrumdevelopment] Re: flattened cost of change curve

I find it non-stop fascinating that people are so willing to say something
that is exactly wrong, and cover it over by saying, "but let's not get into fine
details about it." I find this happens in discussions of Liskov Substitution
Principle and the exponential cost of change curve. Both are mathematical, and
therefore are Right or Wrong.

Barry Boehm declared that the cost of finding and fixing a mistake grows
expentially as it remains undetected through requirements, coding, testing and
deployment. Kent wrote: "...the curve is no longer valid. Or rather, with a
combination of technology and programming practices, it is possible to
experience a
curve that is really quite the opposite." and then drew a log(n) curve.

Both Barry and Kent are talking math here. The math is right or wrong. If
someone chooses to ask the question: "Is the curve still exponential?" then the
answer is Yes or No, and not "Not really, because there are lots of little
exponential curves, so it's not really exponential."

I wrote up and Ron published my defense of it being exponential even in XP. I
stand by those arguments.

Ilya asks: <<the thing XP calls into question, as I understand it, is that
"bugs" introduced in the *requirements* are much easier to fix (when
working XP-style). That's why we don't try to get them all "correct"
before starting with implementation.... What do you think?>>

You can see the analysis in the article, but the main point for Ilya's
question is that he is only comparing requirements and coding. He left out the
next
two steps - acceptance testing by the customer and field deployment. I don't
think there are yet any XP advocates who argue that we shouldn't bother to get
requirements or code correct because it is so much easier to correct it in the
field. We hear quite the opposite: get the bugs out before deploying. Why?
Because it is so much more expensive.  The reason requirements can overlap
coding has to do with cost of communications (still inside the exponential
curve),
and the value of feedback from coding to requirements (a different cost factor
altogether, and worthy of its own scrutiny).

Ron correctly writes: <<The biggest question may be whether one seeks to
understand what is
meant by the notion of flattening the change curve,>>

He is correct -- and we don't gain that understanding by pretending an
exponential curve is logarithmic and then working to explain away the funny
mismatch
in shape, but rather by first establishing which curve we are dealing with
and then mapping that back to project life, and back again to math.

I already toook my stab at this, and I stand by my conclusion in the absence
of counter data or a better explanation:
<<
"Bureaucratic Specification (BS) methodologies . . .also try
to handle the exponential growth, but basically their cost for everything is
a Very Large constant greater than XP's. So much greater that XP is running
the cost-to-fix in "p" (programming time) while the BS methodologies are
still running in "r" (requirements time). So XP has the program written and
fixed by the time the BS methodology has just the spec written and studied.
Keep the exponential curve, but drop everything by very large constants, and
XP is likely to have shipped the product while the other one is still being
designed.
>>

Given that Ron has already agreed that the curve is exponential, can we stop
trying to pretend it isn't, and get on with the "understanding" part?

Alistair

I wonder if some of the apparently reduced cost of change in XP is instead
an illusion resulting from moving those costs to the test development area -
and then not measuring them as carefully. When we calculate the "cost of
change", do we include the cost of maintaining or extending the test
suite/harness in our calculations?

Could some of the practitioners perhaps contribute a brief summary of how
they measure this "cost of change", and what costs are included? For
example - do we include the cost of enumerating the user stories, then
collecting and structuring the example data used to generate test cases in
our cost summaries? Do we include the cost of refactoring the test harnesses
when a new story or bug reveals fundamental errors or omissions?

Perhaps because agile methods are still new to me, I find my head-scratching
time often moves from the code under test to the test harness, rather than
disappearing altogether. I find that an important part of my development
time for new functionality is determining how to think about it. I have
always rejected waterfall-style "white-board engineering" or "word
engineering", in favor of Smalltalk-style prototyping. I'm not sure whether
or how much XP has improved my prototyping - but I'm pretty sure it has NOT
been exponential. Perhaps this falls under the umbrella of "spike
solutions"?

For example, I'm just rediscovering the power of streams, sequences, and
delayed evaluation as described in the Wizard Book, because I'm working on a
project where I'm manipulating LOTS of hierarchical structures. I can write
the code (often just transcribing it from the Scheme examples in the text)
easily enough - but how do I even think about *testing* it? Do I bother?

I wonder how the *total* cost of new functionality (including
headscratching, prototyping, and test development time) using XP compares
to, for instance, my habit of prototyping and iterating until it works "good
enough". I ask this because I find myself just as able to get lost in
designing and coding unit tests as I used to get doing "prototypes".

Thanks,
Tom



----- Original Message -----
From: "Steven Gordon" <sagordon@...>
To: <scrumdevelopment@yahoogroups.com>
Sent: Tuesday, September 30, 2003 12:06 PM
Subject: RE: [scrumdevelopment] Re: flattened cost of change curve


> Alistair,
>
> This still does not address my earlier observation:
>
> 1. This analysis addresses mistaken or misunderstood or misimplemented
requirements -the much shorter feedback mechanisms of XP and other agile
software development methodologies catch these kinds of problems much
sooner, vastly shortening the time over which the exponential costs
accelerate.
>
> 2. This analysis does not address the cost of changes not due to mistakes,
but due to evolving requirements, new requirements, market changes, etc.
>
> Refactoring and simple design are a very minor contributor in the first
case compared to time lag, but anecdotally they give XP a huge advantage
over phasist approaches in dealing with the second kinds of changes.  If the
change curve is also exponential for this second type of changes, then how
does refactoring and simple design lessen their costs?  Or do they really?
>
> Steven Gordon
>
> -----Original Message-----
> From: acockburn@... [mailto:acockburn@...]
> Sent: Tue 9/30/2003 8:30 AM
> To: scrumdevelopment@yahoogroups.com
> Cc:
> Subject: Re: [scrumdevelopment] Re: flattened cost of change curve
>
> I find it non-stop fascinating that people are so willing to say something
> that is exactly wrong, and cover it over by saying, "but let's not get
into fine
> details about it." I find this happens in discussions of Liskov
Substitution
> Principle and the exponential cost of change curve. Both are mathematical,
and
> therefore are Right or Wrong.
>
> Barry Boehm declared that the cost of finding and fixing a mistake grows
> expentially as it remains undetected through requirements, coding, testing
and
> deployment. Kent wrote: "...the curve is no longer valid. Or rather, with
a
> combination of technology and programming practices, it is possible to
experience a
> curve that is really quite the opposite." and then drew a log(n) curve.
>
> Both Barry and Kent are talking math here. The math is right or wrong. If
> someone chooses to ask the question: "Is the curve still exponential?"
then the
> answer is Yes or No, and not "Not really, because there are lots of little
> exponential curves, so it's not really exponential."
>
> I wrote up and Ron published my defense of it being exponential even in
XP. I
> stand by those arguments.
>
> Ilya asks: <<the thing XP calls into question, as I understand it, is that
> "bugs" introduced in the *requirements* are much easier to fix (when
> working XP-style). That's why we don't try to get them all "correct"
> before starting with implementation.... What do you think?>>
>
> You can see the analysis in the article, but the main point for Ilya's
> question is that he is only comparing requirements and coding. He left out
the next
> two steps - acceptance testing by the customer and field deployment. I
don't
> think there are yet any XP advocates who argue that we shouldn't bother to
get
> requirements or code correct because it is so much easier to correct it in
the
> field. We hear quite the opposite: get the bugs out before deploying. Why?
> Because it is so much more expensive.  The reason requirements can overlap
> coding has to do with cost of communications (still inside the exponential
curve),
> and the value of feedback from coding to requirements (a different cost
factor
> altogether, and worthy of its own scrutiny).
>
> Ron correctly writes: <<The biggest question may be whether one seeks to
> understand what is
> meant by the notion of flattening the change curve,>>
>
> He is correct -- and we don't gain that understanding by pretending an
> exponential curve is logarithmic and then working to explain away the
funny mismatch
> in shape, but rather by first establishing which curve we are dealing with
> and then mapping that back to project life, and back again to math.
>
> I already toook my stab at this, and I stand by my conclusion in the
absence
> of counter data or a better explanation:
> <<
> "Bureaucratic Specification (BS) methodologies . . .also try
> to handle the exponential growth, but basically their cost for everything
is
> a Very Large constant greater than XP's. So much greater that XP is
running
> the cost-to-fix in "p" (programming time) while the BS methodologies are
> still running in "r" (requirements time). So XP has the program written
and
> fixed by the time the BS methodology has just the spec written and
studied.
> Keep the exponential curve, but drop everything by very large constants,
and
> XP is likely to have shipped the product while the other one is still
being
> designed.
> >>
>
> Given that Ron has already agreed that the curve is exponential, can we
stop
> trying to pretend it isn't, and get on with the "understanding" part?
>
> Alistair
>
>
>
>
>
> To Post a message, send it to:   scrumdevelopment@eGroups.com
> To Unsubscribe, send a blank message to:
scrumdevelopment-unsubscribe@eGroups.com
>
> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
>
>

acockburn@... wrote:
> I find it non-stop fascinating that people are so willing to say something
> that is exactly wrong, and cover it over by saying, "but let's not get into
fine
> details about it." I find this happens in discussions of Liskov Substitution
> Principle and the exponential cost of change curve. Both are mathematical, and
> therefore are Right or Wrong.
>
> Barry Boehm declared that the cost of finding and fixing a mistake grows
> expentially as it remains undetected through requirements, coding, testing and
> deployment. Kent wrote: "...the curve is no longer valid. Or rather, with a
> combination of technology and programming practices, it is possible to
experience a
> curve that is really quite the opposite." and then drew a log(n) curve.

I haven't got Kent's White Book in front of me, and I don't have Barry's
paper anywhere, so I'm speaking from oft-fragile memory here, but ...

AFAIK, Barry Boehm's research showed the cost of *fixing defects* grew
exponentially.

AFAIK, Kent Beck's assertion is that the cost of *making changes* under
XP does not grow exponentially.

AFAIK, *fixing defects* != *making changes*. More specifically, there
exist changes to software that do not involve fixing existing defects.
(Obviously, fixing a defect involves making a change. Somewhere.)

Thus, AFAIK, both gentlemen may be correct.

Unfortunately, the exponential cost curve that Barry Boehm described has
spent the last several decades labeled a cost of *change* curve, rather
than a cost of *bugfix* curve. If it is in fact a cost of bugfix curve,
then the "cost of change curve" could easily be dramatically less than
exponential for agile methods which emphasize short iterations, and thus
detect defects very quickly while fixing them is still cheap.

[ much deletia ]

> Given that Ron has already agreed that the curve is exponential, can we stop
> trying to pretend it isn't, and get on with the "understanding" part?

Allow me to graciously suggest that we first recognize that "cost of
change" and "cost of fixing defects" are different beasts. I'm not at
all sure from the conversation that everybody's talking about the same
curve ...

--
Edmund Schweppe -- schweppe@... -- http://schweppe.home.tiac.net
The opinions expressed herein are at best coincidentally related to
those of any past, present or future employer.

On Tuesday, September 30, 2003, at 11:52:41 AM, acockburn@... wrote:

> In a message dated 9/30/2003 9:47:00 AM Mountain Daylight Time,
> ronjeffries@... writes:
> <<Paraphrasing from a Wendy's commercial that I like:
>   Your concept of Right or Wrong in human discourse: very narrow.>>
> For better or worse (both, actually) mathematics doesn't count as "human"
> discourse in that sense :-)

Yes. However, the statement that XP (or agile methods for that matter)
"flatten the exponential cost curve" is not a mathematical statement. It is
a statement of a kind of higher or alternate truth regarding the impact of
short iterations and rapid feedback.

In short, it's a metaphor. Doing math on a metaphor is blowing air up one's
own skirt. QED.

Ron Jeffries
www.XProgramming.com
You have to either laugh or cry.  -- Bill Rogers

Thanks for the insight. I've always had serious heartburn about the
"flattening" statement. And just blown it off as something that can't
get rationalized in my math/physics pea-brain.

From a metaphor view it now makes sense.

Glen B. Alleman
CH2M HILL
Rocky Flats Environmental Technology Site
glen.alleman@...
glen.alleman@...

>-----Original Message-----
>From: Ron Jeffries [mailto:ronjeffries@...]
>Sent: Tuesday, September 30, 2003 11:38 AM
>To: scrumdevelopment@yahoogroups.com
>Subject: Re: [scrumdevelopment] Re: flattened cost of change curve
>
>On Tuesday, September 30, 2003, at 11:52:41 AM, acockburn@...
wrote:
>
>> In a message dated 9/30/2003 9:47:00 AM Mountain Daylight Time,
>> ronjeffries@... writes:
>> <<Paraphrasing from a Wendy's commercial that I like:
>>   Your concept of Right or Wrong in human discourse: very narrow.>>
>> For better or worse (both, actually) mathematics doesn't count as
"human"
>> discourse in that sense :-)
>
>Yes. However, the statement that XP (or agile methods for that matter)
>"flatten the exponential cost curve" is not a mathematical statement.
It is
>a statement of a kind of higher or alternate truth regarding the impact
of
>short iterations and rapid feedback.
>
>In short, it's a metaphor. Doing math on a metaphor is blowing air up
one's
>own skirt. QED.
>
>Ron Jeffries
>www.XProgramming.com
>You have to either laugh or cry.  -- Bill Rogers
>
>
>------------------------ Yahoo! Groups Sponsor
---------------------~-->
>KnowledgeStorm has over 22,000 B2B technology solutions. The most
>comprehensive IT buyers' information available. Research, compare,
decide.
>E-Commerce | Application Dev | Accounting-Finance | Healthcare |
Project
>Mgt | Sales-Marketing | More
>http://us.click.yahoo.com/IMai8D/UYQGAA/cIoLAA/9EfwlB/TM
>---------------------------------------------------------------------~-
>
>
>To Post a message, send it to:   scrumdevelopment@eGroups.com
>To Unsubscribe, send a blank message to: scrumdevelopment-
>unsubscribe@eGroups.com
>
>Your use of Yahoo! Groups is subject to
http://docs.yahoo.com/info/terms/
>

On Tuesday, September 30, 2003, at 12:12:59 PM, Tom Stambaugh wrote:

> I wonder if some of the apparently reduced cost of change in XP is instead
> an illusion resulting from moving those costs to the test development area -
> and then not measuring them as carefully. When we calculate the "cost of
> change", do we include the cost of maintaining or extending the test
> suite/harness in our calculations?

I doubt whether anyone has done (or could do) a comparison, as no two
projects are comparable.

However, those of who do more testing than we used to are quite sure that
our total time for development is reduced thereby, and that conformance to
requirements is increased.

> Could some of the practitioners perhaps contribute a brief summary of how
> they measure this "cost of change", and what costs are included? For
> example - do we include the cost of enumerating the user stories, then
> collecting and structuring the example data used to generate test cases in
> our cost summaries? Do we include the cost of refactoring the test harnesses
> when a new story or bug reveals fundamental errors or omissions?

>From this list, I'd note:

   - All the information in the user stories must be created in any case,
   and there is no reason to think that doing it in story style is more
   costly. If anything, owing to reduced formality, I would expect it to be
   less.

   - Unless we are comparing a tested project to an untested one, all the
   information used for tests needs to be created somewhere in the project
   context in any case. Are you thinking that perhaps agile projects test
   "too much"?

   - <sigh> Refactoring as part of incremental development is not a matter
   of doing large amounts of work over. It is more like sanding a surface
   prior to painting. First the coarse sandpaper takes down the rough spots,
   then finer and finer grades until the surface is as smooth as we want.
   People who sand surfaces don't think of sanding multiple times as doing
   things over again. Neither should people who develop software, or test
   harnesses, incrementally, think of refactoring as doing things over
   again. It's doing the thing first roughly, then refining it.

> Perhaps because agile methods are still new to me, I find my head-scratching
> time often moves from the code under test to the test harness, rather than
> disappearing altogether.

I may not be correctly apprehending your use of the word "harness".
Certainly one thinks about testing. If one is spending much time thinking
about the harness in the sense of framework, it might be too much. We don't
need a testing framework, we need tests.

More thinking about the tests seems to reduce the time spend coding by more
than the same amount.

> I find that an important part of my development
> time for new functionality is determining how to think about it. I have
> always rejected waterfall-style "white-board engineering" or "word
> engineering", in favor of Smalltalk-style prototyping. I'm not sure whether
> or how much XP has improved my prototyping - but I'm pretty sure it has NOT
> been exponential. Perhaps this falls under the umbrella of "spike
> solutions"?

I'm not sure that XP claims to improve anything exponentially, most
especially not a good evolution-from-prototype approach. What XP claims is
that the old saw about the cost of change being exponential gets translated
in the minds of people as "change is bad, very bad", and that in fact, in
incremental design supported by tests and refactoring, change is not all
that bad after all.

> For example, I'm just rediscovering the power of streams, sequences, and
> delayed evaluation as described in the Wizard Book, because I'm working on a
> project where I'm manipulating LOTS of hierarchical structures. I can write
> the code (often just transcribing it from the Scheme examples in the text)
> easily enough - but how do I even think about *testing* it? Do I bother?

I don't know. The first time I spent more than five minutes in the
debugger, I'd start trying to figure it out.

> I wonder how the *total* cost of new functionality (including
> headscratching, prototyping, and test development time) using XP compares
> to, for instance, my habit of prototyping and iterating until it works "good
> enough". I ask this because I find myself just as able to get lost in
> designing and coding unit tests as I used to get doing "prototypes".

It could happen. XP and Scrum and so on are probably more about taming the
effects of change, and making results more predictable thereby, than about
reducing the overall cost of projects. We get projects done that would
otherwise not get done; we provide information to enable knowing when
things will be done; and we generally produce results that satisfy the
customers.

If you're already doing those things, XP or Scrum might help you do them
more easily, but Done At Right Quality As Predicted is what it's really all
about.

Ron Jeffries
www.XProgramming.com
No one expects the Spanish Inquisition ...

On Tue, Sep 30, 2003 at 07:26:08PM -0600, Mike Cohn wrote:
>    So, how would Scrum alter bus scheduling?

Okay, I'll give it a try.

I don't think the constant distance idea is good. It's a planned
approach and would just slow the buses down even more (e.g. all
buses would be delayed by any stoplight any bus hit instead of just
their individual stoplights).

   [Okay, upon rereading this, the constant distance might actually
   improve human flow, despite the slower bus speed; it seems
   unintuitive, so would have to be simulated or what not]

Scrum is an empirical process: frequent inspection and adjustment.

So, my rough guess is to let the buses go freely and then every
10/15 minutes take a poll (either manually or via GPS), see where
the buses are at, and create a new solution specific to that
situation.

There are probably typical problems, e.g. too big of a gap between
individual buses or a group of buses traveling together.

For each typical problem, you could probably develop a set of common
solutions for each problem, just as a ScrumMaster has a general set
of common solutions for problems in software development (though
granted the ScrumMaster does a lot of tailoring to specific
situations and adding of their own solutions).

So lets say several typical solutions are:

- If you have two buses half-full lined up behind each other, have
   one unload (giving its passengers to the other bus) and have
   the unloaded bus turn around to deal with demand in the other
   direction.

- Or have one of the buses sit a stop for 5/10 minutes.

- If you have a large gap between buses, perhaps have a variable
   number of buses in service and bring in more to respond to demand.
   E.g. 2 or 3 buses sit out at certain spots along the route and
   then can dynamically be brought in to respond to gaps between
   other buses.

I dunno; I hesitate in suggesting an algorithm could be developed.
An algorithm could equate to a plan, which is bad in Scrum. But the
algorithm could also be equated to a process, e.g. Scrum itself, and
be something the bus system carries out to psuedo-empirically
respond to situations.

Granted the job of a ScrumMaster can't be put in an algorithm, but
I'm tempted to think that the bus-routing domain is a lot simpler
than software development (even though it is chaotic as well).

I'm sure others have thought lots more about bus-routing than I, and
had much better solutions. That's just my Scrum-based take, per
your original question.

- Stephen