Andrew,

One interesting question is how did all those config options get there?
Theoretically, because you had a failing test, right? And, without meaning to,
some of your configuration options were not independent from each other, so you
created unexpected interactions. So that eventually you had things happening
outside of your test suite boundary.

Personally, I think we create systems which are complex, too complex to expect
100% coverage and so we live with the holes in the coverage (we can't cover the
whole ;-)). We use some kind of judgment or intuition or experience to tell us
where we need to add some tests because we suspect we've created a hole. So, use
the unexpected failure to create some new tests, and take that experience to
your next project.

I suppose if you are worried about other undiscovered failures, you could create
a test tool to cycle through all the possible combinations.

John D.

-----Original Message-----
From: testdrivendevelopment@yahoogroups.com
[mailto:testdrivendevelopment@yahoogroups.com] On Behalf Of Andrew Wagner
Sent: 09 July 2009 17:03
To: testdrivendevelopment@yahoogroups.com
Subject: [TDD] Agile response to combinatorial explosion

I'm trying to figure out the right approach in my thinking here. Suppose you
have an application which has 10 different configuration options. Each
option has 3 different settings. It doesn't take rocket science to figure
out that this means 3^10 or over 59000 configurations. And those are small
numbers! Certainly we can't write a test for every single one of these
cases. Yet it's also hard to be absolutely sure that they're completely
independent. I recently had a UI bug that was unexpectedly caused by bad
data in a completely different area of the site, and I really don't think it
was caused by poor design. So what do we make of this? How shall we then
test?


[Non-text portions of this message have been removed]



------------------------------------

Yahoo! Groups Links