On Thu, Jul 9, 2009 at 9:49 PM, Dale Emery<dale@...> wrote:
> Hi Andrew,
>
> I'm trying to figure out the right approach in my thinking here. Suppose you
>> have an application which has 10 different configuration options. Each
>> option has 3 different settings. It doesn't take rocket science to figure
>> out that this means 3^10 or over 59000 configurations. And those are small
>> numbers! Certainly we can't write a test for every single one of these
>> cases. Yet it's also hard to be absolutely sure that they're completely
>> independent. I recently had a UI bug that was unexpectedly caused by bad
>> data in a completely different area of the site, and I really don't think it
>> was caused by poor design. So what do we make of this? How shall we then
>> test?
>
>
> One common approach is to make this simplifying assumption: Most
> combination-related problems are caused by interactions of PAIRs of
> variables, and very few are caused by larger tuples.
>
> I have no idea under what circumstances that assumption is warranted. I do
> know that making the assumption greatly reduces the number of combinations
> you need. There's a technique called All Pairs Analysis (aka Pairwise
> Analysis) that yields a minimal set of combinations that includes every pair
> of values for every pair of variables.

I read an article on testing quite a while ago that has stuck with me.
It discusses pairwise testing, n-wise testing for the larger tuples
as mentioned above, and then some random sampling:

http://www.developerdotstar.com/mag/articles/test_smarter_not_harder.html

> If you're willing to make the pairwise assumption (and accept the attendant
> risks), give All Pairs Analysis a try. James Bach offers a free tool on his
> web site: http://www.satisfice.com/tools.shtml

Nice tools. I'll keep that in mind.

Thanks.

--Kaleb