If you're working in the .NET world, either NUnit or MbUnit
have a Pairwise attribute used to generate test cases from
a set of parameter values.

Charlie

> -----Original Message-----
> From: testdrivendevelopment@yahoogroups.com
> [mailto:testdrivendevelopment@yahoogroups.com] On Behalf Of Dale Emery
> Sent: Thursday, July 09, 2009 9:50 PM
> To: testdrivendevelopment@yahoogroups.com
> Subject: Re: [TDD] Agile response to combinatorial explosion
>
> Hi Andrew,
>
> I'm trying to figure out the right approach in my thinking
> here. Suppose you
> > have an application which has 10 different configuration
> options. Each
> > option has 3 different settings. It doesn't take rocket science to
> > figure out that this means 3^10 or over 59000 configurations. And
> > those are small numbers! Certainly we can't write a test for every
> > single one of these cases. Yet it's also hard to be absolutely sure
> > that they're completely independent. I recently had a UI
> bug that was
> > unexpectedly caused by bad data in a completely different
> area of the
> > site, and I really don't think it was caused by poor
> design. So what
> > do we make of this? How shall we then test?
>
>
> One common approach is to make this simplifying assumption:
> Most combination-related problems are caused by interactions
> of PAIRs of variables, and very few are caused by larger tuples.
>
> I have no idea under what circumstances that assumption is
> warranted. I do know that making the assumption greatly
> reduces the number of combinations you need. There's a
> technique called All Pairs Analysis (aka Pairwise
> Analysis) that yields a minimal set of combinations that
> includes every pair of values for every pair of variables.
>
> If you're willing to make the pairwise assumption (and accept
> the attendant risks), give All Pairs Analysis a try. James
> Bach offers a free tool on his web site:
> http://www.satisfice.com/tools.shtml
>
> Dale
>
> --
> Dale Emery, Consultant
> Inspiring Leadership for Software People
> Web: http://dhemery.com
> Weblog: http://cwd.dhemery.com
>
>
> [Non-text portions of this message have been removed]
>
>
>
> ------------------------------------
>
> Yahoo! Groups Links
>
>
>
>

Islator.



From: testdrivendevelopment@yahoogroups.com
[mailto:testdrivendevelopment@yahoogroups.com] On Behalf Of George Dinwiddie
Sent: Friday, July 10, 2009 1:08 AM
To: testdrivendevelopment@yahoogroups.com
Subject: [TDD] quick poll on C# mocking tools








What's most widely used C# mocking tool?

What's your current recommendation to a newby?

--
----------------------------------------------------------
* George Dinwiddie * http://blog.gdinwiddie.com
Software Development http://www.idiacomputing.com
Consultant and Coach http://www.agilemaryland.org







[Non-text portions of this message have been removed]

Hi Andrew,

I'm trying to figure out the right approach in my thinking here. Suppose you
> have an application which has 10 different configuration options. Each
> option has 3 different settings. It doesn't take rocket science to figure
> out that this means 3^10 or over 59000 configurations. And those are small
> numbers! Certainly we can't write a test for every single one of these
> cases. Yet it's also hard to be absolutely sure that they're completely
> independent. I recently had a UI bug that was unexpectedly caused by bad
> data in a completely different area of the site, and I really don't think it
> was caused by poor design. So what do we make of this? How shall we then
> test?


One common approach is to make this simplifying assumption: Most
combination-related problems are caused by interactions of PAIRs of
variables, and very few are caused by larger tuples.

I have no idea under what circumstances that assumption is warranted. I do
know that making the assumption greatly reduces the number of combinations
you need. There's a technique called All Pairs Analysis (aka Pairwise
Analysis) that yields a minimal set of combinations that includes every pair
of values for every pair of variables.

If you're willing to make the pairwise assumption (and accept the attendant
risks), give All Pairs Analysis a try. James Bach offers a free tool on his
web site: http://www.satisfice.com/tools.shtml

Dale

--
Dale Emery, Consultant
Inspiring Leadership for Software People
Web: http://dhemery.com
Weblog: http://cwd.dhemery.com


[Non-text portions of this message have been removed]

Good day,

I tried it first with just a simple Java Tag.

public class MyJsTag extends TagSupport {
>
>     ...
>
>     public void doStartTag() {
>         new Processor(super.getValue())
>             .insert(super.pageContext);
>         return Tag.SKIP_BODY;
>     }
>     ...
> }
>

Thus, in my test, I have something like

String givenValue = "dummy value"
> myJsTag.setValue(givenValue)
> myJsTag.setPageContext(pageContextSpy);
>
> myJsTag.doStartTag();
>
> assertEquals("Should have processed the dummy value.", "formatted dummy
> vaule", pageContextSpy.getOutputString());
>

Instead of verifying one of the behaviors pageContext.getOut().write(...) or
pageContext.getOut().print(...) were executed, I just verified that
resulting state of pageContext.getOut().

Cheers,
Franz

On Wed, Jul 8, 2009 at 11:09 PM, Alan Baljeu <alanbaljeu@...> wrote:

>
>
> Franz, can you illustrate what you do with an example?
>
> Alan Baljeu
>
> ________________________________
> From: Franz Allan Valencia See <franz.see@...<franz.see%40gmail.com>
> >
>
>
> Hi Manuel,
>
> Thanks, I am currently doing #1 as you mentioned. But for behavior
> verification of my inputs or my dependencies, I currently prefer using
> Spies
> / Fakes instead of Mocks so that I could verify the verification by
> verifying the resulting state ( I am currently minimizing my use of Mocks
> since resulting code seems to be too brittle ).
>
> Thanks,
> Franz
>
> On Wed, Jul 8, 2009 at 1:37 PM, Manuel Klimek <klimek@box4. net> wrote:
>
> >
> >
> > I'd not just change all tests from state to interaction testing.
> >
> > For each test, look whether:
> > 1) the object under test can be set up by passing only value objects into
> > it: use state based testing
> > 2) you have to create other behavioral objects to create the object under
> > test, and you don't want to test the behavior in the other object: you
> hit
> > a
> > test smell; change the test to interaction testing by mocking out the
> > classes the object under test depends upon.
> >
> > In a nutshell: use both styles, depending on what you actually want to
> > test.
> >
> > /Manuel
>
> __________________________________________________________
> Looking for the perfect gift? Give the gift of Flickr!
>
> http://www.flickr.com/gift/
>
> [Non-text portions of this message have been removed]
>
>
>



--
Franz Allan Valencia See | Java Software Engineer
franz.see@...
LinkedIn: http://www.linkedin.com/in/franzsee


[Non-text portions of this message have been removed]

I'm trying to figure out the right approach in my thinking here. Suppose you
have an application which has 10 different configuration options. Each
option has 3 different settings. It doesn't take rocket science to figure
out that this means 3^10 or over 59000 configurations. And those are small
numbers! Certainly we can't write a test for every single one of these
cases. Yet it's also hard to be absolutely sure that they're completely
independent. I recently had a UI bug that was unexpectedly caused by bad
data in a completely different area of the site, and I really don't think it
was caused by poor design. So what do we make of this? How shall we then
test?


[Non-text portions of this message have been removed]

RhinoMocks or Moq. Moq seems more intuitive, though.

Tuna Toksöz
Eternal sunshine of the open source mind.

http://devlicio.us/blogs/tuna_toksoz
http://tunatoksoz.com
http://twitter.com/tehlike




On Fri, Jul 10, 2009 at 1:08 AM, George Dinwiddie
<lists@...>wrote:

> hat's most widely used C# mocking tool?


[Non-text portions of this message have been removed]

Hi All,

*i hope this formats ok, each time I press preview it is just a single
line*

I have sent a new example to Michael Feathers in the hope that he will
upload it to the files section here.

With this example I have tried to make it more clear how to go East
whilst showing that with the introduction of a facade/adaptor you

could combine logic together to make new logic out of existing
tested classes.

For example, making a CustomerSalutationPrinter print a salutation

in text and with the injection of another class print the
salutation in XML.  While keeping the responsibilities simple
and clear.

I'm not sure when the file upload will happen, so below are a few
test classes to wet your appetite.


Rgs, James.
-------------------

public class CustomersTest {

      private Customers customers;
      private Customer firstCustomer;
      private Customer secondCustomer;
      private CustomerVisitor customerVisitor;

      @Test
      public void shouldBeAbleToAddCustomerToCustomers() {
          customers.add(firstCustomer);
          assertTrue(customers.hasCustomerCount(1));
      }

      @Test
      public void shouldBeAbleToVisitEachCustomer() {
          customers.add(firstCustomer);
          customers.add(secondCustomer);

          customers.visitEachWith(customerVisitor);
          verify(customerVisitor).visit(firstCustomer);
          verify(customerVisitor).visit(secondCustomer);
      }

      @Before
      public void setup() {
          firstCustomer = mock(Customer.class);
          secondCustomer = mock(Customer.class);
          customerVisitor = mock(CustomerVisitor.class);
          customers = new Customers();
      }
}

public class CustomerPrinterTest {

      private CustomerPrinter customerPrinter;
      private Customer customer;
      private CustomerClient customerClient;

      @Test
      public void shouldBeAbleToPrintVisitedCustomers() {
          customerPrinter.visit(customer);
          verify(customer).printOn(customerClient);
      }

      @Before
      public void setup() {
          customer = mock(Customer.class);
          customerClient = mock(CustomerClient.class);
          customerPrinter = new CustomerPrinter(customerClient);
      }
}





[Non-text portions of this message have been removed]

We use moq in our C# testing and TDD classes and eLearning.   Best jk
Sent via BlackBerry from T-Mobile

-----Original Message-----
From: George Dinwiddie <lists@...>

Date: Thu, 09 Jul 2009 18:08:19
To: <testdrivendevelopment@yahoogroups.com>
Subject: [TDD] quick poll on C# mocking tools


What's most widely used C# mocking tool?

What's your current recommendation to a newby?

--
   ----------------------------------------------------------------------
    * George Dinwiddie *                      http://blog.gdinwiddie.com
    Software Development                    http://www.idiacomputing.com
    Consultant and Coach                    http://www.agilemaryland.org
   ----------------------------------------------------------------------



------------------------------------

Yahoo! Groups Links

Moq .

It has a clean interface and good documentation.

On Fri, Jul 10, 2009 at 12:20 AM, Adam Dymitruk <adam@...> wrote:

>
>
> RhinoMocks
>
>
> On Thu, Jul 9, 2009 at 3:08 PM, George
Dinwiddie<lists@...<lists%40idiacomputing.com>>
> wrote:
> >
> >
> > What's most widely used C# mocking tool?
> >
> > What's your current recommendation to a newby?
> >
> > --
> > ----------------------------------------------------------
> > * George Dinwiddie * http://blog.gdinwiddie.com
> > Software Development http://www.idiacomputing.com
> > Consultant and Coach http://www.agilemaryland.org
> > ----------------------------------------------------------
> >
> >
>
> --
> Adam Dymitruk
> Director of IT
> SCANit International (a division of APARA Systems)
> mobile: 604.831.7804
>
>



--
Sidar Ok

http://www.sidarok.com
http://www.twitter.com/sidarok


[Non-text portions of this message have been removed]

RhinoMocks

On Thu, Jul 9, 2009 at 3:08 PM, George Dinwiddie<lists@...> wrote:
>
>
> What's most widely used C# mocking tool?
>
> What's your current recommendation to a newby?
>
> --
> ----------------------------------------------------------
> * George Dinwiddie * http://blog.gdinwiddie.com
> Software Development http://www.idiacomputing.com
> Consultant and Coach http://www.agilemaryland.org
> ----------------------------------------------------------
>
>



--
Adam Dymitruk
Director of IT
SCANit International (a division of APARA Systems)
mobile: 604.831.7804

What's most widely used C# mocking tool?

What's your current recommendation to a newby?

--
   ----------------------------------------------------------------------
    * George Dinwiddie *                      http://blog.gdinwiddie.com
    Software Development                    http://www.idiacomputing.com
    Consultant and Coach                    http://www.agilemaryland.org
   ----------------------------------------------------------------------

Hi Carlos,

> > I think the point is that one needs to know the purpose of
> every test.
> > One should not write a test without some purpose.
> >
> > But having a fixed set of names is not a requirement for
> understanding
> > and - especially - failing to use some naming system advocated by
> > another person or group does not show a lack of understanding.
> >
> > So, yes, I can't fit my hypothetical test above into
> > *your* naming system, but it fits quite well in my own:
> > it's a customer test.
>
> Ok I see. I leave the idea of the code smell upon naming
> conventions. The purpose is the key.

I sometimes tell teams I work with that the three rules
of test-writing are:
  1. Know what you're testing
  2. Know what you're testing
  3. Know what you're testing

I think it *is* reasonable to say "If you don't know what
to call it, it *may* mean you don't know why you are
doing it."

> > I'd say a functional test is the most common kind of
> customer test (I
> > don't actually use "acceptance" any more) but there can be
> others. If
> > the customer has a performance requirement, then a performance test
> > could be a customer test.
> >
> > Functional versus "quality attributes" is a different axis from
> > customer/developer. So saying a functional test is an
> acceptance test
> > is like saying "big" is "red"
>
> That is a bit confusing, let me rebuild this in my mind:
>
> A functional test can be any kind of test as every test is
> checking out some kind of functionality. A class method in
> the finest case.

Sure. But they bring to mind the english term "functional
requirement" which is higher level.

> However we use functional test for customer tests which
> excersice some kind of functional business requirement.
> Customer tests are broader than functional tests, as they can
> be performance or load tests, if performance and load are
> requirements.

> Better?

For me, yes. BUT... I'm a 66 year old anglophone who remembers a
lot of old terminology. You need to think of what will work for
*your* audience. That said, I attended a meeting of an architect
group not too long ago and they still talk about functional
and non-functional requirements.

I wonder, is there a Spanish-language group on which you can
raise some of these questions of terminology?

>
> >
> >> The acceptance test could be smaller than a system test, that is,
> >> could just test a business requirement which is smaller
> than a screen
> >> use case or any other system function. When the acceptance test is
> >> end to end, it is also a system test.
> >
> > I don't use the term system test that way. In my lexicon, a system
> > test exercises the system from end to end and from side to
> side - that
> > is it exercises all or a major subset of the system's functionality.
> >
>
> Aren't we saying the same thing? Do you mean that a system
> test is not just like... "fill this form, click on the send
> button and validate that page says, 'form sent'"? Do you mean
> that a single system test should test the whole application?
> Because I wrote the system test is and end to end test and
> you did so too.
> That confuses me.

In medical equipment development, we didn't consider something
a system test unless it used the actual physical device. The
entire system had to be exercised.

In other kinds of work, I think it means using an actual
database, web farm, etc. Normally, when you do a system
test - since it's quite expensive - you run an entire
suite of tests rather than testing one bit of functionality.

Running the whole application to test a single story would
be a customer test in my mind. Whether we use the whole
system or a part of it is simply an implementation issue.

BTW, I mentioned that I no longer use the terminology
"acceptance test." That's because of having worked in some
safety critical applications where certain people were
offended by the idea that merely passing an automated
test would be sufficient to "accept" the product.

So, I adapted my language to the culture of the company.
In some cases, it was a functional test. In others a
customer test. In one case it was called something like
an "pre-qualification test" meaning that the work could
now be passed on to the separate quality assurance group. :-)

Charlie

> >> Eventually we could state again that a funtional test is
> not a system
> >> test because it could be smaller in scope than a system test.
> >>
> >> Thanks a ton Charlie :-)
> >
> > I hope it's helping. I think it's very important to acknowledge in
> > your book that there are multiple ways to characterize tests. We're
> > calling those dimensions or axes, but they could be thought
> of simply
> > as attributes as well. In theory, you could combine the values of
> > these attributes in a test in any way you like. But in practice,
> > experienced developers only use certain combinations.
>
> I'll do so
>
> >
> > For example, customer tests are usually bigger in scope
> than a single
> > object, load tests usually cover the entire system and mocking
> > techniques are more likely to be used in tests that are relatively
> > smaller in scope. Of course, there are exceptions to each of these
> > examples, but it seems to me that the overall patterns are
> much more
> > intereseting than the exceptions. Just as modern physics
> can only tell
> > us where an object is likely to be, so we programmers can only say
> > that such and such a test is likely to be a unit test - or
> a customer
> > test - or some other kind. ;-)
> >
> > Charlie
> >
> >> > Food for thought anyway. :-)
> >> >
> >> > Charlie
> >> >
> >> >
> >>
> >>
> >> ------------------------------------
> >>
> >> Yahoo! Groups Links
> >>
> >>
> >>
> >>
> >
> >
>
>
> ------------------------------------
>
> Yahoo! Groups Links
>
>
>
>

2009/7/9 Charlie Poole <cpoole@...>:
>
>
> Hi Carlos,
>
>> > Sure... something I've done is to test a desktop gui using
>> a fake back
>> > end. Still, it has been a rare for me to do that.
>> > But note that I'm writing about what I've done - not about
>> what might
>> > be done. :-)
>> >
>>
>> This is exactly what I wanted to point out with code smell.
>
> I don't think we're pointing out the same thing. :-)
>
>> One can write any kind of test, of course. But it does not
>> mean that the test is meaningful or useful. The fact that you
>> wouldn't know how to call that test might mean you should not
>> write it. Maybe because the functionality is already covered
>> with a bunch of fine grained unit tests, so that a coarse
>> grained one don't pay its maintainability cost. Or maybe
>> because the tests is showing the SUT is not accomplishing the
>> single responsibility principle, or any other bad design. It
>> is hard to me to explain, specially in a language that is not
>> my mother tongue, but I hope you get the point.
>
> I think the point is that one needs to know the purpose of
> every test. One should not write a test without some purpose.
>
> But having a fixed set of names is not a requirement for
> understanding and - especially - failing to use some
> naming system advocated by another person or group does
> not show a lack of understanding.
>
> So, yes, I can't fit my hypothetical test above into
> *your* naming system, but it fits quite well in my own:
> it's a customer test.

Ok I see. I leave the idea of the code smell upon naming
conventions. The purpose is the key.


>
>> >> > Another distinction that seems important is between tests
>> >> of correct
>> >> > function and tests of performance, behavior under load,
>> >> etc. I'm not
>> >> > sure what to call this axis.
>> >> >
>> >>
>> >> I used an ellipsis (...) to indicate that I thought there
>> were more,
>> >> but hadn't enumerated them. Here are some ideas:
>> >
>> > Yes, I saw the ellipsis - to be clear, I didn't intend my offer of
>> > another dimension as a criticism of your list. :-)
>> >
>> >> There is a time dimension. We talk in terms of First vs.
>> >> Last, but there are others. There are initial tests that
>> describe the
>> >> behavior we are looking for. There are just-in-time tests that
>> >> elaborate that.
>> >> There are tests for bugs/regressions/clarifications that we put in
>> >> after the fact - sometimes even after release.
>> >>
>> >> The distinctions between performance-based tests and
>> behavior-based
>> >> tests and others ??? are significant, but I don't know if
>> that can be
>> >> delineated the same way (i.e. is it a dimension with two
>> extremes and
>> >> some gradations in the
>> >> middle?) It is certainly an important distinction and
>> would have to
>> >> be a part of any complete taxonomy (Should such a thing
>> ever exist.)
>> >
>> > I wanted to focus on functional versus ... non-functional? ...
>> > tests because the OP asked about functional testing. Folks still
>> > distinguish functional versus non-functional requirements in many
>> > companies and projects. I don't much like calling the tests of
>> > non-functional requrements non-functional tests though. :-)
>> > Performance and load tests are the only kind that come to
>> mind right
>> > now... Ah, here's wikipedia with a longer list than I had imagined:
>> > http://en.wikipedia.org/wiki/Non-functional_requirement
>> >
>>
>> I am with you. Tell me if you consider the next statement right:
>> A functional test is just an acceptance test. An acceptance
>> test might or might not be a system test. A system test is an
>> end to end test, a test of part of the application (system)
>> which uses it as the human user, but automatically.
>
> I'd say a functional test is the most common kind of customer
> test (I don't actually use "acceptance" any more) but there
> can be others. If the customer has a performance requirement,
> then a performance test could be a customer test.
>
> Functional versus "quality attributes" is a different axis
> from customer/developer. So saying a functional test is an
> acceptance test is like saying "big" is "red"

That is a bit confusing, let me rebuild this in my mind:

A functional test can be any kind of test as every test is checking out
some kind of functionality. A class method in the finest case.
However we use functional test for customer tests which excersice
some kind of functional business requirement. Customer tests are
broader than functional tests, as they can be performance or load
tests, if performance and load are requirements.
Better?


>
>> The acceptance test could be smaller than a system test, that is,
>> could just test a business requirement which is smaller than
>> a screen use case or any other system function. When the
>> acceptance test is end to end, it is also a system test.
>
> I don't use the term system test that way. In my lexicon,
> a system test exercises the system from end to end and from
> side to side - that is it exercises all or a major subset
> of the system's functionality.
>

Aren't we saying the same thing? Do you mean that a system
test is not just like... "fill this form, click on the send button and
validate that page says, 'form sent'"? Do you mean that a single
system test should test the whole application? Because I wrote
the system test is and end to end test and you did so too.
That confuses me.


>> Eventually we could state again that a funtional test is not
>> a system test because it could be smaller in scope than a system test.
>>
>> Thanks a ton Charlie :-)
>
> I hope it's helping. I think it's very important to acknowledge in
> your book that there are multiple ways to characterize tests. We're
> calling those dimensions or axes, but they could be thought of simply
> as attributes as well. In theory, you could combine the values of
> these attributes in a test in any way you like. But in practice,
> experienced developers only use certain combinations.

I'll do so

>
> For example, customer tests are usually bigger in scope than
> a single object, load tests usually cover the entire system
> and mocking techniques are more likely to be used in tests
> that are relatively smaller in scope. Of course, there are
> exceptions to each of these examples, but it seems to me
> that the overall patterns are much more intereseting than
> the exceptions. Just as modern physics can only tell us
> where an object is likely to be, so we programmers can
> only say that such and such a test is likely to be a
> unit test - or a customer test - or some other kind. ;-)
>
> Charlie
>
>> > Food for thought anyway. :-)
>> >
>> > Charlie
>> >
>> >
>>
>>
>> ------------------------------------
>>
>> Yahoo! Groups Links
>>
>>
>>
>>
>
>

Hi Adam,

> If we're discussion taxonomy it seems that we need to be able
> to categorize whatever might be legitimately useful. We don't
> need to include antipatterns, but we probably do need to
> include things we might do but haven't needed/chosen to do.

Sure. I'm just not sure how much effort I'd want to spend on
taxonomy before moving on to testing. :-)

In replying to Carlos, I got into the question of patterns
of usage. That is, once we have a set of dimensions, where
do the best teams and developers cluster?

Charlie

Mike Vizdos and I would like to invite you spend a few moments taking the 2009
Agile Practice Adoption survey at
http://www.surveymonkey.com/s.aspx?sm=3lZZ3fYRxvV3y6XsnXPDEw_3d_3d .  The goal
of this survey is to find out what practices agile practitioners find useful,
which they're finding difficult to adopt, and which they're abandoning.  There
are 17 questions in this survey, although you may not be asked all of them, and
it should take you at most 5 minutes to complete.

To entice you to fill out the survey we'll be drawing for 10 copies of "Stand
Back and Deliver: Accelerating Business Agility" by Pollyanna Pixton, Niel
Nickolaisen, Todd Little, and Kent McDonald published in June 2009 by Addison
Wesley.  I've been reading through this book the past few days and I'm finding a
lot of great insights in it.  More information can be found at
http://www.informit.com/store/product.aspx?isbn=0321572882

The results of this survey will be summarized in my "Agile by the Numbers"
presentation at Agile 2009 in Chicago (Aug 24-28). Furthermore, this is an open
survey, so the source data (without identifying information to protect your
privacy), a summary slide deck, and the original source questions will be posted
at www.ambysoft.com/surveys/ so that others may analyze the data for their own
purposes. Data from previous surveys have been used by university students and
professors for their research papers, and hopefully the same will be true of the
data from this survey. The results from several other surveys are already posted
there, so please feel free to take advantage of this resource.

We apologize if you've received several copies of this request.

- Scott
Scott W. Ambler
Chief Methodologist/Agile, IBM Rational
Agile at Scale: http://www.ibm.com/developerworks/blogs/page/ambler


       __________________________________________________________________
Yahoo! Canada Toolbar: Search from anywhere on the web, and bookmark your
favourite sites. Download it now
http://ca.toolbar.yahoo.com.

On Wed, Jul 8, 2009 at 4:08 PM, Charlie Poole<cpoole@...> wrote:
>
>
> Hi Adam,
>
>> An example of interaction-based techniques for a Customer test:
>>
>> The user fills in a form on a web page which generates an
>> image that gets transmitted to a fax service such as eFax. We
>> could create an acceptance test that exercised a complete UI
>> but called a mocked service.
>
> That's a good technique. I wouldn't have called this an
> interaction test, but if that's what you call it then
> we're in agreement. :-)
>

I wouldn't have called it an interaction test. As I described the
axis, there is a spectrum between tests that verify state and tests
that specify messages to be passed (And many tests do a bit of both.)
In this case, the part of the test that mocks the service is purely
the latter, and the part of the test that verifies the interface is
more of the former.

On Wed, Jul 8, 2009 at 3:58 PM, Charlie Poole<cpoole@...> wrote:
>
>
> Hi Adam,
>
>> >> unit in isolation <---> system as a whole for the benefit of the
>> >> developer <---> for the benefit of the customer verifying
>> state <--->
>> >> specifying messages to be passed ("behavior") ...
>> >
>> > The last seems to me to be non-orthogonal. Normally, I would use
>> > interaction tests in order to test isolated components.
>> > And I'd rarely expect to use them in customer tests. This
>> seems more
>> > like an attribute of the tests - albeit an important one -
>> than a part
>> > of the taxonomy we're looking for.
>> >
>>
>> Probably not orthogonal, but a dimension along which they can
>> be described. There are cases where interaction-based
>> techniques are useful for customer tests. There are plenty of
>> cases where interaction-based techniques can be used in
>> partially integrated systems. As a matter of fact, the first
>> descriptions of Mock objects that I ever encountered
>> discussed mocking at "system boundaries" such as file system
>> or database and weren't isolation tests per se.
>
> Sure... something I've done is to test a desktop gui using a
> fake back end. Still, it has been a rare for me to do that.
> But note that I'm writing about what I've done - not about
> what might be done. :-)
>

If we're discussion taxonomy it seems that we need to be able to
categorize whatever might be legitimately useful. We don't need to
include antipatterns, but we probably do need to include things we
might do but haven't needed/chosen to do.

>> > Another distinction that seems important is between tests
>> of correct
>> > function and tests of performance, behavior under load,
>> etc. I'm not
>> > sure what to call this axis.
>> >
>>
>> I used an ellipsis (...) to indicate that I thought there
>> were more, but hadn't enumerated them. Here are some ideas:
>
> Yes, I saw the ellipsis - to be clear, I didn't intend my
> offer of another dimension as a criticism of your list. :-)
>

I didn't mean to suggest that either.

>> There is a time dimension. We talk in terms of First vs.
>> Last, but there are others. There are initial tests that
>> describe the behavior we are looking for. There are
>> just-in-time tests that elaborate that.
>> There are tests for bugs/regressions/clarifications that we
>> put in after the fact - sometimes even after release.
>>
>> The distinctions between performance-based tests and
>> behavior-based tests and others ??? are significant, but I
>> don't know if that can be delineated the same way (i.e. is it
>> a dimension with two extremes and some gradations in the
>> middle?) It is certainly an important distinction and would
>> have to be a part of any complete taxonomy (Should such a
>> thing ever exist.)
>
> I wanted to focus on functional versus ... non-functional? ...
> tests because the OP asked about functional testing. Folks
> still distinguish functional versus non-functional requirements
> in many companies and projects. I don't much like calling the
> tests of non-functional requrements non-functional tests though. :-)
> Performance and load tests are the only kind that come to mind
> right now... Ah, here's wikipedia with a longer list than I
> had imagined:
> http://en.wikipedia.org/wiki/Non-functional_requirement
>

That's a good thought. I remember first encountering the term
"non-functional requirement" and thinking, "who wants non-functional
software?" But, assuming you know what a "non-functional requirement"
is you still want the test for it to "function."

> Food for thought anyway. :-)
>

Definitely.

On Wed, Jul 8, 2009 at 2:30 PM, Dale Emery<dale@...> wrote:
>
>
> Hi Adam
>
> The distinctions between performance-based tests and behavior-based tests
>> and others ??? are significant, but I don't know if that can be delineated
>> the same way (i.e. is it a dimension with two extremes and some gradations
>> in the middle?) It is certainly an important distinction and would have to
>> be a part of any complete taxonomy
>
> My choice of the word "dimension" in my blog post may have been a bad idea.
> Better would be the more general "variable." A dimension might be a kind of
> variable whose values can meaningfully be arranged on a line. But not all
> variables are like that.
>

Hi Dale:

I actually never read the blog post. I don't know whether I was
picking up on your use of the word "dimension" or whether I arrived at
it on my own. I think it is a useful concept to talk about various
dimensions/axes/spectra but it isn't sufficient for categorizing /all/
of the various tests that we might choose to create in all of the
different ways that we might choose to categorize them.

Hi Carlos,

> > Sure... something I've done is to test a desktop gui using
> a fake back
> > end. Still, it has been a rare for me to do that.
> > But note that I'm writing about what I've done - not about
> what might
> > be done. :-)
> >
>
> This is exactly what I wanted to point out with code smell.

I don't think we're pointing out the same thing. :-)

> One can write any kind of test, of course. But it does not
> mean that the test is meaningful or useful. The fact that you
> wouldn't know how to call that test might mean you should not
> write it. Maybe because the functionality is already covered
> with a bunch of fine grained unit tests, so that a coarse
> grained one don't pay its maintainability cost. Or maybe
> because the tests is showing the SUT is not accomplishing the
> single responsibility principle, or any other bad design.  It
> is hard to me to explain, specially in a language that is not
> my mother tongue, but I hope you get the point.

I think the point is that one needs to know the purpose of
every test. One should not write a test without some purpose.

But having a fixed set of names is not a requirement for
understanding and - especially - failing to use some
naming system advocated by another person or group does
not show a lack of understanding.

So, yes, I can't fit my hypothetical test above into
*your* naming system, but it fits quite well in my own:
it's a customer test.

> >> > Another distinction that seems important is between tests
> >> of correct
> >> > function and tests of performance, behavior under load,
> >> etc. I'm not
> >> > sure what to call this axis.
> >> >
> >>
> >> I used an ellipsis (...) to indicate that I thought there
> were more,
> >> but hadn't enumerated them. Here are some ideas:
> >
> > Yes, I saw the ellipsis - to be clear, I didn't intend my offer of
> > another dimension as a criticism of your list. :-)
> >
> >> There is a time dimension. We talk in terms of First vs.
> >> Last, but there are others. There are initial tests that
> describe the
> >> behavior we are looking for. There are just-in-time tests that
> >> elaborate that.
> >> There are tests for bugs/regressions/clarifications that we put in
> >> after the fact - sometimes even after release.
> >>
> >> The distinctions between performance-based tests and
> behavior-based
> >> tests and others ??? are significant, but I don't know if
> that can be
> >> delineated the same way (i.e. is it a dimension with two
> extremes and
> >> some gradations in the
> >> middle?) It is certainly an important distinction and
> would have to
> >> be a part of any complete taxonomy (Should such a thing
> ever exist.)
> >
> > I wanted to focus on functional versus ... non-functional? ...
> > tests because the OP asked about functional testing. Folks still
> > distinguish functional versus non-functional requirements in many
> > companies and projects. I don't much like calling the tests of
> > non-functional requrements non-functional tests though. :-)
> > Performance and load tests are the only kind that come to
> mind right
> > now... Ah, here's wikipedia with a longer list than I had imagined:
> > http://en.wikipedia.org/wiki/Non-functional_requirement
> >
>
> I am with you. Tell me if you consider the next statement right:
> A functional test is just an acceptance test. An acceptance
> test might or might not be a system test. A system test is an
> end to end test, a test of part of the application (system)
> which uses it as the human user, but automatically.

I'd say a functional test is the most common kind of customer
test (I don't actually use "acceptance" any more) but there
can be others. If the customer has a performance requirement,
then a performance test could be a customer test.

Functional versus "quality attributes" is a different axis
from customer/developer. So saying a functional test is an
acceptance test is like saying "big" is "red"

> The acceptance test could be smaller than a system test, that is,
> could just test a business requirement which is smaller than
> a screen use case or any other system function. When the
> acceptance test is end to end, it is also a system test.

I don't use the term system test that way. In my lexicon,
a system test exercises the system from end to end and from
side to side - that is it exercises all or a major subset
of the system's functionality.

> Eventually we could state again that a funtional test is not
> a system test because it could be smaller in scope than a system test.
>
> Thanks a ton Charlie :-)

I hope it's helping. I think it's very important to acknowledge in
your book that there are multiple ways to characterize tests. We're
calling those dimensions or axes, but they could be thought of simply
as attributes as well. In theory, you could combine the values of
these attributes in a test in any way you like. But in practice,
experienced developers only use certain combinations.

For example, customer tests are usually bigger in scope than
a single object, load tests usually cover the entire system
and mocking techniques are more likely to be used in tests
that are relatively smaller in scope. Of course, there are
exceptions to each of these examples, but it seems to me
that the overall patterns are much more intereseting than
the exceptions. Just as modern physics can only tell us
where an object is likely to be, so we programmers can
only say that such and such a test is likely to be a
unit test - or a customer test - or some other kind. ;-)

Charlie


> > Food for thought anyway. :-)
> >
> > Charlie
> >
> >
>
>
> ------------------------------------
>
> Yahoo! Groups Links
>
>
>
>

Great answers guys, you definitively will be named in the book with
links to your
blogs.
See some more comments below.

2009/7/8 Charlie Poole <cpoole@...>:
>
>
> Hi Adam,
>
>> >> unit in isolation <---> system as a whole for the benefit of the
>> >> developer <---> for the benefit of the customer verifying
>> state <--->
>> >> specifying messages to be passed ("behavior") ...
>> >
>> > The last seems to me to be non-orthogonal. Normally, I would use
>> > interaction tests in order to test isolated components.
>> > And I'd rarely expect to use them in customer tests. This
>> seems more
>> > like an attribute of the tests - albeit an important one -
>> than a part
>> > of the taxonomy we're looking for.
>> >
>>
>> Probably not orthogonal, but a dimension along which they can
>> be described. There are cases where interaction-based
>> techniques are useful for customer tests. There are plenty of
>> cases where interaction-based techniques can be used in
>> partially integrated systems. As a matter of fact, the first
>> descriptions of Mock objects that I ever encountered
>> discussed mocking at "system boundaries" such as file system
>> or database and weren't isolation tests per se.
>
> Sure... something I've done is to test a desktop gui using a
> fake back end. Still, it has been a rare for me to do that.
> But note that I'm writing about what I've done - not about
> what might be done. :-)
>

This is exactly what I wanted to point out with code smell. One
can write any kind of test, of course. But it does not mean that
the test is meaningful or useful. The fact that you wouldn't know
how to call that test might mean you should not write it. Maybe
because the functionality is already covered with a bunch of
fine grained unit tests, so that a coarse grained one don't pay
its maintainability cost. Or maybe because the tests is showing
the SUT is not accomplishing the single responsibility principle,
or any other bad design.  It is hard to me to explain, specially
in a language that is not my mother tongue, but I hope you
get the point.




>> > Another distinction that seems important is between tests
>> of correct
>> > function and tests of performance, behavior under load,
>> etc. I'm not
>> > sure what to call this axis.
>> >
>>
>> I used an ellipsis (...) to indicate that I thought there
>> were more, but hadn't enumerated them. Here are some ideas:
>
> Yes, I saw the ellipsis - to be clear, I didn't intend my
> offer of another dimension as a criticism of your list. :-)
>
>> There is a time dimension. We talk in terms of First vs.
>> Last, but there are others. There are initial tests that
>> describe the behavior we are looking for. There are
>> just-in-time tests that elaborate that.
>> There are tests for bugs/regressions/clarifications that we
>> put in after the fact - sometimes even after release.
>>
>> The distinctions between performance-based tests and
>> behavior-based tests and others ??? are significant, but I
>> don't know if that can be delineated the same way (i.e. is it
>> a dimension with two extremes and some gradations in the
>> middle?) It is certainly an important distinction and would
>> have to be a part of any complete taxonomy (Should such a
>> thing ever exist.)
>
> I wanted to focus on functional versus ... non-functional? ...
> tests because the OP asked about functional testing. Folks
> still distinguish functional versus non-functional requirements
> in many companies and projects. I don't much like calling the
> tests of non-functional requrements non-functional tests though. :-)
> Performance and load tests are the only kind that come to mind
> right now... Ah, here's wikipedia with a longer list than I
> had imagined:
> http://en.wikipedia.org/wiki/Non-functional_requirement
>

I am with you. Tell me if you consider the next statement right:
A functional test is just an acceptance test. An acceptance test
might or might not be a system test. A system test is an end to end
test, a test of part of the application (system) which uses it as
the human user, but automatically. The acceptance test could
be smaller than a system test, that is, could just test a business
requirement which is smaller than a screen use case or any other
system function. When the acceptance test is end to end, it is
also a system test. Eventually we could state again that a funtional
test is not a system test because it could be smaller in scope
than a system test.

Thanks a ton Charlie :-)



> Food for thought anyway. :-)
>
> Charlie
>
>

Charlie Poole wrote:
>> I like the term "quality attribute" because it tells me what
>> they are, instead of what they aren't.
>
> I like that. It fits with the common jargon of calling
> all these things "ilities"

So a test for one of these would be a QA ility?

   - George

--
   ----------------------------------------------------------------------
    * George Dinwiddie *                      http://blog.gdinwiddie.com
    Software Development                    http://www.idiacomputing.com
    Consultant and Coach                    http://www.agilemaryland.org
   ----------------------------------------------------------------------

Hi Dale,

> I wanted to focus on functional versus ... non-functional?
> ... tests because
> > the OP asked about functional testing. Folks  still distinguish
> > functional versus non-functional requirements in many companies and
> > projects. I don't much like calling the tests of non-functional
> > requrements non-functional tests though. :-) Performance and load
> > tests are the only kind that come to mind right now...
>
>
> I like the term "quality attribute" because it tells me what
> they are, instead of what they aren't.

I like that. It fits with the common jargon of calling
all these things "ilities"

Charlie

Hi Charlie,

I wanted to focus on functional versus ... non-functional? ... tests because
> the OP asked about functional testing. Folks  still distinguish functional
> versus non-functional requirements in many companies and projects. I don't
> much like calling the tests of non-functional requrements non-functional
> tests though. :-) Performance and load tests are the only kind that come to
> mind right now...


I like the term "quality attribute" because it tells me what they are,
instead of what they aren't.

Dale

--
Dale Emery, Consultant
Inspiring Leadership for Software People
Web: http://dhemery.com
Weblog: http://cwd.dhemery.com


[Non-text portions of this message have been removed]

Hi Adam,

> An example of interaction-based techniques for a Customer test:
>
> The user fills in a form on a web page which generates an
> image that gets transmitted to a fax service such as eFax. We
> could create an acceptance test that exercised a complete UI
> but called a mocked service.

That's a good technique. I wouldn't have called this an
interaction test, but if that's what you call it then
we're in agreement. :-)

Charlie

Hi Adam,

> >> unit in isolation <---> system as a whole for the benefit of the
> >> developer <---> for the benefit of the customer verifying
> state <--->
> >> specifying messages to be passed ("behavior") ...
> >
> > The last seems to me to be non-orthogonal. Normally, I would use
> > interaction tests in order to test isolated components.
> > And I'd rarely expect to use them in customer tests. This
> seems more
> > like an attribute of the tests - albeit an important one -
> than a part
> > of the taxonomy we're looking for.
> >
>
> Probably not orthogonal, but a dimension along which they can
> be described. There are cases where interaction-based
> techniques are useful for customer tests. There are plenty of
> cases where interaction-based techniques can be used in
> partially integrated systems. As a matter of fact, the first
> descriptions of Mock objects that I ever encountered
> discussed mocking at "system boundaries" such as file system
> or database and weren't isolation tests per se.

Sure... something I've done is to test a desktop gui using a
fake back end. Still, it has been a rare for me to do that.
But note that I'm writing about what I've done - not about
what might be done. :-)

> > Another distinction that seems important is between tests
> of correct
> > function and tests of performance, behavior under load,
> etc. I'm not
> > sure what to call this axis.
> >
>
> I used an ellipsis (...) to indicate that I thought there
> were more, but hadn't enumerated them. Here are some ideas:

Yes, I saw the ellipsis - to be clear, I didn't intend my
offer of another dimension as a criticism of your list. :-)

> There is a time dimension. We talk in terms of First vs.
> Last, but there are others. There are initial tests that
> describe the behavior we are looking for. There are
> just-in-time tests that elaborate that.
> There are tests for bugs/regressions/clarifications that we
> put in after the fact - sometimes even after release.
>
> The distinctions between performance-based tests and
> behavior-based tests and others ??? are significant, but I
> don't know if that can be delineated the same way (i.e. is it
> a dimension with two extremes and some gradations in the
> middle?) It is certainly an important distinction and would
> have to be a part of any complete taxonomy (Should such a
> thing ever exist.)

I wanted to focus on functional versus ... non-functional? ...
tests because the OP asked about functional testing. Folks
still distinguish functional versus non-functional requirements
in many companies and projects. I don't much like calling the
tests of non-functional requrements non-functional tests though. :-)
Performance and load tests are the only kind that come to mind
right now... Ah, here's wikipedia with a longer list than I
had imagined:
  http://en.wikipedia.org/wiki/Non-functional_requirement

Food for thought anyway. :-)

Charlie

Hi Adam

The distinctions between performance-based tests and behavior-based tests
> and others ??? are significant, but I don't know if that can be  delineated
> the same way (i.e. is it a dimension with two extremes and some gradations
> in the middle?) It is certainly an important  distinction and would have to
> be a part of any complete taxonomy


My choice of the word "dimension" in my blog post may have been a bad idea.
Better would be the more general "variable." A dimension might be a kind of
variable whose values can meaningfully be arranged on a line. But not all
variables are like that.

Dale

--
Dale Emery, Consultant
Inspiring Leadership for Software People
Web: http://dhemery.com
Weblog: http://cwd.dhemery.com


[Non-text portions of this message have been removed]

An example of interaction-based techniques for a Customer test:

The user fills in a form on a web page which generates an image that
gets transmitted to a fax service such as eFax. We could create an
acceptance test that exercised a complete UI but called a mocked
service.

On Wed, Jul 8, 2009 at 2:09 PM, Adam Sroka<adam.sroka@...> wrote:
> On Wed, Jul 8, 2009 at 8:49 AM, Charlie Poole<cpoole@...>
wrote:
>
>>> unit in isolation <---> system as a whole for the benefit of
>>> the developer <---> for the benefit of the customer verifying
>>> state <---> specifying messages to be passed ("behavior") ...
>>
>> The last seems to me to be non-orthogonal. Normally, I would
>> use interaction tests in order to test isolated components.
>> And I'd rarely expect to use them in customer tests. This
>> seems more like an attribute of the tests - albeit an
>> important one - than a part of the taxonomy we're looking
>> for.
>>
>
> Probably not orthogonal, but a dimension along which they can be
> described. There are cases where interaction-based techniques are
> useful for customer tests. There are plenty of cases where
> interaction-based techniques can be used in partially integrated
> systems. As a matter of fact, the first descriptions of Mock objects
> that I ever encountered discussed mocking at "system boundaries" such
> as file system or database and weren't isolation tests per se.
>
>> Another distinction that seems important is between tests
>> of correct function and tests of performance, behavior
>> under load, etc. I'm not sure what to call this axis.
>>
>
> I used an ellipsis (...) to indicate that I thought there were more,
> but hadn't enumerated them. Here are some ideas:
>
> There is a time dimension. We talk in terms of First vs. Last, but
> there are others. There are initial tests that describe the behavior
> we are looking for. There are just-in-time tests that elaborate that.
> There are tests for bugs/regressions/clarifications that we put in
> after the fact - sometimes even after release.
>
> The distinctions between performance-based tests and behavior-based
> tests and others ??? are significant, but I don't know if that can be
> delineated the same way (i.e. is it a dimension with two extremes and
> some gradations in the middle?) It is certainly an important
> distinction and would have to be a part of any complete taxonomy
> (Should such a thing ever exist.)
>

On Wed, Jul 8, 2009 at 8:49 AM, Charlie Poole<cpoole@...> wrote:

>> unit in isolation <---> system as a whole for the benefit of
>> the developer <---> for the benefit of the customer verifying
>> state <---> specifying messages to be passed ("behavior") ...
>
> The last seems to me to be non-orthogonal. Normally, I would
> use interaction tests in order to test isolated components.
> And I'd rarely expect to use them in customer tests. This
> seems more like an attribute of the tests - albeit an
> important one - than a part of the taxonomy we're looking
> for.
>

Probably not orthogonal, but a dimension along which they can be
described. There are cases where interaction-based techniques are
useful for customer tests. There are plenty of cases where
interaction-based techniques can be used in partially integrated
systems. As a matter of fact, the first descriptions of Mock objects
that I ever encountered discussed mocking at "system boundaries" such
as file system or database and weren't isolation tests per se.

> Another distinction that seems important is between tests
> of correct function and tests of performance, behavior
> under load, etc. I'm not sure what to call this axis.
>

I used an ellipsis (...) to indicate that I thought there were more,
but hadn't enumerated them. Here are some ideas:

There is a time dimension. We talk in terms of First vs. Last, but
there are others. There are initial tests that describe the behavior
we are looking for. There are just-in-time tests that elaborate that.
There are tests for bugs/regressions/clarifications that we put in
after the fact - sometimes even after release.

The distinctions between performance-based tests and behavior-based
tests and others ??? are significant, but I don't know if that can be
delineated the same way (i.e. is it a dimension with two extremes and
some gradations in the middle?) It is certainly an important
distinction and would have to be a part of any complete taxonomy
(Should such a thing ever exist.)

Franz, can you illustrate what you do with an example?

  Alan Baljeu





________________________________
From: Franz Allan Valencia See <franz.see@...>



Hi Manuel,

Thanks, I am currently doing #1 as you mentioned. But for behavior
verification of my inputs or my dependencies, I currently prefer using Spies
/ Fakes instead of Mocks so that I could verify the verification by
verifying the resulting state ( I am currently minimizing my use of Mocks
since resulting code seems to be too brittle ).

Thanks,
Franz

On Wed, Jul 8, 2009 at 1:37 PM, Manuel Klimek <klimek@box4. net> wrote:

>
>
> I'd not just change all tests from state to interaction testing.
>
> For each test, look whether:
> 1) the object under test can be set up by passing only value objects into
> it: use state based testing
> 2) you have to create other behavioral objects to create the object under
> test, and you don't want to test the behavior in the other object: you hit
> a
> test smell; change the test to interaction testing by mocking out the
> classes the object under test depends upon.
>
> In a nutshell: use both styles, depending on what you actually want to
> test.
>
> /Manuel


       __________________________________________________________________
Looking for the perfect gift? Give the gift of Flickr!

http://www.flickr.com/gift/

[Non-text portions of this message have been removed]

Hi Nat,

> > If all acceptance tests were system tests, then it would be
> hard for
> > XP projects to have acceptance tests for each story. :-)
>
> Do you mean because of the long time it would take to run the
> acceptance tests?

No, because the system has to be built before you can run
the whole system. :-)

As I've always used it, a system test is one that exercises
the whole system and tests it's full functionality. You can't
have such a test till pretty near the end of the project.

If instead we mean testing some piece or pieces
of functionality by running the whole system, that's
a reasonable thing to do - but I don't call it a
system test.

Charlie