Good question.

I'm trying to find out whether you can only be agile when you are doing
projects of small sizes (5 manyears or less).

Why wouldn't being agile not be important for bigger projects?  We develop
in an iterative and incremental way.  Increments are delivered each 6 to 8
weeks.

We are not doing XP.  I don't think there is anyone in our development shop
that believes that we are doing one or another process as documented in a
book.  It's mix and match.

I don't believe so much in name spaces anyway (in terms of processes).  Who
cares whether or not someone is doing XP, SCRUM, etc.  Finally you have to
be able to decide to do what is right for your business.  Probably doing
less rather than doing more for large organisations and projects, probably
doing more for small organisations and projects.
XP is a collection of a set of practices.  Some practices may be sufficient,
some are mandatory, some are perhaps insufficient.  It will all depend on
context.

Look at all the possible practices, take those that fit your business and
your local culture, and try to get better.  It takes lots of knowledge and
experience before you can do it.

Perhaps being agile can also mean, learn how to be efficient and effective.
Scalability is what I am looking for and a decision process to know what you
have to do depending on what the scale is.

How do you define context and come up with the right solution.  If this is
not about being agile, what is?

How would you implement XP in a Component Based Development environment?

Stefan

> -----Original Message-----
> From: Lisa Crispin [mailto:lisa.crispin@...]
> Sent: Wednesday, November 07, 2001 5:59 PM
> To: agile-testing@yahoogroups.com
> Subject: [agile-testing] Re: testing non-web GUI
>
>
> Well said.
>
> I'm curious, though, is the project you are talking about (50 man
> years) using XP or some other agile processes?
>
>

Yes, I am completely aware that Lisa asked this question in the context of
XP and I am very aware of the descriptions of acceptance testing.
Ultimately, the product has to actually serve the needs of the customer and
I think the relevant testing goes far beyond what I've seen described as
story-based acceptance testing.

The points that I made about the GUI tools are specifically with reference
to Lisa's question as to whether these approaches can be agile or not.
Vendors selling the tools typically advocate use under development
processes that control maintenance cost by controlling change of the
application under test. Obviously, this isn't compatible with any of the
agile approaches. But depending on the project duration, it might be
possible to build GUI-level test suites that are less perturbed by rapid
change in the UI. On the other hand, for adequate testing in a fast paced
environment, the automated GUI tests will address only a portion of the
tests that are needed. Not all tests should become frequently reused
regression tests. A wide range of other tests, including some of the
complex scenarios that would derive from user stories, and many exploratory
tests that derive from the tester's insights about the customer, are not
going to be worth automating.

-- Cem Kaner


At 11:59 PM 11/7/2001 -0600, Erik Meade wrote:
>Cem,
>
>You do realize that Lisa asked this question in the context of XP?
>
>Acceptance tests in XP really are there to define what is acceptable
>to the customer.  As well as to let the developers know when that
>story is done in the iteration.  Additionally they serve to prove that
>the functionality remains done.
>
>--
>Erik Meade             emeade@...
>Senior Consultant      Object Mentor, Inc.
>http://www.junit.org
>
>
> > -----Original Message-----
> > From: Cem Kaner [mailto:kaner@...]
> > Sent: Wednesday, November 07, 2001 11:05 PM
> > To: agile-testing@yahoogroups.com
> > Subject: Re: [agile-testing] Re: testing non-web GUI
> >
> >
> > The first problem lies in calling these "acceptance tests."
> >
> > The task of the testing group is to discover the ways in which
> > the product
> > will be unacceptable to the customer. This is a fundamentally different
> > viewpoint from the programming team and leads testers to a different
> > analysis. It's the difference that adds value.
> >
> > There are a lot of different tests of the integrated product.
> > Some of them
> > are worth repeating many times, but many, many others are not.
> > For example,
> > if you run an end-to-end task to see how easy it is to realize a certain
> > benefit, and you find out that it is easy enough, you don't have
> > to run the
> > same test again because those aspects of the design that are under test
> > have shown that they work. There's a cost to automating a test.
> > If the test
> > won't be reused enough times, the cost isn't worth the benefit.
> >
> > There's a maintenance cost with GUI-level automation. To the extent that
> > changes to the design of the product will trigger a need for test case
> > maintenance, the product becomes inflexible (the incremental cost of late
> > change is higher than the cost of early change). If you design a good
> > interpreter (whether you embed it in a table or implement it some other
> > way) that separates your test cases from the details of the application
> > under test, the incremental cost of change is lower. However, it takes
> > up-front design to develop that interpreter.
> >
> > Testability is a broad concept. It's a class of features that help the
> > tester test, primarily features that increase the tester's
> > control over the
> > app or the visibility of the app's behavior. Testability features might
> > support automated regression testing, but they might be more
> > important for
> > high volume tests that are designed to expose delayed-reaction
> > bugs (stack
> > corruption, memory leaks, wild pointers, inappropriate sharing of data
> > areas or variables, etc.). They might be more important for supporting
> > configuration testing, or for helping the tester troubleshoot a problem
> > that is too hard to reproduce.
> >
> > -- cem
> >
> > At 04:56 PM 11/7/2001 +0000, Lisa Crispin wrote:
> >
> > > >
> > > > Add a task "Automate acceptance test" to every story.
> > > >
> > >Yep, we do that.  Now, in real life we are automating acceptance tests
> > >for web apps, which are somewhat easier even if the app hasn't been
> > >designed to be very testable.  But my hypothetical question was
> > >automating non-web GUI tests, which I think is harder, and am thinking
> > >there is no other good approach other than a testable app.  Then it's
> > >kind of a catch 22.  In my experience, it has been hard to get
> > >developers to see the value of acceptance tests.  They have the idea
> > >that it's something you do for this release and never again.  The
> > >concept of regression acceptance tests for some reason is hard to get
> > >across to some developers.  So, if they don't think regression testing
> > >is important, or automation is important, they aren't going to think
> > >about testability enough.
> > >
> > >
> > >
> > >To unsubscribe from this group, send an email to:
> > >agile-testing-unsubscribe@yahoogroups.com
> > >
> > >
> > >
> > >Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
> >
> > ========================================
> > "An eye for an eye makes the whole world blind"
> >   --- Mahatma Gandhi.
> >
> > Cem Kaner
> > Professor of Computer Sciences
> > Florida Institute of Technology
> > www.kaner.com  www.badsoftware.com
> >
> >
> >
> > To unsubscribe from this group, send an email to:
> > agile-testing-unsubscribe@yahoogroups.com
> >
> >
> >
> > Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
> >
>
>
>To unsubscribe from this group, send an email to:
>agile-testing-unsubscribe@yahoogroups.com
>
>
>
>Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

========================================
"An eye for an eye makes the whole world blind"
   --- Mahatma Gandhi.

Cem Kaner
Professor of Computer Sciences
Florida Institute of Technology
www.kaner.com   www.badsoftware.com

Hi,

Developers who haven't cycled through releases and haven't seen the kind of
problems that regression brings to operational systems are harder to
convince about the value of non-regression testing.

In our organisation there is an older generation of developers that is very
serious about regression testing.  They will think about regression testing
from the first test onwards.  These developers think about software test
automation from day 1.  They are not so concerned about the GUI though but
more about the robustness and reliability of the system as a whole (nearly
all our systems are built as client/server systems, some thin clients, some
fat).

Developers that we have hired more recently and who have not seen the
consequences or regressions are harder to convince.  Some are very
disciplined but not enough.

It's even nice to see how this 'older' generation of developers gets worried
when some of the test framework they are supposed to use for regression
testing is not ready on time.

What I don't understand about these stories Linda referes to is that you
have to add generic things like 'automate acceptance test' to it.

Why can't there be a general rule that each use case or story is acceptance
tested.

I also don't believe that all acceptance testing can be automated and I
think it's even not a good idea.  Every use case/story has usually some
aspects that can be automated very well, but the more complex the business,
the harder the automation, because there is too much intellect in the head
of the user that cannot be captured in any test.
Software of any kind still comes short when compared to any human mind.

Regression testing also has its limitations.  There are plenty of reports
that illustrate that regression testing is good but that it also creates
false confidence.  The value of regression testing is as great as the
variation in the tests.  No variation makes regression tests of less and
less value.  For more info see the Pesticide Paradox written by Boris
Beizer.

The main reason is that dynamic tests only cover certain categories of
defects and only part of the code.  A fixed test set wil therefor have soon
killed all the defects that it can detect and will not detect any new
defects.

There is also some false confidence that regression tests can proof that
what worked before will still work later on.

This is a separate discussion that perhaps is worth doing, but I don't know
if anyone on this list is interested.

Regards, Stefan.
> -----Original Message-----
> From: Lisa Crispin [mailto:lisa.crispin@...]
> Sent: Wednesday, November 07, 2001 5:56 PM
> To: agile-testing@yahoogroups.com
> Subject: [agile-testing] Re: testing non-web GUI
>
>
>
> >
> > Add a task "Automate acceptance test" to every story.
> >
> Yep, we do that.  Now, in real life we are automating
> acceptance tests
> for web apps, which are somewhat easier even if the app hasn't been
> designed to be very testable.  But my hypothetical question was
> automating non-web GUI tests, which I think is harder, and am
> thinking
> there is no other good approach other than a testable app.  Then it's
> kind of a catch 22.  In my experience, it has been hard to get
> developers to see the value of acceptance tests.  They have the idea
> that it's something you do for this release and never again.  The
> concept of regression acceptance tests for some reason is hard to get
> across to some developers.  So, if they don't think
> regression testing
> is important, or automation is important, they aren't going to think
> about testability enough.

Cem,

You do realize that Lisa asked this question in the context of XP?

Acceptance tests in XP really are there to define what is acceptable
to the customer.  As well as to let the developers know when that
story is done in the iteration.  Additionally they serve to prove that
the functionality remains done.

--
Erik Meade             emeade@...
Senior Consultant      Object Mentor, Inc.
http://www.junit.org


> -----Original Message-----
> From: Cem Kaner [mailto:kaner@...]
> Sent: Wednesday, November 07, 2001 11:05 PM
> To: agile-testing@yahoogroups.com
> Subject: Re: [agile-testing] Re: testing non-web GUI
>
>
> The first problem lies in calling these "acceptance tests."
>
> The task of the testing group is to discover the ways in which
> the product
> will be unacceptable to the customer. This is a fundamentally different
> viewpoint from the programming team and leads testers to a different
> analysis. It's the difference that adds value.
>
> There are a lot of different tests of the integrated product.
> Some of them
> are worth repeating many times, but many, many others are not.
> For example,
> if you run an end-to-end task to see how easy it is to realize a certain
> benefit, and you find out that it is easy enough, you don't have
> to run the
> same test again because those aspects of the design that are under test
> have shown that they work. There's a cost to automating a test.
> If the test
> won't be reused enough times, the cost isn't worth the benefit.
>
> There's a maintenance cost with GUI-level automation. To the extent that
> changes to the design of the product will trigger a need for test case
> maintenance, the product becomes inflexible (the incremental cost of late
> change is higher than the cost of early change). If you design a good
> interpreter (whether you embed it in a table or implement it some other
> way) that separates your test cases from the details of the application
> under test, the incremental cost of change is lower. However, it takes
> up-front design to develop that interpreter.
>
> Testability is a broad concept. It's a class of features that help the
> tester test, primarily features that increase the tester's
> control over the
> app or the visibility of the app's behavior. Testability features might
> support automated regression testing, but they might be more
> important for
> high volume tests that are designed to expose delayed-reaction
> bugs (stack
> corruption, memory leaks, wild pointers, inappropriate sharing of data
> areas or variables, etc.). They might be more important for supporting
> configuration testing, or for helping the tester troubleshoot a problem
> that is too hard to reproduce.
>
> -- cem
>
> At 04:56 PM 11/7/2001 +0000, Lisa Crispin wrote:
>
> > >
> > > Add a task "Automate acceptance test" to every story.
> > >
> >Yep, we do that.  Now, in real life we are automating acceptance tests
> >for web apps, which are somewhat easier even if the app hasn't been
> >designed to be very testable.  But my hypothetical question was
> >automating non-web GUI tests, which I think is harder, and am thinking
> >there is no other good approach other than a testable app.  Then it's
> >kind of a catch 22.  In my experience, it has been hard to get
> >developers to see the value of acceptance tests.  They have the idea
> >that it's something you do for this release and never again.  The
> >concept of regression acceptance tests for some reason is hard to get
> >across to some developers.  So, if they don't think regression testing
> >is important, or automation is important, they aren't going to think
> >about testability enough.
> >
> >
> >
> >To unsubscribe from this group, send an email to:
> >agile-testing-unsubscribe@yahoogroups.com
> >
> >
> >
> >Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
>
> ========================================
> "An eye for an eye makes the whole world blind"
>   --- Mahatma Gandhi.
>
> Cem Kaner
> Professor of Computer Sciences
> Florida Institute of Technology
> www.kaner.com   www.badsoftware.com
>
>
>
> To unsubscribe from this group, send an email to:
> agile-testing-unsubscribe@yahoogroups.com
>
>
>
> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
>

So they haven't had an occasion where some story which has already
been completed suddenly is no longer complete?  If the project goes
on long enough I bet it will happen.  Unfortunately at that time it
will probably be a pretty big feat to go back and do all the
acceptance tests.

I've seen the same difficulty in getting developers (and in many
case even the customers) to understand the importance of acceptance
tests, at least the first time around...

--
Erik Meade             emeade@...
Senior Consultant      Object Mentor, Inc.
http://www.junit.org


> -----Original Message-----
> From: Lisa Crispin [mailto:lisa.crispin@...]
> Sent: Wednesday, November 07, 2001 10:56 AM
> To: agile-testing@yahoogroups.com
> Subject: [agile-testing] Re: testing non-web GUI
>
>
>
> >
> > Add a task "Automate acceptance test" to every story.
> >
> Yep, we do that.  Now, in real life we are automating acceptance tests
> for web apps, which are somewhat easier even if the app hasn't been
> designed to be very testable.  But my hypothetical question was
> automating non-web GUI tests, which I think is harder, and am thinking
> there is no other good approach other than a testable app.  Then it's
> kind of a catch 22.  In my experience, it has been hard to get
> developers to see the value of acceptance tests.  They have the idea
> that it's something you do for this release and never again.  The
> concept of regression acceptance tests for some reason is hard to get
> across to some developers.  So, if they don't think regression testing
> is important, or automation is important, they aren't going to think
> about testability enough.

The first problem lies in calling these "acceptance tests."

The task of the testing group is to discover the ways in which the product
will be unacceptable to the customer. This is a fundamentally different
viewpoint from the programming team and leads testers to a different
analysis. It's the difference that adds value.

There are a lot of different tests of the integrated product. Some of them
are worth repeating many times, but many, many others are not. For example,
if you run an end-to-end task to see how easy it is to realize a certain
benefit, and you find out that it is easy enough, you don't have to run the
same test again because those aspects of the design that are under test
have shown that they work. There's a cost to automating a test. If the test
won't be reused enough times, the cost isn't worth the benefit.

There's a maintenance cost with GUI-level automation. To the extent that
changes to the design of the product will trigger a need for test case
maintenance, the product becomes inflexible (the incremental cost of late
change is higher than the cost of early change). If you design a good
interpreter (whether you embed it in a table or implement it some other
way) that separates your test cases from the details of the application
under test, the incremental cost of change is lower. However, it takes
up-front design to develop that interpreter.

Testability is a broad concept. It's a class of features that help the
tester test, primarily features that increase the tester's control over the
app or the visibility of the app's behavior. Testability features might
support automated regression testing, but they might be more important for
high volume tests that are designed to expose delayed-reaction bugs (stack
corruption, memory leaks, wild pointers, inappropriate sharing of data
areas or variables, etc.). They might be more important for supporting
configuration testing, or for helping the tester troubleshoot a problem
that is too hard to reproduce.

-- cem

At 04:56 PM 11/7/2001 +0000, Lisa Crispin wrote:

> >
> > Add a task "Automate acceptance test" to every story.
> >
>Yep, we do that.  Now, in real life we are automating acceptance tests
>for web apps, which are somewhat easier even if the app hasn't been
>designed to be very testable.  But my hypothetical question was
>automating non-web GUI tests, which I think is harder, and am thinking
>there is no other good approach other than a testable app.  Then it's
>kind of a catch 22.  In my experience, it has been hard to get
>developers to see the value of acceptance tests.  They have the idea
>that it's something you do for this release and never again.  The
>concept of regression acceptance tests for some reason is hard to get
>across to some developers.  So, if they don't think regression testing
>is important, or automation is important, they aren't going to think
>about testability enough.
>
>
>
>To unsubscribe from this group, send an email to:
>agile-testing-unsubscribe@yahoogroups.com
>
>
>
>Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/

========================================
"An eye for an eye makes the whole world blind"
   --- Mahatma Gandhi.

Cem Kaner
Professor of Computer Sciences
Florida Institute of Technology
www.kaner.com   www.badsoftware.com

Also look at HttpUnit/ServletUnit (they're bundled together). You can find
HttpUnit at Sourceforge.

Steve Hayes
steve@...
http://www.khatovartech.com
(03) 9489 0860 (home)
(0403) 902 431 (mobile)

On Wednesday, 7 November 2001 9:18 PM, ejgroene@...
[SMTP:ejgroene@...] wrote:
> Hi,
>
> We have a Servlet that uses Servlet Context Parameters.  The Servlet
> has a method getMyParameters() that returns the parameters from
> getServletContext().getInitParameter( "..." ).
>
> How do you test this method?
>
> The Servlet Context is only present within a running Servlet engine,
> and you cannot test individual pieces like getting the parameters.
> The only thing you can do is sending HTML to it.
>
> I devised the following method.  We create a special HTML page with a
> specific title like "test get parameter" and send this to the Servlet
> using HTTPunit.  The servlet has special testing code that checks for
> the test HTML page.  When it encouters the test HTML page, it calls
> getMyParameters() and returns the result as a HTML page.  The result
> page is then inspected in HTTPunit.
>
> I don't think it is a beautiful approach.  Any comments on this
> method?
>
> --Erik
>
>
>
> To unsubscribe from this group, send an email to:
> agile-testing-unsubscribe@yahoogroups.com
>
>
>
> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
>
>

At 10:56 AM 11/7/01, you wrote:

>[...]  In my experience, it has been hard to get
>developers to see the value of acceptance tests.  They have the idea
>that it's something you do for this release and never again.  The
>concept of regression acceptance tests for some reason is hard to get
>across to some developers.  So, if they don't think regression testing
>is important, or automation is important, they aren't going to think
>about testability enough.

I wonder if "acceptance test" isn't too overloaded a term.

I'm inclined to present the issue this way. Suppose I'm a programmer. At
any given moment, I'll be working on a subset of the code. As I touch code,
I will want to run the existing unit tests that target that code often,
every few minutes.

There are a bunch of unit tests for the parts of the system I don't touch.
If they took no time to run, I would run them every few minutes too,
because I know that sometimes a change I make *here* will break something
way over *there*, refactoring notwithstanding. I want to know instantly
when I've done that. Those unit tests for the code over there might tell me.

If those unit tests do take substantial time to run, I won't run them as
often. It's more important to me to be able to move quickly than to know
whether a particular move has broken something. Running ten minutes of
tests every five minutes is not acceptable.

I might run those tests every hour or two, while taking a break, certainly
before I integrate. Finding about a problem hours after I created it is not
ideal, but it doesn't happen so often that I'm willing to slow down and run
more tests.

Now, the full set of unit tests is not sufficient to catch all bugs. That's
because they're focused on clarifying API and exercising subsets of the
system. For example, I believe unit tests will catch some interaction
("change here breaks something over there") bugs, unless mock objects /
stubs are overused. But they won't catch enough of them, so end-to-end
tests are good insurance. Also, unit tests won't catch performance
degradations. And they won't catch you-know-this-GUI-is-just-plain-stupid bugs.

So there might be an additional set of automated tests targeted at these
slippery bugs. The question for the developers is how long should it be
between the time they introduce such a bug and the time they find out about
it? Someone who is properly "test infected" will want that time to be
small. The next question is how worried are they about such bugs? Have they
had bad experiences already?

Now you've established the value of the tests to the programmers. It's time
to talk about what tests could be automated and run overnight (or whatever)
if someone added such-and-so testability hooks to the code. Are the hooks
worth it?

Sorry to be long-winded. What I'm trying to do is tap into the developers'
(presumed) affection for unit testing, and show that your testing is
nothing but an extension of that.

Note that not all tests will be automated. The this-GUI-is-stupid bugs
can't be. Such tests are most properly done by the customer - are some of
the real tests-used-to-decide-whether-to-accept-the-release. Independent
testers, especially ones with domain knowledge, can often also do a
reasonable job. Not that many developers can (alas, fewer than think they
can).

And some tests that can be automated won't be. It's not worth the cost; the
time to do it would be better spent on other things. That's the gist of the
"When Should Tests Be Automated?" paper Erik mentioned.

--
Brian Marick, marick@...
www.testing.com - Software testing services and resources
www.testingcraft.com - Where software testers exchange techniques
www.visibleworkings.com - Adequate understanding of system internals

Well said.

I'm curious, though, is the project you are talking about (50 man
years) using XP or some other agile processes?


--- In agile-testing@y..., STEURS Stefan <stefan.steurs@e...> wrote:
> I am using WinRunner to test in an iterative/incremental software
> development environment.
>
> Using a keyword driven/data driven approach, I think it is possible
to
> implement quite some tests in a reasonable time.
>
> I must add we have quite a structured approach to development and
testing.
> The software we are writing is high-availability, distributed
processing in
> nature.  The biggest project for which WinRunner is employed is in
the range
> of 50 manyears from the the first iteration to the first full scale
> deployment (the project replaces gradually a Mainframe solution by a
Unix
> Server based solution to put it simply, but it's a lot more).
>
> There are 5 levels of testing and test automation that we practice
>
> Unit testing - unfortunately not at the level of discipline of XP
and done
> at object/method level - based on SWIG, CppUnit, PerlUnit
> Integration testing Corba component/service IDL level - based on
COPE,
> PerlUnit
> System testing - JAVA GUI client level - based on WinRunner
> Acceptance testing - Procedure/Organisation/End user level -
currently
> manual
> Systems Integration testing - distributed processing/system
interfaces with
> internal/external clients and servers - WinRunner but also a lot
manual
>
> Who are the people who are writing the WinRunner/SilkTest tests?
Are they
> programmers?  Do they test the right things at the right level?
Perhaps the
> use of the tools depends more on the user than on the tool itself.
>
> Automating a bad test is always a waste of time.
> Poorly automating a test is also a waste of time.
>
> I think being agile should mean:
> 1. having a way to design those tests that should be done and reject
those
> that do not add value
> 2. having a way to automate tests that is efficient and maintainable
> (datadriven, keyworddriven and framework based)
>
> I would conclude that to perform agile testing:
> 1. testers should have good developer skills
> 2. test automation projects are treated like any other software
project
>
> Regards, Stefan Steurs.
>
> > -----Original Message-----
> > From: Lisa Crispin [mailto:lisa.crispin@a...]
> > Sent: Tuesday, November 06, 2001 4:15 PM
> > To: agile-testing@y...
> > Subject: [agile-testing] Re: testing non-web GUI
> >
> >
> > I totally agree with your suggestions, although this doesn't seem
> > specific to non-web GUI (Webload and Silktest are for Web apps).
Do
> > you think WinRunner is lightweight enough for use in an XP
project?
> > I've had some people using SilkTest and WinRunner tell me that it
> > takes them so long to automate they can't keep up.
> >
> > --- In agile-testing@y..., bret@p... wrote:
> > > 1. Make sure that your tests are stored in the same
configuration
> > > management environment as the development code. This makes
> > it easier
> > > for the developers to access.
> > > 2. Use a tool that stores test and test information in files
rather
> > > than databases. This will make it easier for the developers to
> > > access.
> > > This means NO to Robot, QA Run. Possibilities include SilkTest,
> > > WinRunner.
> > > 3. Try to use a test tool that uses a language that the
developers
> > > already know. This means avoiding proprietary testing
> > languages like
> > > 4test (SilkTest) and TSL (WinRunner). Radview's WebLoad FT uses
> > > JavaScript, which might fill the bill. (Notice how you are
running
> > > out
> > > of suitable tools?)
> > > 4. Make sure you have a copy of the tool available for all the
> > > developers. This way they can run your acceptance tests and
> > determine
> > > whether they have broken anything. And then they have to fix it!
> > > Otherwise, you can just give up. Maintainability will be a
problem,
> > > but once they feel the pain, they will help you out, either by
> > > managing necessary changes better or by helping to make the
> > automated
> > > tests easier to maintain. XP won't work without this!
> > >
> > > Let me know if you have more questions. I'm sorry i didn't get a
> > > chance to talk to you last week at STAR.
> > >
> > > Bret
> >
> >
> > To unsubscribe from this group, send an email to:
> > agile-testing-unsubscribe@y...
> >
> >
> >
> > Your use of Yahoo! Groups is subject to
> > http://docs.yahoo.com/info/terms/
> >
> >

>
> Add a task "Automate acceptance test" to every story.
>
Yep, we do that.  Now, in real life we are automating acceptance tests
for web apps, which are somewhat easier even if the app hasn't been
designed to be very testable.  But my hypothetical question was
automating non-web GUI tests, which I think is harder, and am thinking
there is no other good approach other than a testable app.  Then it's
kind of a catch 22.  In my experience, it has been hard to get
developers to see the value of acceptance tests.  They have the idea
that it's something you do for this release and never again.  The
concept of regression acceptance tests for some reason is hard to get
across to some developers.  So, if they don't think regression testing
is important, or automation is important, they aren't going to think
about testability enough.

I have also used support and documentation be 'dummy users'. It has
been very effective in catching the types of issues that are often
overlooked by people who have been testing the application for a
while. You know the issues are there, but have seen them so
often, they become just another 'feature'. No amount of automated
testing can replace this.

There are two times when I would not use automated testing for GUI -
these are:

1. If the GUI is not stable and undergoes major changes with each
release, it is hard to automate and manual user testing is about the
only way I have found around the problem. Of course, I haven't run
into this problem with XP (yet...)

2. If the GUI is a very small part of the application and is just
used for admin or as a demo, it is not worth while spending the
time or money on automation. You can catch any problems during each
iteration that touches it.

Janet Gregory



--- In agile-testing@y..., "Erik Meade" <emeade@g...> wrote:
> First thing I would have unit tests that show when the create,
> read, update, and delete methods used by the GUI do the correct
thing
> to the model.
>
> Once that is done I only know of a couple of options.  The options
are
> not exclusive.  Use tools like WinRunner, homegrown, etc, to
automate
> the tests that these tools can do well, brute force kinda stuff.
Use
> testing people to do exploratory testing (as Brian Marick says).
People
> are much more intuitive then tools.
>
> Brian Marick has a paper I've found an interesting read about
> When Should a Test be Automated?
http://www.testing.com/writings.html
>
> Another thing I have seen is an example of eat your own dogfood.
Have
> your people who use/support/document the tool, use it.  These folks
can
> be a good example of things your user is gonna do, without the
expense of
> having users finding your bugs.  In house beta testing.
>
> --
> Erik Meade             emeade@o...
> Senior Consultant      Object Mentor, Inc.
> http://www.junit.org
>
>
> > -----Original Message-----
> > From: Lisa Crispin [mailto:lisa.crispin@a...]
> > Sent: Sunday, November 04, 2001 12:27 PM
> > To: agile-testing@y...
> > Subject: [agile-testing] Re: testing non-web GUI
> >
> >
> > I'm not actually trying to test a non-web GUI app, this is a
general
> > question because people ask me all the time what tools I
recommend
for
> > non-web GUI testing.  Let's say we have a Java Swing timesheet
> > application.  It allows you to create, read, update and delete
> > timesheets, and lets managers get reports of timesheets.  Are
there
> > any tools you'd recommend to test this.
> > -- Lisa
> >
> > --- In agile-testing@y..., emeade@g... wrote:
> > > --- In agile-testing@y..., "Lisa Crispin" <lisa.crispin@a...>
wrote:
> > > > Does anyone have experience automating testing of non-web GUI
apps
> > > in
> > > > an XP environment?  Everyone says test it just under the GUI,
> > which
> > > > is the best idea, but what if you can't get your team to think
> > > enough
> > > > about testability?  Anyway, you do need to test the GUI
sooner
or
> > > > later.  Any tools/approaches that work for you?
> > > > -- Lisa
> > >
> > > What kinds of things do you want to test?
> > >
> > > --
> > > Erik Meade             emeade@o...
> > > Senior Consultant      Object Mentor, Inc.
> > > http://www.junit.org
> >
> >
> > To unsubscribe from this group, send an email to:
> > agile-testing-unsubscribe@y...
> >
> >
> >
> > Your use of Yahoo! Groups is subject to
http://docs.yahoo.com/info/terms/
> >

> press_tab();
> press_tab();
> press_enter();

> goto_channel_menu();
> goto_secondary_channel()
> enter("5");

We played around with several options. The one that turned out the easiest
to maintain was (java)

// somewhere in an interface
public static final String DOWN = "d";
public static final String ENTER = "e";


// in the test (a test class which implements the interface above)
pressKeySequence(DOWN + DOWN + ENTER + DOWN + ENTER);

// the same code, when number navigation finally worked
pressKeySequence("32");



Maintainability had little to do with writing code once. Because all test
scripts

have sort-of the same navigation, they all have the navigation sequence
somewhere

in the beginning of the script. Changing the scripts meant replacing

"pressKeySequence("32");" with "pressKeySequence("42");" when a submenu was
added

to the main menu. Great job for a friday afernoon, which was lost anyhow.



>> These tests do not cover placement of components on the screen,

>> colors, fonts, an so on.
> So how do you test these things?

You got me... we didn't. They are different on a TV than on the PC simulation
anyhow. We thought (and still think) that manual verification of this part is
faster than the effort of automating it.

>> They also rely on a correctly working navigation mechanism (If many tests
>> suddenly fail, start looking there). They do just test if the user interface
>> actions (I press those and those keys) lead to the desired results (The box
>> changes this and that).

> I don't understand the navigation mechanism, can you provide more
> details on how this works?


In Java, you can use Toolkit.getSystemEventQueue(), and post KeyEvents this
queue.
The AWT implementation treats this as if the user presses keys (first a press
event, then release, then typed).

By specifying the correct (VK_) key codes, you can let your test act as if the
user pressed those keys.

Regards,
Walter van Iterson

I've put links to papers and articles relevant to agile testing here:
<http://testing.com/agile/>
Let me know if there's anything else I should include. Thanks.

--
Brian Marick, marick@...
www.testing.com - Software testing services and resources
www.testingcraft.com - Where software testers exchange techniques
www.visibleworkings.com - Adequate understanding of system internals

ejgroene@... said:
> We have a Servlet that uses Servlet Context Parameters.  The Servlet
> has a method getMyParameters() that returns the parameters from
> getServletContext().getInitParameter( "..." ).
>
> How do you test this method?

our approach is to mock up the servlet request and response and exercise the
servlet straight in junit. See www.mockobjects.com for more info.

Steve

Erik,

You can mock it or you may want to take a look at JUnitEE or
Cactus.  You can find urls from http://junit.org/extensions.htm

--
Erik Meade             emeade@...
Senior Consultant      Object Mentor, Inc.
http://www.junit.org


> -----Original Message-----
> From: ejgroene@... [mailto:ejgroene@...]
> Sent: Wednesday, November 07, 2001 4:18 AM
> To: agile-testing@yahoogroups.com
> Subject: [agile-testing] Testing Servlet Context Parameters
>
>
> Hi,
>
> We have a Servlet that uses Servlet Context Parameters.  The Servlet
> has a method getMyParameters() that returns the parameters from
> getServletContext().getInitParameter( "..." ).
>
> How do you test this method?
>
> The Servlet Context is only present within a running Servlet engine,
> and you cannot test individual pieces like getting the parameters.
> The only thing you can do is sending HTML to it.
>
> I devised the following method.  We create a special HTML page with a
> specific title like "test get parameter" and send this to the Servlet
> using HTTPunit.  The servlet has special testing code that checks for
> the test HTML page.  When it encouters the test HTML page, it calls
> getMyParameters() and returns the result as a HTML page.  The result
> page is then inspected in HTTPunit.
>
> I don't think it is a beautiful approach.  Any comments on this
> method?
>
> --Erik
>
>
>
> To unsubscribe from this group, send an email to:
> agile-testing-unsubscribe@yahoogroups.com
>
>
>
> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
>

Hi,

We have a Servlet that uses Servlet Context Parameters.  The Servlet
has a method getMyParameters() that returns the parameters from
getServletContext().getInitParameter( "..." ).

How do you test this method?

The Servlet Context is only present within a running Servlet engine,
and you cannot test individual pieces like getting the parameters.
The only thing you can do is sending HTML to it.

I devised the following method.  We create a special HTML page with a
specific title like "test get parameter" and send this to the Servlet
using HTTPunit.  The servlet has special testing code that checks for
the test HTML page.  When it encouters the test HTML page, it calls
getMyParameters() and returns the result as a HTML page.  The result
page is then inspected in HTTPunit.

I don't think it is a beautiful approach.  Any comments on this
method?

--Erik

I am using WinRunner to test in an iterative/incremental software
development environment.

Using a keyword driven/data driven approach, I think it is possible to
implement quite some tests in a reasonable time.

I must add we have quite a structured approach to development and testing.
The software we are writing is high-availability, distributed processing in
nature.  The biggest project for which WinRunner is employed is in the range
of 50 manyears from the the first iteration to the first full scale
deployment (the project replaces gradually a Mainframe solution by a Unix
Server based solution to put it simply, but it's a lot more).

There are 5 levels of testing and test automation that we practice

Unit testing - unfortunately not at the level of discipline of XP and done
at object/method level - based on SWIG, CppUnit, PerlUnit
Integration testing Corba component/service IDL level - based on COPE,
PerlUnit
System testing - JAVA GUI client level - based on WinRunner
Acceptance testing - Procedure/Organisation/End user level - currently
manual
Systems Integration testing - distributed processing/system interfaces with
internal/external clients and servers - WinRunner but also a lot manual

Who are the people who are writing the WinRunner/SilkTest tests?  Are they
programmers?  Do they test the right things at the right level?  Perhaps the
use of the tools depends more on the user than on the tool itself.

Automating a bad test is always a waste of time.
Poorly automating a test is also a waste of time.

I think being agile should mean:
1. having a way to design those tests that should be done and reject those
that do not add value
2. having a way to automate tests that is efficient and maintainable
(datadriven, keyworddriven and framework based)

I would conclude that to perform agile testing:
1. testers should have good developer skills
2. test automation projects are treated like any other software project

Regards, Stefan Steurs.

> -----Original Message-----
> From: Lisa Crispin [mailto:lisa.crispin@...]
> Sent: Tuesday, November 06, 2001 4:15 PM
> To: agile-testing@yahoogroups.com
> Subject: [agile-testing] Re: testing non-web GUI
>
>
> I totally agree with your suggestions, although this doesn't seem
> specific to non-web GUI (Webload and Silktest are for Web apps).  Do
> you think WinRunner is lightweight enough for use in an XP project?
> I've had some people using SilkTest and WinRunner tell me that it
> takes them so long to automate they can't keep up.
>
> --- In agile-testing@y..., bret@p... wrote:
> > 1. Make sure that your tests are stored in the same configuration
> > management environment as the development code. This makes
> it easier
> > for the developers to access.
> > 2. Use a tool that stores test and test information in files rather
> > than databases. This will make it easier for the developers to
> > access.
> > This means NO to Robot, QA Run. Possibilities include SilkTest,
> > WinRunner.
> > 3. Try to use a test tool that uses a language that the developers
> > already know. This means avoiding proprietary testing
> languages like
> > 4test (SilkTest) and TSL (WinRunner). Radview's WebLoad FT uses
> > JavaScript, which might fill the bill. (Notice how you are running
> > out
> > of suitable tools?)
> > 4. Make sure you have a copy of the tool available for all the
> > developers. This way they can run your acceptance tests and
> determine
> > whether they have broken anything. And then they have to fix it!
> > Otherwise, you can just give up. Maintainability will be a problem,
> > but once they feel the pain, they will help you out, either by
> > managing necessary changes better or by helping to make the
> automated
> > tests easier to maintain. XP won't work without this!
> >
> > Let me know if you have more questions. I'm sorry i didn't get a
> > chance to talk to you last week at STAR.
> >
> > Bret
>
>
> To unsubscribe from this group, send an email to:
> agile-testing-unsubscribe@yahoogroups.com
>
>
>
> Your use of Yahoo! Groups is subject to
> http://docs.yahoo.com/info/terms/
>
>

First thing I would have unit tests that show when the create,
read, update, and delete methods used by the GUI do the correct thing
to the model.

Once that is done I only know of a couple of options.  The options are
not exclusive.  Use tools like WinRunner, homegrown, etc, to automate
the tests that these tools can do well, brute force kinda stuff.  Use
testing people to do exploratory testing (as Brian Marick says).  People
are much more intuitive then tools.

Brian Marick has a paper I've found an interesting read about
When Should a Test be Automated? http://www.testing.com/writings.html

Another thing I have seen is an example of eat your own dogfood.  Have
your people who use/support/document the tool, use it.  These folks can
be a good example of things your user is gonna do, without the expense of
having users finding your bugs.  In house beta testing.

--
Erik Meade             emeade@...
Senior Consultant      Object Mentor, Inc.
http://www.junit.org


> -----Original Message-----
> From: Lisa Crispin [mailto:lisa.crispin@...]
> Sent: Sunday, November 04, 2001 12:27 PM
> To: agile-testing@yahoogroups.com
> Subject: [agile-testing] Re: testing non-web GUI
>
>
> I'm not actually trying to test a non-web GUI app, this is a general
> question because people ask me all the time what tools I recommend for
> non-web GUI testing.  Let's say we have a Java Swing timesheet
> application.  It allows you to create, read, update and delete
> timesheets, and lets managers get reports of timesheets.  Are there
> any tools you'd recommend to test this.
> -- Lisa
>
> --- In agile-testing@y..., emeade@g... wrote:
> > --- In agile-testing@y..., "Lisa Crispin" <lisa.crispin@a...> wrote:
> > > Does anyone have experience automating testing of non-web GUI apps
> > in
> > > an XP environment?  Everyone says test it just under the GUI,
> which
> > > is the best idea, but what if you can't get your team to think
> > enough
> > > about testability?  Anyway, you do need to test the GUI sooner or
> > > later.  Any tools/approaches that work for you?
> > > -- Lisa
> >
> > What kinds of things do you want to test?
> >
> > --
> > Erik Meade             emeade@o...
> > Senior Consultant      Object Mentor, Inc.
> > http://www.junit.org
>
>
> To unsubscribe from this group, send an email to:
> agile-testing-unsubscribe@yahoogroups.com
>
>
>
> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
>

> -----Original Message-----
> From: Lisa Crispin [mailto:lisa.crispin@...]
> Sent: Sunday, November 04, 2001 12:29 PM
> To: agile-testing@yahoogroups.com
> Subject: [agile-testing] Re: testing non-web GUI
>
>
> I agree that testability issues are more important, but I'm a tester,
> not the boss, and I can only suggest that the team think more about
> testability.  As long as the developers can do their unit tests, they
> aren't real motivated to make it easier to do acceptance tests, even
> when they are responsible for running and automating the acceptance
> tests.  At least, that has been my experience.  Suggestions?

Add a task "Automate acceptance test" to every story.

--
Erik Meade             emeade@...
Senior Consultant      Object Mentor, Inc.
http://www.junit.org

>
> These tests do not cover placement of components on the screen,
colors, fonts, an so on.

So how do you test these things?

> They also rely on a correctly working navigation mechanism (If many
tests suddenly fail,
> start looking there). They do just test if the user interface
actions (I press those and those
> keys) lead to the desired results (The box changes this and that).
>

I don't understand the navigation mechanism, can you provide more
details on how this works?

thanks
Lisa

I totally agree with your suggestions, although this doesn't seem
specific to non-web GUI (Webload and Silktest are for Web apps).  Do
you think WinRunner is lightweight enough for use in an XP project?
I've had some people using SilkTest and WinRunner tell me that it
takes them so long to automate they can't keep up.

--- In agile-testing@y..., bret@p... wrote:
> 1. Make sure that your tests are stored in the same configuration
> management environment as the development code. This makes it easier
> for the developers to access.
> 2. Use a tool that stores test and test information in files rather
> than databases. This will make it easier for the developers to
> access.
> This means NO to Robot, QA Run. Possibilities include SilkTest,
> WinRunner.
> 3. Try to use a test tool that uses a language that the developers
> already know. This means avoiding proprietary testing languages like
> 4test (SilkTest) and TSL (WinRunner). Radview's WebLoad FT uses
> JavaScript, which might fill the bill. (Notice how you are running
> out
> of suitable tools?)
> 4. Make sure you have a copy of the tool available for all the
> developers. This way they can run your acceptance tests and
determine
> whether they have broken anything. And then they have to fix it!
> Otherwise, you can just give up. Maintainability will be a problem,
> but once they feel the pain, they will help you out, either by
> managing necessary changes better or by helping to make the
automated
> tests easier to maintain. XP won't work without this!
>
> Let me know if you have more questions. I'm sorry i didn't get a
> chance to talk to you last week at STAR.
>
> Bret

At 02:59 AM 11/6/01, you wrote:
>The issues we encountered were:
>[snip]
>- I didn't like people asking for new submenus, as this meant rewriting
>the navigation part of a lot of tests

Did you end up refactoring your tests? I think it's common to start with
tests like this:

press_tab();
press_tab();
press_enter();
press_tab();
press_enter();
enter("5")
...

and end up with tests like this:

goto_channel_menu();
goto_secondary_channel()
enter("5");

where goto_channel_menu() is defined as press_tab(); press_tab();
press_enter();

Then, if a new submenu is added that changes the tab order on the main
menu, you only have to change the main menu's goto methods, not all the tests.

The standard programming rules ("once and only once") apply to
programs-that-are-tests, though with qualifications. One is that it's often
useful for tests to clearly document the behavior of the thing they're
testing. Sometimes too much abstraction can make tests hard to understand.
(This may be more important when testing classes or small APIs.)

--
Brian Marick, marick@...
www.testing.com - Software testing services and resources
www.testingcraft.com - Where software testers exchange techniques
www.visibleworkings.com - Adequate understanding of system internals

In my last project, I used a set of UI test cases to promote JUnit within the
organisation.
No licencing issues, tests and development code in the same language, etc.
Just had to make clear what could be tested, and what couldn't.

Btw, the product we work on is a set top box, which can be fully operated by a
single remote
control. Capture and replay tools don't exist (unless we make them ourselves)
and there is
no mouse involved.

The main structure of our tests was:
- look up the original value in a database
- write key events to the event queue (tabs and spaces in a PC world) to go to a
certain ui component
- enter the new value (most of the time different from the original value)
- accept (or cancel) the current dialog, go back to the initial position
- look up the new value in a database
- check if the new value is the same as the entered value
   (or the same as the original, if the dialog was cancelled)
- if the test fails, restart the application, to make sure you're again in the
initial position

These tests do not cover placement of components on the screen, colors, fonts,
an so on.
They also rely on a correctly working navigation mechanism (If many tests
suddenly fail,
start looking there). They do just test if the user interface actions (I press
those and those
keys) lead to the desired results (The box changes this and that).

The issues we encountered were:
- How fast can we generate key presses without entering keys before a dialog
appears on the screen
   As an indication: In the PC simulation environment, 200 ms for navigation, and
1000 ms to accept a dialog
   were the fastest we could get. Even this fast, the tests still took 15 minutes
- I didn't like people asking for new submenus, as this meant rewriting the
navigation part of a lot of tests
- These tests didn't test one class at a time, but the entire UI at once. It
took quite some talking to
    convince people that it still isn't really hard to locate a problem.

Walter van Iterson

Points of agreement and disagreement highlighted below.

Stefan

> -----Original Message-----
> From: bret@... [mailto:bret@...]
> Sent: Tuesday, November 06, 2001 1:05 AM
> To: agile-testing@yahoogroups.com
> Subject: [agile-testing] Re: testing non-web GUI
>
>
> 1. Make sure that your tests are stored in the same configuration
> management environment as the development code. This makes it easier
> for the developers to access.

Agree.  But we have found it also useful to make sure that test framework
code is implemented in a seperate view or area so the new developments in
the test code do not interfere with the tests that need to be run by the
developers.  One would defend of course the point of view that test
framework code has to be backward compatible but we've experienced it is not
that easy to make the changes invisible.
This implies also that we have to make releases and baselines of our test
framework code, moreover, the test framework code is used for many different
projects so it needs its own configuration management space and it is also
necessary to identify the dependencies between the test framework code and
the software under test.

> 2. Use a tool that stores test and test information in files rather
> than databases. This will make it easier for the developers to
> access.
> This means NO to Robot, QA Run. Possibilities include SilkTest,
> WinRunner.

Agree.  Tests should be plain text files.  It is of course a pain that
WinRunner uses the name 'script' without extension for each test that you
write so you cannot use meaninful file names (only directory names are
somewhat meaningful).  Think about it in relation to configuration
management.  Having all files named script, header, and etc. doesn't make
Configuration Management easier.

> 3. Try to use a test tool that uses a language that the developers
> already know. This means avoiding proprietary testing languages like
> 4test (SilkTest) and TSL (WinRunner). Radview's WebLoad FT uses
> JavaScript, which might fill the bill. (Notice how you are running
> out
> of suitable tools?)

Small disagreement.

It's perhaps not that bad.  TSL is very C like.  It has limitations but I
can live with them.  Anyway, most developers don't write tests at this
level, they don't code for application/GUI level tests.  They shouldn't be.
We have locally standardised on Perl as a test scripting language.  Half of
the developers at least did not know it so the Test Team (us) organised two
courses for developers to get trained in the use of Perl (3 days).  If the
developers don't know a common language for testing, educate them.

> 4. Make sure you have a copy of the tool available for all the
> developers. This way they can run your acceptance tests and determine
> whether they have broken anything. And then they have to fix it!
> Otherwise, you can just give up. Maintainability will be a problem,
> but once they feel the pain, they will help you out, either by
> managing necessary changes better or by helping to make the automated
> tests easier to maintain. XP won't work without this!

Question?  Does this have anything to do with XP?  I don't think so.

Agree as far as licenses being available to developers goes but.
1. usually you have floating licenses - you don't need as many licenses as
you have developers
2. most developers should not be testing GUI stuff since in many projects
more than half of the developers won't be programming any GUI stuff anyway
(in our case the ratio is closer to 75% or higher not doing any GUI stuff
ever).  These people will be testing using test frameworks developed inhouse
on the back of open source software or fully proprietary.  Tools like
WinRunner don't come into the picture, licenses usually don't come into the
picture either.
3. we have to be careful using the term acceptance tests.  Acceptance tests
are tests done by the user to accept software.  I assume that Bret means
functional system level tests through the GUI and other external interfaces.
Acceptance tests should not be technical in nature but operational.  They
should focus on procedures, organisation, etc (at least in our case).
XP assumes only two levels of testing, in our environment we have 4 levels.
Overkill?  We don't think so, but this is another issue.

If developers run acceptance tests they must know the business of the user
inside and out.  In contexts like ours this is out of the question.  I work
in an Air Traffic Control/Management software development environment.
Developers don't have years of experience in this very specialised business
area.  That's why developers can get it very wrong trying to run acceptance
tests.  Anyway, even if they could simulate the software behaviour, they
wouldn't be able to simulate the operational procedures which requires a lot
of interaction and synchronisation with different types of actors/end-users.

I think agile should also mean "Don't do the thinks that you cannot do right
if someone else can do it better."  Agile should also mean taking as many
steps as it needs to go from A to B and not a fixed number of steps.  I am
referring here to the XP two level approach of testing.  I assume we will
get back to this later.  Sometimes 1 step can be enough, sometimes you may
need 5.  Mostly we need 3 or 4, but we have done 5 and we have done just 1
as well.

Shouldn't agile also mean scalable?

>
> Let me know if you have more questions. I'm sorry i didn't get a
> chance to talk to you last week at STAR.
>
> Bret

Regards, Stefan Steurs.

1. Make sure that your tests are stored in the same configuration
management environment as the development code. This makes it easier
for the developers to access.
2. Use a tool that stores test and test information in files rather
than databases. This will make it easier for the developers to
access.
This means NO to Robot, QA Run. Possibilities include SilkTest,
WinRunner.
3. Try to use a test tool that uses a language that the developers
already know. This means avoiding proprietary testing languages like
4test (SilkTest) and TSL (WinRunner). Radview's WebLoad FT uses
JavaScript, which might fill the bill. (Notice how you are running
out
of suitable tools?)
4. Make sure you have a copy of the tool available for all the
developers. This way they can run your acceptance tests and determine
whether they have broken anything. And then they have to fix it!
Otherwise, you can just give up. Maintainability will be a problem,
but once they feel the pain, they will help you out, either by
managing necessary changes better or by helping to make the automated
tests easier to maintain. XP won't work without this!

Let me know if you have more questions. I'm sorry i didn't get a
chance to talk to you last week at STAR.

Bret

I agree that testability issues are more important, but I'm a tester,
not the boss, and I can only suggest that the team think more about
testability.  As long as the developers can do their unit tests, they
aren't real motivated to make it easier to do acceptance tests, even
when they are responsible for running and automating the acceptance
tests.  At least, that has been my experience.  Suggestions?


--- In agile-testing@y..., j.c.yip@c... wrote:

>
> Can't say that I've done it but everyone I talk to suggest using
> screen scraper type tools if you absolutely have to test at the GUI
> level.  On the other hand, if you "can't get your team to think
> enough about testability" perhaps there's a more important problem
to
> deal with.

I'm not actually trying to test a non-web GUI app, this is a general
question because people ask me all the time what tools I recommend for
non-web GUI testing.  Let's say we have a Java Swing timesheet
application.  It allows you to create, read, update and delete
timesheets, and lets managers get reports of timesheets.  Are there
any tools you'd recommend to test this.
-- Lisa

--- In agile-testing@y..., emeade@g... wrote:
> --- In agile-testing@y..., "Lisa Crispin" <lisa.crispin@a...> wrote:
> > Does anyone have experience automating testing of non-web GUI apps
> in
> > an XP environment?  Everyone says test it just under the GUI,
which
> > is the best idea, but what if you can't get your team to think
> enough
> > about testability?  Anyway, you do need to test the GUI sooner or
> > later.  Any tools/approaches that work for you?
> > -- Lisa
>
> What kinds of things do you want to test?
>
> --
> Erik Meade             emeade@o...
> Senior Consultant      Object Mentor, Inc.
> http://www.junit.org

--- In agile-testing@y..., "Lisa Crispin" <lisa.crispin@a...> wrote:
> Does anyone have experience automating testing of non-web GUI apps
in
> an XP environment?  Everyone says test it just under the GUI, which
> is the best idea, but what if you can't get your team to think
enough
> about testability?  Anyway, you do need to test the GUI sooner or
> later.  Any tools/approaches that work for you?
> -- Lisa

What kinds of things do you want to test?

--
Erik Meade             emeade@...
Senior Consultant      Object Mentor, Inc.
http://www.junit.org

--- In agile-testing@y..., "Lisa Crispin" <lisa.crispin@a...> wrote:
> Does anyone have experience automating testing of non-web GUI apps
in
> an XP environment?  Everyone says test it just under the GUI, which
> is the best idea, but what if you can't get your team to think
enough
> about testability?  Anyway, you do need to test the GUI sooner or
> later.  Any tools/approaches that work for you?

Can't say that I've done it but everyone I talk to suggest using
screen scraper type tools if you absolutely have to test at the GUI
level.  On the other hand, if you "can't get your team to think
enough about testability" perhaps there's a more important problem to
deal with.

Does anyone have experience automating testing of non-web GUI apps in
an XP environment?  Everyone says test it just under the GUI, which
is the best idea, but what if you can't get your team to think enough
about testability?  Anyway, you do need to test the GUI sooner or
later.  Any tools/approaches that work for you?
-- Lisa