On Tuesday, December 10, 2002, at 07:59 AM, Ron Jeffries wrote:
>
> The rule in XP is to test everything that could possibly break.
> Therefore if things could break in integration or in the system, they
> must be tested.

I'd like to suggest that this slogan is another way of saying "test
that every case you thought of has been implemented as you intended".
Put that way, it's kind of mundane. And, for novices to test-first
design, I suspect it shifts the attention away from the "design" to the
"test". I think the design is the cool part. The design consists of
thinking of those cases worth thinking of, and deciding what should be
intended in those cases.

Let me give an example.

My time-tracking program has the notion of 'a job' in its domain model.
One of my jobs is 'stqe', for my magazine editing. When I start
editing, I start the 'stqe' job. When I start doing something else, I
pause it. At the end of the day, I stop work, and a new record of how
much I did for STQE that day gets saved.

I'm just now finding a need to be able to forget jobs. There are some
mundane tests:
- can you really forget a job?
- what happens if you try to forget a job that doesn't exist?

What's more interesting are the less obvious tests:
- what happens to old records when you forget the job they record time
for?
    (answer: they're still there)
- can you create a job with the same name as the one you forgot?
    (answer: yes)
- You can summarize the amount of time spent on a job. What happens if
both
    the old version of the job and the new version of the job have
records?
    (tentative answer: you get a summary that includes both versions.
That's kind
     of dubious, but it's not likely to happen much and I'm sort of
unclear about what
     else you could do. However, let's modify the previous answer to say
that you
     can create a job with the same name as a now-deleted one, but you'll
be told
     you've just done that. If you don't like that, undo and pick a
different name, or
     delete the old records. Maybe there should be a way to rename old
records?
- What happens if you try to forget a job that's currently accumulating
time?
    (answer: I believe you could actually get away with that, sort of the
way you can
     delete an open file in Unix and the program that has it open doesn't
notice. But
     it's not something a normal person would do, so it's best to throw
an error.
     Probably less work that way, since the test for the case where you
could delete
     would be more complicated, and I think I'd have to test it.

-----
Brian Marick
Consulting, training, contracting, and research
Focused on the intersection of testing, programming, and design
marick@..., marick@...
www.testing.com, www.visibleworkings.com

"Act always so as to increase the number of choices." -- Heinz von
Foerster

I've written a position paper for a workshop. It touches on
architectures to support product-level tests, the way I use test
automation to explore design, and my increasing inability to tell the
difference between acceptance and unit tests. Your comments would help
me improve my writing. Thanks.

<http://www.testing.com/writings/Marick-AWTA-position.pdf>

-----
Brian Marick
Consulting, training, contracting, and research
Focused on the intersection of testing, programming, and design
marick@..., marick@...
www.testing.com, www.visibleworkings.com

"Act always so as to increase the number of choices." -- Heinz von
Foerster

Shine Technologies is running a survey on the adoption and use of
Agile Methodologies and processes in the IT industry.  If you have
heard of, investigated or used any or all of the Agile processes,
please take a couple of minutes to complete the survey and give us
your valued input.  By taking part you are able to select to receive
the results, which are certain to make interesting reading!  Please
go to:

http://www.shinetech.com/survey.jsp

Regards,
Mark

Hi Brian,
I like the paper, I think it does a good job of demonstrating your
position.  I'm not sure if I would have understood it as well if I
hadn't seen you doing these tests in person.  I think if it were the
first time I saw these examples it might take more work to read.  But
maybe not, I can't say.  It wouldn't be a problem for people who do
development every day.

I had a question on the example in the middle of Page 6.  After the
undo you have a statement "stop 'jobber'".  Why do you do that?  You
mention you were trying other user commands to see how they were
affected, I guess that's why but I don't totally understand it.
Which of the asserts after the stop 'jobber' test the stop 'jobber'?

In general I like your position, I think some people will have a hard
time with it.  It fits with my position that testers should be on the
same team with developers and everyone should work on testing.  The
blurred line between unit tests and other kinds of tests can make
things tricky even in an integrated team, but I think it is the best
way, once we all figure out how to do it.  I'm still a ways from that
in my current position but we're making progress. The developers are
starting to write some automated unit tests and help us learn Tcl so
we can use it for test scripts.  They are starting to talk about
testing issues. The longest journey begins with a single step!

I'm interested to know what other members of this list think of
Brian's position.

thanks,
Lisa

--- In agile-testing@yahoogroups.com, Brian Marick <marick@t...>
wrote:
> I've written a position paper for a workshop. It touches on
> architectures to support product-level tests, the way I use test
> automation to explore design, and my increasing inability to tell
the
> difference between acceptance and unit tests. Your comments would
help
> me improve my writing. Thanks.
>
> <http://www.testing.com/writings/Marick-AWTA-position.pdf>
>
> -----
> Brian Marick
> Consulting, training, contracting, and research
> Focused on the intersection of testing, programming, and design
> marick@t..., marick@v...
> www.testing.com, www.visibleworkings.com
>
> "Act always so as to increase the number of choices." -- Heinz von
> Foerster




-----
Brian Marick
Consulting, training, contracting, and research
Focused on the intersection of testing, programming, and design
marick@..., marick@...
www.testing.com, www.visibleworkings.com

"Act always so as to increase the number of choices." -- Heinz von
Foerster

On Wednesday, December 18, 2002, at 11:51 AM, Lisa Crispin
<lisa.crispin@...> wrote:

> I had a question on the example in the middle of Page 6.  After the
> undo you have a statement "stop 'jobber'".  Why do you do that?  You
> mention you were trying other user commands to see how they were
> affected, I guess that's why but I don't totally understand it.
> Which of the asserts after the stop 'jobber' test the stop 'jobber'?

That isn't clear, is it?

I started by showing this test code as a sample of what a user might do:

        def test_undo_ineffectual_background_setting
            job 'jobber'
            start 'jobber'
            background 'jobber'  # this takes effect only after start_day

            undo
            # there is no background job


Then I added this, describing it as "some checks that the behavior is
what I intended it to be."

            stop 'jobber'

            assert_equal(false, @session.jobs['jobber'].is_background?)
            message = start_day
            assert_match(/But there is no background job/m, message)
        end

The 'stop', to be clear, should come after the first assert_equal. The
first assert_equal checks that the 'background' property has been
removed from the job. Then the test tries the 'start_day' command,
expecting it will fail because there is no background job (because I
just undid making 'jobber' a background job). The stop is necessary
because you can't start a day if something's currently running. If you
do, you'll get an error message - but not one that has anything to do
with what this test is about. So I stop the only running job so I get
an error message that suggests that the undo really worked.

-----
Brian Marick
Consulting, training, contracting, and research
Focused on the intersection of testing, programming, and design
marick@..., marick@...
www.testing.com, www.visibleworkings.com

"Act always so as to increase the number of choices." -- Heinz von
Foerster

Brian,

Thank you for a thought provoking paper. I have been using the resources on
your site for years now and I still haven'y explored them all. A worthy
effort.

Now, I have tried and I cannot really see the point of accessing the
business layer via a RPC interface. Why not DTSTTCPW and just have an API?

It seems to be adding complexity for no purpose, even if Ruby gives you
distributed objects for free. The use of RPC still has issues (methods
defined in two places, speed, security) which all have a cost. Your paper
mentioned accessing the business layer via sessions. Does this allow
multiple concurrent sessions? Was this an original requirement for the app?
Multiple sessions looks like more complexity that is not really required.

A command line interface to an app is a great idea, I do it myself. We would
not be able to do our acceptance tests without it. None of our users ever
use it, but thats OK. However, how does a command line mandate RPC and two
processes?

Using RPC to allow the test to peek into the business layer is useful, but
can be acheived in a number of other ways. I use a little app that formats a
variety of database queries, so that the tester can check that a GUI gesture
in the main app has the correct effect on the correct table and maintains
database consistancy. This bypasses the business layer, and requires a
knowledge of the DB schema, but makes a good solid acceptance test.

I am probably missing your point: adding RPC gave you some enormous benefit,
which I cannot see. I am trying to teach myself some better testing
practices, which is how I came upon the paper. I would be most grateful if
you could let me in.

As a suggestion, I found the sample business, presentation and product tests
most unclear about what sort of session they were using, whether a mock or
not. This would have helped me understand it a lot better.

Tom Harris <tomh@...>

I've joined a small company as the lead tester (and really only
tester)that is looking to move toward an XP based process, I've
volunteered to help drive this movement and I was wondering if anyone
in the group would have any relevant advice about the first steps down
the Agile road (what worked well, what did not, pitfalls to avoid
etc...), especially dealing with the initial release planning and
kick-off.  Any advice and stories would be appreciated.

Jamie
jamiehosley@...

Jamie,

My first question is: "How familiar are you or anyone else in the
company with Agile methods?" The answer to that question will help
determine the best way to start.

As a tester, you do want to read "Testing Extreme Programming" by
Lisa Crispin and Tip House. It will give you some great ideas on how
to approach your job.

I started in a small company about 6 months ago and introduced XP to
the team. The first thing I did was start using the terminology to
get every using the same vocabulary. There are still some practices
that are hard to implement because of the size of the team, but we
found it has helped us to react to the ever changing landscape that
is our product.

I look forward to hearing how you do with your new project.

Janet Gregory

--- In agile-testing@yahoogroups.com, "jamiehosley1
<jamiehosley@m...>" <jamiehosley@m...> wrote:
> I've joined a small company as the lead tester (and really only
> tester)that is looking to move toward an XP based process, I've
> volunteered to help drive this movement and I was wondering if
anyone
> in the group would have any relevant advice about the first steps
down
> the Agile road (what worked well, what did not, pitfalls to avoid
> etc...), especially dealing with the initial release planning and
> kick-off.  Any advice and stories would be appreciated.
>
> Jamie
> jamiehosley@m...

Brian,

I have reviewed your paper, and think you have some good ideas, but
I have a few questions.

Are you saying that all the tests, unit tests included, can be run
from a separate process? If so, doesn't that start to separate the
developers from the responsibility of the unit tests - especially in
a large application?

I like the idea of having the product / acceptance / "black box"
tests as a separate process. In fact, we had to build our own J-Unit
test framework once to address the problem. Using Ruby or something
like that may be another answer, especially if the current framework
is easily converted.  We are exploring the use of Ruby right now and
hope to be able to use it.

Janet


--- In agile-testing@yahoogroups.com, Brian Marick <marick@t...>
wrote:
> I've written a position paper for a workshop. It touches on
> architectures to support product-level tests, the way I use test
> automation to explore design, and my increasing inability to tell
the
> difference between acceptance and unit tests. Your comments would
help
> me improve my writing. Thanks.
>
> <http://www.testing.com/writings/Marick-AWTA-position.pdf>
>
> -----
> Brian Marick
> Consulting, training, contracting, and research
> Focused on the intersection of testing, programming, and design
> marick@t..., marick@v...
> www.testing.com, www.visibleworkings.com
>
> "Act always so as to increase the number of choices." -- Heinz von
> Foerster

Hi Jamie

I'm working on an 'agile' project for a large organisation that has always
followed the classic V model/waterfall approach.   The challenges I've
faced are more human than technical.
You need answers for ... "but we've always done it this way....".
I get to see a lot of 'superstition testing' where people are following
procedures that they dont fully understand. You're in a small team, so
maybe they're not weighed down by 'tradition'.

Communication is of paramount importance. It goes well beyond good
documentation, precise notes etc. You actually have to be there. Most good
testers develop 'radar' after a while.... what might sound like a bit of
developers' techno-gossip can have major implications for testing.  If
you're in the planning stage, definiitely put co-location on the agenda.
Good fault tracking is important too... whether you use a piece of paper or
some fancy 'Wiz-O-trak' proprietary program.  Fault tracking depends on all
the stakeholders being in the loop, and responding quickly and efficiently.
If you're still planning, make sure that  you communicate with everybody
you need to, to make them understand what's needed.

Anyway, good luck, it sounds fun.

Peter Horsam









                       "jamiehosley1
                       <jamiehosley@mac.        To:      
agile-testing@yahoogroups.com
                       com>"                    cc:
                       <jamiehosley             Subject:  [agile-testing] Advice
on kicking off XP project and Release Planning Meeting

                       19/12/2002 13:56
                       Please respond to
                       agile-testing






I've joined a small company as the lead tester (and really only
tester)that is looking to move toward an XP based process, I've
volunteered to help drive this movement and I was wondering if anyone
in the group would have any relevant advice about the first steps down
the Agile road (what worked well, what did not, pitfalls to avoid
etc...), especially dealing with the initial release planning and
kick-off.  Any advice and stories would be appreciated.

Jamie
jamiehosley@...



To unsubscribe from this group, send an email to:
agile-testing-unsubscribe@yahoogroups.com



Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/









Important:  This e-mail is intended for the use of the addressee and may contain
information that is confidential, commercially valuable or subject to legal or
parliamentary privilege.  If you are not the intended recipient you are notified
that any review, re-transmission, disclosure, use or dissemination of this
communication is strictly prohibited by several Commonwealth Acts of Parliament.
If you have received this communication in error please notify the sender
immediately and delete all copies of this transmission together with any
attachments.

On Wednesday, December 18, 2002, at 10:51 PM, Janet Gregory
<janet_gregory@...> wrote:

> Brian,
>
> I have reviewed your paper, and think you have some good ideas, but
> I have a few questions.
>
> Are you saying that all the tests, unit tests included, can be run
> from a separate process? If so, doesn't that start to separate the
> developers from the responsibility of the unit tests - especially in
> a large application?

Unit tests could be run from a separate process, I suppose, but I
wouldn't. I think my point is that putting a remote procedure call
interface on an app makes some kinds of testing easier. For those
tests, you should use it. For others, you needn't. I have things set up
so that most tests and most product code are completely oblivious to
whether they're talking to a remote session or to one living in the
same process.

Responsibility... I wish programmers to have their traditional
responsibility for unit tests. I also wish to give them great
flexibility about how those tests look and where they live (possibly
like acceptance tests, possibly with the acceptance tests) - at least
as an experiment.

I also wish to free us from the assumption that certain people are
responsible for certain types of tests, at least to the degree that
that inhibits them from thinking about *other* kinds of tests. If a
programmer is elbow-deep in the code and has a thought about an
important product-level statement of fact, she should have absolutely
no hesitation about discussing that statement with a "goal donor" and
turning it into an executable test that lives with tests written by
customers, testers, or the person who waters the plants.

I am wary of boundaries between roles. Assigning people to roles makes
it more likely certain things will be taken care of. (If there is a
designated plant waterer, the plants will get watered.) But
rigid-seeming boundaries make it too easy for things to fall through
the cracks. (The programmer is thinking "X isn't a unit test, so I
don't have to worry about it." while the customer or tester is thinking
"X isn't an acceptance test so I don't have to worry about it.")

A project's infrastructure, it seems to me, should help people cross
boundaries. Having a customer on site does that. Blurring the
distinction between acceptance and unit tests might do it too.


> I like the idea of having the product / acceptance / "black box"
> tests as a separate process. In fact, we had to build our own J-Unit
> test framework once to address the problem. Using Ruby or something
> like that may be another answer, especially if the current framework
> is easily converted.  We are exploring the use of Ruby right now and
> hope to be able to use it.

Let me know how it goes, please.

-----
Brian Marick
Consulting, training, contracting, and research
Focused on the intersection of testing, programming, and design
marick@..., marick@...
www.testing.com, www.visibleworkings.com

"Act always so as to increase the number of choices." -- Heinz von
Foerster

Hi Guys

Is anyone else receiving spam from IIST?  They seem to have obtained my
e-mail address from somewhere (this list, perhaps) and keep sending me
unsolicited e-mails, plus a pile of randomly generated e-mail addresses
in hex.  I've written to ask them to remove me, but no reply or action.
Any one else on the receiving end of this?  Do they realise how bad this
makes them look?  They also appear to be sharing addresses with other
organisations (psqtconference.com, testinginstitute.com) as I have
received a number of spams from similar organisations with the same
bizarre hex usernames.

--
Hubert Matthews         http://www.oxyware.com/
Software Consultant     hubert@...

On Wednesday, December 18, 2002, at 07:16 PM, Tom Harris wrote:

> Brian,
>
> Thank you for a thought provoking paper. I have been using the
> resources on
> your site for years now and I still haven'y explored them all. A worthy
> effort.

You're welcome. And thank you.

>
> Now, I have tried and I cannot really see the point of accessing the
> business layer via a RPC interface. Why not DTSTTCPW and just have an
> API?

(DTSTTCPW = do the simplest thing that can possibly work.)

A remote procedure call / remote method invocation interface is an API
to an object that happens to live in another process. I wanted to write
tests for the behavior of the app when it was shut down and restarted
and when it crashed and restarted. How would you write that with a
plain API, where the tests have to be part of the same process as the
code under test (and will hence exit when the app does)? I can think of
ways to do it, but the remote object way seems simpler, actually.

An alternative would be to embed a command-line interface to the app
and drive that interface remotely with something like Expect. But to me
it seems simpler just to expose the app's interface objects to the
tests than to write command-line commands that reach into them. (I did
write a command line and use it in tests, but I don't have to write
commands for everything my tests do. I can mix and match as is
convenient.)

I do confess that my bias is toward making all applications scriptable.
You can do that by embedding a scripting language or by exposing a
remotely invocable API. The problem with embedding a scripting language
is that you can only script that one app. It's harder to write scripts
that choreograph multiple apps. And you restrict script-writers to
writing in one scripting language.

Because of my bias, I probably leap at the opportunity to put in an RPC
interface even some other way is somewhat simpler. But I don't believe
the simplest way that can possibly work is necessarily the right way.
In that, I deviate from the XP orthodoxy, probably more toward Dick
Gabriel's notion of habitable software (from _Patterns of Software_)
and Christopher Alexander's notion of design sequences (that Ron
Goldman has been talking about recently).

> It seems to be adding complexity for no purpose, even if Ruby gives you
> distributed objects for free. The use of RPC still has issues (methods
> defined in two places, speed, security) which all have a cost. Your
> paper
> mentioned accessing the business layer via sessions. Does this allow
> multiple concurrent sessions? Was this an original requirement for the
> app?
> Multiple sessions looks like more complexity that is not really
> required.

In my setup, defining a remotely-callable method is pretty trivial.
Speed isn't an issue for this app. Security will be eventually, since I
do intend to stick the server on my web site so that I can track my
time from anywhere I have a net connection, and it will be interesting
to see how I handle it.

I originally didn't have sessions. But once I started using the app to
track my time, the simplest way to protect my time records from the
tests was to add sessions. Otherwise, I would have had to separate
development and deployment versions, which would mean figuring out the
right way to package things on the Mac, and I didn't want to bother
with that yet. It would have slowed me down, and it was important to
get quickly to "eating my own dog food" (using it to track billable
hours).

> A command line interface to an app is a great idea, I do it myself. We
> would
> not be able to do our acceptance tests without it. None of our users
> ever
> use it, but thats OK. However, how does a command line mandate RPC and
> two
> processes?

It doesn't - see above.

Another way to look at it. Consider conventional GUI testing. There,
you have two processes. One is running the test tool. The other is
running the app. Or consider testing something with a command line
using Expect. You have two processes. One is running TCL/Expect; the
other is running the app. Maybe you always have two processes. I've
just chosen to put their boundary in an odd place.

>
> Using RPC to allow the test to peek into the business layer is useful,
> but
> can be acheived in a number of other ways. I use a little app that
> formats a
> variety of database queries, so that the tester can check that a GUI
> gesture
> in the main app has the correct effect on the correct table and
> maintains
> database consistancy. This bypasses the business layer, and requires a
> knowledge of the DB schema, but makes a good solid acceptance test.

Yes. I think this is a good idea. If my database were more complicated,
I would probably do this too. But this is reaching "around" the app to
check something that stands "behind" it, normally invisible to the
user. It is also useful to reach "into" the app.

> I am probably missing your point: adding RPC gave you some enormous
> benefit,
> which I cannot see. I am trying to teach myself some better testing
> practices, which is how I came upon the paper. I would be most
> grateful if
> you could let me in.

I don't know that I'd call it an enormous benefit to testing.

> As a suggestion, I found the sample business, presentation and product
> tests
> most unclear about what sort of session they were using, whether a
> mock or
> not. This would have helped me understand it a lot better.

Thanks. I don't use any mock Sessions. Every @session variable is a
real Session object. You can't tell, without looking at some test
support code, whether they live remotely or locally. Which I think is a
good thing.

-----
Brian Marick
Consulting, training, contracting, and research
Focused on the intersection of testing, programming, and design
marick@..., marick@...
www.testing.com, www.visibleworkings.com

"Act always so as to increase the number of choices." -- Heinz von
Foerster

For those interested in more along the same lines as my position paper,
here's a writeup of an interview I did with Sam Guckheimer for Rational
Edge magazine:

<http://www.therationaledge.com/content/oct_02/f_testFirstDesign_sg.jsp>

-----
Brian Marick
Consulting, training, contracting, and research
Focused on the intersection of testing, programming, and design
marick@..., marick@...
www.testing.com, www.visibleworkings.com

"Act always so as to increase the number of choices." -- Heinz von
Foerster

> An alternative would be to embed a command-line interface to the app
> and drive that interface remotely with something like Expect. But to
> me it seems simpler just to expose the app's interface objects to
> the tests than to write command-line commands that reach into
> them. (I did write a command line and use it in tests, but I don't
> have to write commands for everything my tests do. I can mix and
> match as is convenient.)

In fact if your application is written in any language that has
"eval", i.e. Smalltalk, Lisp/Scheme, Ruby, etc. then the easy thing to
do is support at least one remote API: remote-eval, that takes a
string and interprets it. I have done this several times, but it is a
security hole, so you want to leave it out of production systems if
there is much at stake.

> I do confess that my bias is toward making all applications
> scriptable.

Me too. Same with the UI and Model process separation, even when they
run on the same machine.

Junit is free. Check out Junit.org.

At 10:41 AM 12/9/2002, Radu Ciprian <Ciprian.Radu@...> wrote:
>where can I find free testing tools ? Trial versions should be ok.
>Thanks

_____________________________________
   Bret Pettichord
     Book - www.testinglessons.com
     Hotlist - www.testinghotlist.com
     Consulting - www.pettichord.com

-----Original Message-----
From: Logan, Patrick D [mailto:patrick.d.logan@...]

> An alternative would be to embed a command-line interface to the app
> and drive that interface remotely with something like Expect. But to
> me it seems simpler just to expose the app's interface objects to
> the tests than to write command-line commands that reach into
> them. (I did write a command line and use it in tests, but I don't
> have to write commands for everything my tests do. I can mix and
> match as is convenient.)

In fact if your application is written in any language that has
"eval", i.e. Smalltalk, Lisp/Scheme, Ruby, etc. then the easy thing to
do is support at least one remote API: remote-eval, that takes a
string and interprets it. I have done this several times, but it is a
security hole, so you want to leave it out of production systems if
there is much at stake.

Given an application development environment that supports evaluation, such as Smalltalk, how much opportunity for testing are you missing out on by staying within the environment? The reason I ask is that we've been selling a Smalltalk testing tool (www.silvermark.com) since 1997, and though I've often considered the idea of adding a remote interface to it, it's never really seemed worth the bother, and nobody's asked for it anyway. Sure you miss out on testing startup/shutdown, and a catastrophic meltdown will kill the test, but in the grand scheme of things, those turn out to be minor issues weighed against the added complexity of testing through a remote interface, constrained scripting, and the loss of the tools that the development environment provides.

 

- Mike

 

-----------------------------
Mike Silverstein
SilverMark, Inc.
The Object Testing Company

On Friday, December 20, 2002, at 03:32 PM, Michael Silverstein wrote:
>
> Given an application development environment that supports evaluation,
> such as Smalltalk, how much opportunity for testing are you missing
> out on by staying within the environment? The reason I ask is that
> we've been selling a Smalltalk testing tool (www.silvermark.com) since
> 1997, and though I've often considered the idea of adding a remote
> interface to it, it's never really seemed worth the bother, and
> nobody's asked for it anyway. Sure you miss out on testing
> startup/shutdown, and a catastrophic meltdown will kill the test, but
> in the grand scheme of things, those turn out to be minor issues
> weighed against the added complexity of testing through a remote
> interface, constrained scripting, and the loss of the tools that the
> development environment provides.

I think these are good points. I wouldn't draw the line between
"language that supports eval" and "language that doesn't", though.
Thinking about it, I think maybe the line is between "IDE in the same
address space as the app" and "IDE in a different address space".

For those who haven't used Smalltalk, everything is integrated. You
edit a bit of a program in one window. You mouse over to another window
and start the program. If something bad happens, you get a window that
shows the code where the problem happened. That's the same kind of
editor window. You can fix the code right there, and continue on. The
editor, debugger, and all similar tools have complete access to
everything that's going on. They're in the same process as the app. (In
fact, making a distinction between "the tools" and "the app" is a bit
artificial.) Moving tests to a separate process takes away a lot of
power.

(But it would be interesting to get the opinions of people developing
distributed applications with OpenTalk, a Smalltalk distributed object
system. How much more difficult is it than developing a non-distributed
application? That's probably a clue to the difficulty of developing
tests in a separate image/environment from the application. I used
OpenTalk a little, maybe two years ago, and found it pretty painful.
But I'm a Smalltalk dabbler, at best.)

In Ruby or Python or other more "conventional" languages with eval,
many pieces of the puzzle are already assumed to be separate. When you
edit Ruby code, you're not using a program in a Ruby process - you're
using Emacs, or vi, or Eclipse. And you're editing a file, not (as in
Smalltalk) the actual running code. Unit testing is assumed to be a
matter of invoking a script that runs a bunch of tests - it's not done
by issuing a command in the interpreter. (At least I hope not -
otherwise, I've missed something basic.) Given a pre-existing
assumption of separation, it makes more sense to do as I've done. I
hope.

--

But, Michael: most people are working in languages like Java and C++.
Languages that are more rigid, without interpreters and without eval.
Faster to run, slower to program. In this case, I really do think you
want to write your (acceptance) tests in a different language, be it
something like Python or Ruby (my preference), a tabular language (like
Ward Cunningham's FIT or Hans Buwalda's action words) or the
vendorscripts in the GUI testing tools. My guess is that it's easier to
run that language in a separate process than it is to embed it into the
app. What do you think?


-----
Brian Marick
Consulting, training, contracting, and research
Focused on the intersection of testing, programming, and design
marick@..., marick@...
www.testing.com, www.visibleworkings.com

"Act always so as to increase the number of choices." -- Heinz von
Foerster

Brian,
I have included by thoughts below.

--- In agile-testing@yahoogroups.com, Brian Marick <marick@t...>
wrote:
<snip>
> Responsibility... I wish programmers to have their traditional
> responsibility for unit tests. I also wish to give them great
> flexibility about how those tests look and where they live
(possibly like acceptance tests, possibly with the acceptance
tests) - at least > as an experiment.

I thought the whole purpose of using J-Unit, n-unit etc. was to give
the programmers a mechanism to do their unit tests in the same
language and structure that they were programming in. The
environment is set up, they can run them in the same IDE as they are
using to program.

The disadvantage to some of these languages is they are designed for
unit tests and not the system as a whole. If there is any easy
mechanism (such as RUBY) or other language that can be use to
program the acceptance level tests, it is helpful to the customers
and the testers who look at the big picture on a regular basis.

> I also wish to free us from the assumption that certain people are
> responsible for certain types of tests, at least to the degree
that > that inhibits them from thinking about *other* kinds of
tests. If a
> programmer is elbow-deep in the code and has a thought about an
> important product-level statement of fact, she should have
absolutely > no hesitation about discussing that statement with
a "goal donor" and
> turning it into an executable test that lives with tests written
by > customers, testers, or the person who waters the plants.
> I am wary of boundaries between roles. Assigning people to roles
makes > it more likely certain things will be taken care of. (If
there is a  designated plant waterer, the plants will get watered.)
But > rigid-seeming boundaries make it too easy for things to fall
through
> the cracks. (The programmer is thinking "X isn't a unit test, so I
> don't have to worry about it." while the customer or tester is
thinking
> "X isn't an acceptance test so I don't have to worry about it.")

Roles can change, but I do believe it is enevitable that people are
slotted in roles. It's good to have people who are designated to
consider other options, and look at things from a different point of
view. However, that doesn't absolve anyone from taking
responsibility to make the product the best it can be.

I encourage all programmers to think about the acceptance tests and
it would be great if they could write them, but it may just not be
the most efficient way to do it. It may be good enough to
communicate their thoughts to the tester or customer on the team. I
have also had developers write the acceptance tests as a separate
story, which works as well.

Janet
>

> Given an application development environment that supports
> evaluation, such as Smalltalk, how much opportunity for testing are
> you missing out on by staying within the environment? The reason I
> ask is that we've been selling a Smalltalk testing tool
> (www.silvermark.com) since 1997, and though I've often considered
> the idea of adding a remote interface to it, it's never really
> seemed worth the bother, and nobody's asked for it anyway. Sure you
> miss out on testing startup/shutdown, and a catastrophic meltdown
> will kill the test, but in the grand scheme of things, those turn
> out to be minor issues weighed against the added complexity of
> testing through a remote interface, constrained scripting, and the
> loss of the tools that the development environment provides.

The times I have used a "remote-eval" it's been for server-based
systems. In a development context, the ability to connect to any
server, inspect, modify, etc. As for "loss of tools"... Gemstone
Smalltalk never had many useful tools for server testing, even
in-house! Most customers accessed it via the client Smalltalk, but we
had these promising things like GemEnterprise federation and the
System Management Framework which only were used by a few customers
before the move to Java killed them.

But that's another story...

Same thing with Lisp, though: testing a distributed system for factory
planning and monitoring.

As a testing consultant, I've worked with a couple teams using XP. And i've
also talked to a number of other testers on XP projects.

The teams that i worked with had the most trouble with the user stories.
They had trouble defining and planning their tests and managing progress.
Ultimately i attributed to the problem to the way that the managers and
developers had defined the user stories -- or rather not defined. In once
case, they labelled features as "user stories," in the other case, they
labelled tasks as "user stories." In both cases, this gave the programmers
enough information to go ahead a do their programming, but the testers
didn't really know how to test. The stories were either too general
(features) or not described in user terms (tasks), so it took a lot of work
for the testers to figure out what good tests looked like.  Also, not all
of the programming work was driven by user stories.

It wasn't until i'd had a chance to confer with others using XP, that i
came to realize that this was the core of their problems. It's very
important for the user stories to be concrete, meaningful to the customer
and specific. I can recommend two books on this topic, *Planning Extreme
Programming* by Beck and Fowler, and *Writing Effective Use Cases* by
Cockburn, which describes the relationship between use cases and user
stories (they are very similar).

Another problem that we had was that the use of unit testing was
inconsistent. It's important to have some means to make sure that all the
programmers are writing their unit tests. (Pair programming is one means,
but both my teams were weak on this.)

Bret

At 08:56 PM 12/18/2002, jamiehosley1 <jamiehosley@...> wrote:
>I've joined a small company as the lead tester (and really only
>tester)that is looking to move toward an XP based process, I've
>volunteered to help drive this movement and I was wondering if anyone
>in the group would have any relevant advice about the first steps down
>the Agile road (what worked well, what did not, pitfalls to avoid
>etc...), especially dealing with the initial release planning and
>kick-off.  Any advice and stories would be appreciated.
>
>Jamie
>jamiehosley@...

_____________________________________
   Bret Pettichord
     Book - www.testinglessons.com
     Hotlist - www.testinghotlist.com
     Consulting - www.pettichord.com

Hi Jamie,
It sounds like your company has made a commitment to move to an agile
process, so you've overcome what I think is the biggest hurdle.

Apart from getting commitment from management and from developers to
try agile practices, my own experience has been that one of the
biggest challenges is getting the developers comfortable with
automated unit testing, especially test-first design.

If you're starting from scratch with a brand new team, in other words
not trying to introduce XP or whatever into an existing culture, this
is easier to deal with.  Get someone who is an expert unit tester to
come show your developers how it's done.  Have the developers get
together and brainstorm ways to unit test pieces of code where the
approach is not obvious.

My current situation is that I'm a lead tester in an organization
which wants to adopt XP or some other agile methodology but for
various reasons is having a hard time getting there.  Our development
director has wanted to implement test-first automated unit testing
but just had too many fires blazing, plus it's difficult to start
doing test-first when all you have is legacy code.  I gave him some
good articles on automated unit testing and took advantage of an
opportunity to have Brian Marick come give a short but inspiring demo
of test-first design.  He (our development director) has finally come
up with a framework to do the automated unit testing (they'll use
Tcltest and jUnit as appropriate).  The developers are going to start
slowly, writing tests before any new code, taking on the easier
testing tasks first and building up to the trickier ones.  We'll see
how it goes!

There are many, many other challenges and potential pitfalls to XP
and other agile processes, but if your team can't nail unit testing,
I don't think anything else will work very well.  In all the teams
I've worked on, it took strong leadership to implement really
effective unit testing.  Even though developers want to do their best
work, unit testing is difficult at first, and they are also pressured
to make deadlines.

One other practice I think is essential whether you're using XP or
something else is to hold retrospectives after each iteration.  Pick
just 2 or 3 things to work on for the next iteration.  The teams I
worked on made huge improvements in all the other practices this way.

-- Lisa


--- In agile-testing@yahoogroups.com, "jamiehosley1
<jamiehosley@m...>" <jamiehosley@m...> wrote:
> I've joined a small company as the lead tester (and really only
> tester)that is looking to move toward an XP based process, I've
> volunteered to help drive this movement and I was wondering if
anyone
> in the group would have any relevant advice about the first steps
down
> the Agile road (what worked well, what did not, pitfalls to avoid
> etc...), especially dealing with the initial release planning and
> kick-off.  Any advice and stories would be appreciated.
>
> Jamie
> jamiehosley@m...

Hi Jamie,
It sounds like your company has made a commitment to move to an agile
process, so you've overcome what I think is the biggest hurdle.

Apart from getting commitment from management and from developers to
try agile practices, my own experience has been that one of the
biggest challenges is getting the developers comfortable with
automated unit testing, especially test-first design.

If you're starting from scratch with a brand new team, in other words
not trying to introduce XP or whatever into an existing culture, this
is easier to deal with.  Get someone who is an expert unit tester to
come show your developers how it's done.  Have the developers get
together and brainstorm ways to unit test pieces of code where the
approach is not obvious.

My current situation is that I'm a lead tester in an organization
which wants to adopt XP or some other agile methodology but for
various reasons is having a hard time getting there.  Our development
director has wanted to implement test-first automated unit testing
but just had too many fires blazing, plus it's difficult to start
doing test-first when all you have is legacy code.  I gave him some
good articles on automated unit testing and took advantage of an
opportunity to have Brian Marick come give a short but inspiring demo
of test-first design.  He (our development director) has finally come
up with a framework to do the automated unit testing (they'll use
Tcltest and jUnit as appropriate).  The developers are going to start
slowly, writing tests before any new code, taking on the easier
testing tasks first and building up to the trickier ones.  We'll see
how it goes!

There are many, many other challenges and potential pitfalls to XP
and other agile processes, but if your team can't nail unit testing,
I don't think anything else will work very well.  In all the teams
I've worked on, it took strong leadership to implement really
effective unit testing.  Even though developers want to do their best
work, unit testing is difficult at first, and they are also pressured
to make deadlines.

One other practice I think is essential whether you're using XP or
something else is to hold retrospectives after each iteration.  Pick
just 2 or 3 things to work on for the next iteration.  The teams I
worked on made huge improvements in all the other practices this way.

-- Lisa


--- In agile-testing@yahoogroups.com, "jamiehosley1
<jamiehosley@m...>" <jamiehosley@m...> wrote:
> I've joined a small company as the lead tester (and really only
> tester)that is looking to move toward an XP based process, I've
> volunteered to help drive this movement and I was wondering if
anyone
> in the group would have any relevant advice about the first steps
down
> the Agile road (what worked well, what did not, pitfalls to avoid
> etc...), especially dealing with the initial release planning and
> kick-off.  Any advice and stories would be appreciated.
>
> Jamie
> jamiehosley@m...

-----Original Message-----
From: Brian Marick [mailto:marick@...]

On Friday, December 20, 2002, at 03:32 PM, Michael Silverstein wrote:
>
> Given an application development environment that supports evaluation,
> such as Smalltalk, how much opportunity for testing are you missing
> out on by staying within the environment? The reason I ask is that
> we've been selling a Smalltalk testing tool (www.silvermark.com) since
> 1997, and though I've often considered the idea of adding a remote
> interface to it, it's never really seemed worth the bother, and
> nobody's asked for it anyway. Sure you miss out on testing
> startup/shutdown, and a catastrophic meltdown will kill the test, but
> in the grand scheme of things, those turn out to be minor issues
> weighed against the added complexity of testing through a remote
> interface, constrained scripting, and the loss of the tools that the
> development environment provides.

I think these are good points. I wouldn't draw the line between
"language that supports eval" and "language that doesn't", though.
Thinking about it, I think maybe the line is between "IDE in the same
address space as the app" and "IDE in a different address space".
 

I agree. The distinction really hinges on the way the language, runtime execution support (VM) and the development environment play off each other to enable the developer to dynamically interact with the code as it comes into existence.

 
For those who haven't used Smalltalk, everything is integrated. You
edit a bit of a program in one window. You mouse over to another window
and start the program. If something bad happens, you get a window that
shows the code where the problem happened. That's the same kind of
editor window. You can fix the code right there, and continue on. The
editor, debugger, and all similar tools have complete access to
everything that's going on. They're in the same process as the app. (In
fact, making a distinction between "the tools" and "the app" is a bit
artificial.) Moving tests to a separate process takes away a lot of
power.
 

Yes. To expand on that, in Smalltalk the application you develop and the development tools are all live objects in the same address space, which is called an 'image'. It is only when you are ready to deploy (ship) your application that you create a copy of the image with the development tools stripped out.

 

In Ruby or Python or other more "conventional" languages with eval,
many pieces of the puzzle are already assumed to be separate. When you
edit Ruby code, you're not using a program in a Ruby process - you're
using Emacs, or vi, or Eclipse. And you're editing a file, not (as in
Smalltalk) the actual running code. Unit testing is assumed to be a
matter of invoking a script that runs a bunch of tests - it's not done
by issuing a command in the interpreter. (At least I hope not -
otherwise, I've missed something basic.) Given a pre-existing
assumption of separation, it makes more sense to do as I've done. I
hope.

--

But, Michael: most people are working in languages like Java and C++.
Languages that are more rigid, without interpreters and without eval.
Faster to run, slower to program.  

 

Rewind 100 years: "most people are using horses to get around and we just need to learn to accept their droppings" :-)

Anyway, perhaps this points to the fundamental problem (in my opinion) that most people are still using sub-optimal languages and development environments. In over 20 years of software development I've tried most mainstream languages, development environments, assemblers, interpreters, etc. and have never been more productive as when using Smalltalk. And yes, you are right-it is more than just syntax.

 

Not to turn this into a language debate but over the years, the leading Smalltalk VMs have become very well optimized. A good Smalltalk VM such as IBM's VisualAge or Cincom's VisualWorks will perform very favorably compared to Java - especially in the GUI responsiveness, so the statement that they are faster to run isn't necessarily true under all circumstances. An interesting side note before this line goes completely off on a tangent is that the IBM/OTI people who created the speedy SWT for Eclipse used many of the same principles they employed years earlier when developing the GUI framework for VisualAge Smalltalk, which is still much more responsive than AWT or Swing.

 

 In this case, I really do think you
want to write your (acceptance) tests in a different language, be it
something like Python or Ruby (my preference), a tabular language (like
Ward Cunningham's FIT or Hans Buwalda's action words) or the
vendorscripts in the GUI testing tools. My guess is that it's easier to
run that language in a separate process than it is to embed it into the
app. What do you think?
 

I'll go out on a limb and say it depends :-)

 

First of all, there seem to be two issues here that I'll address: 

1) Whether to write tests in the same language as the code under development

2) Whether to run tests within the same or a different address space via some remote (RPC) interface

 

As a developer I prefer to stay within the language I'm developing in. One of the reasons our Smalltalk GUI testing tool is very successful with developers is that it generates playback statements in Smalltalk. On the other hand, my company has been a proponent of providing a means for non-developers to compose component tests from objects that know how to test, which clearly departs from the idea of writing test code in the target application's language. For information in this see http://www.silvermark.com/documentation/developerQACollaboration.pdf and http://servlet.java.sun.com/javaone/javaone2000/pdfs/TS-708.pdf, which we've embodied in a Java component testing tool at http://www.silvermark.com/Product/java/tmJava.html.

 

So in answer to number #1, I'd say it depends on who's developing the test. In general, I tend to see the following mapping:

Developer --> Same development language

Tester* --> light-weight, interpretive language or alternate (visual) test composition

Business analyst/customer --> Table driven tests, such as provided by FIT or action words

 

* Someone not well versed in heavy-duty development but with enough skills to automate tests

 

Since the subject of your paper is acceptance testing, which typically involves latter two groups, then yes I agree that the implementation language is not always the best test script language.

 

Regarding #2, I like the idea of being able to dynamically interact with code and the work you did helps in that regard for those languages that don't support it. I think anything that streamlines the compile/link steps is a good thing. Whether the test scripting has to run in a separate address space or through an interpreter, etc. that's linked in to the application under test is an implementation detail, although the ability to test startup and exit is a net add. So it depends whether on balance, using a remote interfaced adds to or subtracts from the development environment, I agree with you that in cases such as C++ it adds to it. In cases such as Smalltalk, it is likely to subtract, unless there are other extenuating circumstances.

 

- Mike

 
-----------------------------
Mike Silverstein
SilverMark, Inc.
The Object Testing Company
www.javatesting.com

On Sunday, December 22, 2002, at 02:16 PM, Janet Gregory
<janet_gregory@...> wrote:

> Brian,
> I have included by thoughts below.
>
> --- In agile-testing@yahoogroups.com, Brian Marick <marick@t...>
> wrote:
> <snip>
>> Responsibility... I wish programmers to have their traditional
>> responsibility for unit tests. I also wish to give them great
>> flexibility about how those tests look and where they live
> (possibly like acceptance tests, possibly with the acceptance
> tests) - at least > as an experiment.
>
> I thought the whole purpose of using J-Unit, n-unit etc. was to give
> the programmers a mechanism to do their unit tests in the same
> language and structure that they were programming in. The
> environment is set up, they can run them in the same IDE as they are
> using to program.

Yes, but this is only (I think) a matter of programmer convenience:
- she's guaranteed to know the language she's writing the app in.
- it's often hard to do unit-test-ey detail-level manipulations in
    a "foreign" language that calls the app's language.
- as you correctly point out, the unit tests are typically set up to
    be "ready to hand" for the programmer. She can just reach out
    and run them as easily as she reaches out and edits the source
    or reaches out and builds the app.

Those remain entirely true with my scheme, so I expect an awful lot of
"unit testing" will happen in the app language. But suppose (as is the
case for my app) that the programmer knows the language used for
acceptance tests, and suppose some thoughts about what the program
should do seem more helpfully expressed in that language. Then the
programmer should, I think, write her tests in that language - and not
worry about whether they're unit or acceptance tests.

That flows from my notion that the point of testing is to clarify
thinking before programming. Programmers - and everyone else - writes
tests to help them think through what's coming, and then later to raise
their confidence that they did as they intended. What's important is
amplifying that thinking, not observing distinctions.

> The disadvantage to some of these languages is they are designed for
> unit tests and not the system as a whole. If there is any easy
> mechanism (such as RUBY) or other language that can be use to
> program the acceptance level tests, it is helpful to the customers
> and the testers who look at the big picture on a regular basis.

A disadvantage of my scheme is that a language more helpful to
programmers is probably less helpful to customers. There's a balancing
act here. Ward Cunningham is exploring a different pivot point with
FIT. I'm exploring this one, and thinking about how face-to-face
conversation might compensate for the (over)emphasis on programmer
usability.

[snip]

>
> Roles can change, but I do believe it is enevitable that people are
> slotted in roles. It's good to have people who are designated to
> consider other options, and look at things from a different point of
> view. However, that doesn't absolve anyone from taking
> responsibility to make the product the best it can be.
>
> I encourage all programmers to think about the acceptance tests and
> it would be great if they could write them, but it may just not be
> the most efficient way to do it. It may be good enough to
> communicate their thoughts to the tester or customer on the team. I
> have also had developers write the acceptance tests as a separate
> story, which works as well.

I agree with all of this. In any given established project, people will
have roles, tests will fall into categories. The noodling about I'm
doing in testing follows the motto in my signature: "Act always so as
to increase the number of choices". When projects are in the mood for
thinking about roles and kinds of tests, it will help them if they have
examples of alternatives, worked out in at least a little detail. They
might make their choices after thoroughly understanding "book" XP
practice, or they might be more Crystal/Scrum-like and do more up-front
tailoring. That, I think, is a matter of temperament.

What I really would like to see are examples: stories of agile projects
where users wrote acceptance tests, stories of projects where
programmers wrote acceptance tests, stories of projects where testers
wrote acceptance tests. And stories of projects where testers within
the project maintained some of their conventional distance in order to
(as you say) look at things from a different point of view, plus
stories of projects that took alternate paths toward getting that
different point of view.

I'm toying with the idea of a book or other project that will somehow
collect and present such stories.

-----
Brian Marick
Consulting, training, contracting, and research
Focused on the intersection of testing, programming, and design
marick@..., marick@...
www.testing.com, www.visibleworkings.com

"Act always so as to increase the number of choices." -- Heinz von
Foerster

On Friday, December 27, 2002, at 11:27 AM, Michael Silverstein wrote:
> But, Michael: most people are working in languages like Java and C++.
> Languages that are more rigid, without interpreters and without eval.
> Faster to run, slower to program.  
>  
> Not to turn this into a language debate but over the years, the
> leading Smalltalk VMs have become very well optimized. A good
> Smalltalk VM such as IBM's VisualAge or Cincom's VisualWorks will
> perform very favorably compared to Java - especially in the GUI
> responsiveness, so the statement that they are faster to run isn't
> necessarily true under all circumstances. An interesting side note
> before this line goes completely off on a tangent is that the IBM/OTI
> people who created the speedy SWT for Eclipse used many of the same
> principles they employed years earlier when developing the GUI
> framework for VisualAge Smalltalk, which is still much more responsive
> than AWT or Swing.

You're right. I was wrong. The snarky "faster to run, slower to
program" crack is facile in many ways.

> As a developer I prefer to stay within the language I'm developing in.
> One of the reasons our Smalltalk GUI testing tool is very successful
> with developers is that it generates playback statements in
> Smalltalk. On the other hand, my company has been a proponent of
> providing a means for non-developers to compose component tests from
> objects that know how to test, which clearly departs from the idea of
> writing test code in the target application's language. For
> information in this see
> http://www.silvermark.com/documentation/
> developerQACollaboration.pdf and
> http://servlet.java.sun.com/javaone/javaone2000/pdfs/TS-708.pdf, which
> we've embodied in a Java component testing tool at
> http://www.silvermark.com/Product/java/tmJava.html.

I hope you'll consider coming to the workshop.

---
Brian Marick
Consulting, training, contracting, and research
Focused on the intersection of testing, programming, and design
marick@..., marick@...
www.testing.com, www.visibleworkings.com

"Act always so as to increase the number of choices." -- Heinz von
Foerster



-----
Brian Marick
Consulting, training, contracting, and research
Focused on the intersection of testing, programming, and design
marick@..., marick@...
www.testing.com, www.visibleworkings.com

"Act always so as to increase the number of choices." -- Heinz von
Foerster

Call for Participation

Fourth Annual Austin Workshop on Test Automation (AWTA)
January 25-26, 2003 - Austin, Texas
http://www.pettichord.com/awta4.html

Building and Using Test Interfaces

This is a revised call that includes an expansion of scope from the
previous call.

What interfaces should we use to test software? One tradition uses
Graphical User Interfaces (GUIs), typically with GUI testing
tools. Another tradition uses the programming interfaces provided by
the methods and functions of the software under test -- often used for
unit testing. This workshop will explore other methods for providing
interfaces to software to support testing. One alternative we'll
explore is creating and utilizing interfaces for inter-process
communication (IPC) using the many interface technologies that are
maturing, such as XML-RPC, XMI, SOAP, COM, or Java RMI. A related area
of interest is the the practice of embedding interpreters into
products to support testing scripts. We are also interested in the use
of API's, test parsers, and embedded test fixtures.

Possible topics include:

- How have you created and used such interfaces?

- What are technical challenges to creating such interfaces?

- What are social or organizational challenges to utilizing such
    interfaces?

- What are the benefits and drawbacks to these interfaces, as compared
    to the alternatives?

Brian Marick's position paper:
http://www.testing.com/writings/Marick-AWTA-position.pdf


Workshop Goals

1. Exchange concrete techniques and approaches regarding creating and
     utilizing such interfaces.

2. Encourage the development of published reference implementations.

3. Understand how using such interfaces for testing affects the roles
     of developers, testers and other members of a development team.


Attending the Workshop

Participation in the workshop is by invitation based on a position
paper. The workshop is limited to 15 participants. Your position paper
should have two parts.

1. Experience Report. Describe your experience using IPC or
     embedded-language interfaces for testing. Relevent experience may
     include other kinds of interface technologies than those listed in
     this call.

2. Position Statement. State something that you think people should
     know. This may be a technique for providing test interfaces, a
     reference implementation you would like to demonstrate, a challenge
     that such interfaces may run into, or a reason for why further
     investigation is warranted.

So far we have received inquiries both from talented testers who are
concerned that they may not have sufficient technical background and
from perceptive developers who are concerned whether they will fit
in. The topic of this workshop is challenging because it deliberately
cuts across traditional boundaries. We encourage submissions from
interested testers and developers who are open-minded and interested
in learning from others. We expect that some participants will
only have tangential experience with the topic under discussion.

Position papers should be between one and three pages long. After
acceptance, they will be posted to a private web site that will be
shared by other participants. Thus, a web-based format is
preferred. We can publish text, HTML and PDF. We can also generate PDF
from MS Word files.

Position papers should be submitted to Bret Pettichord
(bret@...). Papers will be reviewed on a rolling basis,
with replies in three days or less. The workshop website will
indicate when the workshop has been filled.


What Will Happen at the Workshop

The workshop will be organized as a moderated discussion, following
the format of the Los Altos Workshop on Software Testing
<http://www.kaner.com/lawst.htm>.

Participants will be selected to present particular techniques and
experiences of using test interfaces. The workshop will explore the
techniques and the conditions that favor or disfavor the technique.

We expect that subthemes will emerge from this discussion. If
suitable, we hope to for subgroups to explore these themes and report
back to the larger group.


Expenses

Participants are responsible for own travel and lodging.  The workshop
expenses, including some meals, will be covered the workshop sponsors.


The Organizers

Bret Pettichord is the founder of the Austin Workshops on Test
Automation. He is an independent consulting specializing in testing
and test automation. He has had several opportunities to test software
that had test interfaces and would like to see more developers provide them
and more testers use them. He is co-author of *Lessons Learned in
Software Testing* and editor of the TestingHotlist.com.

Brian Marick specializes in code-based software testing. He is
currently developing a reference application that has an IPC interface
for testing. He is the author of *The Craft of Software Testing* and
technical editor for STQE magazine.

Cem Kaner is professor of Computer Sciences at Florida Tech, where
he's developing a curriculum for training test architects. He is the
founder of the Los Altos Workshop for Software Testing (LAWST), and
lead author of *Testing Computer Software* and *Lessons Learned in
Software Testing.*

Barton Layne is a testing consultant based in Austin, Texas.


Time Frame

Sat Jan 25, 9 am to 5 pm
Sun Jan 26, 9 am to 3 pm

The workshop will start promptly at 9 am on Sat Jan 25. Participants
from out of town should plan to arrive the night before. There will be
a welcoming dinner on Friday Jan 24; participants are welcome to
invite family, friends and colleagues to dinner.

We expect all participants to attend for Saturday from 9 to 5 and
Sunday from 9 till noon. Prospective participants who won't be able to
attend for this time should so indicate when they submit their
position papers. Usually most participants plan to attend until 3 pm
on Sunday. We'll have the space until 5 pm; often a subgroup will be
active until then.


Location

The workshop will be located at the Homewood Suites in Austin,
Texas. We used the same location for AWTA3 and the participants really
enjoyed the cozy setting.

Homewood Suites
10925 Stonelake Blvd
Austin, TX 78759
512-349-9966
homewoodsuitesaustin.com

Upon invitation, mention Pettichord Consulting to get a group rate of
$79 a night, available through Jan 5.

On Thursday, December 26, 2002, at 3:47:04 PM, Bret Pettichord wrote:

> As a testing consultant, I've worked with a couple teams using XP. And i've
> also talked to a number of other testers on XP projects.

> The teams that i worked with had the most trouble with the user stories.
> They had trouble defining and planning their tests and managing progress.
> Ultimately i attributed to the problem to the way that the managers and
> developers had defined the user stories -- or rather not defined. In once
> case, they labelled features as "user stories," in the other case, they
> labelled tasks as "user stories." In both cases, this gave the programmers
> enough information to go ahead a do their programming, but the testers
> didn't really know how to test. The stories were either too general
> (features) or not described in user terms (tasks), so it took a lot of work
> for the testers to figure out what good tests looked like.  Also, not all
> of the programming work was driven by user stories.

> It wasn't until i'd had a chance to confer with others using XP, that i
> came to realize that this was the core of their problems. It's very
> important for the user stories to be concrete, meaningful to the customer
> and specific. I can recommend two books on this topic, *Planning Extreme
> Programming* by Beck and Fowler, and *Writing Effective Use Cases* by
> Cockburn, which describes the relationship between use cases and user
> stories (they are very similar).

> Another problem that we had was that the use of unit testing was
> inconsistent. It's important to have some means to make sure that all the
> programmers are writing their unit tests. (Pair programming is one means,
> but both my teams were weak on this.)

Good points all. I'd not be surprised to find that a team trying XP was having
trouble if its stories were not customer-value focused, if they were doing
things that were not story driven, if they weren't writing tests, and they
weren't pairing.

Similarly, I would expect that a team would have trouble playing football
without goals, with some of the people running off in their own direction, and
with had no concept of how to score the game. ;->

In spite of all the words we all write, it seems to be hard for people to learn,
by reading, all the things we can so easily show them. Maybe that's why real
change takes so long.

Ron Jeffries
www.XProgramming.com
It's easy to have a complicated idea. It's very very hard to have a simple idea.
   -- Carver Mead