This month at Extreme Programming San Diego (XPSD), Dan and Llewellyn will
teach us how to write Approval Tests. The meeting will take place at
Cardinal Health at 6pm on Thursday, November 6th.

** Approval Based Tests **

Write your code, run it, see that it works. This is a familiar process to
all programmers. The addition of 1 line of code can now make that a
repeatable automated regression test!

Well tested code has been shown, over and over, to improve the quality and
maintainability of software. Yet most projects are not well tested. Why is
this? Too often programmers feel that testing imposes too high of an
overhead on writing code.

In this talk, our goal is to make it easier and faster for you to write unit
tests.

We will show how using approval tests, in addition to asserts, will great
increase not only the ease and speed of writing a test, but the completeness
and maintainability of those tests; bridging the gap between people who
don't write tests, and test driven developers.

"I feel like I have to write my code twice." Right now, unit tests impose a
difficult step upfront of knowing exactly what you want. Asserting this
outcome then becomes cumbersome the more robust & detailed it is. After
that, maintaining the test becomes equally cumbersome. Because of this,
tests either get written, or get discarded, leaving untested code which is
fragile, error prone and leads to the nightmare of maintenance so many
experience.

After this talk, you should find yourself spending less time writing tests,
yet having better tested code, a proven way to increase successfulness of
your software.

"After I heard Dan & Llewellyn talk about approval tests, I tried it out. in
the first 4 minutes I was testing my code" - C. Monkey

"I find myself writing more code just so I can write more approval tests!" -
Script Kiddie

"All my life I'd sought approval from other developers, now I can approve on
my own" - Noself E. Steam

"With all the time I saved, I actually learned to talk to women" - Anonymous


The meeting will be held at Cardinal Health San Diego Headquarters (
http://xpsd.org/CardinalHealthDirections ) from 6-7:30 PM on Thursday
November 6th.


http://xpsd.org
http://llewellynfalco.blogspot.com/2008/10/approval-tests.html

This month at Extreme Programming San Diego (XPSD), Dan and Llewellyn will
teach us how to write Approval Tests. The meeting will take place at
Cardinal Health at 6pm on Thursday, November 6th.

** Approval Based Tests **

Write your code, run it, see that it works. This is a familiar process to
all programmers. The addition of 1 line of code can now make that a
repeatable automated regression test!

Well tested code has been shown, over and over, to improve the quality and
maintainability of software. Yet most projects are not well tested. Why is
this? Too often programmers feel that testing imposes too high of an
overhead on writing code.

In this talk, our goal is to make it easier and faster for you to write unit
tests.

We will show how using approval tests, in addition to asserts, will great
increase not only the ease and speed of writing a test, but the completeness
and maintainability of those tests; bridging the gap between people who
don't write tests, and test driven developers.

"I feel like I have to write my code twice." Right now, unit tests impose a
difficult step upfront of knowing exactly what you want. Asserting this
outcome then becomes cumbersome the more robust & detailed it is. After
that, maintaining the test becomes equally cumbersome. Because of this,
tests either get written, or get discarded, leaving untested code which is
fragile, error prone and leads to the nightmare of maintenance so many
experience.

After this talk, you should find yourself spending less time writing tests,
yet having better tested code, a proven way to increase successfulness of
your software.

"After I heard Dan & Llewellyn talk about approval tests, I tried it out. in
the first 4 minutes I was testing my code" - C. Monkey

"I find myself writing more code just so I can write more approval tests!" -
Script Kiddie

"All my life I'd sought approval from other developers, now I can approve on
my own" - Noself E. Steam

"With all the time I saved, I actually learned to talk to women" - Anonymous


The meeting will be held at Cardinal Health San Diego Headquarters (
http://xpsd.org/CardinalHealthDirections ) from 6-7:30 PM on Thursday
November 6th.


http://xpsd.org
http://llewellynfalco.blogspot.com/2008/10/approval-tests.html

On Oct 11, 2008, at 7:41 AM, ae1fric wrote:

> I'm considering switching my development over to Python under
> Apple's Xcode development
> environment -- Cocoa and Core Data provide some significant
> advantages for the type of
> development I'm doing. The one downside so far is that I haven't
> been able to figure out how
> to do TDD under Xcode.
>
> Has anyone successfully developed with TDD under Xcode?

Yes, there are actually quite a few people who do extensive TDD on Mac
OS X using Cocoa and Xcode — it turns out that it's actually a great
platform for doing so, thanks to its objects-all-the-way-down (for the
most part) design.

Disclosure: I work on the Xcode IDE at Apple, but I don't speak for
Apple or the Xcode team here or anywhere else online.  (Unless you
happen to be watching me in a WWDC video...)

> I think the reason I'm struggling to see how to do TDD under Xcode
> is that the development
> environment does so much automagically: you define your data model,
> draw your user
> interface, define bindings between the data model and the UI and
> suddenly you have a
> working application, without writing a line of code.
>
> Any suggestions for using TDD in this kind of environment would be a
> great help.

You can actually still do TDD by writing unit tests that specify how
your data model should be set up, and how the connections between
objects in your nib files -- whether those connections are target-
action, outlet or binding connections -- are set up.

Virtually everything that Xcode, Interface Builder and Cocoa let you
do graphically can also be done in code.  None of them *generate* code
to do so, but the objects that are created are the same either way, so
you can just introspect them.

For example, lets say you want to create a View Controller that has a
Name field for an employee, which is managed by an object controller.
You can then write a test that you want the NSTextField's value
binding to be connected to the "selection" controller key path and
"name" model key path of an Employee Controller.

Here's an Objective-C test method that does just that written against
Sen:te's OCUnit framework that's included with Xcode.  Using PyObjC
and a Python testing framework should be the similar except for the
obvious differences in syntax and setting up your own type of test rig.

// This assumes viewController has nameField and employeeController
properties for
// the appropriate IBOutlets, and is configured in -setUp
- (void)testNameFieldBoundToName {
      NSDictionary *info = [viewController.nameField
infoForBinding:NSValueBinding];
      STAssertNotNil(info,
          @"The name field's value should be bound.");

      NSObjectController *observedController = [info
objectForKey:NSObservedObjectKey];
      STAssertEquals(observedController,
viewController.employeeController,
          @"The name field's value should be bound to the employee
controller.");

      NSString *observedKeyPath = [info
objectForKey:NSObservedKeyPathKey];
      STAssertEqualObjects(observedKeyPath, @"selection.name",
          @"The name field's value should be bound to the 'selection'
controller key "
          @"and 'name' model key of the employee controller.");
}

I've usually created my own assertion macros to encapsulate this kind
of test, of course; then the test winds up looking like this:

- (void)testNameFieldBoundToNameWithBindingAssertionMacro {
      MyAssertObjectIsBoundThroughKeyPath(viewController.nameField,
NSValueBinding,
          viewController.employeeController, @"selection", @"name",
          @"The employee name field should be bound to the employee
controller.");
}

Note that this isn't specifying the layout of the text field in my nib/
xib file, its font, or anything like that. You can specify those in
tests too, if you want to, but you don't have to.  Whether you do so
or not depends on the level of detail you need for your UI and the
level of churn that it's likely to go through.

For example, if you're changing the look and feel or layout of your
UI, you might not want to specify any of that, but the above test
would probably be very useful in ensuring you don't forget to bind up
a text field you've just replaced as part of iterating on your UI
design.

On the other hand, I wrote some code a while back to create a "glass
bar" like at the bottom of the message list in Mail.app on Leopard.  I
actually did write tests to specify the layout of that interface down
to the point, despite creating it in Interface Builder, because it
needed a very specific layout and I didn't want to accidentally change
it as I adjusted other things in the xib.

One other thing this doesn't do is "push" events through controls to
test that your controller-level code reacts to them.  In general you
don't need to write such tests except for your own custom controls.
Otherwise, you can trust that controls will work correctly as long as
they're connected correctly, and just verify that those connections
are set up right in your tests.

The same philosophy can be followed when writing tests that describe
your data model.  An NSManagedObjectModel can be loaded
programmatically and you can just ask for its entities and their
properties, and walk across the entire thing looking at their values
in code.  That makes it very easy to TDD because you can just create
an infrastructure of tests that say "this model should have an entity
named Employee" and "the Employee entity should have a required name
attribute" and "the Employee entity should have a to-one 'department'
relationship to the Department entity, with an 'employees' inverse
relationship."

Here are some of the blog posts I've written on the topic of writing
tests for Cocoa over the years:

	 Xcode: Unit testing Cocoa frameworks
	 http://chanson.livejournal.com/119303.html

	 Xcode: Debugging Cocoa framework unit tests
	 http://chanson.livejournal.com/119578.html

	 Xcode: Unit testing Cocoa applications
	 http://chanson.livejournal.com/120263.html

	 Xcode: Debugging Cocoa application unit tests
	 http://chanson.livejournal.com/120740.html

	 Trust, but verify.
	 http://chanson.livejournal.com/118380.html

	 Unit testing Cocoa user interfaces: Target-Action
	 http://chanson.livejournal.com/148204.html

	 Unit testing Cocoa user interfaces: Cocoa Bindings
	 http://chanson.livejournal.com/172390.html

	 Unit testing and Core Data
	 http://chanson.livejournal.com/115621.html

	 My blog's "unit testing" tag
	 http://chanson.livejournal.com/tag/unit+testing

Erik Doernenburg, the creator of the useful OCMock mock-objects
framework, has also written a bit about how to use OCMock to test
Cocoa controllers.

	 Testing Cocoa Controllers with OCMock
	 http://erik.doernenburg.com/2008/07/testing-cocoa-controllers-with-ocmock/

His approach is very appropriate for testing that your own controller
objects react to changes in your UI correctly, again without having to
"push events" through controls.

Let me know if there's anything else you'd like me to write about!

And if there are any ways you'd like to see Xcode, Interface Builder,
or Cocoa improved, or if you run into any bugs or other problems,
please file an enhancement request or bug report at http://bugreport.apple.com/
   — bugs and feature/enhancement requests go into Apple engineering's
Radar system and are evaluated by engineering and management.  Even
"duplicate" bug reports are extremely useful because they can wind up
having the critical nugget of information that is needed to fix a bug
(by making it reproducible or revealing a common thread), or can be
used to gauge demand for specific kinds of enhancement.

    -- Chris



[Non-text portions of this message have been removed]

George Dinwiddie wrote:

> Phlip wrote:

>> I thought the industry would support more XPath testing by now. What
libraries
>> are out there (besides mine?)?

> HtmlUnit, for one.  You won't like it, though, as it's written in Java.

That wouldn't be very politic of me!

Here's the first hit:

http://www.google.com/codesearch?q=package%3Ahtmlunit+xpath

htmlunit-plugin-0.3-src/.../plugin/htmlunit/HtmlUnitFunctionTagTest.java

      75:   functionTag.navigate("file:./tst/html/tables.html");
            String xpath = "//head[title='tables page']";
            functionTag.assertXPathMatches(xpath);

      85:   MockHtmlUnitHelper helper = setMockHelper();
            String xpath = "//head[title='tables page']";
            functionTag.clickElementWithXPath(xpath);

Firstly, where did the @ go? That should be @title = 'tables page' in the
predicate.

Next, I want to nest them. Suppose we have a report table which must contain a
certain HD. I want this:

    assert_xpath :table, :report do  #  hits //table[@id='report']
      assert_xpath 'tbody/tr/td[ . = "SubTotal" ]'
    end

The do-end lets us avoid writing a run-on XPath expression. And if the inner
assert_xpath fails, its error diagnostic will only print out its current context
- the outer <table> - not the whole page.

Then you can stack up more assertions inside the block. The goal is not to
mirror your HTML in the tests, but to efficiently skip over the fuzzy details
that might change.

--
    Phlip

TFUI:

How many here are using XPath to test into HTML, to detect its generated
details?

I think the book /Working Effectively with Legacy Code/ would call that
"sensing"...

--
    Phlip
    http://www.oreillynet.com/ruby/blog/2008/09/polygamous_assert_latest.html

Pekka Laukkanen wrote:

> 1) How do you automate testing browser compatibility through the
> business layer? Or do you plan to do that manually for all the
> browsers on all operating systems?

Then you are testing the GUI directly - the test case is close to the tested
case.

Then, if such a test fails (conceptually or literally), you fix it by changing
code in your real unit tests. For example, suppose your HTML accidentally
duplicates IDs - which is forbidden - and automated (or manual) testing
discovers your Javascript does not get the right object thru its ID on some
browsers.

You naturally improve the GUI tests in that area. But the bug is not fixed until
you have improved the unit tests. By "unit" I mean tests that...

   - mock the server, so no HTTP or browser get in the way
   - interpret the HTML that would have gone out the wire
   - parse the HTML with queries, such as XPath
   - assert the HTML does not contain various tell-tale signatures

That way, you improve the confidence that a developer's instant unit test
successfully defends behavior real browsers depend on - even if test finds that
actual behavior inconceivable.

So an HTML test could validate entire pages, such as with Tidy, to fail if any
IDs get duplicated. Then an XPath test could find the specific IDs that matter -
no matter where they moved on a page - and could verify their internal details.

> 2) How do you automate full end-to-end tests if you only talk to
> business layer? End users don't really care if the business logic
> works flawlessly if the UI to it doesn't work.

Sometimes a GUI test covers business logic. I took care not to say they never
should. But, once again, there had better be a unit-level test that covers the
exact same business cycle.

> For me the ideal situation is where requirements can be expressed in
> natural language, and executed as acceptance tests against different
> interfaces by changing the driver that is used. This way same
> requirements/tests can be executed both against the business layer
> (fast, might be ready before the UI) and the actual user interface
> (real end-to-end test). In web-testing world this can even be extended
> so that same tests can be executed using a browser driver (e.g.
> Selenium RC, Watir) or a driver working on HTTP/HTML level (e.g.
> HTMLUnit).

Awesome idea - I didn't even think of it like that.

I suspect that some people abuse GUI tests to cover internal business logic.
That's an antipattern...

--
    Phlip

aidy lewis wrote:

>  True, but if we implement story or scenario driven development then
> these tests lead development. A change in the tests means a change in
> development - not the other way round!  I know for sure you are not
> saying; not automate through the browser and validate user
> requirements?

Two big points:

   - the first tier of tests must cover as many features as possible, and
       be super-fast to run

   - test cases must be as close as possible to the tested code.

Put them together, and you should only test GUI things thru the GUI. Never test
biz logic only thru the GUI.

These rules imply "acceptance tests" are _not_ "thru the GUI tests". Pro testers
must work the test rig itself to add their coverage.

--
    Phlip

Shadowfirebird wrote:

> I've found this lovely bit of example code for mocha, though -- see
> below.  If I understand you correctly, you seem to be saying that you
> *don't* test the output routine; you use a mock to fake it so that you
> can test the rest of the code?

Consider a test T that calls A, which calls B.

Sometimes, a B is cheap to assemble, so T can assemble B, activate A, and test
that A did its thing. T considers B's behavior just a side-effect.

Most of the time, B should be real. And its state should be static (such as a
Rails "fixture", with static data.) B should not have runaway dependencies. The
test T should be easy to set-up.

You should mock B if it's too expensive to set up. For example, if B reads the
system clock, and if T would prefer the date is 2008 July 31, the test T should
not wait an infinite amount of time, until the cosmos oscillates and 2008 July
31 occurs again. Tests should run as fast as possible, to avoid any hesitation
running them.

You should mock B, so it behaves as if the date is correct. Or you should mock
(in Ruby) Time.now, so all Ruby methods that use the date will read the mocked
date.

Other examples of things too expensive to directly test:

   - live users
   - random numbers
   - hardware - networks, robots, tape drives, the clock, etc
   - system errors

If your B object is not on the list, you should not mock it. Unit tests work
best when they cross-test everything. The only thing better than a test that
fails because A broke is many tests that all accurately fail because B broke. If
your B is too expensive to assemble, you should refactor it, so it bypasses the
behavior that T and A did not care about.

> class Enterprise
>   def initialize(dilithium);  @dilithium = dilithium;  end
>
>   def go(warp_factor);  warp_factor.times { @dilithium.nuke(:anti_matter) };
end
> end

Very nice. And note it obeys my list, by mocking both hardware and space-time
distortions.

--
    Phlip

David Mitchell wrote:

> The reason I need to do this is that we've got a small Watir-based DSL
> written to allow us to drive an app through code that looks sort of
> like:
> login("fred", "password")
> click_tab("Reports")
> click_drilldown("Asia")
> open_report("Some report title")
> ..
>
> It essentially lets us construct test cases in something like plain English.

Do your programmers run that after every edit?

> We built a few test cases using the DSL with a "normal"
> Test::Unit::TestCase approach, then showed them to our testers.
> Everyone was pretty excited about it; we can generate our own test
> data using the DSL, the testers can comprehend the DSL without having
> to dig into the nuts and bolts of the application itself, we can
> finally build a full regression test suite for an application that's
> basically a pig to drive using normal automation testing tools like
> QTP, the scripts we write using the DSL are easy to maintain over
> time, and everyone's happy.

Asking the question another way - do your developers write any tests?

> Once our testers got a look at that, they pointed out what should've
> been obvious all along: we can now get actual business users to write
> a lot of the test cases using the DSL, rather than using specialist
> testers.  Rather than writing huge business requirements documents
> that have a habit of getting misinterpreted, we can get the business
> users to create what are essentially test cases using the DSL, and
> that gives the developers a reasonably unambiguous description of how
> things are supposed to work - we'll save a whole lot of time and money
> we're currently wasting translating between business-speak and
> developer-speak.

Awesome! Now, can your business side actually run the tests - such as thru a web
site with a "what if" interface?

> The only problem was the "scaffolding" code; apparently business users
> are incapable of writing/extending code that looks like
>
> class Testcases < Test::Unit::TestCase
>   def test_1
>     <<DSL stuff here>>
>   end
>   def test_2
>     <<more DSL stuff here>>
>   end
>   ...
> end
>
> but they are capable of creating a bunch of test cases in individual
> files that contain nothing but the DSL commands.  They'll use e.g.
> Notepad to create test cases in individual files that look like:
> login('fred', 'password')
> click_tab('Reports')

That sounds like Fitnesse's territory. It does the Notepad thing, but with a
real GUI around it. Your customer team writes the test criteria in a DSL, and
FIT acts as a test runner.

> Fine with me - I just work here...
>
> So now I've got a situation where we're going to have business users
> generating loads of test cases using our DSL (without any of that
> nasty complicated Test::Unit::TestCase stuff), saving them in flat
> files, and we need to run be able to run some unspecified number of
> test cases that will change over time.  What I need to be able to do
> is something like:
>
> Dir.glob("app_test_cases/**/*.app).each do |test_script_file|
>   <<grab the content of each file, build a new Test::Unit::TestCase
> wrapper around it and eval it>>
> end

Now the slight problem is you are using TestCase as your runner, when it's full
of features you don't need, and thin on features you actually do need. More
below.

> That's no problem - I've got most of this working already; all I
> needed was the way to dynamically add new test cases, and you've now
> given me a way to do that.  I'll have a play with it tomorrow.

I use define_method, in Rails, like this:

    [all this controller's actions].each do
       define_method test_one_action_#{action} do
         # test one common thing
       end
    end

The first thing you need to look for is if your failures are humane, or if they
are a huge mass of developer-friendly diagnostics and stack traces. The great
thing about a DSL (per RSpec) is (reputedly!) that faults can lead with clear
English too: "The frob should have returned 42 but it returned 43".

--
    Phlip

[CCd to testfirstuserinterfaces@yahoogroups.com ]

Baris Evrim Demiröz wrote:

> From the first time I saw XP I thought it is a very good methodology.
> But I have a questions about unit testing.
> As you may already know, in image processing most of the things work
> with a probability. Let's say you have a method that finds the
> location of faces in a given image, and this algorithm works well 98%
> of the time, and gives false trues (a location that doesn't contain
> any face). How do you test such a method?

Study the source to Cairo (part of Gtk+). It uses a system called "golden master
test". Each test case (at this level) draws a little graphic, such as one
diagonal line. Then the test case byte-compares the image to a png file.

If the test output is right and the png file is wrong, engineers replace the png
file with the new output. That's why the master files are "golden" - they are
part of your versioned source.

This system won't prove you match faces. TDD cannot test that crypto makes your
app bulletproof, or that compression absolutely minimizes your files, or that a
Turing loop will never halt.

This system should stop you if you make a change too big for its tolerances.
This frees your edits up to make whatever change you like. Get a battery of
known faces, and test that they all get recognized to within their expected
variations.

When you actually TDD, you are simply matching tests to low-level code, to check
it manipulates its bits correctly. Most of your tests should run below the level
of the faces.

--
    Phlip

I think neither, but perhaps focus on problem forms (?) first.

Actually, I would focus on new stories first, then problem forms second.

I would work new stories "presenter-first" and get my TDD working from the
start.
For problem forms, I would get my unit tests in order including refactoring
to MVP if required.

Lastly, I would get my acceptance/story tests running in the same area...
plugging in at the presenter.

.rob.

On Mon, May 26, 2008 at 11:37 PM, Stephan Meyn <smeyn@...> wrote:

>   Just a question to the group - if you are looking to start automating
> testing, you may not have the opportunity to invest in automating all
> you would like to test. So you look what kind of tests you'd want to
> invest in to optimize the benefits vs effort.
>
> So would you rather automate
> 1) simple tests, such as testing all the navigation works, or
> 2) complex test cases, such as processing a withdrawal of a payment
> from a superannuation account
>
> 1) would give you the ability to create a large number of small test
> cases, which could then be run after a build as a smoke test. Catches
> errors before the test team gets involved
>
> 2) automates stuff that are error prone to do manually. But it
> requires more effort to build and so you would create less tests with
> the same budget.
>
> given the two choices, which way would you go?
>
> Stephan
>
>
>


[Non-text portions of this message have been removed]

Just a question to the group - if you are looking to start automating
testing, you may not have the opportunity to invest in automating all
you would like to test. So you look what kind of tests you'd want to
invest in to optimize the benefits vs effort.

So would you rather automate
1) simple tests, such as testing all the navigation works, or
2) complex test cases, such as processing a withdrawal of a payment
from a superannuation account

1) would give you the ability to create a large number of small test
cases, which could then be run after a build as a smoke test. Catches
errors before the test team gets involved

2) automates stuff that are error prone to do manually. But it
requires more effort to build and so you would create less tests with
the same budget.

given the two choices, which way would you go?

Stephan

> Thanks Steve, that's a great point! A training that really focus in an
> improvement for the organization.
> We have had several meetings for identifying the more reiterative
> problems and found between them:
> - Too long times writing test cases.

Do your developers run the tests after every 1 to 10 edits, with the highest
odds on 1?

Do your developers prefer to revert when a test fails unexpectedly (and the
reason is not simple & obvious, preferably appearing in the fault diagnostic)?

> - Too formal test cases that include redundant GUI testing (this
> increases the writing time).

Are the majority of your cases all unit tests that run 1 or 2 calls above the
tested code?

Could you write a new test to reach any line of code, anywhere, at any time,
without excessive research to determine how to reach that line?

> - Not much use of automation tools (only 5 of the team use QTP), this
> is mostly because of budget limits (that's why I´m trying to find an
> OpenSource alternative).

Can you test the GUI from within, by writing tests in the same language as the
GUI?

> - We don't have knowledge on security testing.

Do you TDD your security code?

No test can "prove" your code is secure. TDD tests should spot-check that your
algorithm works as designed. For example, at my day-job, we run an MD5 checksum
hash on today's date with a pass phrase and a >cough<. To test this, we
trivially mock the Date module to return a fixed date. Then we trivially compare
the result to a hard-coded checksum. As copied from the fault diagnostic on the
day we wrote this test.

Yes, the NSA could crack this with one hemisphere tied behind their backs. Our
tests allow us to freely refactor all the code in and around the security
system, without changing those odds.

aidy lewis wrote:

> Could anyone suggest any low-weight tools for .Net WinForm GUI
> testing, either open or closed source?

Can you treat the forms as objects, and just access their properties from normal
unit tests?

--
    Phlip

TFUIers:

Ideally, when your developer tests call production code, it should not spew
output to your STDOUT stream. If it must emit strings, it should return them
to its caller. That way, your main() methods have the option to spew the
strings, and your tests can instead intercept the strings and assert_match
their contents.

Real life is often less than ideal. Your production code might couple with
legacy code. Or your production code might require some out-of-band spew,
such as a "syntax" or "usage" message.

Ruby makes this situation delightfully easy to test, by intercepting the
STDOUT itself. Some languages would force you to manipulate the raw STDOUT
file descriptor. Ruby simply permits you to replace your $stdout object with
one that implements .write.

So, without further ado, here's the assertion to trap and match your STDOUT
stream:

   class BufferStdout
     def write(stuff)
       (@output ||= '') << stuff
     end
     def output;  @output || ''  end
   end

   def assert_stdout(matcher = nil, diagnostic = nil)
     waz = $stdout
     $stdout = BufferStdout.new
     yield
     assert_match matcher, $stdout.output, diagnostic  if matcher
     return $stdout.output  # for if you need the whole string
   ensure
     $stdout = waz
   end

   def deny_stdout(unmatcher, diagnostic = nil, &block)
     got = assert_stdout(nil, nil, &block)
     assert_no_match unmatcher, got, diagnostic
   end

With further ado, here's the assertion in action:

   def test_help
     assert_stdout /invalid.*argument.*
                    verbose.*=\>.*:false/mx do
       assert_efficient_sql(:help){}
     end
   end

That test case demonstrates that another assertion, assert_efficient_sql,
will spew a helpful error to $stdout if you call it with an invalid
argument.

Modulo my typical life-issues, we will soon see more of that
assert_invalid_sql. But it only works with MySQL.

If you use it with another database adapter, it should not fail; it should
simply warn (spew) that it honestly cannot determine your SQL's efficiency.

To test that, we write a case which installs another adapter, then detects
that the warning only appears once:

     assert_stdout /requires MySQL/, 'warn'  do  assert_efficient_sql  end
       deny_stdout /requires MySQL/, 'once'  do  assert_efficient_sql  end

That shows deny_stdout in action, and the 'diagnostic' parameters.

--
  Phlip
  http://www.oreilly.com/catalog/9780596510657/
  ^ assert_xpath

Mark Waite wrote:
> --- George Dinwiddie <lists@...> wrote:
>
>> Ulrich Freyer-Hirtz wrote:
>>> As I understood your GUI is a Swing GUI. Therefore you can try
>>> jfcUnit (open source). I love to use qfTest from Quality First
>>> Software (http://www.qfs.de/) with Swing-GUIs. It's not costly and
>>> the money comes back soon through the efftive way of desinging the
>>> tests.
>> For Swing, I recommend Abbot over jfcUnit (see
>> http://blog.gdinwiddie.com/2007/04/25/tdd-and-java-swing/).  You can
>> also use the Swing wrapper of JBehave
>>
> (http://blog.gdinwiddie.com/2007/05/09/making-tdd-and-java-swing-behave/).
>>   It's very nice, but not as full-featured as Abbot.  Abbot has a bit
>> more learning curve (at least for me) because some techniques weren't
>> obvious to me.  Be sure to see the comments on those entries.
>
> I used Abbot to create a regression test for a Swing application
> whose user interface was not changing but whose implementation
> was changing.  While developing the Abbot tests, I often found that
> I wanted to "program" something, but because I had initially chosen
> to record the tests, I had tests written in XML which were not ready
> to be refactored, extended, or changed.
>
> We left those tests alone, grateful that they continued running.  They
> continued that way for a year or more.
>
> When we switched to JDK 6, the tests stopped working.  Rather than
> dig deeply into the cause of the failure, I decided to try a different
> automation tool in a scripting language I prefer.  I switched to
> Marathon (http://sourceforge.net/projects/marathonman) because it
> lets me record scripts, the recorded scripts are written in Python,
> and I'm comfortable with Python.  Some of my initial observations are
> http://blog.360.yahoo.com/blog-4B.afOU8erSVU28a2FPKRNU-?cq=1&p=215
>
> The switch was about 2 days worth of work, including recording and
> refactoring the tests to keep them simple, and switching our test
> infrastructure to run the Marathon based tests instead of the Abbot
> based tests.
>
> Others will probably have different experiences than I had.  I know
> that Abbot has a programming interface which allows Java programmers
> to code their UI tests in Java.  I assume that is preferable for Java
> programmers in test, since they can use a familiar language.  In my
> case, Marathon was a better fit because I was more comfortable with
> Python than with Java, and I didn't want to get any "closer" to the
> Java GUI implementation than was absolutely necessary.
>
> The switch to Marathon has not been trouble free, but the two
> Marathon questions I've asked on the sourceforge forums have been
> addressed promptly and very well.
>
> This is not any statement against Abbot's style of testing, I think
> it is a great program, and my first experiences with it a year or
> two ago were quite positive.  As I worked with it further, I found
> that I wanted more programmable tests than I was getting with my
> chosen implementation path on Abbot.

Mark, that's something I hadn't thought about as I've never used the
record/playback form of testing with Abbot.  I used it for TDDing the
GUI, so the tests were all programmatic.

I'm curious, have you tried TDD using a testing framework in a different
language than the implementation?  I haven't done so, but I can imagine
it might cause me difficulty.

   - George

[cross-posted, because I think this might be an interesting question on
TFUI, also]

--
   ----------------------------------------------------------------------
    * George Dinwiddie *                      http://blog.gdinwiddie.com
    Software Development                    http://www.idiacomputing.com
    Consultant and Coach                    http://www.agilemaryland.org
   ----------------------------------------------------------------------

Hi,
Please try WebAii from www.artoftest.com
I have used it and it is really good.It can be run inside the NUnit
framework,supports AJAX testing.
And it can be integrated into build process too.

Take care
Ketan

----- Original Message ----
From: Daniel Wellman <wellman@...>
To: TestFirstUserInterfaces@yahoogroups.com
Sent: Thursday, 8 November, 2007 5:36:08 PM
Subject: [TFUI] Re: GWT and Fit (Was: Test-driving ASP.NET)













             GWT code looks like regular Java code except in a few conditions: it

can inline native JavaScript within your Java source, or it can use a

compile-time deferred binding technique to pick code based on your

browser type (Firefox, IE, etc.).  You can write regular Java JUnit or

Fit tests for your application, but as soon as your code invokes

Javascript (instantiating any widget does this) or uses the deferred

binding, you need to run your code in a browser.



GWT ships with a modified browser that can essentially run your GWT

Java code in a JVM instead of JavaScript.  Whenever you need to test

code that needs a browser, Google provides an extension to TestCase

called GWTTestCase.  This class essentially starts up a hidden web

browser and runs your test case in this browser -- as Java or

Javascript.  This works very well, but it is very slow - it takes

several seconds to run a test as opposed to milliseconds.



For my project, I've used JUnit tests with JMock for the presenter

layers above the UI, and when I really wanted to test actual widget or

JavaScript code (JSON parsing is another case that needs a browser),

used GWTTestCase.  I haven't used Fit tests, but I think some folks on

the GWT mailing list may have; the only issue would be getting that

headless browser up when your code required it.



Dan



--- In TestFirstUserInterf aces@yahoogroups .com, Rick Mugridge

<rick@...> wrote:

>

> Hi Dan,

>

> No, I haven't, but I've been keen to try out GWT for some time. I

really

> like the way they manage Javascript. I've been waiting for the widget

> set to become more comprehensive - perhaps it is now?

>

> Now that you mention it, I can see that hooking into the Java-based

> browser runtime is the interesting bit. Is the source available for

> that? Is there a useful testing api?

>

> How have you approached it yourself?

>

> Cheers, Rick

>

> Daniel Wellman wrote:

> >

> > Hi Rick,

> >

> > Have you written Fit tests for GWT? If so, did your tests require one

> > of the fake hosted mode browsers to run to handle widgets or deferred

> > binding?

> >

> > I'm working on a GWT project right now and am interested to share

> > experience stories with other folks.

> >

> > Cheers,

> > Dan

> >

> > --- In TestFirstUserInterf aces@yahoogroups .com

> > <mailto:TestFirstUs erInterfaces% 40yahoogroups. com>, Rick Mugridge

> > <rick@> wrote:

> > > PS, if GWT was used, the Java code intended for translation to

> > > Javascript could be tested with Fit

> >

> >

>

>

> [Non-text portions of this message have been removed]

>














<!--

#ygrp-mkp{
border:1px solid #d8d8d8;font-family:Arial;margin:14px 0px;padding:0px 14px;}
#ygrp-mkp hr{
border:1px solid #d8d8d8;}
#ygrp-mkp #hd{
color:#628c2a;font-size:85%;font-weight:bold;line-height:122%;margin:10px 0px;}
#ygrp-mkp #ads{
margin-bottom:10px;}
#ygrp-mkp .ad{
padding:0 0;}
#ygrp-mkp .ad a{
color:#0000ff;text-decoration:none;}
-->



<!--

#ygrp-sponsor #ygrp-lc{
font-family:Arial;}
#ygrp-sponsor #ygrp-lc #hd{
margin:10px 0px;font-weight:bold;font-size:78%;line-height:122%;}
#ygrp-sponsor #ygrp-lc .ad{
margin-bottom:10px;padding:0 0;}
-->



<!--

#ygrp-mlmsg {font-size:13px;font-family:arial, helvetica, clean, sans-serif;}
#ygrp-mlmsg table {font-size:inherit;font:100%;}
#ygrp-mlmsg select, input, textarea {font:99% arial, helvetica, clean,
sans-serif;}
#ygrp-mlmsg pre, code {font:115% monospace;}
#ygrp-mlmsg * {line-height:1.22em;}
#ygrp-text{
font-family:Georgia;
}
#ygrp-text p{
margin:0 0 1em 0;}
#ygrp-tpmsgs{
font-family:Arial;
clear:both;}
#ygrp-vitnav{
padding-top:10px;font-family:Verdana;font-size:77%;margin:0;}
#ygrp-vitnav a{
padding:0 1px;}
#ygrp-actbar{
clear:both;margin:25px 0;white-space:nowrap;color:#666;text-align:right;}
#ygrp-actbar .left{
float:left;white-space:nowrap;}
.bld{font-weight:bold;}
#ygrp-grft{
font-family:Verdana;font-size:77%;padding:15px 0;}
#ygrp-ft{
font-family:verdana;font-size:77%;border-top:1px solid #666;
padding:5px 0;
}
#ygrp-mlmsg #logo{
padding-bottom:10px;}

#ygrp-vital{
background-color:#e0ecee;margin-bottom:20px;padding:2px 0 8px 8px;}
#ygrp-vital #vithd{
font-size:77%;font-family:Verdana;font-weight:bold;color:#333;text-transform:upp\
ercase;}
#ygrp-vital ul{
padding:0;margin:2px 0;}
#ygrp-vital ul li{
list-style-type:none;clear:both;border:1px solid #e0ecee;
}
#ygrp-vital ul li .ct{
font-weight:bold;color:#ff7900;float:right;width:2em;text-align:right;padding-ri\
ght:.5em;}
#ygrp-vital ul li .cat{
font-weight:bold;}
#ygrp-vital a{
text-decoration:none;}

#ygrp-vital a:hover{
text-decoration:underline;}

#ygrp-sponsor #hd{
color:#999;font-size:77%;}
#ygrp-sponsor #ov{
padding:6px 13px;background-color:#e0ecee;margin-bottom:20px;}
#ygrp-sponsor #ov ul{
padding:0 0 0 8px;margin:0;}
#ygrp-sponsor #ov li{
list-style-type:square;padding:6px 0;font-size:77%;}
#ygrp-sponsor #ov li a{
text-decoration:none;font-size:130%;}
#ygrp-sponsor #nc{
background-color:#eee;margin-bottom:20px;padding:0 8px;}
#ygrp-sponsor .ad{
padding:8px 0;}
#ygrp-sponsor .ad #hd1{
font-family:Arial;font-weight:bold;color:#628c2a;font-size:100%;line-height:122%\
;}
#ygrp-sponsor .ad a{
text-decoration:none;}
#ygrp-sponsor .ad a:hover{
text-decoration:underline;}
#ygrp-sponsor .ad p{
margin:0;}
o{font-size:0;}
.MsoNormal{
margin:0 0 0 0;}
#ygrp-text tt{
font-size:120%;}
blockquote{margin:0 0 0 4px;}
.replbq{margin:4;}
-->








       Now you can chat without downloading messenger. Go to
http://in.messenger.yahoo.com/webmessengerpromo.php

[Non-text portions of this message have been removed]

GWT code looks like regular Java code except in a few conditions: it
can inline native JavaScript within your Java source, or it can use a
compile-time deferred binding technique to pick code based on your
browser type (Firefox, IE, etc.).  You can write regular Java JUnit or
Fit tests for your application, but as soon as your code invokes
Javascript (instantiating any widget does this) or uses the deferred
binding, you need to run your code in a browser.

GWT ships with a modified browser that can essentially run your GWT
Java code in a JVM instead of JavaScript.  Whenever you need to test
code that needs a browser, Google provides an extension to TestCase
called GWTTestCase.  This class essentially starts up a hidden web
browser and runs your test case in this browser -- as Java or
Javascript.  This works very well, but it is very slow - it takes
several seconds to run a test as opposed to milliseconds.

For my project, I've used JUnit tests with JMock for the presenter
layers above the UI, and when I really wanted to test actual widget or
JavaScript code (JSON parsing is another case that needs a browser),
used GWTTestCase.  I haven't used Fit tests, but I think some folks on
the GWT mailing list may have; the only issue would be getting that
headless browser up when your code required it.

Dan

--- In TestFirstUserInterfaces@yahoogroups.com, Rick Mugridge
<rick@...> wrote:
>
> Hi Dan,
>
> No, I haven't, but I've been keen to try out GWT for some time. I
really
> like the way they manage Javascript. I've been waiting for the widget
> set to become more comprehensive - perhaps it is now?
>
> Now that you mention it, I can see that hooking into the Java-based
> browser runtime is the interesting bit. Is the source available for
> that? Is there a useful testing api?
>
> How have you approached it yourself?
>
> Cheers, Rick
>
> Daniel Wellman wrote:
> >
> > Hi Rick,
> >
> > Have you written Fit tests for GWT? If so, did your tests require one
> > of the fake hosted mode browsers to run to handle widgets or deferred
> > binding?
> >
> > I'm working on a GWT project right now and am interested to share
> > experience stories with other folks.
> >
> > Cheers,
> > Dan
> >
> > --- In TestFirstUserInterfaces@yahoogroups.com
> > <mailto:TestFirstUserInterfaces%40yahoogroups.com>, Rick Mugridge
> > <rick@> wrote:
> > > PS, if GWT was used, the Java code intended for translation to
> > > Javascript could be tested with Fit
> >
> >
>
>
> [Non-text portions of this message have been removed]
>

Hi Dan,

No, I haven't, but I've been keen to try out GWT for some time. I really
like the way they manage Javascript. I've been waiting for the widget
set to become more comprehensive - perhaps it is now?

Now that you mention it, I can see that hooking into the Java-based
browser runtime is the interesting bit. Is the source available for
that? Is there a useful testing api?

How have you approached it yourself?

Cheers, Rick

Daniel Wellman wrote:
>
> Hi Rick,
>
> Have you written Fit tests for GWT? If so, did your tests require one
> of the fake hosted mode browsers to run to handle widgets or deferred
> binding?
>
> I'm working on a GWT project right now and am interested to share
> experience stories with other folks.
>
> Cheers,
> Dan
>
> --- In TestFirstUserInterfaces@yahoogroups.com
> <mailto:TestFirstUserInterfaces%40yahoogroups.com>, Rick Mugridge
> <rick@...> wrote:
> > PS, if GWT was used, the Java code intended for translation to
> > Javascript could be tested with Fit
>
>


[Non-text portions of this message have been removed]

Hi Rick,

Have you written Fit tests for GWT?  If so, did your tests require one
of the fake hosted mode browsers to run to handle widgets or deferred
binding?

I'm working on a GWT project right now and am interested to share
experience stories with other folks.

Cheers,
Dan


--- In TestFirstUserInterfaces@yahoogroups.com, Rick Mugridge
<rick@...> wrote:
> PS, if GWT was used, the Java code intended for translation to
> Javascript could be tested with Fit

Thanks!

.rob.


On 11/7/07, Rick Mugridge <rick@...> wrote:
>
>   Hi Rob,
> > On the agile-testing list, I've been gathering info WRT acceptance
> > tests for this and comparing WatiR, WatiN, Selenium, etc...
> >
> > So now I've noticed that our WatiN tests are very slow. (We have some
> > NUnitAsp tests too which are worse.) We have 100s+ tests in FitNesse
> > that take about 80s to run. And <10 in WatiN that take about 80s to
> > run.
> Yes, UI tests are very slow, whether it's web-based or GUI. That's one
> of the big problems with them. Sometimes you're forced to decide on
> timeout periods, and so have to balance speed against spurious fails.
> The other problem is that the UI tests are verbose and tend to be
> fragile under changes to the UI (I'm working on solving this problem).
> >
> > So I'm theorizing that Fit tests are really going to be the stronger
> > set. It'll be 1 layer down, but as long as we do some sort of MVC/MVP
> > or simple command pattern, we only won't be testing the client-side
> > validation. I'm debating whether or not those tests are even worth
> > writing? Especially if for example, we submit some form and the server
> > side validates fields as well.
> I recommend using Fit for testing below the UI layer. Then have some
> UI-based tests that verify end-to-end behavior, so that pages are
> 'wired" together correctly, etc.
>
> If the UI tests are too slow, you can run a two-stage build process,
> with fast feedback from Fit storytests. I've also had success with
> running lots of Selenium tests in parallel.
> >
> > Furthermore, for reuse of the validation logic, I'm leaning towards
> > less JavaScript in the page for AJAX calls that get their results from
> > the server.
> That's a tricky one, as Fit is only good for validating logic on the
> server. I'd use Javascript unit tests for testing the AJAX and
> validation logic in Javascript. And, again, have some UI tests to check
> end-to-end.
>
> Cheers, Rick
>
> PS, if GWT was used, the Java code intended for translation to
> Javascript could be tested with Fit
> >
> > Thoughts?
> > Comments?
> >
> > .rob.
>
>
>


[Non-text portions of this message have been removed]

Hi Rob,
> On the agile-testing list, I've been gathering info WRT acceptance
> tests for this and comparing WatiR, WatiN, Selenium, etc...
>
> So now I've noticed that our WatiN tests are very slow. (We have some
> NUnitAsp tests too which are worse.) We have 100s+ tests in FitNesse
> that take about 80s to run. And <10 in WatiN that take about 80s to
> run.
Yes, UI tests are very slow, whether it's web-based or GUI. That's one
of the big problems with them. Sometimes you're forced to decide on
timeout periods, and so have to balance speed against spurious fails.
The other problem is that the UI tests are verbose and tend to be
fragile under changes to the UI (I'm working on solving this problem).
>
> So I'm theorizing that Fit tests are really going to be the stronger
> set. It'll be 1 layer down, but as long as we do some sort of MVC/MVP
> or simple command pattern, we only won't be testing the client-side
> validation. I'm debating whether or not those tests are even worth
> writing? Especially if for example, we submit some form and the server
> side validates fields as well.
I recommend using Fit for testing below the UI layer. Then have some
UI-based tests that verify end-to-end behavior, so that pages are
'wired" together correctly, etc.

If the UI tests are too slow, you can run a two-stage build process,
with fast feedback from Fit storytests. I've also had success with
running lots of Selenium tests in parallel.
>
> Furthermore, for reuse of the validation logic, I'm leaning towards
> less JavaScript in the page for AJAX calls that get their results from
> the server.
That's a tricky one, as Fit is only good for validating logic on the
server. I'd use Javascript unit tests for testing the AJAX and
validation logic in Javascript. And, again, have some UI tests to check
end-to-end.

Cheers, Rick

PS, if GWT was used, the Java code intended for translation to
Javascript could be tested with Fit
>
> Thoughts?
> Comments?
>
> .rob.

On the agile-testing list, I've been gathering info WRT acceptance
tests for this and comparing WatiR, WatiN, Selenium, etc...

So now I've noticed that our WatiN tests are very slow.  (We have some
NUnitAsp tests too which are worse.)  We have 100s+ tests in FitNesse
that take about 80s to run.  And <10 in WatiN that take about 80s to
run.

So I'm theorizing that Fit tests are really going to be the stronger
set.  It'll be 1 layer down, but as long as we do some sort of MVC/MVP
or simple command pattern, we only won't be testing the client-side
validation.  I'm debating whether or not those tests are even worth
writing?  Especially if for example, we submit some form and the server
side validates fields as well.

Furthermore, for reuse of the validation logic, I'm leaning towards
less JavaScript in the page for AJAX calls that get their results from
the server.

Thoughts?
Comments?

.rob.

Agilitas:

Here's a blog entry on how to use Why's Hpricot library inside assertions,
to constrain your Rails views:

http://www.oreillynet.com/onlamp/blog/2007/08/assert_hpricot_1.html

--
   Phlip
   http://www.oreilly.com/catalog/9780596510657/
   "Test Driven Ajax (on Rails)"
   assert_xpath, assert_javascript, & assert_ajax

I finally found a good XPath exploring tool, so I wrote it up here:

http://www.oreillynet.com/onlamp/blog/2007/08/xpath_checker_and_assert_xpath.htm\
l

--
  Phlip
  http://www.oreilly.com/catalog/9780596510657/
  ^ assert_xpath
  http://tinyurl.com/yrc77g  <-- assert_latest Model

At this month's Extreme Programming San Diego (XPSD) meeting, Phlip
Plumlee will teach us how to perform Test-Driven Development for Ajax
under Rails.

Phlip will give a tutorial on how to efficiently test Web pages, using
Ruby on Rails and assert_xpath, leading to tests on Ajax. assert_xpath
is a Rails plugin built around Rails assertions which Phlip has created
to enable testing of Ajax and Javascript. Those Rails developers among
you will gain a valuable addition to your testing toolset. Those of you
who are not using Rails should be.
http://assertxpath.rubyforge.org/

Phlip applies his compulsion for art and logic to complex problems in
games, linguistics, bioinformatics, statistical process control,
e-commerce, and software visualization. Advocating Agile methodologies
increases the odds he can actually get to use them himself at work.

The meeting will be held at SAIC on Campus Point Dr. (
http://xpsd.org/SAICDirections ) from 6-7:30 PM on Thursday August 2nd.
Please RSVP to joonspoon@... so that we can have a badge ready
for you. If you are not a U.S. citizen, bring your passport with you to
the meeting.

http://xpsd.org
http://www.oreilly.com/catalog/9780596510657/

I'm in rather a new position: I've actually gotten a colleague
enthused about TDD, and even NMock! He's an engineer, w/o much
experience writing GUIs, and now he's looking to me for advice about
the MVP pattern.

So much for the preamble. We have a requirement for (relatively
simple) drag-and-drop, simply from one control on the app to another.
How should I think of this in the context of MVP? As you may remember,
the control from which an item is dragged has to handle the ItemDrag
event, and the target control has to handle DragDrop. Now, I could
certainly have the presenter provide the object passed to DoDragDrop()
by the source, and the same or another presenter actually deal with
it, once the handler for DragDrop has pulled it out of
DragEventArgs.GetData(). Is this a case where, as with data binding, I
basically entrust the mechanics to .NET, and simply verify the results?

Greetings,

I'm interested to learn how folks approach TDDing views which contain
tables.  I've read about using various styles of presenters (Supervising
Controller, Passive View, etc.) and was interested to hear what experiences
folks have had.  I've just been reading Jeremy D. Miller's series on TDDing
GUIs
<http://codebetter.com/blogs/jeremy.miller/archive/2007/07/25/the-build-your-own\
-cab-series-table-of-contents.aspx>and
they were quite good.

In my particular case, I'm using the Google Web Toolkit (GWT) and need to
display a table of data.  The interesting bits are:
- The table has a variable number of columns based on the data set being
displayed
- The table only shows certain columns (and variations of widgets) based
upon the user's permissions
- Each row contains widgets which may have different states (for example,
one cell might contain a combobox with a button, or might be a string with
no widgets)


So far I've started to try out a Passive View approach, where the controller
tells the view to add each cell individually.  This is fairly verbose, and
one way requires the controller to tell the view the specific row and
column, e.g.

...
view.addLabelWithTextAt(5, 10, "Hello");  // row 5, column 10
view.addTitleSelectBoxAt(5, 11, listOfStringTitles, indexOfSelectedItem);
view.setReadOnlyList(5, 12, anotherListOfOptions);
...

This feels fairly chatty, and sometimes slightly awkward --- e.g. creating a
Select Box with a list of all valid options and the index/value selected in
one call.  The advantage here is that it's very explicit in the view and I
could test this behavior.

Then, the view calls back to the controller with an index of the row that
received an event, so the controller can find the corresponding model in its
list.  e.g. in the view

checkbox.addChangeListener() {
...
   controller.titleChanged(11);  // the Title Select Box widget is created
with a row # index
..
}



I know an alternative to Passive View would be a Supervising Controller,
where the View could take the set of model objects and explicitly map and
create all the columns needed.  e.g. in the controller:

view.setPeopleList(people);

Since there are requirements that the table column length varies based upon
the data set, and some controls vary based upon permissions, this seems to
encode a lot of responsibility in the view... and that responsibility will
be trickier to test.  (However, GWT does provide a test case which lets you
instantiate widgets in isolation, so perhaps this is another way to test.)


Can anyone comment on their experiences with this kind of work?

Thanks,

Dan


[Non-text portions of this message have been removed]

Alan Cyment wrote:

> Shall automated acceptance tests use the GUI the app provides? Selenium
> takes this approach whereas I reckon FIT does not...I'm currently
> developing
> an ad-hoc engine for my app (Win32 app with inputs more complex than those
> that FIT can handle) and I'm not sure which path I should take: tests that
> make use of the GUI are more realistic, but on the other hand are a lot
> slower and fragile.
>
> Thoughts on this point?

Treat the GUI as just another library, and _start_ with fearless unit tests
on it. If by Win32 you mean its widgets layer - buttons, edit fields, and
other ready-made controls - you can generally test those by populating them
in production code, and querying their states back in test code. This
requires taking control of the event loop, and forcing virtual updates
without displaying a window (or by quickly displaying it), yet the result is
much more efficient than driving the GUI thru its input system.

--
   Phlip
   http://www.oreilly.com/catalog/9780596510657/
   "Test Driven Ajax (on Rails)"
   assert_xpath, assert_javascript, & assert_ajax

I've written a couple of posts on my weblog describing how to do unit
testing — and test-driven development — of user interfaces using the
Cocoa application framework that's part of Mac OS X, and the OCUnit
unit testing framework that's included with the Xcode IDE that's also
included with Mac OS X.  I thought that at least some members of the
TDD, XP, and test-first UI communities would find them interesting.

      Unit testing Cocoa user interfaces: Target-Action
      http://chanson.livejournal.com/148204.html

      Unit testing Cocoa user interfaces: Cocoa Bindings
      http://chanson.livejournal.com/172390.html

(In the interest of full disclosure, I work on Xcode at Apple - though
I've been a member of all three of these communities for several times
as long as I've worked on Xcode or even at Apple, and I'm not here to
evangelize.  I don't speak for my employer; I'm just a tools developer
who's also really passionate about agile development practices and the
ways they can be applied to improve quality and predictability.)

Cocoa has some relatively unique qualities as an application framework
that make it easy to do test-driven development with:  You can follow
a "trust, but verify" approach to test-driving your user interface,
where can write tests to specify (and verify) that your views and
controllers are connected to your model correctly and trust that so
long as it's the case, the framework will behave correctly.
Furthermore, it's built on the Objective-C language, which supports a
lot of introspection features which you can leverage to make writing
your tests easier without violating encapsulation too much in the code
under test.

The upshot of all this is that you don't need to worry about injecting
events to ensure that the right thing happens when a button is
clicked, you don't have to worry about implementing "reveal" methods
to eyeball your interfaces since you're still doing all of your
interface design in Interface Builder (which isn't a code generator
and doesn't have their major drawbacks), or anything cumbersome like
that.

For anyone doing Mac development, I hope these posts provide a useful
starting point for writing tests for your user interfaces.  For anyone
not doing Mac development, I hope these provide a useful perspective
on your own frameworks and ways that you may be able to approach test-
driving your user interface development in the future.  I'm sure
Cocoa's not the only MVC-oriented framework where these techniques can
be applied; I'd love to hear from others who are doing similar things
with other frameworks and languages.

    -- Chris