--- In agile-testing@yahoogroups.com, John Goodsen <jgoodsen@...> wrote:
>
>with over 50% of your code in the GUI layers, the argument to not test thru
>the GUI becomes a cop out for writing and maintaining good acceptance
>tests... google "declarative vs imperative cucumber" for more on that...
>
>yes, I know I'm on the controversial side of that one...
>

I suspect it really comes down to where your bugs come from.  For example, at
Socialtext, we have a 'watch page' feature.  You click a star that says "watch
page", the star lights up, then you clicked the "watched pages" link, the page
now appears in the link.  Go back, unwatch, watched pages, page does *not*
appear.

That's not the kind of thing I want to test through the business logic layer.

Our /whole app/ is like that.

We do have javascript unit tests, but I find they are more helpful to the
developers than to the testers -- a failure doesn't really indicate something is
broken, a pass doesn't indicate it isn't ... they function more like change
detectors.

We have also had a fair amount of success with GUI automation with Selenium, but
a lot of that is due to the nature of our product and development process.  In
general, I recommend exploratory testing the GUI and a few small, light, quick,
build verification GUI-driving tests.

Meanwhile, on the Software-Testing Discussion list, Elisabeth Hendrickson is
making a big deal about how ATDD is /Acceptance/ Test Driven Development - not
"Automated Testing During Development", and Ken Pugh is pointing out that you
can do ATDD without automation:

http://groups.yahoo.com/group/software-testing/

Might be something to consider here, as, well, as I am /certainly/ not picking
up that vibe here.

regards,



--
Matthew Heusser,
Personal Blog: http://xndev.blogspot.com/
Test Community Blog: http://softwaretestpro.com/blog/
Twitter: mheusser

Thanks for this new perspective, Matthew. It looks certainly very
interesting and I am going to check the link out straight away.

I have kind of decided that we will drive the ATDD for my team via
cucumber (which i am currently reading up on now as a direct result of
all the very helpful suggestions I have received from one and all)

I am very grateful for all your responses which has helped me a great
deal in grappling with my 'beast', and a BIG Thank you to everyone that
pitched in to help with their wonderful ideas as always!

I will revert with further questions I am sure, in due course!

Warmest Regards

C O


-----Original Message-----
From: Matthew <matt.heusser@...>
To: agile-testing@yahoogroups.com
Sent: Tue, Mar 1, 2011 2:26 pm
Subject: [agile-testing] Re: Extending TDD to ATDD





--- In agile-testing@yahoogroups.com, John Goodsen <jgoodsen@...> wrote:
>
>with over 50% of your code in the GUI layers, the argument to not test
thru
>the GUI becomes a cop out for writing and maintaining good acceptance
>tests... google "declarative vs imperative cucumber" for more on
that...
>
>yes, I know I'm on the controversial side of that one...
>

I suspect it really comes down to where your bugs come from.  For
example, at Socialtext, we have a 'watch page' feature.  You click a
star that says "watch page", the star lights up, then you clicked the
"watched pages" link, the page now appears in the link.  Go back,
unwatch, watched pages, page does *not* appear.

That's not the kind of thing I want to test through the business logic
layer.

Our /whole app/ is like that.

We do have javascript unit tests, but I find they are more helpful to
the developers than to the testers -- a failure doesn't really indicate
something is broken, a pass doesn't indicate it isn't ... they function
more like change detectors.

We have also had a fair amount of success with GUI automation with
Selenium, but a lot of that is due to the nature of our product and
development process.  In general, I recommend exploratory testing the
GUI and a few small, light, quick, build verification GUI-driving
tests.

Meanwhile, on the Software-Testing Discussion list, Elisabeth
Hendrickson is making a big deal about how ATDD is /Acceptance/ Test
Driven Development - not "Automated Testing During Development", and
Ken Pugh is pointing out that you can do ATDD without automation:

http://groups.yahoo.com/group/software-testing/

Might be something to consider here, as, well, as I am /certainly/ not
picking up that vibe here.

regards,

--
Matthew Heusser,
Personal Blog: http://xndev.blogspot.com/
Test Community Blog: http://softwaretestpro.com/blog/
Twitter: mheusser

John,

I'm not sure what you're saying here.

Are you saying that, if one has 50% of their code in GUI layers, then one should
not just skip GUI tests and do only service layer testing?

Is that what you're saying?

I did google the declarative vs. imperative, but all I came up with was two
different styles of expressing user story acceptance tests, and two different
styles of writing cucumber tests.  Could you expand on how this is relevant to
what I posted?

-------
Charles Bradley, CSM, PSM I
Experienced Scrum Coach
My blog: http://scrumcrazy.wordpress.com/

--- In agile-testing@yahoogroups.com, John Goodsen <jgoodsen@...> wrote:
>
> with over 50% of your code in the GUI layers, the argument to not test thru
> the GUI becomes a cop out for writing and maintaining good acceptance
> tests... google "declarative vs imperative cucumber" for more on that...
>
> yes, I know I'm on the controversial side of that one...
>
> (sent from my droid)
> On Feb 28, 2011 1:29 PM, "charles_bradley_scrum_coach" <
> chuck-lists2@...> wrote:
> >
> > Conrad,
> >
> > You're on the right path, so keep up the good work.
> >
> > First, if you're not already familiar with the "Testing Pyramid" concept,
> you need to be:
> >
>
http://blog.mountaingoatsoftware.com/the-forgotten-layer-of-the-test-automation-\
pyramid
> > Also read the comments specifically the one about the "Cloud" that resides
> above the pyramid of "Exploratory Testing"
> >
> > My advice:
> > Everyone so far has given good advice, of which the theme, I believe, is
> this: "Obtain and use a Service Layer testing tool like Cucumber, Fitnesse,
> etc to support your GUI testing(Sellenium)"
> >
> > My main justification for this, is the same as Cohn's -- once you get over
> the learning curve with your service layer tool, tests to exercise the
> service layer (where a huge percentage of business logic resides... or
> SHOULD reside 'if your team is architecting/designing using best practices')
> are much cheaper and easier to maintain than GUI/Sellenium tests.
> >

Matt,

On 3/1/11 8:26 AM, Matthew wrote:
> I suspect it really comes down to where your bugs come from.  For
> example, at Socialtext, we have a 'watch page' feature.  You click a
> star that says "watch page", the star lights up, then you clicked the
> "watched pages" link, the page now appears in the link.  Go back,
> unwatch, watched pages, page does *not* appear.
>
> That's not the kind of thing I want to test through the business
> logic layer.

If the page watching logic is in the business logic layer and the
star-light and watched pages list are in the model that layer provides
to the GUI, then why not?

I'd also want to test that the GUI is hooked up correctly to the data
model and the actions of the business logic layer, of course.

   - George

--
   ----------------------------------------------------------------------
    * George Dinwiddie *                      http://blog.gdinwiddie.com
    Software Development                    http://www.idiacomputing.com
    Consultant and Coach                    http://www.agilemaryland.org
   ----------------------------------------------------------------------

In case you're looking for differences, you might want to take a look at the
Triangle Test Sampler from Elisabeth Hendrickson. It solves the triangle tests
on a web page using different frameworks. My fork is a bit more ahead right now
also containing concordion tests:
https://github.com/MarkusGaertner/Triangle-Test-Sampler

Personally, I like using Cucumber similarly as Robot Framework (though I don't
have an auto completion in my text editor of choice, which is vim or emacs), but
I also love to do things with FitNesse. There are situations when I prefer one
over the other, but that's rarely an impediment at all. In fact, I solved one
problem in all of the three mentioned ones - each time a little bit different.
All three were fun to deal with.

Best
Markus Gärtner
http://www.shino.de/blog
http://www.it-agile.de
@mgaertne

--- In agile-testing@yahoogroups.com, Rob Park <robert.d.park@...> wrote:
>
> I wouldn't necessarily assume FitNesse as a defacto.
> As a former FitNesse user, I've been loving ATDD with Cucumber.
> I've been using Cucumber with Celerity & Firewatir primarily, but many are
> using Cucumber+Capybara to ATDD with Selenium.
> I've heard good things about Concordian, Robot, and I think Lisa is still a
> fan of Canoo, so there are many choices.
>
> --
> Rob
> --
> http://agileintention.blogspot.com
> http://twitter.com/robpark
> http://leandog.com
>
>
>
> On Mon, Feb 28, 2011 at 2:41 AM, Franz Allan Valencia See <
> franz.see@...> wrote:
>
> >
> >
> > Fitness is the defacto, but I would suggest you guys take a look at
> > Robotframework as well :-)
> >
> > Our testers found  it easier to use than FitNess. One of the major reasons
> > is because it has an auto-complete feature, which means testers do not have
> > to memorize the list of commands/keywords available.
> >
> > Cheers,
> > --
> > Franz Allan Valencia See | Java Software Engineer
> > franz.see@...
> > LinkedIn: http://www.linkedin.com/in/franzsee
> > Twitter: http://www.twitter.com/franz_see
> >
> > On Mon, Feb 28, 2011 at 1:46 PM, Conrad Ononeme <conradononeme@...>wrote:
> >
> >>
> >>
> >> Thank-You everyone for your insightful input as usual.
> >> Fitness is definitely an option to consider....
> >>
> >> Sent from my Blackberry Device
> >> ------------------------------
> >> *From: * "pepijnvandevorst" <pepijn.vandevorst@...>
> >> *Sender: * agile-testing@yahoogroups.com
> >> *Date: *Sun, 27 Feb 2011 19:05:16 -0000
> >> *To: *<agile-testing@yahoogroups.com>
> >> *ReplyTo: * agile-testing@yahoogroups.com
> >> *Subject: *[agile-testing] Re: Extending TDD to ATDD
> >>
> >>
> >>
> >> Hello Conrad,
> >>
> >> Did you consider FitNesse?
> >>
> >> From your post it's not clear to me why you think Selenium isn't good
> >> enough to use for ATDD. But FitNesse is often used to automate acceptance
> >> tests.
> >>
> >> Regards,
> >> Pepijn
> >>
> >> <conradononeme@> wrote:
> >> >
> >> > Hello ALL!
> >> >
> >> > I have a quick question.
> >> > I am the lead embedded tester in my team that has done very well in the
> >> uptake of Agile.
> >> >
> >> > Now having successfully gotten the team to follow best practice in
> >> pretty much every other area of Agile Software development, I now want to
> >> push the boat out a little by seeing if we can actually extend the
> >> developers' tests to form the basis (or at least part of it) of our
> >> Acceptance Tests.
> >> > Currently the practice is that we refine the Acceptance Criteria once a
> >> story is in play, as part of our iteration activities, and that process in
> >> turn serves out or Acceptance tests.
> >> > The technical tester then automates the tests (identified as automation
> >> candidates) while the alternative flows and negative testing scenarios are
> >> manually run and validated as the main part of the Acceptance Tests.
> >> > The tester writes all the tests in Selenium DE only and at the moment
> >> too as far as I am aware the devs do too.
> >> >
> >> > The devs as stated earlier, currently practice TDD.
> >> >
> >> > My challenge is how to make better use of their assets by finding a way
> >> to extend their tests to at least form part of the basis of our Acceptance
> >> Tests.
> >> >
> >> > I suspect however that selenium may have some limitations that may not
> >> make this ambition possible though. So in that case can someone advise what
> >> the best OS tools out there that can help will be? (preferably something
> >> light on tecchy savviness as I am not a `technical' tester at all :-)
> >> >
> >> > I hope I haven't put all to sleep yet with my long message, but I
> >> thought as much background as possible would be germane at this point, so
> >> kindly do forgive me.
> >> >
> >> > On that note u shall look forward to hearing your always illuminating
> >> ideas and experiences!
> >> >
> >> > Many thanks!
> >> >
> >> > CO
> >> > Sent from my Blackberry Device
> >> >
> >>
> >>
> >
> >
> >
> >
>

Hi Charles,

may I cite you on this:
> Another suggestion:  I've found that, when I coach teams on heading down the
"let's merge the testers' and developers' roads" like you are, an interesting
phenomenon happens.  Often times, a particular piece of logic will be well
tested at the integration or unit level(which may or may not be exposed to your
service testing tool).  When developers and testers communicate about this,
testers will find less need to test that logic at the GUI level(which is
essentially also what happens when you do service layer testing -- less need to
do GUI level testing)  They'll still test, but they'll usually just that logic
in an exploratory way(just a few boundary cases, rather than exhaustive testing)
rather than writing GUI tests.  This saves time and money, and helps bring
developers and testers closer together in the communication gap.  The best
solution here, IMO, is to add the service layer tool, which acts as a natural
meeting place for developers and testers, but without the high costs of GUI
tests.
>

???

I love this paragraph.

Best
Markus Gärtner
http://www.shino.de/blog
http://www.it-agile.de
@mgaertne

--- In agile-testing@yahoogroups.com, "charles_bradley_scrum_coach"
<chuck-lists2@...> wrote:
>
>
> Conrad,
>
> You're on the right path, so keep up the good work.
>
> First, if you're not already familiar with the "Testing Pyramid" concept, you
need to be:
>
http://blog.mountaingoatsoftware.com/the-forgotten-layer-of-the-test-automation-\
pyramid
> Also read the comments specifically the one about the "Cloud" that resides
above the pyramid of "Exploratory Testing"
>
> My advice:
> Everyone so far has given good advice, of which the theme, I believe, is this:
"Obtain and use a Service Layer testing tool like Cucumber, Fitnesse, etc to
support your GUI testing(Sellenium)"
>
> My main justification for this, is the same as Cohn's -- once you get over the
learning curve with your service layer tool, tests to exercise the service layer
(where a huge percentage of business logic resides... or SHOULD reside 'if your
team  is architecting/designing using best practices') are much cheaper and
easier to maintain than GUI/Sellenium tests.
>
> Another suggestion:  I've found that, when I coach teams on heading down the
"let's merge the testers' and developers' roads" like you are, an interesting
phenomenon happens.  Often times, a particular piece of logic will be well
tested at the integration or unit level(which may or may not be exposed to your
service testing tool).  When developers and testers communicate about this,
testers will find less need to test that logic at the GUI level(which is
essentially also what happens when you do service layer testing -- less need to
do GUI level testing)  They'll still test, but they'll usually just that logic
in an exploratory way(just a few boundary cases, rather than exhaustive testing)
rather than writing GUI tests.  This saves time and money, and helps bring
developers and testers closer together in the communication gap.  The best
solution here, IMO, is to add the service layer tool, which acts as a natural
meeting place for developers and testers, but without the high costs of GUI
tests.
>
> So, sometimes, just based on trust and communication alone, you can lessen
your GUI testing need for acceptance testing, and instead just communicate with
the developers to make sure they are fully testing/exercising particular
acceptance tests.  A better solution is to downright prove that you don't need
as much GUI testing because you and the devs have a highly readable service
layer test (Cucumber, Fitnesse, etc) that proves that your acceptance tests are
fully automated and exercised.
>
> In short, you can develop and run service layer acceptance
tests(Cucumber/Fitnesse), much cheaper and quicker than you can running GUI
layer tests (Sellenium, etc).  You can also exercise more data scenarios (same
feature, different test inputs) much cheaper and quicker than with GUI test
tools.
>
> This is not to say that GUI test tools are bad, just that they should be the
last resort(assuming you have a service layer tool), and should generally be
used to test logic that resides actually in the GUI layer (which again,
according to arch/design best practices, should be very little logic).
>
> Also, I'm sort of assuming your developers also do unit/integration testing
with JUnit or a similar tool.  You mentioned that they use Sellenium, but I sure
hope they're also doing a lot of JUnit stuff.  Just like service layer testing
is cheaper than GUI testing, unit/integration testing using JUnit is also
cheaper than service layer testing(Cucumber, Fitnesse, etc).
>
> -------
> Charles Bradley, CSM, PSM I
> Experienced Scrum Coach
> My blog: http://scrumcrazy.wordpress.com/
>
>
> --- In agile-testing@yahoogroups.com, "Conrad Ononeme" <conradononeme@> wrote:
> >
> > Hello ALL!
> >
> > I have a quick question.
> > I am the lead embedded tester in my team that has done very well in the
uptake of Agile.
> >
> > Now having successfully gotten the team to follow best practice in pretty
much every other area of Agile Software development, I now want to push the boat
out a little by seeing if we can actually extend the developers' tests to form
the basis (or at least part of it) of our Acceptance Tests.
>

--- In agile-testing@yahoogroups.com, George Dinwiddie <lists@...> wrote:
>
> Matt,
>
> On 3/1/11 8:26 AM, Matthew wrote:
> > I suspect it really comes down to where your bugs come from.  For
> > example, at Socialtext, we have a 'watch page' feature.  You click a
> > star that says "watch page", the star lights up, then you clicked the
> > "watched pages" link, the page now appears in the link.  Go back,
> > unwatch, watched pages, page does *not* appear.
> >
> > That's not the kind of thing I want to test through the business
> > logic layer.
>
> If the page watching logic is in the business logic layer and the
> star-light and watched pages list are in the model that layer provides
> to the GUI, then why not?
>
> I'd also want to test that the GUI is hooked up correctly to the data
> model and the actions of the business logic layer, of course.
>
>

I was really disappointed by this comment, George.  I struggled for some time to
come up with a reply that was helpful.  That said, here goes ...

I think the most polite thing I can say here is that you are speaking in
abstractions that don't map well onto a LAMP/AJAX technology infrastructure.  I
mean, you could /try/ to separate concerns in this way, and just test at the
business logic level, but you'd /fail/ -- consider testing a WYSIWYG customer
control for bold and unbold -- those things don't even /make/ server trips.

Even if you could find a way to test watch, sooner or later, it would break and
the business logic layer would be just fine.  This has actually happened.

Not only that, the idea that one little change "couldn't possible break" the
system and there is "no need to look over there" for the problem is /classic/.  
I believe Jerry Weinberg uses it in "Becoming a Technical Leader", but it might
be "Secrets of Consulting": His general conclusion is that when people say "the
bug couldn't possibly be in there" you should *look in there*.

Another short reason might be because 'just' testing the business logic for a
GUI feature isn't testing the whole black box.  To quote Bob Martin: "End to end
is further than you think."

Of course, it's possible to 'cover' for not writing selenium tests in several
ways.  (or 'test it twice') You could do exploratory tests or have a
'beta'/staging program to generate coverage of the application, etc.

Your email didn't speak to those other ways of covering the app; instead, you
asked "why not" "just" test at the Business Logic Level.

/*Because you'll miss the bugs, dude!*/

Asked and answered.

regards,



--
Matthew Heusser,
Personal Blog: http://xndev.blogspot.com/
Test Community Blog: http://softwaretestpro.com/blog/
Twitter: mheusser

--- In agile-testing@yahoogroups.com, "Markus Gaertner" <shino@...> wrote:
>
> Hi Charles,
>
> may I cite you on this:
> > Another suggestion:  I've found that, when I coach teams on heading down the
>"let's merge the testers' and developers' roads" like you are, an interesting
>phenomenon happens.  Often times, a particular piece of logic will be well
tested
>at the integration or unit level(which may or may not be exposed to your
>service testing tool).  When developers and testers communicate about this,
>testers will find less need to test that logic at the GUI level(which is
essentially
>also what happens when you do service layer testing -- less need to do GUI
>level testing)  They'll still test, but they'll usually just that logic in an
>exploratory way(just a few boundary cases, rather than exhaustive testing)
>

Like I said before, it depends on how much eye-candy is in the GUI, but I /can/
see this working on certain circumstances.  I especially appreciate that you
mentioned 'covering' the risks in other ways, eg exploratory testing.  That's an
important piece that's easy to miss.


regards,


--
Matthew Heusser,
Personal Blog: http://xndev.blogspot.com/
Test Community Blog: http://softwaretestpro.com/blog/
Twitter: mheusser

--- In agile-testing@yahoogroups.com, Josue Barbosa dos Santos <josuesantos@...>
wrote:
>
> May be when he says business logic layer in fact he is meaning the
> possibility to write unit tests (micro tests) to the logic without the need
> of tools like selenium. And using a pattern like MVP it is possible to the
> example you gave. Ex:

Thanks Jose. Yes, that was exactly what I assumed he meant. I'm familiar with
mocks; I gave a talk with Sean McMcmillan on the subject at Google Test
Automation Conference a few years back:

http://www.youtube.com/watch?v=PHtEkkKXSiY

Notice Sean McMillan's concerns about 15:00-16:12 in.  Also notice my comments
immediately following that.  I am still expressing those concerns today.  (As
I've mentioned on or twice all ready,  Micro-tests over time in Javascript tend
to act more like change detectors. "Failure" means some expectation no longer
holds true, "passing" means it still holds true - but both of those cases might
or might involve new defects.)

As to the claim that developer-tests alone can be sufficient for testing, well,
I would assert "not where I work."  I /can/ see /some/ domains where that can
work fine, such as applications with very light javascript designed to run in a
single browser for internal company use, etc.

I work at a company deploying software as a service for a thousands of
professional customers supporting a large variety of browsers and mobile devices
that provides a fair amount of client-side "eye candy."  The context are
different.  (I talk about applying a strategy that works in one context into an
incorrect context at 24:24 :-)

Thank you for the trip down memory lane; I'm pleased that the talk holds up as
well as it does.  :-)

regards,



--
Matthew Heusser,
Personal Blog: http://xndev.blogspot.com/
Test Community Blog: http://softwaretestpro.com/blog/
Twitter: mheusser

Hi folks,

Has anyone used any of the existing ATDD tools such as Robot Framework,
Fitnesse/CSLIM or similar for testing telecom products (in particular
C/C++ systems)?

I'd be very interested in hearing your experiences.

Thanks!
--
Dave Rooney
Westboro Systems
Web: http://www.WestboroSystems.com
Blog: http://practicalagility.blogspot.com
Twitter: daverooneyca

Hi Dave,

we used FitNesse/Slim for testing our customized rating and billing engine for
our clients at my previous jobl. We had a JBoss application server, and used
Java to drive the functions in the system. For the C/C++ components under Unix
we used shell commands to trigger certain operations or get some results.

We couldn't get easy access to expert users, which made the whole benefits of
Specification by Example not apply to us. However, I remember when I presented
once the test system we had built to one of our clients (he asked for it), they
became curious, and eventually asked to get this thing on their machines (though
that never happened, unfortunately).

What are you looking for in particular?

Markus Gaertner
http://www.shino.de/blog
http://www.it-agile.de
Twitter: @mgaertne


----- Original Message -----
From: dave.rooney@...
To: agile-testing@yahoogroups.com
Date: 03.03.2011 12:54:00
Subject: [agile-testing] ATDD in Telecom Environment


> Hi folks,
>
> Has anyone used any of the existing ATDD tools such as Robot Framework,
> Fitnesse/CSLIM or similar for testing telecom products (in particular
> C/C++ systems)?
>
> I'd be very interested in hearing your experiences.
>
> Thanks!
> --
> Dave Rooney
> Westboro Systems
> Web: http://www.WestboroSystems.com
> Blog: http://practicalagility.blogspot.com
> Twitter: daverooneyca
>
>
> ------------------------------------
>
> Yahoo! Groups Links
>
>
>

Hi Markus,

I'm looking for testing of core call-processing functionality, i.e. the
'guts' of a switch.

I have been able to get one team to start using a table-driven approach
to determining what to test, and they have done some automation using
their current homegrown tool.  Their approach, though, is nowhere near
as efficient as I've seen in other environments.

Thanks!

Dave...

On 03/03/2011 6:58 AM, Markus Gaertner wrote:
> Hi Dave,
>
> we used FitNesse/Slim for testing our customized rating and billing engine for
our clients at my previous jobl. We had a JBoss application server, and used
Java to drive the functions in the system. For the C/C++ components under Unix
we used shell commands to trigger certain operations or get some results.
>
> We couldn't get easy access to expert users, which made the whole benefits of
Specification by Example not apply to us. However, I remember when I presented
once the test system we had built to one of our clients (he asked for it), they
became curious, and eventually asked to get this thing on their machines (though
that never happened, unfortunately).
>
> What are you looking for in particular?
>
> Markus Gaertner
> http://www.shino.de/blog
> http://www.it-agile.de
> Twitter: @mgaertne
>
>
> ----- Original Message -----
> From: dave.rooney@...
> To: agile-testing@yahoogroups.com
> Date: 03.03.2011 12:54:00
> Subject: [agile-testing] ATDD in Telecom Environment
>
>
>> Hi folks,
>>
>> Has anyone used any of the existing ATDD tools such as Robot Framework,
>> Fitnesse/CSLIM or similar for testing telecom products (in particular
>> C/C++ systems)?
>>
>> I'd be very interested in hearing your experiences.
>>
>> Thanks!
>> --
>> Dave Rooney
>> Westboro Systems
>> Web: http://www.WestboroSystems.com
>> Blog: http://practicalagility.blogspot.com
>> Twitter: daverooneyca

--
Dave Rooney
Westboro Systems
Web: http://www.WestboroSystems.com
Blog: http://practicalagility.blogspot.com
Twitter: daverooneyca

Hi Dave,

first, we had a mobile telecom system which could be used for fixed line, but
was not very near the hardware.

We set up our calls a few steps behind the hardware-layer based upon
standardized interfaces using xml. We used xml templates to create tickets based
upon the data fed form the table, and send these over a socket into the system.
This got awesome once I discovered apache's velocity library for this.

For the reponses, we simply parsed them, and converted the call results. We also
hid most of the complexity behind a facade structure, which took over to make
start-requests, extend-requests, and end-requests based upon the call duration.

Of course, we could not get all information out of the XML requests and
responses. We also talked to the business part sitting inside the JBoss
application server. There were services to get certain information from the
accounts which we made use of.

In order to create accounts we invented a little DSL, for examples create an
account in this tariff, which was by convention active, and had no money. Any
further changes to this standard account needed to be expressed in the table to
some degree. This worked good considering the trade-off we took.

One thing to point out from my side is that we made use of a migration tool to
express accounts in XML. Using the template approach went very smoothly for
this.

You might want to peek into this paper I wrote for the Agile Testing Days 2009:
http://www.shino.de/publications/Agile_practices_in_a_traditional_environment.pd\
f

It describes how we went from using shell scripts to use FitNesse using Agile
practices.

Best
Markus Gaertner
http://www.shino.de/blog
http://www.it-agile.de
Twitter: @mgaertne


----- Original Message -----
From: dave.rooney@...
To: agile-testing@yahoogroups.com
Date: 03.03.2011 13:04:00
Subject: Re: [agile-testing] ATDD in Telecom Environment


> Hi Markus,
>
> I'm looking for testing of core call-processing functionality, i.e. the
> 'guts' of a switch.
>
> I have been able to get one team to start using a table-driven approach
> to determining what to test, and they have done some automation using
> their current homegrown tool.  Their approach, though, is nowhere near
> as efficient as I've seen in other environments.
>
> Thanks!
>
> Dave...
>
> On 03/03/2011 6:58 AM, Markus Gaertner wrote:
> > Hi Dave,
> >
> > we used FitNesse/Slim for testing our customized rating and billing engine
for our clients at my previous jobl. We had a JBoss application server, and used
Java to drive the functions in the system. For the C/C++ components under Unix
we used shell commands to trigger certain operations or get some results.
> >
> > We couldn't get easy access to expert users, which made the whole benefits
of Specification by Example not apply to us. However, I remember when I
presented once the test system we had built to one of our clients (he asked for
it), they became curious, and eventually asked to get this thing on their
machines (though that never happened, unfortunately).
> >
> > What are you looking for in particular?
> >
> > Markus Gaertner
> > http://www.shino.de/blog
> > http://www.it-agile.de
> > Twitter: @mgaertne
> >
> >
> > ----- Original Message -----
> > From: dave.rooney@...
> > To: agile-testing@yahoogroups.com
> > Date: 03.03.2011 12:54:00
> > Subject: [agile-testing] ATDD in Telecom Environment
> >
> >
> >> Hi folks,
> >>
> >> Has anyone used any of the existing ATDD tools such as Robot Framework,
> >> Fitnesse/CSLIM or similar for testing telecom products (in particular
> >> C/C++ systems)?
> >>
> >> I'd be very interested in hearing your experiences.
> >>
> >> Thanks!
> >> --
> >> Dave Rooney
> >> Westboro Systems
> >> Web: http://www.WestboroSystems.com
> >> Blog: http://practicalagility.blogspot.com
> >> Twitter: daverooneyca
>
> --
> Dave Rooney
> Westboro Systems
> Web: http://www.WestboroSystems.com
> Blog: http://practicalagility.blogspot.com
> Twitter: daverooneyca
>
>
> ------------------------------------
>
> Yahoo! Groups Links
>
>
>

On 03/03/11 11:54, Dave Rooney wrote:
> Has anyone used any of the existing ATDD tools such as Robot Framework,
> Fitnesse/CSLIM or similar for testing telecom products (in particular
> C/C++ systems)?

I once wrote a simple telephony application that accepted incoming
call notifications from a telephony switch using a text-based protocol
over TCP sockets.  It decided what to do with the call: forward, route
to voicemail, IVR, etc.  I wrote this in C++ and drove all development
from dejagnu scripts in an ATDD approach.

Unfortunately for commercial reasons the project was never completed
but I was very happy with the experience of developing from the
outside in.  If I were to do it again I would probably use Robot
rather than Fitnesse or similar because Robot has more system-level
libraries - for starting and stopping processes, connecting over
telnet or SSH, etc - than Fitnesse.  I have successfully used Robot
for unit testing perl scripts that interact with an Asterisk server by
passing in outbound script dependencies and using netcat or socat to
replay known results over a socket.

--
Hubert Matthews         http://www.oxyware.com/
Software Consultant     hubert@...

I’ll run a webinar on the long term value of acceptance testing on 22nd
March.

Most of the discussion on automated acceptance tests focuses on
immediate benefits in development and defect detection or regression
testing. But that’s not nearly all you can get from your tests. While
working on my new book, I’ve interviewed many teams that got big
pay-offs from automated acceptance tests, including some that have been
using agile acceptance testing for six or seven years.

In the long term, most of these teams got quite unexpected benefits,
such as being able to support their system easier, significantly change
their business models or survive the absence of key business people. In
this webinar, I will talk about these long term benefits of acceptance
tests and what you need to do to get them.

Click here for more details and to register:
http://www2.smartbear.com/webinar_unexpected_benefits_smga.html

Gojko

We are looking for presentations on the topics of agile testing processes and
agile testing tool use for the Verify/ATI testing conference (see
http://verifyati.com/) held in Arlington, VA from 9/26 - 9/28, 2011.

Please visit this page
http://verifyati.com/index.php?option=com_jforms&view=form&id=1&Itemid=85 for
more details on the Call for Presentations.

Elfriede Dustin
Verify/ATI Conference Co-Chair
http://www.amazon.com/Elfriede-Dustin/e/B001IO9RTM/ref=ntt_athr_dp_pel_1

Hi Hubert,

> I once wrote a simple telephony application that accepted incoming
> call notifications from a telephony switch using a text-based protocol
> over TCP sockets.  It decided what to do with the call: forward, route
> to voicemail, IVR, etc.  I wrote this in C++ and drove all development
> from dejagnu scripts in an ATDD approach.
>
> Unfortunately for commercial reasons the project was never completed
> but I was very happy with the experience of developing from the
> outside in.  If I were to do it again I would probably use Robot
> rather than Fitnesse or similar because Robot has more system-level
> libraries - for starting and stopping processes, connecting over
> telnet or SSH, etc - than Fitnesse.  I have successfully used Robot
> for unit testing perl scripts that interact with an Asterisk server by
> passing in outbound script dependencies and using netcat or socat to
> replay known results over a socket.

Cool - that's excellent feedback!  Thanks!
--
Dave Rooney
Westboro Systems
Web: http://www.WestboroSystems.com
Blog: http://practicalagility.blogspot.com
Twitter: daverooneyca

Well Robot was developed for a telecom's company: NSN

I don't know about C++, but from recent experience with C# if the binding doesn't exist it implements a simple XML-RPC protocol. Its easy enough to deserialize on the client side and consume. 

I can give you a pointer to a C# implementation offlist.

On 04/03/11 03:05, Dave Rooney wrote:
> I'm looking for something like Robot or Fitnesse/CSLIM that can talk
> to the switch and has all the plumbing already to send values and
> receive results. The tricky part is in wiring the fixtures to the
> switch software, which is where I'm seeking some guidance.

What protocols does the switch use?  If it's TCP or UDP then you can
use netcat or socat to do most of the networking part and all you need
deal with is stdin and stdout and text files.  Then all you need is
some fixtures to turn your test tool's tables into binary packets and
pipe them into netcat.  This has the advantage that you can unit test
the fixtures (table -> binary) independently of the networking aspects.

An alternative would be to use a network packet capture tool such as
wireshark or tcpdump.  You could then do record/playback style testing
and your tests would refer to binary file names.  I recently
implemented a stub for a client that did record/playback simulation of
a mainframe.  All it did was have a map of known inputs and
corresponding outputs read from text files (this was all XML).  The
problem was that the mainframe is shared and they could never get
stable known results for testing.  With the stub they had a known
back-end so we could test the e-commerce front-end checkout using
Selenium.  After automating only two tests we spotted that the
front-end was sending out duplicate stock level queries, something
that made no difference but no-one had noticed!

--
Hubert Matthews         http://www.oxyware.com/
Software Consultant     hubert@...

Matt,

As I'm reading back over your response, it dawns on me.

You say:
> Your email didn't speak to those other ways of covering the app;
> instead, you asked "why not" "just" test at the Business Logic
> Level.

But George did say:
> I'd also want to test that the GUI is hooked up correctly to the
> data model and the actions of the business logic layer, of course.

Does this statement by George above not address your concern that people suggest
to "skip GUI testing" altogether?

I wasn't saying to skip GUI testing, and I don't think George said that, either.
I think what he and I (and Mike Cohn) were saying is that it might be wise to
test *less* at the GUI layer *if* you do good service level testing, since
service level tests are cheaper and easier to maintain(again, once you get over
the not so big learning curve of your business service layer testing tool/s)

-------
Charles Bradley, CSM, PSM I
Experienced Scrum Coach
My blog: http://scrumcrazy.wordpress.com/

> --- In agile-testing@yahoogroups.com, George Dinwiddie <lists@> wrote:
> >
> > Matt,
> >
> > On 3/1/11 8:26 AM, Matthew wrote:
> > > I suspect it really comes down to where your bugs come from.  For
> > > example, at Socialtext, we have a 'watch page' feature.  You click a
> > > star that says "watch page", the star lights up, then you clicked the
> > > "watched pages" link, the page now appears in the link.  Go back,
> > > unwatch, watched pages, page does *not* appear.
> > >
> > > That's not the kind of thing I want to test through the business
> > > logic layer.
> >
> > If the page watching logic is in the business logic layer and the
> > star-light and watched pages list are in the model that layer provides
> > to the GUI, then why not?
> >
> > I'd also want to test that the GUI is hooked up correctly to the data
> > model and the actions of the business logic layer, of course.
> >
> >
>
> I was really disappointed by this comment, George.  I struggled for some time
to come up with a reply that was helpful.  That said, here goes ...
>
> I think the most polite thing I can say here is that you are speaking in
abstractions that don't map well onto a LAMP/AJAX technology infrastructure.  I
mean, you could /try/ to separate concerns in this way, and just test at the
business logic level, but you'd /fail/ -- consider testing a WYSIWYG customer
control for bold and unbold -- those things don't even /make/ server trips.
>
> Even if you could find a way to test watch, sooner or later, it would break
and the business logic layer would be just fine.  This has actually happened.
>
> Not only that, the idea that one little change "couldn't possible break" the
system and there is "no need to look over there" for the problem is /classic/.  
I believe Jerry Weinberg uses it in "Becoming a Technical Leader", but it might
be "Secrets of Consulting": His general conclusion is that when people say "the
bug couldn't possibly be in there" you should *look in there*.
>
> Another short reason might be because 'just' testing the business logic for a
GUI feature isn't testing the whole black box.  To quote Bob Martin: "End to end
is further than you think."
>
> Of course, it's possible to 'cover' for not writing selenium tests in several
ways.  (or 'test it twice') You could do exploratory tests or have a
'beta'/staging program to generate coverage of the application, etc.
>
> Your email didn't speak to those other ways of covering the app; instead, you
asked "why not" "just" test at the Business Logic Level.
>
> /*Because you'll miss the bugs, dude!*/
>
> Asked and answered.
>
> regards,
>
>
>
> --
> Matthew Heusser,
> Personal Blog: http://xndev.blogspot.com/
> Test Community Blog: http://softwaretestpro.com/blog/
> Twitter: mheusser
>

--- In agile-testing@yahoogroups.com, "charles_bradley_scrum_coach"
<chuck-lists2@...> wrote:
>
>
> But George did say:
> > I'd also want to test that the GUI is hooked up correctly to the
> > data model and the actions of the business logic layer, of course.
>

Point taken, Charles.

Then again, when I see an Ad with  30 seconds about sugar smacks and 3 seconds
about "part of a balanced breakfast", I don't see that as actually promoting a
balanced breakfast.

Testing is a /skill/.  It takes practice and learning.  Seeing it minimized as
"oh yeah, you gotta do end to end testing too" is disappointing.

The comparison of breakfast choice to businesss layer and end to end tests is
(mostly) an exercise for the reader. :-)


regards,

--
Matthew Heusser,
Personal Blog: http://xndev.blogspot.com/
Test Community Blog: http://softwaretestpro.com/blog/
Twitter: mheusser