Mark,

It is a tough choice to find the right balance of unit, integration, and
system level testing. You really need to have some type of full end-to-end
suite that is fully automated to really "close the loop" for each build. Of
course, there will likely be some tests you can't automate, but you can
likely make some sacrifices and get something of value automated.

Some things to keep in mind:

- Have a unit testing environment that allows single units to be tested by
themselves. Another great feature, is to be able to track dependencies so
that whenever a dependency is touched that only the dependent tests need to
be rerun. Keeping developer testing quick is a key to ensuring that it is
done often. Hopefully with each change to the source.

- Have your CI system run all unit tests on a good simulator or the real
target. Although, in a medium to large project, the latter usually becomes
impractical, unless you schedule it for an overnight build.

- Integration tests using the simulator can be very valuable for proving out
subsystems before dumping them into the target. As James said, tracking down
these issues outside the real target is much more desirable than fumbling
with printfs and debuggers when you are on the target.

- System tests should not try and beat up the real system to the extent that
your unit tests do. It usually ends up being too much work and too much
duplication. System tests should focus on the high-level validation of
customer features, for the most part. Being high-level tests, they provide
an additional nets to catch behavior changes when a significant refactoring
is done.

Enjoy,
Greg

Greg Williams | Software Developer
Atomic Object | http://atomicobject.com
[Ph] +1 616 776 6020 [Fx] +1 616 776 6015


On Mon, Jun 15, 2009 at 8:07 AM, James Grenning <james@...> wrote:

>
>
> Hi Mark
>
> Integration is a pretty broad term, and there are many levels. Some
> amount can be done off target. But the ultimate, and maybe
> unattainable in some situations, is a fully automated end to end test
> suite. Depending on the application, that could result in
> considerable external instrumentation.
>
> I look at it in these 6 stages of incorporating of TDD and CI.
>
> Here is the CI approach I recommend (CI really begins in step 1):
> 0) Do TDD in host for the most rapid feedback and defect prevention.
> 1) Build and run in the host environment on checkin, check results
> 2) Compile unit test build for target or eval board
> 3) Push the UT build to the target and check results
> - in constrained memory situations you might need to chunk the tests
> 4) Run ATs (maybe in host too) in target (Acceptance tests at the sub-
> system and system level like you mentioned)
> 5) Run manual tests
>
> Some of these are going to be quite challenging. But that should not
> discourage getting this started. Software only integration can often
> be done off-target, at least some can be. When a team begins they can
> probably only practically do #0 and #5, or maybe #2 and #5. I know
> companies doing #0 and #5 and very happy with the improvement and the
> track they are on.
>
> The goal of this is to get warning of problems. I'd rather find a
> problem before it becomes a bug/defect. I'd rather track down the
> problems I can outside the target, so that when we get to the target
> the problems we could get ride of are gone from the code.
>
>
> James
>
> ----------------------
> James Grenning
> james@... <james%40grenning.net>
> www.renaissancesoftware.net
> www.renaissancesoftware.net/blog
> www.twitter.com/jwgrenning
>
> On Jun 13, 2009, at 9:12 AM, marktxx wrote:
>
> >
> >
> > --- In AgileEmbedded@yahoogroups.com <AgileEmbedded%40yahoogroups.com>,
> "davekelloggs"
> > <dave.kellogg@...> wrote:
> > >
> > > I'm working toward compiling my embedded code on a continuous
> > integration server. Any suggestions (for or against) using GCC vs
> > Microsoft Visual C as the compiler in that environment?
> > >
> > > The target is a 16-bit Freescale part, using CodeWarrior.
> >
> > As mentioned in other posts as well, IMHO real integration can only
> > be run on the target or with a target simulation. In the embedded
> > world, running the suite of unit tests on the host is not
> > "integration" but is an extremely important step in becoming agile.
> > Therefore for true integration you need to start developing your
> > system or sub-system level tests.
> >
> > Any other opinions?
> >
> >
> >
>
> [Non-text portions of this message have been removed]
>
>
>


[Non-text portions of this message have been removed]