On Mon, Aug 19, 2002 at 12:00:39PM +0200, Bram Moolenaar wrote:
>
> I ran into this issue and can't find a good solution.  I hope someone
> knows the answer to this one.
>
> When executing a shell command with ":system", the exit value is used to
> decide if the command worked or not.  The output of the shell command is
> redirected, so that it appears in the log file.  To also see the output
> while the command is executing "tee" is used.  The command passed to the
> shell looks like this:
>
>  { the-command; } 2>&1 | tee /tmp/@...
>
> In an ordinary shell, the exit value of the whole is the exit value of
> "tee".  But we need to get the exit value of "the-command".  How can
> this be obtained?  I tried a few things but couldn't make it work.

I've only a pointer for you. Hopefully someone will have a better
answer, but you should be able to make some creative use of the $?
special parameter?

Regards,
Doug

Bram wrote:
> I ran into this issue and can't find a good solution.  I hope someone
> knows the answer to this one.
>
> When executing a shell command with ":system", the exit value is used to
> decide if the command worked or not.  The output of the shell command is
> redirected, so that it appears in the log file.  To also see the output
> while the command is executing "tee" is used.  The command passed to the
> shell looks like this:
>
>    { the-command; } 2>&1 | tee /tmp/@...
>
> In an ordinary shell, the exit value of the whole is the exit value of
> "tee".  But we need to get the exit value of "the-command".  How can
> this be obtained?  I tried a few things but couldn't make it work.

Check google groups (groups.google.com). All solutions there read
something like this:

      { the-command; } 2>&1 ; echo $? > res.tmp | tee /tmp/@...

and then read the result from res.tmp. Some magic can return that as an
exit code in its own right, it seems.

Hope that helps,

Dion

I ran into this issue and can't find a good solution.  I hope someone
knows the answer to this one.

When executing a shell command with ":system", the exit value is used to
decide if the command worked or not.  The output of the shell command is
redirected, so that it appears in the log file.  To also see the output
while the command is executing "tee" is used.  The command passed to the
shell looks like this:

	 { the-command; } 2>&1 | tee /tmp/@...

In an ordinary shell, the exit value of the whole is the exit value of
"tee".  But we need to get the exit value of "the-command".  How can
this be obtained?  I tried a few things but couldn't make it work.

--
You can tune a file system, but you can't tuna fish
							 -- man tunefs

  ///  Bram Moolenaar -- Bram@... -- http://www.moolenaar.net  \\\
///   Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim   \\\
\\\           Project leader for A-A-P -- http://www.a-a-p.org           ///
  \\\ Lord Of The Rings helps Uganda - http://iccf-holland.org/lotr.html ///

Aschwin Marsman wrote:

> Is creating branches possible with the current implementation?

This is currently not supported.  Although you can use the generic:

	 :verscont action {attr = val} ... file ...

How do you create a branch in CVS anyway?  It's not obvious.

A-A-P attempts to provide a uniform interface to the version control
system being used, so that a casual user doesn't need to know all the
details of the commands.  However, for advanced use you still need to
fall back to the specific commands.  Thus the idea is that only a few
people need to know the details of the version control system (e.g., the
project leader or integrator).  All the users and most of the developers
should be able to do their work with A-A-P commands.

> > Current work is the Issue tracker wrapper.  This is supposed to make it
> > easy to submit a bug report.  Not an easy subject, since it appears
> > issue trackers are very different, and web-based ones require manual
> > entry by the user.  Supplying a text with instructions and what to fill
> > in which field seems the best to do.  But then viewing this text isn't
> > simple either, since there is a wide choice of text viewers.  Starting a
> > web browser with the URL of the issue tracker is useful, but this isn't
> > obvious either (except on MS-Windows).
>
> Maybe the BROWSER effort from Eric S. Raymond can help:
> http://tuxedo.org/~esr/BROWSER/
>
> A user can specify a BROWSER variable, which specifies how a browser
> should be started.

Thanks for the hint.  It has one problem though: since ":" is used to
separate alternatives, it is not possible to use a ":" in a browser
command.  Could be solved by using "%:".  He apparently didn't think of
MS-Windows...

> From the page:
>
> Who observes this convention?
>
> In the near future, any Python program that uses the standard-library
> webbrowser.py module. This module, which debuted in Python 2.0, was
> taught how to interpret BROWSER in CVS on 23 Jan 2001, and will be
> released in Python 2.1a2.

I'm still trying to support Python 1.5.  It's simple enough to implement
in A-A-P anyway.

> How many hours do you plan for a week?

Fourty.  But some hours are longer than others :-).

> > - Spend a week on the interface to the cross referencer.
> > - Spend a week on the interface to the GUI IDE
>
> Is it realistic to only do 2 weeks out of 8?

Not for the whole module.  I only plan to work on the interface.  That
means guessing what the module will do inside and not being able to test
the interface...

--
I AM THANKFUL...
...for the piles of laundry and ironing because it means I
have plenty of clothes to wear.

  ///  Bram Moolenaar -- Bram@... -- http://www.moolenaar.net  \\\
///   Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim   \\\
\\\           Project leader for A-A-P -- http://www.a-a-p.org           ///
  \\\ Lord Of The Rings helps Uganda - http://iccf-holland.org/lotr.html ///

On Thu, 1 Aug 2002, Bram Moolenaar wrote:

> Most of July was spend on integrating version control with the existing
> functionality.  Adding more functionality has the problem of making the
> use of A-A-P more complex.  Therefore an attempt has been done to make
> the functionality related to uploading and downloading more consistent.

Always good to keep consistency in mind, in this stage of the project,
those changes can be made without breaking anything important.

...

> The version control functionality has been implemented for CVS.  It has
> been tested by using it for A-A-P itself.  The files are now distributed
> by CVS, using a recipe.  Checking out the project from CVS and updating
> for a new version can be done with a simple recipe, a one-liner if you
> really want (see http://www.a-a-p.org/docs_recipe.html#cvs for an
> example).  The CVS-specific stuff has been put in a separate module.
> It should be possible to implement version control for another system by
> adding a module for it.  That remains to be tried out though, mainly
> because CVS doesn't do file locking while most other version control
> systems do lock files.

Code Manager & Bitkeeper aren't locking files also, and I like that
approach a lot better than other systems which do lock files.

Is creating branches possible with the current implementation?

...

> Current work is the Issue tracker wrapper.  This is supposed to make it
> easy to submit a bug report.  Not an easy subject, since it appears
> issue trackers are very different, and web-based ones require manual
> entry by the user.  Supplying a text with instructions and what to fill
> in which field seems the best to do.  But then viewing this text isn't
> simple either, since there is a wide choice of text viewers.  Starting a
> web browser with the URL of the issue tracker is useful, but this isn't
> obvious either (except on MS-Windows).

Maybe the BROWSER effort from Eric S. Raymond can help:
http://tuxedo.org/~esr/BROWSER/

A user can specify a BROWSER variable, which specifies how a browser
should be started.

From the page:

Who observes this convention?

In the near future, any Python program that uses the standard-library
webbrowser.py module. This module, which debuted in Python 2.0, was
taught how to interpret BROWSER in CVS on 23 Jan 2001, and will be
released in Python 2.1a2.

> Work and progress in July related to the original plan:
> - Two weeks of work on the version control wrapper.  Done enough for
>   version 0.1, just like the plan.
> - A couple of days spent on design; Not all of the planned two weeks
>   have been used up, this appears to go as planned.
> - Half a week of work on the recipe syntax, two of the planned three
>   weeks done.  Leaves a week for cleaning up and writing down the
>   syntax.
> - A week spent on the recipe executive: all of the planned six weeks
>   have been used now, but there is still some work to be done.

What's new in the real world ;-)

> - Half a week of work on the issue tracker, it's about halfway done.
>   Might take a few days more than the planned week.
>
> The original plan was to bring out a version 0.1 by the end of August.
> The work remaining to be done for this (as planned):
> - The cross referencer:  Nothing done yet, 4 weeks planned.
> - GUI IDE:  Nothing done yet, 4 weeks planned.
> - User documentation: Some pieces done, needs to be structured.

How many hours do you plan for a week?

...

> Thus August will include these activities:
> - Finish the issue tracker wrapper in a few days.
> - Spend a week on the interface to the cross referencer.
> - Spend a week on the interface to the GUI IDE
> - Make the documentation ready for version 0.1.  Also functions as a
>   test if the functionality is easy to explain.
> - Do enough testing and bug fixes to be able to release version 0.1.
>   Do a "proof of concept" for an application (e.g. Vim).

Is it realistic to only do 2 weeks out of 8?

..

Thanks for the detailed update, I hope to get more time in the future
to be able to do some of the work.

Aschwin Marsman

--
aYniK Software Solutions         all You need is Knowledge
Bedrijvenpark Twente 305         NL-7602 KL Almelo - the Netherlands
P.O. box 134                     NL-7600 AC Almelo - the Netherlands
telephone: +31 (0)546-581400     fax: +31 (0)546-581401
a.marsman@...              http://www.aYniK.com

Most of July was spend on integrating version control with the existing
functionality.  Adding more functionality has the problem of making the
use of A-A-P more complex.  Therefore an attempt has been done to make
the functionality related to uploading and downloading more consistent.
Fortunately, the version control fits in there quite nicely.  For
example, "aap refresh" is now used to obtain a fresh version of a
project, no matter what method is used to obtain the files.  The recipe
specifies whether to use ftp, http, scp or CVS with the "refresh"
attribute.  Other methods can be added later without much effort.  The
command ":refresh" can be used to download a specific file and
":refreshall" to download all outdated files.  Also see design decision
17: http://www.a-a-p.org/decisions.html#17.  This consistent naming also
applies to "publish" for uploading.  For version control the "commit"
attribute is used for several commands and targets, such as "checkout",
"unlock" and "tag".  This consistency makes it easier to remember the
various targets and commands.

The version control functionality has been implemented for CVS.  It has
been tested by using it for A-A-P itself.  The files are now distributed
by CVS, using a recipe.  Checking out the project from CVS and updating
for a new version can be done with a simple recipe, a one-liner if you
really want (see http://www.a-a-p.org/docs_recipe.html#cvs for an
example).  The CVS-specific stuff has been put in a separate module.
It should be possible to implement version control for another system by
adding a module for it.  That remains to be tried out though, mainly
because CVS doesn't do file locking while most other version control
systems do lock files.

With all these new attributes a recipe starts to look cluttered.  I
added the ":attribute" command to be able to separate specifying the
files and the attributes.  This makes it possible to separate the build
dependencies from specifying the upload and download attributes.  That
looks much better without adding a new mechanism, just one extra command.

Uploading has also been made simpler.  "aap publish" takes a list of
files and uploads the files that have been modified.  The recipe for
this can be as short as two lines.  This is now used to publish the
A-A-P and Vim web pages (see
http://www.a-a-p.org/docs_recipe.html#publish for an example).  For this
to work a few problems with scp and remembering signatures had to be
solved.  Fortunately this had hardly any effect on the recipe syntax:
only added the "signdirectory" attribute to be used for exceptional
situations (design decision 16).

Gradually the program becomes more robust.  A log is now kept for the
last ten invocations, so that it is possible to find out what happened.
The log contains more messages than the output to the screen.  This can
be used to figure out problems in the recipe and bugs in the program.
Thus it's both useful for end users and for A-A-P developers.

Current work is the Issue tracker wrapper.  This is supposed to make it
easy to submit a bug report.  Not an easy subject, since it appears
issue trackers are very different, and web-based ones require manual
entry by the user.  Supplying a text with instructions and what to fill
in which field seems the best to do.  But then viewing this text isn't
simple either, since there is a wide choice of text viewers.  Starting a
web browser with the URL of the issue tracker is useful, but this isn't
obvious either (except on MS-Windows).

Work and progress in July related to the original plan:
- Two weeks of work on the version control wrapper.  Done enough for
   version 0.1, just like the plan.
- A couple of days spent on design; Not all of the planned two weeks
   have been used up, this appears to go as planned.
- Half a week of work on the recipe syntax, two of the planned three
   weeks done.  Leaves a week for cleaning up and writing down the
   syntax.
- A week spent on the recipe executive: all of the planned six weeks
   have been used now, but there is still some work to be done.
- Half a week of work on the issue tracker, it's about halfway done.
   Might take a few days more than the planned week.

The original plan was to bring out a version 0.1 by the end of August.
The work remaining to be done for this (as planned):
- The cross referencer:  Nothing done yet, 4 weeks planned.
- GUI IDE:  Nothing done yet, 4 weeks planned.
- User documentation: Some pieces done, needs to be structured.

Unfortunately, the volunteers that were working on the cross referencer
and the GUI IDE have not produced anything usable yet.  Waiting for them
to produce results makes the 0.1 release date highly uncertain.  And the
0.1 release is needed to get people interested in helping A-A-P
development.  To get out of this deadlock I will do a minimal amount of
work on the recipe syntax part of the cross referencer and the GUI IDE.
This means taking a risc on whether this can be done without fully
exploring the issues involved.  It's possible that the recipe syntax has
to be adjusted later.  Considering the stability of the recipe syntax so
far, this risc is small enough to take now.

Thus August will include these activities:
- Finish the issue tracker wrapper in a few days.
- Spend a week on the interface to the cross referencer.
- Spend a week on the interface to the GUI IDE
- Make the documentation ready for version 0.1.  Also functions as a
   test if the functionality is easy to explain.
- Do enough testing and bug fixes to be able to release version 0.1.
   Do a "proof of concept" for an application (e.g. Vim).

With a bit of luck version 0.1 will be ready by the end of August and
stable enough to be useful.  A bit of promotion needs to be done then.
Hopefully this results in feedback and discussions about the choices.

For September packages and ports are to be worked on.  This isn't in the
original plan and was added because of the feedback from the poster
presentation at SANE and the work for the upcoming presentation at
FreeBSDcon in November about using A-A-P for BSD ports.

--
From "know your smileys":
  !-| I-am-a-Cylon-Centurian-with-one-red-eye-bouncing-back-and-forth

  ///  Bram Moolenaar -- Bram@... -- http://www.moolenaar.net  \\\
///   Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim   \\\
\\\           Project leader for A-A-P -- http://www.a-a-p.org           ///
  \\\ Lord Of The Rings helps Uganda - http://iccf-holland.org/lotr.html ///

In case you didn't notice it yet, the current version of the Recipe
Executive can obtain files from a CVS repository (this requires the
"cvs" command, Vim doesn't do it all by itself!).

Because of the chicken-egg problem you still need to start with a CVS
command:

	 cvs -d:pserver:anonymous@...:/cvsroot/a-a-p checkout Exec

This creates the "Exec" directory and puts the source files in there.
After that Aap can refresh itself with this simple recipe:

	 refresh:
		 :refresh {origin = cvs://} .

Put this recipe as "main.aap" in the Exec directory and do "./Main.py"
(the "main.aap" recipe is the default and "refresh" is the default
target since it comes first).  CVS will think about it and refresh the
changed files for you.


However, I have been reconsidering the names of the targets, commands
and attributes to be used for accessing a Version Control System (VCS).
The basic things to offer to a user are:

	 refresh:  obtain the latest version from the VCS
	 checkout: obtain a file to work on (lock the file)
	 commit:   write changes back to the VCS (keeping it locked)
	 checkin:  write changes back to the VCS (unlocking it)
	 unlock:   unlock without writing changes

The locking isn't done for systems that don't support it, such as CVS.
"refresh" and "checkout" do the same thing then.  "commit" and "checkin"
are also equal for CVS.

Additionally, there needs to be a target for sending out a new release
to the world.  I thought of several names for this:

	 distribute
	 release
	 publish

Currently I think "publish" is the best.  "distribute" is clear but
long.  "release" is confusing: VCS systems use it to undo a checkout
(release a lock).

To make consistent use of these names, I intend to use the same name for
the command that does the work and for the attribute that specifies how
to do it.  For a start, this would mean changing the "origin" attribute
to "refresh".

A few examples to see what this all means:


	 SOURCE = main.c version.c
	 :attr {refresh = ftp://ftp.foo.org/pub/foo/%file%} $SOURCE

	 refresh:
		 :refresh $SOURCE

This says that the source files can be obtained from an ftp server
(%file% is replaced by the file name).  The "refresh" target does the
work of obtaining the files (this happens always, since ftp has no
reliable way to tell if a file actually changed).  The "refresh" target
here actually does the same as what would be done automatically, thus it
could be omitted.


Since "checkout", "commit", "checkin" and "unlock" use the same VCS,
only one of them needs to be used as an attribute.  I think using
"commit" is best.

	 SOURCE = main.c version.c
	 :attr {commit = cvs://:ext:$FOOUSER@cvs.foo.sf.net:/cvsroot/foo}

	 checkout:
		 :checkout $SOURCE
	 commit:
		 :commit $SOURCE
	 checkin:
		 :checkin $SOURCE
	 unlock:
		 :unlock $SOURCE


The difference between "commit" and "publish" is that "commit" mostly
only involve source files and "publish" may also include generated
files, such as executables.  "commit" is for using a Version Control
System while "publish" may upload to an ftp site.

	 SOURCE = main.c version.c
	 TARGET = prog
	 ZIPFILE = prog.zip
	 PUBLISHED = $SOURCE $TARGET

	 :attr {publish = scp://$FOOUSER@ftp.foo.org/foo} $ZIPFILE

	 $ZIPFILE: $PUBLISHED
		 :sys zip $ZIPFILE $PUBLISHED
	 publish: $ZIPFILE
		 :publish $ZIPFILE

Instead of using the ":attr" command to add the "publish" attribute and
then using it for the ":publish" command further on, it's possible to do
this at once by specifying the attribute with the ":publish" command:

	 :publish {publish = scp://$FOOUSER@ftp.foo.org/foo} $ZIPFILE

Putting the "publish" attribute at the ":publish" command looks a bit
strange, but it's consistent with how parameters are specified.


Comments?

--
Sometimes you can protect millions of dollars in your budget simply by buying
a bag of cookies, dropping it on the budget anylyst's desk, and saying
something deeply personal such as "How was your weekend, big guy?"
				 (Scott Adams - The Dilbert principle)

  ///  Bram Moolenaar -- Bram@... -- http://www.moolenaar.net  \\\
///   Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim   \\\
\\\           Project leader for A-A-P -- http://www.a-a-p.org           ///
  \\\  Help me helping AIDS orphans in Uganda - http://iccf-holland.org  ///

Aschwin Marsman wrote:

> > Hmm, I think the program currently requires Python 2.  I might have an
> > attempt to make it work with Python 1.5 again (later).  I'll check to
> > see if it's possible to detect the presence of python2.  However, this
> > won't be possible in the scripts that start with
> > "#!/usr/bin/env python".  Unless someone knows a trick.
>
> Isn't python 2 always installed at python2, or is this a RH invention?

It isn't on my system (FreeBSD): "python" is a hard link to either
"python1.5" or "python2.2".

I have now changed the test script to use "sys.executable".  This means
the same python executable that was used to start the test is used to
execute the individual tests.  Please try if it works now.

I also tried using "execfile()", but soon got bitten by side effects
from the test scripts (they don't end up in the directory they started
in).  It's better to run each test in a clean python interpreter.
Hopefully the slowdown isn't too much.

The problem of using "python" in the first line of the script remains.
I can't put "python2" there, it would not work on my own system.
And I also can't put an command like "if redhat then..." there.  What I
could to is check for running Python 1.5 and then try finding a
"python2" executable.  However, restarting with that executable might
have a few side effects.

> > This really requires Python 2.  The gettext support wasn't very good in
> > 1.5.  I could detect this and skip the gettext support for 1.5 to work
> > around this.  Until running into the next problem...
>
> Doesn't sound like a good idea.

I did avoid the trouble with gettext and indeed ran into the next
problem.  It seems a local method has another way of handling the local
variables of the function it is defined in.  I didn't find an easy way
to solve this, thus it's still not working with Python 1.5.

--
A law to reduce crime states: "It is mandatory for a motorist with criminal
intentions to stop at the city limits and telephone the chief of police as he
is entering the town.
		 [real standing law in Washington, United States of America]

  ///  Bram Moolenaar -- Bram@... -- http://www.moolenaar.net  \\\
///   Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim   \\\
\\\           Project leader for A-A-P -- http://www.a-a-p.org           ///
  \\\  Help me helping AIDS orphans in Uganda - http://iccf-holland.org  ///

On Tue, 9 Jul 2002, Bram Moolenaar wrote:

> Aschwin Marsman wrote:

> > cvs commit is not enough?
>
> That doesn't add new files or remove old files.  The recipe I'm using
> for creating the .zip archive already contains a list of all the files
> that are to be distributed.  I want to edit this list only once, it
> should also be used for CVS.  This also makes sure the zip archive
> contains the same items as what is in CVS.

Good point, should be easy (cvs add, cvs remove)

> > > Having read part of the CVS documentation reminded me how complicated
> > > this is for distributing software.  I'll also try to think of a simple
> > > way that just uses files on an ftp or http server.
> >
> > When you are used to it, it's no problem.
>
> I think it's still a problem.  Example: When I imported the A-A-P files
> to the SourceForge CVS server, after reading their documentation, I
> discovered they had been put in "Exec/Exec", thus one directory too
> deep.  Think I could quickly fix this, I made another mistake and ended
> up with the "Exec" files in the top directory.  The third time I did it
> right, the "Exec" directory had the right files.
>
> Now, I can delete the files from the wrong directories, but I can't
> delete the empty directories.  On a local system you would bypass CVS
> and do this manually.  The SourceForge CVS server doesn't have shell
> access, thus I had to submit a support request (which was handled in
> only a few hours, thankfully to the SourceForge staff!).
>
> Not only did this show there are still things you can't do with CVS, it
> also shows it doesn't have a proper "undo" mechamism (that's not the
> same as rolling back versions).  And it's not too difficult to make
> mistakes.

For a SCM system, you don't want to change history. But I do agree,
that a little misstake can be made, and that it can't be corrected
using cvs commands alone.

> Add to that the magic commands and options and URLs you have to type,
> and you quickly decide to make a script to do this stuff.  Or at least a
> shell alias.

That's what UNIX is all about.

> > I'm having problems with gettings Exec to work.
> >
> > I'm running RH 7.3, which installs the following for Python:
> >
> > $ rpm -q python
> > python-1.5.2-38
> > $ rpm -q python2
> > python2-2.2-16
> >
> > The scripts look for python, which is python 1.5.2 ;-(
>
> Hmm, I think the program currently requires Python 2.  I might have an
> attempt to make it work with Python 1.5 again (later).  I'll check to
> see if it's possible to detect the presence of python2.  However, this
> won't be possible in the scripts that start with
> "#!/usr/bin/env python".  Unless someone knows a trick.

Isn't python 2 always installed at python2, or is this a RH invention?

> > Finding help gives:
> >
> > $ ./Main.py -v -help
> > Traceback (innermost last):
> >   File "./Main.py", line 96, in ?
> >     main()
> >   File "./Main.py", line 48, in main
> >     i18n_init()
> >   File "./Util.py", line 39, in i18n_init
> >     gettext.install("a-a-p")
> > AttributeError: install
> >
> > I haven't done anything with Python yet, so it's probably easy to fix.
>
> This really requires Python 2.  The gettext support wasn't very good in
> 1.5.  I could detect this and skip the gettext support for 1.5 to work
> around this.  Until running into the next problem...

Doesn't sound like a good idea.

> > Maybe a FAQ for problems like this is a good idea.
>
> It's mentioned in the README.txt file.

The requirement for python 2 is in the FAQ, and I had python 2 installed,
but it wasn't used...

> > What are the dependencies of the project?
> > Which tools have to be installed in order for letting it work?
>
> Just Python 2.  And I haven't tried it on anything else but Unix, thus
> it probably doesn't work on MS-Windows yet.
> Obviously you need an "scp" command to be able to use an "scp://" URL for
> ":copy".

Keep up the good work,

Best regards,

Aschwin Marsman

--
aYniK Software Solutions         all You need is Knowledge
Bedrijvenpark Twente 305         NL-7602 KL Almelo - the Netherlands
P.O. box 134                     NL-7600 AC Almelo - the Netherlands
telephone: +31 (0)546-581400     fax: +31 (0)546-581401
a.marsman@...              http://www.aYniK.com

Aschwin Marsman wrote:

> > Today I have started using the CVS repository at SourceForge.  You
> > should be able to checkout the "Exec" module, which is the Recipe
> > Executive.  See http://www.a-a-p.org/download.html for more info.
>
> This works, thanks. For updating, only cvs update in the Exec
> directory is needed, when you've already checked out a version.

Yep, that's easy.

> > I'm currently working on making it easy to upload to the CVS repository,
> > avoiding the need to type all kinds of magic commands and URLs.  Soon
> > uploading should be as simple as "aap upload".  Actually, this already
> > works for existing files, but I also want to automatically add and
> > remove files.  That makes it work just like packing the files in an
> > archive and distributing this.
>
> cvs commit is not enough?

That doesn't add new files or remove old files.  The recipe I'm using
for creating the .zip archive already contains a list of all the files
that are to be distributed.  I want to edit this list only once, it
should also be used for CVS.  This also makes sure the zip archive
contains the same items as what is in CVS.

> > Having read part of the CVS documentation reminded me how complicated
> > this is for distributing software.  I'll also try to think of a simple
> > way that just uses files on an ftp or http server.
>
> When you are used to it, it's no problem.

I think it's still a problem.  Example: When I imported the A-A-P files
to the SourceForge CVS server, after reading their documentation, I
discovered they had been put in "Exec/Exec", thus one directory too
deep.  Think I could quickly fix this, I made another mistake and ended
up with the "Exec" files in the top directory.  The third time I did it
right, the "Exec" directory had the right files.

Now, I can delete the files from the wrong directories, but I can't
delete the empty directories.  On a local system you would bypass CVS
and do this manually.  The SourceForge CVS server doesn't have shell
access, thus I had to submit a support request (which was handled in
only a few hours, thankfully to the SourceForge staff!).

Not only did this show there are still things you can't do with CVS, it
also shows it doesn't have a proper "undo" mechamism (that's not the
same as rolling back versions).  And it's not too difficult to make
mistakes.

Add to that the magic commands and options and URLs you have to type,
and you quickly decide to make a script to do this stuff.  Or at least a
shell alias.

> I'm having problems with gettings Exec to work.
>
> I'm running RH 7.3, which installs the following for Python:
>
> $ rpm -q python
> python-1.5.2-38
> $ rpm -q python2
> python2-2.2-16
>
> The scripts look for python, which is python 1.5.2 ;-(

Hmm, I think the program currently requires Python 2.  I might have an
attempt to make it work with Python 1.5 again (later).  I'll check to
see if it's possible to detect the presence of python2.  However, this
won't be possible in the scripts that start with
"#!/usr/bin/env python".  Unless someone knows a trick.

> Finding help gives:
>
> $ ./Main.py -v -help
> Traceback (innermost last):
>   File "./Main.py", line 96, in ?
>     main()
>   File "./Main.py", line 48, in main
>     i18n_init()
>   File "./Util.py", line 39, in i18n_init
>     gettext.install("a-a-p")
> AttributeError: install
>
> I haven't done anything with Python yet, so it's probably easy to fix.

This really requires Python 2.  The gettext support wasn't very good in
1.5.  I could detect this and skip the gettext support for 1.5 to work
around this.  Until running into the next problem...

> Maybe a FAQ for problems like this is a good idea.

It's mentioned in the README.txt file.

> What are the dependencies of the project?
> Which tools have to be installed in order for letting it work?

Just Python 2.  And I haven't tried it on anything else but Unix, thus
it probably doesn't work on MS-Windows yet.
Obviously you need an "scp" command to be able to use an "scp://" URL for
":copy".

--
Snoring is prohibited unless all bedroom windows are closed and securely
locked.
		 [real standing law in Massachusetts, United States of America]

  ///  Bram Moolenaar -- Bram@... -- http://www.moolenaar.net  \\\
///   Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim   \\\
\\\           Project leader for A-A-P -- http://www.a-a-p.org           ///
  \\\  Help me helping AIDS orphans in Uganda - http://iccf-holland.org  ///

Thorsten -

> On Mon, 08 Jul 2002 22:01:06 +0200
> Bram Moolenaar <bram@...> wrote:
>
> > I haven't yet found a way to avoid having to type the password each
> > time.  Ssh is used for authentication, and it does allow direct login by
> > storing the host key, but the SourceForge CVS server doesn't have shell
> > access.
>
> Sourceforge allows you to upload keys via web interface only.
> 1. Login ( top/left at http://sf.net )
> 2. Choose Account Options (at top of screen)
> 3. Choose "Edit Keys" (at bottom of screen)
> 4. Upload your keys and wait some hours...
>
> (I hope you didnt think of automatic keyupload).

Thanks for the hint!  I didn't know this also worked for the CVS server.

--
Why don't cannibals eat clowns?
Because they taste funny.

  ///  Bram Moolenaar -- Bram@... -- http://www.moolenaar.net  \\\
///   Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim   \\\
\\\           Project leader for A-A-P -- http://www.a-a-p.org           ///
  \\\  Help me helping AIDS orphans in Uganda - http://iccf-holland.org  ///

On Mon, 8 Jul 2002, Bram Moolenaar wrote:

>
> Aschwin Marsman wrote:
>
> > Is there a possibility to create a CVS store (another is also ok),
> > in order to have an easy update possibility for a-a-p?
> >
> > In a fast changing project like this, easy entrance to the latest
> > release is crucial. With CVS, you also get a better look at
> > previous versions.
> >
> > Until a-a-p gives us the desired functionality, a CVS repository
> > would be very welcome.
>
> Today I have started using the CVS repository at SourceForge.  You
> should be able to checkout the "Exec" module, which is the Recipe
> Executive.  See http://www.a-a-p.org/download.html for more info.

This works, thanks. For updating, only cvs update in the Exec
directory is needed, when you've already checked out a version.

> I'm currently working on making it easy to upload to the CVS repository,
> avoiding the need to type all kinds of magic commands and URLs.  Soon
> uploading should be as simple as "aap upload".  Actually, this already
> works for existing files, but I also want to automatically add and
> remove files.  That makes it work just like packing the files in an
> archive and distributing this.

cvs commit is not enough?

> I haven't yet found a way to avoid having to type the password each
> time.  Ssh is used for authentication, and it does allow direct login by
> storing the host key, but the SourceForge CVS server doesn't have shell
> access.
>
> Adding automatic updating from CVS should be the next step (and it's
> probably quite simple).
>
> Having read part of the CVS documentation reminded me how complicated
> this is for distributing software.  I'll also try to think of a simple
> way that just uses files on an ftp or http server.

When you are used to it, it's no problem.

I'm having problems with gettings Exec to work.

I'm running RH 7.3, which installs the following for Python:

$ rpm -q python
python-1.5.2-38
$ rpm -q python2
python2-2.2-16

The scripts look for python, which is python 1.5.2 ;-(

Finding help gives:

$ ./Main.py -v -help
Traceback (innermost last):
   File "./Main.py", line 96, in ?
     main()
   File "./Main.py", line 48, in main
     i18n_init()
   File "./Util.py", line 39, in i18n_init
     gettext.install("a-a-p")
AttributeError: install

I haven't done anything with Python yet, so it's probably easy to fix.
Maybe a FAQ for problems like this is a good idea.

What are the dependencies of the project?
Which tools have to be installed in order for letting it work?

Best regards,

Aschwin Marsman

--
aYniK Software Solutions         all You need is Knowledge
Bedrijvenpark Twente 305         NL-7602 KL Almelo - the Netherlands
P.O. box 134                     NL-7600 AC Almelo - the Netherlands
telephone: +31 (0)546-581400     fax: +31 (0)546-581401
a.marsman@...              http://www.aYniK.com

Hi Bram,

On Mon, 08 Jul 2002 22:01:06 +0200
Bram Moolenaar <bram@...> wrote:

> I haven't yet found a way to avoid having to type the password each
> time.  Ssh is used for authentication, and it does allow direct login by
> storing the host key, but the SourceForge CVS server doesn't have shell
> access.

Sourceforge allows you to upload keys via web interface only.
1. Login ( top/left at http://sf.net )
2. Choose Account Options (at top of screen)
3. Choose "Edit Keys" (at bottom of screen)
4. Upload your keys and wait some hours...

(I hope you didnt think of automatic keyupload).

HTH
--
Thorsten Maerz          <torte@...>
Sylpheed-claws/Win32:   http://claws-w32.sf.net

Aschwin Marsman wrote:

> Is there a possibility to create a CVS store (another is also ok),
> in order to have an easy update possibility for a-a-p?
>
> In a fast changing project like this, easy entrance to the latest
> release is crucial. With CVS, you also get a better look at
> previous versions.
>
> Until a-a-p gives us the desired functionality, a CVS repository
> would be very welcome.

Today I have started using the CVS repository at SourceForge.  You
should be able to checkout the "Exec" module, which is the Recipe
Executive.  See http://www.a-a-p.org/download.html for more info.

I'm currently working on making it easy to upload to the CVS repository,
avoiding the need to type all kinds of magic commands and URLs.  Soon
uploading should be as simple as "aap upload".  Actually, this already
works for existing files, but I also want to automatically add and
remove files.  That makes it work just like packing the files in an
archive and distributing this.

I haven't yet found a way to avoid having to type the password each
time.  Ssh is used for authentication, and it does allow direct login by
storing the host key, but the SourceForge CVS server doesn't have shell
access.

Adding automatic updating from CVS should be the next step (and it's
probably quite simple).

Having read part of the CVS documentation reminded me how complicated
this is for distributing software.  I'll also try to think of a simple
way that just uses files on an ftp or http server.

--
It is illegal for anyone to give lighted cigars to dogs, cats, and other
domesticated animal kept as pets.
		 [real standing law in Illinois, United States of America]

  ///  Bram Moolenaar -- Bram@... -- http://www.moolenaar.net  \\\
///   Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim   \\\
\\\           Project leader for A-A-P -- http://www.a-a-p.org           ///
  \\\  Help me helping AIDS orphans in Uganda - http://iccf-holland.org  ///

Hello,

Is there a possibility to create a CVS store (another is also ok),
in order to have an easy update possibility for a-a-p?

In a fast changing project like this, easy entrance to the latest
release is crucial. With CVS, you also get a better look at
previous versions.

Until a-a-p gives us the desired functionality, a CVS repository
would be very welcome.

Kind regards,

Aschwin Marsman

--
aYniK Software Solutions         all You need is Knowledge
Bedrijvenpark Twente 305         NL-7602 KL Almelo - the Netherlands
P.O. box 134                     NL-7600 AC Almelo - the Netherlands
telephone: +31 (0)546-581400     fax: +31 (0)546-581401
a.marsman@...              http://www.aYniK.com

Steven Knight wrote:

> > > > Furthermore, I have added a mechanism for testing.  Both for testing
> > > > individual modules and executing recipes.  The typeless nature of Python
> > > > requires doing a lot of tests.  Only a few have been added so far, but
> > > > one bug has already been detected.
>
> I'd highly recommend implementing a testing discipline in your process
> up front.  On SCons, we have a policy (adopted from and enforced by
> Aegis) that every checkin must be accompanied by one or more new or
> changed tests that demonstrate the bug or feature by failing on the old
> code base.  By building up our tests incrementally, we now have a very
> complete set of unit tests for our modules, and a regression test suite
> of 163 end-to-end tests.
>
> I've found this way of working to be especially crucial during the early
> stages of development, when you may need to refactor parts of the design
> frequently as you add features.  Being able to do this and *know* you
> haven't broken anything because all of the tests pass (a la eXtreme
> Programming) is invaluable.

Although I agree with doing a lot of testing, A-A-P is still in a phase
where the design is more important than correct code.  The main goal
currently is to quickly get to a stable recipe format and find good
interfaces between modules.  This means I do want to find bugs that
influence the design, but don't care much for details.  Actually, I
should use tests that check if the specifications are right.  That means
using real-life test cases.

> Python will throw an exception if you try to use a function that hasn't
> been imported.  It won't throw an exception just because you pass a
> variable of the wrong type, but it will if/when you try to use that
> variable inappropriately.

Thus this basically means the tests must cause every function to be used
at least once, so that an exception will be thrown when it wasn't
imported.  Would require a coverage checker to check if the tests really
did this.

> In practice, we've found that the ongoing challenge is to not just rely
> on generic exception-handling, which is great for programmer debugging,
> but to catch exceptions at all of the appropriate points and provide
> error messages more helpful for less sophisticated end-users.

That's something I'm doing already, since it greatly helps quickly
locating a problem.  A try/except block around every function that can
throw an IOError, for example.

> > Perhaps some kind of Python-lint exists, I haven't looked for it yet.
>
> Look for PyChecker, available on SourceForge.  It finds a number of
> problems in Python code by performing analysis on the compiled byte
> code.

I'll have a look at it later.

--
CRONE:  Who sent you?
ARTHUR: The Knights Who Say Ni!
CRONE:  Aaaagh!  (she looks around in rear) No!  We have no shrubberies here.
                  "Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

  ///  Bram Moolenaar -- Bram@... -- http://www.moolenaar.net  \\\
///   Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim   \\\
\\\           Project leader for A-A-P -- http://www.a-a-p.org           ///
  \\\  Help me helping AIDS orphans in Uganda - http://iccf-holland.org  ///

> > > Furthermore, I have added a mechanism for testing.  Both for testing
> > > individual modules and executing recipes.  The typeless nature of Python
> > > requires doing a lot of tests.  Only a few have been added so far, but
> > > one bug has already been detected.

I'd highly recommend implementing a testing discipline in your process
up front.  On SCons, we have a policy (adopted from and enforced by
Aegis) that every checkin must be accompanied by one or more new or
changed tests that demonstrate the bug or feature by failing on the old
code base.  By building up our tests incrementally, we now have a very
complete set of unit tests for our modules, and a regression test suite
of 163 end-to-end tests.

I've found this way of working to be especially crucial during the early
stages of development, when you may need to refactor parts of the design
frequently as you add features.  Being able to do this and *know* you
haven't broken anything because all of the tests pass (a la eXtreme
Programming) is invaluable.

> > Isn't the typeless nature of python a problem when you try to build
> > good solid code? I don't like the Java - we do the checks runtime and
> > generate an exception when something goes wrong - attitude. You can't
> > have a 100% test coverage. I like the C++ way - strong typechecking,
> > we don't want to loose performance runtime - better. Do you need
> > lint-like tools to overcome the typeless nature of Python, or is your
> > code 80% error recovery, 20% functionality?
>
> It's true that Python leaves most of the testing to the user.  The
> largest potential problem I see is using a function that wasn't imported
> and passing a variable of the wrong type to a function.  This can only
> be checked with a large set of tests.  Mostly written with one eye on
> the code and making sure each line is used at least once in a test.

Python will throw an exception if you try to use a function that hasn't
been imported.  It won't throw an exception just because you pass a
variable of the wrong type, but it will if/when you try to use that
variable inappropriately.

In practice, we've found that the ongoing challenge is to not just rely
on generic exception-handling, which is great for programmer debugging,
but to catch exceptions at all of the appropriate points and provide
error messages more helpful for less sophisticated end-users.

> Perhaps some kind of Python-lint exists, I haven't looked for it yet.

Look for PyChecker, available on SourceForge.  It finds a number of
problems in Python code by performing analysis on the compiled byte
code.

	 --SK

Aschwin Marsman wrote:

> > Furthermore, I have added a mechanism for testing.  Both for testing
> > individual modules and executing recipes.  The typeless nature of Python
> > requires doing a lot of tests.  Only a few have been added so far, but
> > one bug has already been detected.
>
> Isn't the typeless nature of python a problem when you try to build
> good solid code? I don't like the Java - we do the checks runtime and
> generate an exception when something goes wrong - attitude. You can't
> have a 100% test coverage. I like the C++ way - strong typechecking,
> we don't want to loose performance runtime - better. Do you need
> lint-like tools to overcome the typeless nature of Python, or is your
> code 80% error recovery, 20% functionality?

It's true that Python leaves most of the testing to the user.  The
largest potential problem I see is using a function that wasn't imported
and passing a variable of the wrong type to a function.  This can only
be checked with a large set of tests.  Mostly written with one eye on
the code and making sure each line is used at least once in a test.

Perhaps some kind of Python-lint exists, I haven't looked for it yet.

There is one big "Internal error" exception handler.  Works quite well
to locate a problem and fix it.  User errors are handled separately.

> > Overall the project is very well on schedule, except for the cross
> > referencer and GUI IDE modules.
>
> How critical are those for a 0.1 release? Can work be broken into pieces?

They must be implemented partly to verify the recipe syntax.  Otherwise
there is quite a big risk of changing the recipe syntax later, which
would involve much more work.  And the modules will have to be worked on
anyway, so let's do it in the right order.  I have not worked out
alternate solutions yet, but they probably result in myself doing part
of the work.

> I don't like the Yahoo mail archive with advertisement pages between
> emails, are there alternatives?

The sourceforge maillists add an advertisement to each message...

--
If your life is a hard drive,
Christ can be your backup.

  ///  Bram Moolenaar -- Bram@... -- http://www.moolenaar.net  \\\
///   Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim   \\\
\\\           Project leader for A-A-P -- http://www.a-a-p.org           ///
  \\\  Help me helping AIDS orphans in Uganda - http://iccf-holland.org  ///

On Mon, 1 Jul 2002, Bram Moolenaar wrote:

Thanks for this update.

...

> Furthermore, I have added a mechanism for testing.  Both for testing
> individual modules and executing recipes.  The typeless nature of Python
> requires doing a lot of tests.  Only a few have been added so far, but
> one bug has already been detected.

Isn't the typeless nature of python a problem when you try to build
good solid code? I don't like the Java - we do the checks runtime and
generate an exception when something goes wrong - attitude. You can't
have a 100% test coverage. I like the C++ way - strong typechecking,
we don't want to loose performance runtime - better. Do you need
lint-like tools to overcome the typeless nature of Python, or is your
code 80% error recovery, 20% functionality?

> Ingo Krabbe did not report progress for the cross reference module.
>
> I have asked around to find someone to work on the GUI IDE module.
> Several people responded that it would be a nice job, but they didn't
> have time.  Thomas Scott Urban volunteered to start working on it.

I was one of them ;-(

> He has not reported progress yet.
>
> I have written an abstract for a presentation at the BSDcon Europe
> conference.  An answer can be expected before July 12.  If the paper is
> accepted, I will have to spend time on writing the paper and study
> the ports system and how to replace it with A-A-P.  This will take one
> or two weeks.  Although most of the work would have to be done anyway,
> it was not really required for release 0.1.

The more publicity, the more hands to do the work.

> The time invested this month was about 2 weeks in the recipe executive,
> which means five of the planned six weeks have been done.  But it looks
> like this is going to take a bit more time than planned (there is also
> more functionality than originally planned).  I spent less than the
> planned one week on the dependency checker, but I consider it done for
> version 0.1.  Adding the filetype detection module wasn't planned, this
> took a couple of days.  A few days have been spent on design, updating
> the web site and the recipe syntax.

Nice, a project leader who's giving insight in his own work ;-)

> Overall the project is very well on schedule, except for the cross
> referencer and GUI IDE modules.

How critical are those for a 0.1 release? Can work be broken into pieces?

> The coming weeks I plan to work on version control.  I want to start
> using CVS for distributing A-A-P.  At the same time, the recipe should
> help to upload and download A-A-P with CVS.  This is the version control
> wrapper module.  Related to this is the issue tracker wrapper, thus this
> is probably the next module to be taken care of.

Sounds good to me.

I don't like the Yahoo mail archive with advertisement pages between
emails, are there alternatives?

Kind regards,

Aschwin Marsman

--
aYniK Software Solutions         all You need is Knowledge
Bedrijvenpark Twente 305         NL-7602 KL Almelo - the Netherlands
P.O. box 134                     NL-7600 AC Almelo - the Netherlands
telephone: +31 (0)546-581400     fax: +31 (0)546-581401
a.marsman@...              http://www.aYniK.com

In June the recipe executive module was further enhanced.  Quite a few
tough issues have been dealth with.  The choice for mixing Makefile and
Python syntax has again been proven to be a good one.

One change has been made to simplify the structure: The commands at the
toplevel and the build commands are now the same.  Previously, commands
without a name at the toplevel were assignments and for build commands
they were system (shell) commands.  Now the toplevel syntax is also used
for build commands.  This means that system commands must be preceded
with ":sys" or ":system".  This makes the structure more consistent, but
the disadvantage is that this is different from Makefile syntax.  See
http://www.a-a-p.org/decisions.html#12.

New features:
- Added the ":origin" command: Allow a recipe to update itself from a
   URL automatically.  The "origin" attribute on the ":child" command
   does a similar thing.
- Added pipes: Use the output of one command as the input for the next
   command.  Added the commands ":cat", ":tee", ":assign" and ":filter".
- Improved the caching of remote files.
- Variables with multiple items can be indexed with [n] to obtain one
   item.  Especially useful to get the first source item: "$(source[0])"
- New module defined: Detect the type of a file.  Currently used to
   select a dependency checker.  It can also be used as a stand-alone
   program or from other Python programs.  The principle is based on Vim,
   but implemented differently.  Once the mechanism has proven to be
   good, all Vim's file type detections can be taken over.
- Implemented a first automatic dependency checker.  This defines how the
   recipe executive invokes a dependency checker and how the result is
   interpreted.  It currently only works for C and C++, using gcc.
   Further checkers can be added with the ":autodepend" command.  This
   module is done for version 0.1.

Furthermore, I have added a mechanism for testing.  Both for testing
individual modules and executing recipes.  The typeless nature of Python
requires doing a lot of tests.  Only a few have been added so far, but
one bug has already been detected.

Ingo Krabbe did not report progress for the cross reference module.

I have asked around to find someone to work on the GUI IDE module.
Several people responded that it would be a nice job, but they didn't
have time.  Thomas Scott Urban volunteered to start working on it.
He has not reported progress yet.

I have written an abstract for a presentation at the BSDcon Europe
conference.  An answer can be expected before July 12.  If the paper is
accepted, I will have to spend time on writing the paper and study
the ports system and how to replace it with A-A-P.  This will take one
or two weeks.  Although most of the work would have to be done anyway,
it was not really required for release 0.1.

The time invested this month was about 2 weeks in the recipe executive,
which means five of the planned six weeks have been done.  But it looks
like this is going to take a bit more time than planned (there is also
more functionality than originally planned).  I spent less than the
planned one week on the dependency checker, but I consider it done for
version 0.1.  Adding the filetype detection module wasn't planned, this
took a couple of days.  A few days have been spent on design, updating
the web site and the recipe syntax.

Overall the project is very well on schedule, except for the cross
referencer and GUI IDE modules.

The coming weeks I plan to work on version control.  I want to start
using CVS for distributing A-A-P.  At the same time, the recipe should
help to upload and download A-A-P with CVS.  This is the version control
wrapper module.  Related to this is the issue tracker wrapper, thus this
is probably the next module to be taken care of.

--
FATHER:    You killed eight wedding guests in all!
LAUNCELOT: Er, Well ... the thing is ... I thought your son was a lady.
FATHER:    I can understand that.
                  "Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

  ///  Bram Moolenaar -- Bram@... -- http://www.moolenaar.net  \\\
///   Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim   \\\
\\\           Project leader for A-A-P -- http://www.a-a-p.org           ///
  \\\  Help me helping AIDS orphans in Uganda - http://iccf-holland.org  ///

I have implemented a module to detect the type of a file.  The mechanism
is based on how it works in Vim, since that works quite well and I
couldn't find something else that could be used.

There are four ways to detect the file type:

1. By file name suffix.  "file.c"  gets the type "c"
2. Matching the file name with a regexp.  "*vimrc" gets the type "vim"
    Can also be used to match with the path.
3. Matching the script name in the first line of the file with a regexp.
    "#!/bin/csh" gets the type "csh"
4. Using Python code to do the detection.  This can detect anything, but
    it requires writing a check.

If you get the latest version of the recipe executive (0.021), the
Filetype.py file contains the relevant code and filetype.txt explains
how it works.  Only a few detection rules are currently included.  I did
a few simple tests and it works for those.

Before I start adding all the detection rules from Vim, I would like to
get some feedback about whether the current method will work for
everyone.  I expect this module also to be used for:

- The dependency checker, to select the commands to generate the
   automatic dependencies (this is already implemented!).
- The cross referencer, to decide what parser to use.
- The GUI IDE, to connect each file type with an editor, viewer,
   debugger, etc.

I wonder if we should do something with MIME types?

--
     [clop clop]
ARTHUR:  Old woman!
DENNIS:  Man!
ARTHUR:  Man, sorry.  What knight lives in that castle over there?
DENNIS:  I'm thirty seven.
ARTHUR:  What?
DENNIS:  I'm thirty seven -- I'm not old!
                                   The Quest for the Holy Grail (Monty Python)

  ///  Bram Moolenaar -- Bram@... -- http://www.moolenaar.net  \\\
///   Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim   \\\
\\\           Project leader for A-A-P -- http://www.a-a-p.org           ///
  \\\  Help me helping AIDS orphans in Uganda - http://iccf-holland.org  ///

Steven Knight wrote:

> > I was looking through the source code of SCons, and noticed that only
> > one checksum per target is remembered.  If this target has several
> > source files (dependencies), the checksum is computed from the combined
> > checksum of all source files.  A checksum for the build commands is also
> > combined into it.  I'm not sure if this is exactly how it works.
>
> That's correct, as coded.
>
> > Doesn't this mean that it is not possible to know which of the source
> > files was outdated?  Take this example:
> >
> >  foo.o depends on foo.c and foo.h
> >  the build command for foo.o uses $CFLAGS
> >
> > When SCons decides foo.o needs to be build, how can you tell which of
> > foo.c, foo.h and $CFLAGS caused this?  I would think the checksum of
> > each separate item isn't remembered, thus this would be impossible.  Am
> > I right?
>
> Yes, you are.  This is one problem with distilling the input signatures
> into one signature.  There's information that's lost in the compression.
>
> The solution for this issue would be a more complete record of all the
> input files and their signatures, but that gets pretty unwieldy in the
> context of a simple file format like .sconsign.  Long-term, I'd like to
> add an alternative signature-storage mechanism that would track all of
> that info, but probably using an actual database instead of flat ASCII.

Thanks for the explanation.

I have considered the alternatives, and decided to remember the
signature for each source-target pair.  This does mean that a list of
signatures needs to be stored, but I don't expect this to take a lot of
time and space.  Computing the MD5 checksum of each source likely
consumes most of the time.  I've written it down in a design decision:
http://www.a-a-p.org/decisions.html#13

I also thought about the checksum for the build command itself, as we
discussed before.  I've made the default to use this checksum, thus when
the build commands change (including things like $CFLAGS) they are to be
executed.  But there is a way to specify a different checksum, or avoid
it all together.  Example:

	 foo.o : {buildcheck = $CFLAGS} foo.c
		 $CC $CFLAGS -c -o foo.o foo.c

Here building is forced when $CFLAGS changes, but not when $CC changes
or the command is edited.

I have implemented it and it appears to work just fine (only did a few
simple tests though!).  The signature file has one line for each source
file, with items for each target-checksum combination.  It's also
possible to remember different checksums (and timestamps) for a source
file this way.  Example:

test/common.h<ESC>test/foo.o/c_md5=56c5672f786f2d506e6860d15c486d8d<ESC>test/bar\
.o/c_md5=56c5672f786f2d506e6860d15c486d8d<ESC>

(I'm using an ESC character to separate fields).

Here the source file "test/common.h" is used for targets "test/foo.o"
and "test/bar.o", both with a checksum type "c_md5".  This is simple
enough not to use a database for.  Once the file has been read it's in a
Python dictionary anyway.

This is all for A-A-P, of course, but perhaps you will find a few ideas
to use in SCons.

--
hundred-and-one symptoms of being an internet addict:
108. While reading a magazine, you look for the Zoom icon for a better
      look at a photograph.

  ///  Bram Moolenaar -- Bram@... -- http://www.moolenaar.net  \\\
///   Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim   \\\
\\\           Project leader for A-A-P -- http://www.a-a-p.org           ///
  \\\  Help me helping AIDS orphans in Uganda - http://iccf-holland.org  ///

[I wonder why I got multiple copies of this message...]

Aschwin Marsman wrote:

> Quoting Bram Moolenaar <bram@...>:
>
> > - Check outdated files by MD5 signature instead of a timestamp.
>
> Can the timestamp be used also? I have some files that are changed
> by several people, and than published on a web site. Can rex help
> in this? (Is my own copy newer or older, and then merge them if they
> are both changed? I know I can use CVS to achieve this, but how does
> the monkey solve this for me?

The MD5 check is the default at the moment.  Later a special check for C
and C++ files would be the default for the appropriate files.  But you
can always overrule the check by specifying an attribute:

	 foo : foo.x {check=time}  foo.p {check=newer}

The "newer" check means that it works like with "make": the file is
outdated if the target is older than the source.

The "time" check means that rebuilding is done when the timestamp is
different from the last time building was done.  That's useful to avoid
computing a checksum while still rebuilding when putting back an older
version of the file.

Both should work already.

> > - Automatic download of source files.  A cache is used to reduce the
> >   number of downloads.
>
> Same as above: how do you know which files to up, and which to download?

URLs identify remote files.  I still have to implement a mechanism to
specify when a cached file becomes outdated.  It should be possible to
specify this from "always download" to "update once a year".  Default is
most likely "update once a day".

Another thing to be implemented is obtaining a timestamp of a remote
file.  That should reduce download times while avoiding to rely on the
cached copy to become outdated.

> > I have presented A-A-P at the SANE 2002 conference.  Quite a few people
> > were interested to find out what A-A-P is about.  You can find a short
> > report here: http://www.a-a-p.org/presentations.html.  Includes pictures!
>
> The pdf file is not available on that page, plus I would like the new
> version ;-)

Oops!  I'm uploading it right now.

I didn't make a new version.  I would have to update it daily to have it
show the current state of affairs.  I rather show the poster as it was
presented at SANE.  Please read the rest of the website for up-to-date
info.

--
From "know your smileys":
  :q vi user saying, "How do I get out of this damn emacs editor?"

  ///  Bram Moolenaar -- Bram@... -- http://www.moolenaar.net  \\\
///   Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim   \\\
\\\           Project leader for A-A-P -- http://www.a-a-p.org           ///
  \\\  Help me helping AIDS orphans in Uganda - http://iccf-holland.org  ///

Quoting Bram Moolenaar <bram@...>:

> - Check outdated files by MD5 signature instead of a timestamp.

Can the timestamp be used also? I have some files that are changed
by several people, and than published on a web site. Can rex help
in this? (Is my own copy newer or older, and then merge them if they
are both changed? I know I can use CVS to achieve this, but how does
the monkey solve this for me?

> - Automatic download of source files.  A cache is used to reduce the
>   number of downloads.

Same as above: how do you know which files to up, and which to download?

> - Several commands, including a ":copy" command that can copy remote
>   files.  This is now used to upload A-A-P to the sourceforge ftp site.
> - Build rules can be specified with a flexible way of matching.
>
> However, many things are still untested. USE WITH GREAT CARE!

I will give it a try the coming days.

> I have presented A-A-P at the SANE 2002 conference.  Quite a few people
> were interested to find out what A-A-P is about.  You can find a short
> report here: http://www.a-a-p.org/presentations.html.  Includes pictures!

The pdf file is not available on that page, plus I would like the new
version ;-)

Keep on going,

Aschwin Marsman

--
aYniK Software Solutions         all You need is Knowledge
Bedrijvenpark Twente 305         NL-7602 KL Almelo - the Netherlands
P.O. box 134                     NL-7600 AC Almelo - the Netherlands
telephone: +31 (0)546-581400     fax: +31 (0)546-581401
a.marsman@...              http://www.aYniK.com

Quoting Bram Moolenaar <bram@...>:

> - Check outdated files by MD5 signature instead of a timestamp.

Can the timestamp be used also? I have some files that are changed
by several people, and than published on a web site. Can rex help
in this? (Is my own copy newer or older, and then merge them if they
are both changed? I know I can use CVS to achieve this, but how does
the monkey solve this for me?

> - Automatic download of source files.  A cache is used to reduce the
>   number of downloads.

Same as above: how do you know which files to up, and which to download?

> - Several commands, including a ":copy" command that can copy remote
>   files.  This is now used to upload A-A-P to the sourceforge ftp site.
> - Build rules can be specified with a flexible way of matching.
>
> However, many things are still untested. USE WITH GREAT CARE!

I will give it a try the coming days.

> I have presented A-A-P at the SANE 2002 conference.  Quite a few people
> were interested to find out what A-A-P is about.  You can find a short
> report here: http://www.a-a-p.org/presentations.html.  Includes pictures!

The pdf file is not available on that page, plus I would like the new
version ;-)

Keep on going,

Aschwin Marsman

--
aYniK Software Solutions         all You need is Knowledge
Bedrijvenpark Twente 305         NL-7602 KL Almelo - the Netherlands
P.O. box 134                     NL-7600 AC Almelo - the Netherlands
telephone: +31 (0)546-581400     fax: +31 (0)546-581401
a.marsman@...              http://www.aYniK.com

Quoting Bram Moolenaar <bram@...>:

> - Check outdated files by MD5 signature instead of a timestamp.

Can the timestamp be used also? I have some files that are changed
by several people, and than published on a web site. Can rex help
in this? (Is my own copy newer or older, and then merge them if they
are both changed? I know I can use CVS to achieve this, but how does
the monkey solve this for me?

> - Automatic download of source files.  A cache is used to reduce the
>   number of downloads.

Same as above: how do you know which files to up, and which to download?

> - Several commands, including a ":copy" command that can copy remote
>   files.  This is now used to upload A-A-P to the sourceforge ftp site.
> - Build rules can be specified with a flexible way of matching.
>
> However, many things are still untested. USE WITH GREAT CARE!

I will give it a try the coming days.

> I have presented A-A-P at the SANE 2002 conference.  Quite a few people
> were interested to find out what A-A-P is about.  You can find a short
> report here: http://www.a-a-p.org/presentations.html.  Includes pictures!

The pdf file is not available on that page, plus I would like the new
version ;-)

Keep on going,

Aschwin Marsman

--
aYniK Software Solutions         all You need is Knowledge
Bedrijvenpark Twente 305         NL-7602 KL Almelo - the Netherlands
P.O. box 134                     NL-7600 AC Almelo - the Netherlands
telephone: +31 (0)546-581400     fax: +31 (0)546-581401
a.marsman@...              http://www.aYniK.com

May was a very good month.  The first version of the "Recipe Executive"
was made available (should find a shorter name!).  In just a few weeks
it has grown into a program that can replace "make".  And then adds
quite a few features:

- Use Python in commands and expressions.
- Check outdated files by MD5 signature instead of a timestamp.
- Automatic download of source files.  A cache is used to reduce the
   number of downloads.
- Several commands, including a ":copy" command that can copy remote
   files.  This is now used to upload A-A-P to the sourceforge ftp site.
- Build rules can be specified with a flexible way of matching.

However, many things are still untested. USE WITH GREAT CARE!

You can find the downloadable files from the "download" entry on the
A-A-P web page.


I have presented A-A-P at the SANE 2002 conference.  Quite a few people
were interested to find out what A-A-P is about.  You can find a short
report here: http://www.a-a-p.org/presentations.html.  Includes pictures!

--
From "know your smileys":
  :-)-O Smiling doctor with stethoscope

  ///  Bram Moolenaar -- Bram@... -- http://www.moolenaar.net  \\\
///   Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim   \\\
\\\           Project leader for A-A-P -- http://www.a-a-p.org           ///
  \\\  Help me helping AIDS orphans in Uganda - http://iccf-holland.org  ///

Aschwin Marsman wrote:

> > The functionality is currently comparable to using "make".  With the
> > added possibility of using Python.
>
> Can it be used to build vim as it currently is?

Probably.  I didn't try it though.

> I guess that you use vim as a test bed for a-a-p?

I will.  But there are many things to do, and I have to set priorities.

> > NOTE: Almost nothing has been tested yet, USE WITH GREAT CARE!
>
> Do you warn for a trojan horse ;-) ?

No, but I did manage to accidentally delete all my *.py files when a
line break was missing.  That's fixed now, but since there is no test
procedure yet, things like this may happen.  Backup files can be very
useful!

> What is your impression of a-a-p, now you have worked on it for a couple
> of weeks? Is it fun, easier or more difficult than expected, ...?

It's working quite well.  Makefile and Python fit well together.  The
problems I have run into are minimal.  For example, a variable in
Makefile lines is referred to with "$VAR", while in a Python expression
it's "VAR".  Hopefully that won't be too confusing.

--
How To Keep A Healthy Level Of Insanity:
19. Tell your children over dinner "due to the economy, we are going
     to have to let one of you go."

  ///  Bram Moolenaar -- Bram@... -- http://www.moolenaar.net  \\\
///   Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim   \\\
\\\           Project leader for A-A-P -- http://www.a-a-p.org           ///
  \\\  Help me helping AIDS orphans in Uganda - http://iccf-holland.org  ///

Quoting Bram Moolenaar <bram@...>:

> Hello all,

> The functionality is currently comparable to using "make".  With the
> added possibility of using Python.

Can it be used to build vim as it currently is? I guess that you use
vim as a test bed for a-a-p?

> NOTE: Almost nothing has been tested yet, USE WITH GREAT CARE!

Do you warn for a trojan horse ;-) ?

> I did manage to execute a recipe to upload this file.  I plan to upload
> a new version almost every day.  I won't send an announcement each time,
> only when a milestone has been reached.

Sounds reasonable to me.

> I hope this is the start of a very useful program!

Me too.

What is your impression of a-a-p, now you have worked on it for a couple
of weeks? Is it fun, easier or more difficult than expected, ...?

Have a nice weekend,

Aschwin Marsman

--
aYniK Software Solutions - all You need is Knowledge
Bedrijvenpark Twente 305 - NL-7602 KL Almelo - the Netherlands
P.O. box 134             - NL-7600 AC Almelo - the Netherlands
telephone: +31 (0)546-581400 fax: +31 (0)546-581401
a.marsman@...        http://www.aYniK.com
aschwin@...        http://www.marsman.org

Hello all,

I have just created the a-a-p-dev and a-a-p-user maillists.
They are to be used for A-A-P developers and A-A-P users.
See http://www.a-a-p.org/maillist.html for more info.


At the same time I have made the first source files available.  This is
the Recipe Executive, the central program of A-A-P.  Look on the web
site for the "download" link.  Currently everything is contained in a
single .zip archive.  You can unpack it where you like.

The functionality is currently comparable to using "make".  With the
added possibility of using Python.

NOTE: Almost nothing has been tested yet, USE WITH GREAT CARE!

I did manage to execute a recipe to upload this file.  I plan to upload
a new version almost every day.  I won't send an announcement each time,
only when a milestone has been reached.


I hope this is the start of a very useful program!

--
How To Keep A Healthy Level Of Insanity:
15. Five days in advance, tell your friends you can't attend their
     party because you're not in the mood.

  ///  Bram Moolenaar -- Bram@... -- http://www.moolenaar.net  \\\
///   Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim   \\\
\\\           Project leader for A-A-P -- http://www.a-a-p.org           ///
  \\\  Help me helping AIDS orphans in Uganda - http://iccf-holland.org  ///