MAIN ACTIVITIES

A large variation of issues have been dealth with in August. This is
the result of attempting to make a 0.1 release with useful functionality
and not too many bugs. Unfortunately, the 0.1 release isn't ready yet.
More about that further on.

The main items worked on:
- Finished the work on the issue tracker.
- Designed the basic interface between the cross referencer and the
recipe executive.
- Designed the basic interface between the GUI IDE and the recipe
executive.
- Further cleaning up of the recipe syntax and improvements in the
recipe executive.
- Proof of concept for downloading, compiling and installing Vim.

Note that "finished" above means that it's finished for release 0.1.
Much more work is to be done in the future.


RECIPE SYNTAX

The complexity of the recipe and the recipe executive is starting to
show. The recipe used to set default values and the recipes used for
Vim are not easy to understand for a beginner. The basic choice for
Makefile syntax plus Python and A-A-P commands is still good. However,
the large amount of functionality makes it unavoidable that the recipe
writer has to know how many things work. Further attempts have been
made to simplify the recipe format:
- Added the ":do" and ":action" commands. This makes it possible to
separate invoking an action from the implementation. For example, in
the recipe the command ":do compile foo.c" can be used to compile a C
file. How the compiler is actually invoked is defined in the startup
recipe, possibly after checking the environment. This makes the
actions portable over many systems and configurable for the user.
- Use the ":do" and ":action" commands for various actions. Actions are
now used for building, compiling, figuring out dependencies, editing
and viewing files, sending an e-mail, etc.
Removed the ":autodepend" command, use ":action depend" instead.
- Support $BROWSER, $MAILER and $PAGER for ":do" command defaults.
- Added $?var: when $var is not defined the result is an empty string
and not an error message. Avoids the need for a Python "if" command
to check if the variable is defined.
- Rc-style expansion was causing a lot of trouble in ":sys" and ":print"
commands. Now don't use rc-style expansion for $var. Use $*var to
explicitly enable rc-style expansion. The star indicates
multiplication is being done.
- Some commands used "-o" style options, like shell commands, others
used attributes such as {force}. Made them all use attributes for
consistency. This also makes clear that these commands do not work
exactly the same as similar shell commands.

More simplifications need to be done. Since this requires doing
"inventions" this will have to be done while working on other issues and
constantly keeping the goal of simplification at the back of the mind.


PERFORMANCE

The performance of Python is starting to become a problem in a few
areas. When building an application like Vim it takes about five
seconds to figure out the dependencies. When files need to be compiled
this overhead isn't much, but when nothing is to be done the user has to
wait while "nothing" is happening. For now this isn't too bad, since
the extra checks take some work away from the developer. It does mean
that we need to be careful about performance. A bit of profiling has
been done to locate "hot spots" and these have been solved. This
doubled the speed. If performance really becomes a problem some parts
will have to be re-implemented in C. Python supports integration with
C, thus this will be possible. It's still work to be done and reduces
the portability, therefore using C will be avoided.

The command-line argument "--profile=fname" has been added. This means
profiling can be done any moment. This was easy since Python provides
all the functionality and SCons provided an example how to use it.

The original idea was to filter out comments and other irrelevant items
from C code before computing the MD5 signature. Doing this with Python
was too slow. A C program could do this about 40 times faster, but this
creates portability problems. After considering alternatives the
decision was to drop this idea of filtering and use ordinary MD5
signatures for C programs. See design decision 20:
http://http://www.a-a-p.org/decisions.html#20.

Another performance problem was encountered after adding the filetype
detection rules from Vim. After profiling it became apparent that the
new Python module "sre" is the bottleneck. Using the old "pre" module
made compiling regular expressions about twenty times faster. Several
other optimisations were done to make the startup time acceptable and
the actual filetype detection quite fast.


PROOF OF CONCEPT

Vim is not an easy application to compile and install, thus provides a
good test for the functionality of A-A-P. While writing the recipes for
Vim several issues were encountered that called for changes in the
recipe executive.

Not everything in the Vim Makefile was made portable. Using Autoconf
and shell scripts for linking and figuring out function prototypes
remain for now. The method to use autoconf in a recipe is explained in
docs.txt. This works quite well. Vim makes it a bit complex, because
the generated files are moved to the "auto" subdirectory.

Automatically downloading the original Vim files through CVS was simple,
since CVS support has already been implemented. Using the distributed
Vim archives and applying patches requires several non-obvious lines in
the recipe. This should be simplified when working on ports the next
month.


OTHER ISSUES

Figuring out the dependencies between C files (#include'd files) was
using GCC until now. When GCC is not available this would fail. A
Python function was added for this: aap_depend_c(). This is according
to design decision 22 (http://www.a-a-p.org/decisions.html#22). Its
speed isn't great, but since the results are cached it only needs to be
used when a file was actually modified. The reliability is not as good
as using the actual compiler, because the Python code can't do
preproccessing exactly like the compiler. Therefore a configuration
check was added to use GCC when it's available and the Python function
otherwise. More of these configuration choices will probably be added
in the future.

Created a web page on the interfaces:
http://www.a-a-p.org/interfaces.html. The already existing interfaces
are explained here: Version Control Wrapper, Filetype Detection, Issue
Tracker and Dependency Checker. Added the first version of the Cross
Referencer and GUI IDE interfaces. These were designed in a generic
way, so that they can also be used for other purposes. This also
provides the flexibility needed to compensate for the lack of work done
on the Cross Referencer and GUI IDE.

Some generated files need to be distributed. For example the
"configure" script generated by autoconf. The use of signatures would
cause these distributed files to be generated anyway. To avoid this the
signatures need to be distributed as well. This has been implemented by
providing the "signfile" attribute to be able to specify using a
specific file for storing signatures. This is explained in design
decision 23: http://www.a-a-p.org/decisions.html#23.


PROGRESS AND PLAN

Work and progress in August related to the original plan:
- A couple of days spent on the issue tracker. This was finished within
the planned week.
- A week spent on the interfaces. This was planned to be done by
someone else. Added a web page for all the interfaces.
- A couple of days spent on design; all but a few days of the planned
two weeks done, this appears to go as planned.
- A couple of days on the recipe syntax, all but a few days of the
planned three weeks done. Still needs about a week for cleaning up
and writing down the syntax: A couple of days more than planned.
- A week spent on the recipe executive: seven weeks used while six weeks
were planned. And still a bit more work to be done.
- A week spent on making a proof of concept with Vim. This was missing
in the plan.

The original plan was to bring out a version 0.1 by the end of August.
The work remaining to be done for this is making the documentation
readable for first-time users. This requires chosing the format for the
documentation (currently it is a plain text file). A bit more cleaning
up and streamlining of the recipe syntax is also desirable.

The volunteers that were working on the cross referencer and the GUI IDE
have not produced anything this month. These modules will not be part
of the 0.1 release.


COMING MONTH

In September the paper for the BSD conference has to be written, there
is a fixed deadline. To be able to write a good paper, research and
prototyping needs to be done for ports and packages. On the other hand
version 0.1 is to be released. An important part of this is the proof
of concept with Vim, since the Vim developers can be easily approached.
This means a choice has to be made: Quickly release version 0.1 after
writing some documentation, or postpone it until the work on ports has
progressed. Since the coming month it is very likely that the recipe
used for Vim will change drastically, it would be very strange for the
audience that so many changes happen right after the first release.
Additionally, someone who wants to try out A-A-P right now can obtain
the current version anyway.

Version 0.1 should be promoted to attract attention for the A-A-P
project. Since there currently is so much functionality already,
presenting it in a nice way will be more work than originally planned.
I consider this to be important, the first impression someone gets of
A-A-P must be good. Therefore I think it is not a good idea to rush out
version 0.1 right now. After working out the main known issues, version
0.1 will make a much better impression to first-time users.

September will include these activities:
- Research on existing ports and package systems.
- Design and prototyping for handling ports and packages in a recipe.
- Writing the paper for euroBSDcon.
- Updating the proof of concept with Vim and some cleaning up.
- Making the documentation ready for release 0.1.

This will probably take more than one month of work. All of this work
was not in the original plan. The resulting functionality is more than
planned, thus it's not that more time was needed for the planned work.


MINOR ISSUES

Minor issues handled in August:

- Publish uploads to all locations specified with the "publish"
attribute, not just the first one.
- Added a bootstrap recipe to have the recipe executive update itself
automatically.
- Added ":start command": Run a shell command in the background.
- Computing buildcheck signatures has been improved. Expand ":do"
commands recursively.
- Bugfix: logging of system command output didn't work.
- Improved quoting for Unix shell command arguments: also quote '&', ';'
'<', '>' and '|'. Don't add quotes for ":sys cmd $VAR; cmd", the ';'
ends the command, it's not part of the argument.
- When using --nobuild or --touch, do execute build commands, but don't
execute system commands, don't create files, skip version control
commands, etc. Now signatures for ":publishall" in build commands
will be updated with --touch.
- Fixed a bug in parsing a scp:// URL, was using the machine name in
place of the path.
- Implemented redirection to a URL for commands used in a pipe. Can now
directly write a message into a remote file.
- Implemented the {exist} option for ":touch".
- Fixed: When a build dependency has two targets, it could be build
twice, once for each target.
- Added the file2string() function: read a file and turn it into a
string. Used for a file with configure arguments.
- When defining $SOURCE and $TARGET, now really use $SOURCE. Previously
this depended on default rules.
- When $TARGET and $SOURCE are defined and there are dependencies, but
not one that produces $TARGET, add a default dependency for generating
$TARGET from $SOURCE.
- When adding a default build rule for $TARGET and $SOURCE, make a
separate dependency for each compile command. Avoids rebuilding all
the object files each time.
- The default dependency created from $SOURCE and $TARGET is now done
for each recipe where it is appropriate, also in child recipes.
- Fixed: When generating autodependencies the signatures for newly
discovered dependencies were not stored, causing the file to be
generated again the next time.
- Old signatures for a file starting with ".." were not looked up
correctly and always caused a rebuild.
- Fixed an endless recursion for computing the build signature when an
action invokes itself.
- Fixed: an empty auto-dependency file caused updating the dependency
file to fail.
- Fixed: Generated auto-dependencies included a path relative to the
recipe but it's stored relative to the target. Store the path
relative to the target and expand it when reading the dependency file.
- The automatic dependencies of a child recipe were stored in the $BDIR
directory of the parent. Now use the local $BDIR, so that it works
the same when the child recipe is executed by itself.
- Expand "~/dir/file" and "~user/file" in relevant commands and dependencies.
- Fixed: A target starting with "~/" was not handled properly.
- Default for $var with ":print" and ":error" is to include attributes.
Avoids that removing attributes causes an error or white space to be
changed.
- Added the "$targetarg" variable: targets from the command line. Can
be used to execute ":update" when a certain target is specified.
- Added the {force} argument to ":update": Forcefully build the target.
- If an ":update" command changes directory, change back to the previous
directory, so that following build commands are executed in the right
directory.
- Obtain the exit status of the executed shell command instead of the
"tee" command (used to log the output). Previously a failed shell
command wasn't noticed when the output was logged.
- Fixed: "aap --version" gave an "Aborted" message.
- Added ":quit" and ":exit" commands.
- Fixed: A variant with two alternatives and a "*" alternative would use
the commands of the "*" alternative unless the last but one
alternative was selected.
- Write the autodepend files in the build directory instead of "AAP".
This allows a different one to be generated for each variant.
- Generate the autodepend files without changing directory to the source
file. Avoids that include paths are wrong.
- Support the {directory} and {mode} attributes on arguments of ":touch".
- Implemented the ":execute recipe" command.
- Cleaned up use of work.globals. It can be a local dictionary in build
commands and child recipes. Rename from "globals" to "recdict" to
avoid confusion with the global() function. Pass "recdict" on to all
functions that need it.
- Added ":deldir" command: delete a directory.
- Added ":chmod" command: change file protection.
- Changed "%s" argument of ":filter" to "stdin".
- Made it possible to use ":filter" without stdin to catch the output of
a Python expression. Renamed ":filter" to ":eval".
- Add ":syseval" command: filter text through a shell command.
- Fixed: Comments and empty lines were not counted when reporting the
line number for an error in build commands of a dependency.
- Fixed: Deleting a symbolic link didn't work.
- Added the has_target() function: check if a target has a dependency
with or without build commands.
- Added a level number to dependency messages, makes it easier to
understand what is going on.
- Added aap_abspath(), to be used for ":export" commands.
- Removed quotes from $DATESTR, it's not what you would expect.
- Added "strip" action: strip executable.
- Added a default "preproc" action. Used in the detection of $CC being GCC.
- Added $empty to be able to produce leading white space with ":print".
- For a dependency in a startup recipe, don't change to its directory
when executing the commands.
- When computing a buildcheck, make $source and $target empty, so that
they are not accidentally replaced. Collapse spans of white space so
that trailing spaces don't change the signature.
- When {constant} attribute used and file exists it was still refreshed.
- Fixed: Finding a rule for "patches/%.done" didn't work. Now also try
matching with the short name of a node.


- Bram

--
ARTHUR: Will you ask your master if he wants to join my court at Camelot?!
GUARD #1: But then of course African swallows are not migratory.
GUARD #2: Oh, yeah...
GUARD #1: So they couldn't bring a coconut back anyway...
The Quest for the Holy Grail (Monty Python)

/// Bram Moolenaar -- Bram@... -- http://www.moolenaar.net \\\
/// Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim \\\
\\\ Project leader for A-A-P -- http://www.a-a-p.org ///
\\\ Lord Of The Rings helps Uganda - http://iccf-holland.org/lotr.html ///