RfD - Enhanced local variable syntax, v5b
====================================
Stephen Pelc - 22 October 2009

20091022 Discussion about { and {:.
20090830 Tidied.
20090325 Updated with some renaming.
20080811 Removed references to local buffers as appropriate.
20070914 Moved local buffers to separate proposal.
20070607 Wordsmithing. Corrected reference implementation.
20060822 Added explanatory text.
          Corrected reference implementation.
          Updated ambiguous conditions.

Problem
=======
1) The current LOCALS| ... | notation explicitly forces all
locals
to be initialised from the data stack.
2) 1) The current LOCALS| ... | notation defines locals in
reverse
order to the normal stack notation.

This proposal is derived from implementations that have existed
for
more than 15 years.


Solution
========
Base version
------------
The following syntax for local arguments and local values is
proposed. The sequence:
   {: arg1 arg2 ... | lv1 lv2 ... -- o1 o2 ... :}
declares local arguments, local values, and dummy outputs.

The local arguments are automatically initialised from the data
stack on entry, the rightmost being taken from the top of the
data
stack. Local arguments and local values can be referenced by
name
within the word during compilation. The output names are dummies
to allow a local declaration to be read as a stack comment.
   The items between {: and | are local arguments.
   The items between | and -- are local values.
   The items between -- and :} are outputs for formal comments
only.

The -- to :} section is optional, i.e. | or {: direct to :} is
permitted.

The outputs are provided in the notation so that complete stack
comments can be produced. However, all text between -- and } is
ignored. This facility is there to permit the notation to form a
complete stack comment, which eases documentation.

Local arguments and values return their values when referenced,
and must be preceded by TO to perform a store.

In the example below, a and b are local arguments, and c and
d are local values.

: foo  {: a b | c d -- :}
   a b + to c
   a b * to d
   cr c .  d .
;

Local types and extensions
--------------------------
Although out of the scope of this proposal, it should be noted
that some current Forth systems use indicators to define local
values of sizes other than a cell. To avoid issues when porting
code to such systems, names ending in a ':' (colon) should be
avoided.

: foo  {: a b | F: f1 F: f2 -- c :}
   ...
;

At least one Forth implementation uses local value names ending
in
the '[' character to indicate local buffers. This character
should
be avoided to prevent disenfranchising implementations that
implement
the behaviour. For similar reasons the use of '[' and '\' as
local
argument or value names should also be avoided.


Discussion
==========
The phrase { ... } rather than {: ... :} has been used by
systems
for over 15 years. However, it conflicts with existing practise
in
other Forth systems. Choosing the new name allows both practises
to coexist.

The '|' (ASCII 0x7C) character is widely used as the separator
between local arguments and local values. Other characters
accepted in current Forth implementations are '\' (ASCII 0x5C)
and
'¦' (0xA6). Since the ANS standard is defined in terms of 7-bit
ASCII, and with regard to internationalistion, we propose only
to
consider the '|' and '\' characters further. Only recognition of
the '|' separator is mandatory.

Some current systems permit TO to be used with floats (children
of
FVALUE) and other data types. Such systems often provide
additional
operators such as +TO (add from stack to item) for children of
VALUE
and FVALUE.


Forth 200x text
===============
13.6.2.xxxx {:
brace-colon LOCAL EXT

Interpretation: Interpretation semantics for this word are
undefined.

Compilation:
( ... "<spaces>:}" -- )

Create local arguments by repeatedly skipping leading spaces,
parsing name (arg1 to argn), and executing implementation
defined
actions. The list of local arguments to be defined is terminated
by "|", "--" or ":}". Append the run-time semantics for local
arguments given below to the current definition.

If a space delimited '|' (ASCII code 0x7C) is encountered,
create
local values by repeatedly skipping leading spaces, parsing name
(lv1 to lvm), and creating the local element. The list of local
values to be defined is terminated by "--" or ":}". Append the
run-time semantics for local values given below to the current
definition. If "--" has been encountered, further text between "-
-"
and ":}" is ignored.

Systems should permit at least eight local arguments and eight
local values to be declared. A program that requires more has an
environmental dependency.

Local argument declaration run-time: ( x1 ... xn -- )
Local value declaration run-time: ( -- )

Initialise local arguments from the data stack. Local argument
arg1 is initialized with x1, arg2 with x2 up to argn from xn,
which is on the top of the data stack. When invoked, each local
argument returns its contents. The contents of a local argument
may be
changed using 13.6.1.2295 TO.

Set up local values. The initial contents of local values are
undefined. When invoked, each local value returns its contents.
The
contents of a local value may be changed using 13.6.1.2295 TO.
The
size of a local value is a cell.

Local argument run-time: ( -- x )
Local value run-time: ( -- x )
TO local argument run-time: ( x -- )
TO local value run-time: ( x -- )

Ambiguous conditions:
a) The {: ... :} text extends over more than one line.
b) {: ... :} is used more than once in a word.


Reference implementation
=========================

0 [if]
BUILDLV    c-addr u +n mode
When executed during compilation, BUILDLV passes a message to
the
system identifying a new local argument whose definition name is
given by the string of characters identified by c-addr u. The
size
of the data item is given by +n address units, and the mode
identifies the construction required as follows:
   0 - finish construction of initialisation and data storage
       allocation code. C-addr and u are ignored. +n is 0
       (other values are reserved for future use).
   1 - identify a local argument, +n = cell
   2 - identify a local value, +n = cell
   3+ - reserved for future use
   -ve - implementation specific values

The result of executing BUILDLV during compilation of a
definition
is to create a set of named local arguments and values, each of
which is a definition name, that only have execution semantics
within the scope of that definition's source.

Note that it is often useful to accumulate and store the size of
locals storage (0=none), so that compilers for EXIT can easily
detemine if locals clean up code is required.
[then]

VARIABLE #LVS   \ -- addr
\ Holds size of locals storage required.

: BUILDLV \ c-addr u +n mode --
\ Dummy for testing
   OVER #LVS +!
   CR 2SWAP TYPE  SPACE  SWAP . .
;

: TOKEN         \ -- caddr u
\ Get the next space delimited token from the input stream.
\ Can be extended to permit multiple line declarations.
   PARSE-NAME
;

: LTERM?        \ caddr u -- flag
\ Return true if the string caddr/u is "--" or ":}"
   2DUP S" --" COMPARE 0= >R
   S" :}" COMPARE 0=  R> OR
;

: LSEP?         \ caddr u -- flag
\ Return true if the string caddr/u is the separator between
\ local arguments and local values or buffers.
   2DUP S" |" COMPARE 0= >R
   S" \" COMPARE 0=  R> OR
;

: {:            \ --
\ Parse the locals declaration up to the closing ":}".
   0 #LVS !                              \ indicate no locals yet
   0 >R                                  \ indicate arguments
   BEGIN
     TOKEN 2DUP LTERM? 0=
   WHILE                                 \ -- caddr len
     2DUP LSEP? IF                       \ if '|'
       R> DROP  1 >R                     \ change to vars and
buffers
     ELSE
       R@ 0= IF                          \ argument?
         CELL 1
       ELSE                              \ value
         CELL 2
       THEN
       BUILDLV
     THEN
   REPEAT
   BEGIN
     S" :}" COMPARE
    WHILE
     TOKEN
   REPEAT
   0 0 0 0 BUILDLV
   R> DROP
; IMMEDIATE

Tests
=====

: TEST1  {: a b | c -- f :}
   CR ." Hello1 " CR  a  ;
\ 3 4 TEST1 -> 3

: TEST2  {: a | b c e :}
   CR ." Hello2 " CR  ;
\ 3 TEST2 ->

: TEST3  {: a b c -- :}
   CR ." Hello3 " CR  ;
\ 3 4 5 TEST3 ->

: TEST4  {: a b c :}
   CR ." Hello4 " CR  ;
\ 3 4 5 TEST4 ->

: TEST5  {: a | b c d -- e f g :}
   CR ." Hello5 " CR
   a 2* to b  b 2* to c  c 2* to d
   b c d  ;
\ 3 TEST2 -> 6 12 24

Stephen
--
Stephen Pelc, stephen@...
MicroProcessor Engineering Ltd - More Real, Less Time
133 Hill Lane, Southampton SO15 5AF, England
tel: +44 (0)23 8063 1441, fax: +44 (0)23 8033 9691
web: http://www.mpeforth.com - free VFX Forth downloads

Stephen Pelc wrote:
> Posted by Leon in response to Mitch
> > Very well stated.  And things that seem experimental
> > should not be considered for inclusion in the standard
> > until they are proven (and more widely used).
>
> I agree. Note also that the two words I proposed were simply a
> reaction to code ported from bigForth to VFX. I now regret
> making the comment.
>
> There may be value in a documented pre-standards activity
> called, say, EXPERIMENTAL, which serves to indicate the
> directions in which people are going.

The RfD/CfV process seems to be appropriate for this purpose.  A
proponent can write an RfD, incorporate the first rounds of feedback,
then the proposal can mature by people implementing and using it, then
a few more RfD rounds can be made to nail down the exact specifics of
the standard, incorporating experience that has been made in the
meantime, and finally the specification can be frozen, and the
proposal can proceed to CfV and eventually to discussion in the
Forth200x committee.

This has worked for the xchars proposal (ok, we are still not at the
CfV stage, but I expect that we will get there).

- anton

Posted by Leon in response to Mitch

> > Correct.  I don't have fundamental problems with any
> > of these features.  I'm just pointing out that such
> > features have implications that may extend rather far
> > into various corners of an implementation.  So it
> > shouldn't be surprising that implementors - especially
> > those with substantial customer bases - would want to
> > consider the tradeoffs carefully before agreeing to
> > such features.  Furthermore, interactions with other
> > portions of the standard's text must be checked.

> Very well stated.  And things that seem experimental
> should not be considered for inclusion in the standard
> until they are proven (and more widely used).

I agree. Note also that the two words I proposed were simply a
reaction to code ported from bigForth to VFX. I now regret
making the comment.

There may be value in a documented pre-standards activity
called, say, EXPERIMENTAL, which serves to indicate the
directions in which people are going. My "ample spare time" is
already full.

Stephen
--
Stephen Pelc, stephen@...
MicroProcessor Engineering Ltd - More Real, Less Time
133 Hill Lane, Southampton SO15 5AF, England
tel: +44 (0)23 8063 1441, fax: +44 (0)23 8033 9691
web: http://www.mpeforth.com - free VFX Forth downloads

> Correct.  I don't have fundamental problems with any of these
> features.
> I'm just pointing out that such features have implications that may
> extend rather far into various corners of an implementation.  So it
> shouldn't be surprising that implementors - especially those with
> substantial customer bases - would want to consider the tradeoffs
> carefully before agreeing to such features.  Furthermore,
> interactions
> with other portions of the standard's text must be checked.

Very well stated.  And things that seem experimental should not be considered
for inclusion in the standard until they are proven (and more widely used).

On Thu, Oct 15, 2009 at 7:36 PM, Elizabeth D Rather <erather@...> wrote:
> Mitch Bradley wrote:
>> ...
>>> Anyway, if there is enough user demand, then a Forth implementor will
>>> invest the resources to make the decompiler work well.  For
>>> SAVE-COMPILATION ... RESTORE-COMPILATION this may mean extra effort or
>>> extra memory consumption, but I don't see this feature to be any worse
>>> in this respect than some others that are already standard.
>>>
>>
>> Correct.  I don't have fundamental problems with any of these features.
>> I'm just pointing out that such features have implications that may
>> extend rather far into various corners of an implementation.  So it
>> shouldn't be surprising that implementors - especially those with
>> substantial customer bases - would want to consider the tradeoffs
>> carefully before agreeing to such features.  Furthermore, interactions
>> with other portions of the standard's text must be checked.
>>
> My concern is that during the last round, Greg Bailey and several others
> were absolutely adamant that there should be no nested compilation
> allowed.  I no longer remember the rationale, but their vehemence was
> such that I'd want to understand their concern before putting this in.
>
> Also, AFAIK we haven't really had a groundswell of demand for this, have
> we?  Who besides Hugh is dying for it?
>
> It seems to me this is a feature that folks who are interested should
> implement and experiment with, and if it really does turn out to be
> useful in the Forth context, it can be considered for standardization in
> a future round.
>
> Cheers,
> Elizabeth
> (temporarily on a borrowed computer till mine is fixed :-((( )

Sorry for not reading all my email before sending my earlier email,
but as I said there, I suspect that Forth, Inc. and its customers have
been using this for longer than I have been using Forth (i.e. >30
years).  Although ASSIGN does not nest the compiler, I think it could
be made to accomplish what Hugh is asking for.  It could probably even
be done without nesting the compiler.  Someone (ducking quickly) just
needs to standardize what ASSIGN does and then expand the concept so
Hugh could also use it.

It did always seem too restrictive anyway. ;)

DaR

I had a thought today, as I am driving across country, that this
discussion is very similar in intent as the Forth, Inc. concept called
ASSIGN.  Elizabeth will have to confirm or deny this, but if this is
the concept that Hugh is asking for, then it does, in fact, have quite
a long history of usage.

DaR

Mitch Bradley wrote:
> ...
>> Anyway, if there is enough user demand, then a Forth implementor will
>> invest the resources to make the decompiler work well.  For
>> SAVE-COMPILATION ... RESTORE-COMPILATION this may mean extra effort or
>> extra memory consumption, but I don't see this feature to be any worse
>> in this respect than some others that are already standard.
>>
>
> Correct.  I don't have fundamental problems with any of these features.
> I'm just pointing out that such features have implications that may
> extend rather far into various corners of an implementation.  So it
> shouldn't be surprising that implementors - especially those with
> substantial customer bases - would want to consider the tradeoffs
> carefully before agreeing to such features.  Furthermore, interactions
> with other portions of the standard's text must be checked.
>
My concern is that during the last round, Greg Bailey and several others
were absolutely adamant that there should be no nested compilation
allowed.  I no longer remember the rationale, but their vehemence was
such that I'd want to understand their concern before putting this in.

Also, AFAIK we haven't really had a groundswell of demand for this, have
we?  Who besides Hugh is dying for it?

It seems to me this is a feature that folks who are interested should
implement and experiment with, and if it really does turn out to be
useful in the Forth context, it can be considered for standardization in
a future round.

Cheers,
Elizabeth
(temporarily on a borrowed computer till mine is fixed :-((( )

>
> The way I heard it was that Fcode is a tokenized representation of
> source code, so I would expect it to be easier to recreate something
> source-like from Fcode in this context than from the compiled code in
> any Forth system (including an Open Firmware system).

It's true that FCode is easy to "detokenize" back to source, but that
fact is not necessarily salient because FCode comprises only a part of
Open Firmware.  FCode is used for portable device drivers for plug-in
devices, but the main system and soldered-down devices are often not
compiled from FCode.

>  In particular, I would not expect any problems from features like :[...]: and
> SAVE-COMPILATION ... RESTORE-COMPILATION; but my knowledge of Fcode is
> limited, and I may be wrong.
>

Most of my use of the decompiler is in the context of already-compiled
dictionaries, so the characteristics of FCode don't really matter.  It's
not just the decompiler; the source-level debugger is coupled to the
decompiler so it's potentially affected too.

> Anyway, if there is enough user demand, then a Forth implementor will
> invest the resources to make the decompiler work well.  For
> SAVE-COMPILATION ... RESTORE-COMPILATION this may mean extra effort or
> extra memory consumption, but I don't see this feature to be any worse
> in this respect than some others that are already standard.

Correct.  I don't have fundamental problems with any of these features.
I'm just pointing out that such features have implications that may
extend rather far into various corners of an implementation.  So it
shouldn't be surprising that implementors - especially those with
substantial customer bases - would want to consider the tradeoffs
carefully before agreeing to such features.  Furthermore, interactions
with other portions of the standard's text must be checked.

Mitch Bradley wrote:
>
> --GDTejWdHbo0cOwNdSbPhttjz97VFpzifIGpFq-D
> Content-Type: text/plain; charset=ISO-8859-1
> Content-Transfer-Encoding: 7bit
>
> >
> > if I want to see the source, I look at it in an editor.
>
> That works great if you have the source handy.
>
> I have been supporting Open Firmware implementations on various
> platforms for almost 20 years now.  I often have to work with systems
> where I don't have the source handy, either because it's someone else's
> implementation, or it's an ancient version for which I might have the
> source in some dusty archive, if only I had time to dig it up.  In such
> cases the decompiler is often the difference between timely success and
> failure.

The way I heard it was that Fcode is a tokenized representation of
source code, so I would expect it to be easier to recreate something
source-like from Fcode in this context than from the compiled code in
any Forth system (including an Open Firmware system).  In particular,
I would not expect any problems from features like :[...]: and
SAVE-COMPILATION ... RESTORE-COMPILATION; but my knowledge of Fcode is
limited, and I may be wrong.

Anyway, if there is enough user demand, then a Forth implementor will
invest the resources to make the decompiler work well.  For
SAVE-COMPILATION ... RESTORE-COMPILATION this may mean extra effort or
extra memory consumption, but I don't see this feature to be any worse
in this respect than some others that are already standard.

The worst offender is obviously IMMEDIATE; compared to that stuff like
SAVE-COMPILATION and RESTORE-COMPILATION is small fish.

- anton

>
> if I want to see the source, I look at it in an editor.

That works great if you have the source handy.

I have been supporting Open Firmware implementations on various
platforms for almost 20 years now.  I often have to work with systems
where I don't have the source handy, either because it's someone else's
implementation, or it's an ancient version for which I might have the
source in some dusty archive, if only I had time to dig it up.  In such
cases the decompiler is often the difference between timely success and
failure.

Anton Ertl wrote:
> Stephen Pelc wrote:
> > Define
> >   SAVE-COMPILATION ( -- n1..nn n )
> > and
> >   RESTORE-COMPILATION ( n1..nn n )
>
> That's an interesting suggestion.  I have my doubts that it is the
> right level of abstraction, though.

I thought about it some more:

> 1) Apart from anonymous colon definitions inside colon definitions
>    ("quotations" in Factor, "blocks" in Smalltalk), is there anything
>    else we can and would like to to with it?  If not, why not just
>    define words like :[ and ]: for the blocks/quotations directly.

IIRC there has been a discussion not that long ago where people wanted
to define a table in the middle of a colon definition.  So that would
be another application of SAVE-COMPILATION and RESTORE-COMPILATION
that would not be covered by :[ ... ]:.  More generally, sometimes one
wants to lay down data in the dictionary while compiling a word; e.g.,
SLITERAL and ." are standard words that are usually implemented in
that way, and a programmer may want to write words of this kind in
standard Forth.

So it may be a good idea to define SAVE-COMPILATION and
RESTORE-COMPILATION as factors of :[ and ]: (in addition to defining
:[ and ]:).

Example uses would be:

: SLITERAL ( addr u -- )
   2>r save-compilation 2r@ here swap chars move
   2r> nip here swap 2>r restore-compilation 2literal ;

: :[ ( compilation: -- some-sys )
   save-compilation :noname ;

: ]: ( compilation: some-sys -- ; run-time: -- xt )
   postpone ; >r restore-compilation r> postpone literal ;

> 2) RESTORE-COMPILATION will have a long list of ambiguous conditions
>    (or worse, it won't have an standard-defined list, but an
>    implementation-dependent one), which will make it hard to define
>    and hard to use in a standard way.  We see this problem with
>    SAVE-INPUT and RESTORE-INPUT, which are pretty useless, because
>    RESTORE-INPUT can fail for any reason, and at least on Gforth it
>    does fail except in a few select circumstances.
>
>    The problem here is that this kind of interface seems to support a
>    lot of things that don't fit in most systems, e.g., restoring the
>    same compilation several times, or non-nested restoring of
>    compilations.

I think we can fix this by making the data that SAVE-COMPILATION puts
on the stack an opaque data type, say "comp-sys".  This approach works
well for, e.g., do-sys, which has exactly the same nesting behaviour
that seems appropriate for SAVE-COMPILATION ... RESTORE-COMPILATION,
and there it has not caused any trouble.


Concerning the issue of decompilers: Decompilation in Forth is
generally not perfect, but is done on a best-effort basis.

If a system implementor values decompilation highly, they can put in
extra resources (extra work on the decompiler and/or extra data in the
code when using SAVE-COMPILATION and RESTORE-COMPILATION) to make it
work well; or they can choose not to implement this extension.

If Forth systems have problems decompiling code that uses
SAVE-COMPILATION ... RESTORE-COMPILATION or :[ ... ]:, and a
programmer values nice decompilation highly, he can choose to program
without these words.  Or he can choose to use a system that puts in
the extra effort necessary to decompile code nicely that uses these
words.

BTW, VFX seems to just disassemble the resulting native code, so
trying hard at a reproduction of the source code is obviously not high
on every systems' list.  And while Gforth has a SEE that tries hard, I
have also implemented SIMPLE-SEE and SEE-CODE to look at what the
compiler has made out of that, because I find that much more
interesting; if I want to see the source, I look at it in an editor.

- anton

From: Elizabeth D Rather <erather@...>
To: forth200x@yahoogroups.com
Sent: Tue, October 13, 2009 1:26:11 AM
Subject: Re: [forth200x] Re: RfD: XT

 

Similarly, FORTH, Inc. has a large body of users of SwiftForth on
Windows, and a rapidly growing number of users of our Linux-based
SwiftForth. And there's no C under SwiftForth at all. The concerns
folks have expressed about your proposals have nothing to do with C.

What??? If their concerns have nothing to do with C, then what are their concerns? I have no idea what you're talking about --- the whole issue here is that switch-threaded systems don't support XT. Both machine-language and DTC support XT just fine. ITC will too, although this involves compiling a custom do-colon in code memory for every quotation (impossible on Harvard machines such as the 8-bit AVR, except with a cross-compiler that is generating both code and data images). Primarily, the problem comes down to switch-threaded C-based systems not being able to support XT. SwiftForth actually supports XT easily though, so you of all people should be on my side. You work as a salesperson. When the customer asks: "Why should I buy SwiftForth rather than use GFORTH, that is free?" --- you reply "Quotations!" When the customer asks: "Why should I buy SwiftForth rather than use Factor, that is free and has quotations?" --- you reply: "Speed!" I'm no Zig Ziglar, but that sounds like good marketing to me. If the customer still has doubts, tell them that SwiftForth supports ICON-style generators too, and that neither GFORTH nor Factor do at all. Let them find out after they buy the product that your XT-based generators can't be recursive. :-)

BTW --- Did you ever find out what quotations are? You were asking earlier. Maybe when you are teaching your novice-Forth class at Forth Inc., you can ask your students to explain quotations to you. If they have prior experience in any modern language, then they should be able to do this. Essentially every modern language supports quotations/closures, so you could also just go to the computer section of a bookstore and pick a book off the shelf at random, and it will most likely explain the concept. Even languages that don't support closures (Java), will typically provide a chapter discussing closures and desperately explaining that closures will be supported in the next release of the language (Java 6), and so the reader shouldn't consider the language to be obsolete just because it doesn't support closures yet. You could get Halloway's book on the Clojure language --- the name of the language itself is a big hint that it supports closures. There aren't any books on Forth at the bookstore however, because Forth has been considered obsolete since 1994 when the disappointing ANS-Forth standard came out. Forth just didn't keep up with the modern technology. Many Forth programmers don't even know what quotations are! 

Stephen Pelc wrote:
> ...
>> I don't care about gforth and other C-based Forth
>> systems, and I don't think anybody else does either. I
>> don't see a whole lot of wordprocessors and spreadsheets
>> getting written in Forth.
>>
> Then why are you proposing XT to a standards process? See
>   http://www.ccssa.com
> This is hosted on VFX Forth (not a C-based Forth) and involves
> about 850,000 lines of Forth source code. The app plans
> commercial construction all over the world, including the Hong
> Kong airport and metro. There are commercial applications in
> Forth - the implementers of such applications just don't
> frequent this list or c.l.f.
>
Similarly, FORTH, Inc. has a large body of users of SwiftForth on
Windows, and a rapidly growing number of users of our Linux-based
SwiftForth.  And there's no C under SwiftForth at all.  The concerns
folks have expressed about your proposals have nothing to do with C.
> ....
>> When I first joined the ANS Forth standards team about 17 years ago, I
>> had a rather narrow view of how people use and implement Forth.  I
>> eventually realized that my viewpoint was parochial.  After that I was
>> able to understand the seeming recalcitrance of people who pushed back
>> when I proposed this and that.  It was because I was making assumptions
>> that were false in the context of the other people's usage model.
>>
>> Based on many statements you have made, it seems to me that you are
>> making the same mistake that I made.
>>
>
> Hugh, Mitch is right - he usually is.
>
Agreed.

Cheers,
Elizabeth

--
==================================================
Elizabeth D. Rather   (US & Canada)   800-55-FORTH
FORTH Inc.                         +1 310-491-3356
5155 W. Rosecrans Ave. #1018  Fax: +1 310-978-9454
Hawthorne, CA 90250
http://www.forth.com

"Forth-based products and Services for real-time
applications since 1973."
==================================================

Anton said:

> Stephen Pelc wrote:
> > Define
> >   SAVE-COMPILATION ( -- n1..nn n )
> > and
> >   RESTORE-COMPILATION ( n1..nn n )

> 2) RESTORE-COMPILATION will have a long list of ambiguous conditions
>    (or worse, it won't have an standard-defined list, but an
>    implementation-dependent one), which will make it hard to define
>    and hard to use in a standard way.  We see this problem with
>    SAVE-INPUT and RESTORE-INPUT, which are pretty useless, because
>    RESTORE-INPUT can fail for any reason, and at least on Gforth it
>    does fail except in a few select circumstances.

In our systems, SAVE-INPUT is usually followed by N>R. I'm
inclined to agree with you.

Hugh said:

> Alignment is trivial. XT will just assemble NOP
> instructions as necessary so that the xt address that it
> returns is aligned properly, either because this is
> required, or just because it boosts the speed of the
> call In the case of AHEAD, these NOP instructions will
> never get executed, so they don't harm the execution
> speed at all, but just consume a little bit of memory.

So XT just does what :NONAME does? Note that some CPUs *require*
entry code to a colon definition in the general case, including
ARM, Cortex and MIPS.

I can live with :[ ... ]: much more easily than XT. I also
suspect that it has been implemented more widely than XT and so
satisfies the the common practice requirement better than XT.

> I don't care about gforth and other C-based Forth
> systems, and I don't think anybody else does either. I
> don't see a whole lot of wordprocessors and spreadsheets
> getting written in Forth.

Then why are you proposing XT to a standards process? See
   http://www.ccssa.com
This is hosted on VFX Forth (not a C-based Forth) and involves
about 850,000 lines of Forth source code. The app plans
commercial construction all over the world, including the Hong
Kong airport and metro. There are commercial applications in
Forth - the implementers of such applications just don't
frequent this list or c.l.f.

Mitch said:
> ITC is also nice for resident source-level debugging, a feature that is
> key to productivity in most of my usage scenarios.  Implementation
> techniques that compile machine code fragments inside colon definitions
> tend to make it quite a bit trickier to "hook" all of the places where
> the debugger needs to get involved.

We (MPE) still support a couple of threaded-code systems for
clients who value debugging above all else.

> When I first joined the ANS Forth standards team about 17 years ago, I
> had a rather narrow view of how people use and implement Forth.  I
> eventually realized that my viewpoint was parochial.  After that I was
> able to understand the seeming recalcitrance of people who pushed back
> when I proposed this and that.  It was because I was making assumptions
> that were false in the context of the other people's usage model.
>
> Based on many statements you have made, it seems to me that you are
> making the same mistake that I made.

Hugh, Mitch is right - he usually is.

Stephen
--
Stephen Pelc, stephen@...
MicroProcessor Engineering Ltd - More Real, Less Time
133 Hill Lane, Southampton SO15 5AF, England
tel: +44 (0)23 8063 1441, fax: +44 (0)23 8033 9691
web: http://www.mpeforth.com - free VFX Forth downloads

> It is true that some systems use direct-threaded code for memory
> conservation, but XT will work on those as well. These systems
> typically have code and data in separate places, with the threaded
> Forth words being in the data space and their code-fields (and the
> CODE words) being in the code space.

Several of my Forth implementations, dating back to my first 32-bit
680x0 Forth circa 1982, are direct-threaded.  None have separated data
and code space.  So I don't accept the truth of the "typically" part of
the statement above.  Furthermore, the choice of DTC in those cases was
not because of memory conservation, although memory conservation is
still often important to me.  So your understanding of the reasons for
things does not agree with my experience.

> [Implementation prescription for XT ...]  This can be done, but it is
> in contradiction to the purpose of indirect-threading, which is
> extreme memory conservation

This also disagrees with my experience and choices.  ITC itself is not
particularly space-efficient.  On some processors, it is no more
efficient than DTC.  Several processors can do in-line "docolon" in the
same space as an indirect thread.  Depending on the processor and
application, native compilation can compare favorably in code size.

When I need memory efficiency, I pay careful attention to

a) Size of reference tokens - can they be represented in 1 or 2 bytes?
b) Wasted space due to avoidably alignment padding
c) The set of words than must be present to accomplish the system's
goals (with debugability and long-term maintenance usually in my goal set)

In the FIG-Forth era, one of the main advantages of ITC was the fact
that the basic word layout is processor-independent, increasing the
portability of a model implementation.  It's easier to port to a new CPU
if you don't have to deal with variable-length code fields.

On modern CPUs with Harvard caches, ITC has some nice cache attributes,
since colon definitions are pure data.  You don't have to worry about
cache synchronization when creating new colon definitions.

ITC is also nice for resident source-level debugging, a feature that is
key to productivity in most of my usage scenarios.  Implementation
techniques that compile machine code fragments inside colon definitions
tend to make it quite a bit trickier to "hook" all of the places where
the debugger needs to get involved.

So, while I agree that it is possible to support XT on those systems, I
completely disagree with your assessment of the reasons for
implementation choices.

> Nobody really cares that much about memory conservation nowadays though,

Yeah, well, it is pretty important to me when I need to fit a live Forth
interpreter plus application code plus application data in the internal
ROM and RAM on a low-cost system-on-chip.  I've done that at both my
previous job and my current one.

Call me nobody if you must ...

> so indirect-threading is pretty much obsolete.

Perhaps it is obsolete, but the system that I have been using every day
for the past 3 years in a professional capacity is ITC.


> With the exception of the One-Laptop-per-Child system that you
> mentioned, how many real-world systems use a C-based Forth system?

I have no idea - and neither do you.

When I first joined the ANS Forth standards team about 17 years ago, I
had a rather narrow view of how people use and implement Forth.  I
eventually realized that my viewpoint was parochial.  After that I was
able to understand the seeming recalcitrance of people who pushed back
when I proposed this and that.  It was because I was making assumptions
that were false in the context of the other people's usage model.

Based on many statements you have made, it seems to me that you are
making the same mistake that I made.

> Not too many [use a C-based Forth system] I would think, considering
> the size and speed problems involved.

Size problem?  My C Forth fits on a one-chip ARM SOC with lots of room
left over for application code and data, and includes a live Forth
interpreter with a resident source-level debugger.

Speed problems?  That C Forth (which happens to be switch-threaded) is
not noticeably slower than DTC or ITC implementations on a given
processor.  It measures a bit slower on some isolated benchmarks, but
for the sorts of applications that I normally do, the speed difference
is insignificant.  Application speed problems are solved by identifying
and fixing system bottlenecks, rarely by working on the overall speed of
threaded execution.

> Nowadays however, Lua (non-Forth) is widely considered to be the best
> choice for a scripting language --- because it has closures!

I find it strange that the presence of closures would be a first-order
criterion for the choice of a scripting language.  I can think of
several other factors that I would consider first, with closures being
so far down the list that the choice would already be made before that
one came up.

> This really contradicts you guys' assertion that Forth can't support
> quotations (by way of XT) because Forth is C-based.

First of all, I personally never asserted that.  I asserted that

a) Writing an implementation-neutral specification for how to save and
restore compilation state is likely to be tricky.  I base that on my
experience in wordsmithing the ANS FORTH document.  Many facilities that
were easy to think about in the context of one or two implementations
proved surprisingly difficult to describe accurately in a
implementation-neutral fashion.

b) Your assertions about current uses of Forth directly contradict my
experience.

I'm quite sure that I could indeed support "quotations" in any of my
Forth systems.  The question is - do I want to?  I first considered the
question circa 1985 when I implemented Forth in PostScript.  In
PostScript (as in LISP) a procedure is a manipulable data type - an
explicit array of execution objects.  Closures figure prominentlyt.
PostScript's "if", for example, is postfix - you push an executable
array on the stack and either execute it or not.  I thought about adding
a similar feature to Forth.  I concluded that, for what I do, the
feature would not make my job significantly easier, and the impact on
other aspects of the system would be detrimental.  The assumption that
colon definitions are not statically nested is useful from the
perspective of system maintenance tools like the decompiler and
debugger, tools that I use everyday.

In conclusion,

1) I think it is probably *possible* to specify something like XT in the
context of the standard, but that it is likely to be tricky to do so in
the implementation-neutral framework that ANS FORTH established.

2) I personally like the indivisible nature of Forth colon definitions -
within the context of the set of tradeoffs that makes Forth what it is.
Other languages make different tradeoffs and are thus advantageous or
not in different circumstances.

3) I think you would get more traction by refraining from sweeping - and
incorrect - generalizations about what other people do with Forth.  To
the extent that your conclusions depend on such statements, your
suggestions will be rejected based on the falsehood of the premises.  To
the extent that the conclusions do not depend on those premises,
arguments based on them are redundant and unnecessary.

Hugh said:

> I don't care about gforth and other C-based Forth systems, and I don't
> think anybody else does either. I don't see a whole lot of
> wordprocessors and spreadsheets getting written in Forth. The only
> thing Forth has been used for since the early 1990s is
> micro-controllers, and all of these systems compile to machine-code.
> That's the real world._ _

What about very PowerPC Macintosh, every PowerPC server, every SPARC
server and desktop machine, and every One Laptop Per Child computer?
They all have Forth in the firmware, with a live Forth interpreter.

The One Laptop Per Child multi-battery charging station is also
programmed in Forth, using a C-based Forth with a live interpreter.

So the above assertion about what Forth has been used for is
demonstrably false.

Stephen Pelc wrote:
> Define
>   SAVE-COMPILATION ( -- n1..nn n )
> and
>   RESTORE-COMPILATION ( n1..nn n )

That's an interesting suggestion.  I have my doubts that it is the
right level of abstraction, though.  There are two aspects to this:

1) Apart from anonymous colon definitions inside colon definitions
    ("quotations" in Factor, "blocks" in Smalltalk), is there anything
    else we can and would like to to with it?  If not, why not just
    define words like :[ and ]: for the blocks/quotations directly.

2) RESTORE-COMPILATION will have a long list of ambiguous conditions
    (or worse, it won't have an standard-defined list, but an
    implementation-dependent one), which will make it hard to define
    and hard to use in a standard way.  We see this problem with
    SAVE-INPUT and RESTORE-INPUT, which are pretty useless, because
    RESTORE-INPUT can fail for any reason, and at least on Gforth it
    does fail except in a few select circumstances.

    The problem here is that this kind of interface seems to support a
    lot of things that don't fit in most systems, e.g., restoring the
    same compilation several times, or non-nested restoring of
    compilations.

    This kind of interface is a bad idea, and we should have learned
    this lesson from the failure of SAVE-INPUT and RESTORE-INPUT.

    Plus, it's non-idiomatic and cumbersome by having a variable stack
    effect.

- anton

Hugh said:

> There is a lack of support for programmers to write
> control structures similar to IF, BEGIN, etc..

Wrong. We've discussed this before. If you still believe this,
give us an example.

> This also includes a lack of support for quotations (ala Factor and Joy).

If by this, you should not define definitions inside
definitions, I agree.

> Introduce a new word:  XT ( -- xt )
> -----------------------------------
> This word will provide the current compilation address,
> such that it can later be given to EXECUTE.

No. Whatever compilation address means, it won't be correct in
all cases, for example:
1) Byte-addressed CPU that requires branches to 2 or 4 byte
target addresses.
2) Some Forth systems require *data* before an xt for
compilation control.
3) Some Forth systems align word entry addresses for performance
reasons.

This is why :NONAME returns an xt. Assuming that a current
address can be used as an EXECUTE address in the future is just
wrong because it make assumptions about compilation.

> Modify all of the control-flow words (IF, AHEAD, etc.)
> so that their orig parameter is the code-address after
> the branch-code. This code-address could be used as an
> xt parameter for EXECUTE to call the code there.

See above.

We really do not have enough time to do your R&D for you! You
really should install and test your proposals on several modern
Forths before you make the proposal!

The valid point that has emerged is that being able to save and
restore compilation state can be useful to permit currently non-
standard behaviour. Bernd's code fragments illustrate the
problem and solution. You still appear not to understand the
idea that a standard encapsulates common practice.

Define
   SAVE-COMPILATION ( -- n1..nn n )
and
   RESTORE-COMPILATION ( n1..nn n )
with sample implementations and you may be better received. With
these two, I think that everything you need can be implemented.
Even if you cannot get approval this time round, you will at
least have tools for implementing what you want and suggestions
for common practice to emerge.

Stephen

--
Stephen Pelc, stephen@...
MicroProcessor Engineering Ltd - More Real, Less Time
133 Hill Lane, Southampton SO15 5AF, England
tel: +44 (0)23 8063 1441, fax: +44 (0)23 8033 9691
web: http://www.mpeforth.com - free VFX Forth downloads

Hugh Aguilar wrote:
>
> Quotations are functions without names; you only get the xt. They are
> compiled within other functions however, rather than outside by
> themselves (as done with :NONAME). They are "first-class datums." They
> are similar to numeric literals that are created inside of a function
> and can then be passed around to other functions, except that instead
> of using them in arithmetic, you use them by executing them.
Why would you do this, instead of making a definition in the normal way?

Cheers,
Elizabeth
>
> *From:* Elizabeth D Rather <erather@...>
> *To:* forth200x@yahoogroups.com
> *Sent:* Saturday, October 10, 2009 3:10:29 PM
> *Subject:* Re: [forth200x] RfD: XT
>
>
>
>
> I'd actually be grateful for a simple English explanation of what
> "quotations" are, as I've heard the term used in various ways in this
> context, and have never seen an explanation that isn't along the lines
> of, "Well, this is how {whatever} does it..."
>
> Cheers,
> Elizabeth
>
>
>
>
>


--
==================================================
Elizabeth D. Rather   (US & Canada)   800-55-FORTH
FORTH Inc.                         +1 310.999.6784
5959 West Century Blvd. Suite 700
Los Angeles, CA 90045
http://www.forth.com

"Forth-based products and Services for real-time
applications since 1973."
==================================================

Hugh Aguilar wrote:
>
>
> Here are some answers to Leon Wagner's concerns:
>
> 1.) I think you're talking about switch-threaded implementations. I've
> never used switch-threaded systems, so you might be right that XT is
> impossible in that environment. Perhaps XT could be an optional
> extension that the switch-threaded systems just don't support. I am
> only interested in micro-controllers that compile to machine-code for
> speed, and these will support XT. Honestly, I don't consider desktop
> Forth systems to be useful for anything except writing cross-compilers
> for micro-controllers --- and cross-compilers don't need XT.
The thing is, when you're writing a Standard, you can't consider only
your own needs and practices.  There are token-threaded systems, and
various other strategies wherein an XT is not a simple address.  This is
why Standards efforts always take multiple years, face-to-face meetings
(even nowadays with the internet) and as much public exposure as you can
get, to ensure that all interests get heard.  It's never possible to
please everyone, but at least decisions can be made in as much fullness
of knowledge as possible about other users' and implementors' needs and
practices.
> 2.) Going to quotations should not be difficult. If you provide an XT
> for SwiftForth, I will write a simple implementation of quotations
> written entirely in ANS-Forth --- without any "under the hood
> machinery" except for your implementation-specific XT. If you don't
> have time for this, I can likely figure out how to write the
> SwiftForth XT myself.
I'd actually be grateful for a simple English explanation of what
"quotations" are, as I've heard the term used in various ways in this
context, and have never seen an explanation that isn't along the lines
of, "Well, this is how {whatever} does it..."

Cheers,
Elizabeth
>
> 3.) I agree with this and have dropped BRANCH, from my RfD. That was
> just an afterthought that I should have given more thought to before
> including in the RfD.
>
> ------------------------------------------------------------------------
> *From:* Leon Wagner <leon@...>
> *To:* forth200x@yahoogroups.com
> *Sent:* Saturday, October 10, 2009 10:09:18 AM
> *Subject:* RE: [forth200x] RfD: XT
>
>
>
> I have some problems with this:
>
> 1) An xt (execution token) is *not* necessarily an address.  So saying
> this returns an xt when what you really want is an address is wrong.
> (And which DO you want?  Maybe both, so there's another set of issues.)
>
> 2) Not sure how you make the leap from this to quotations when a lot
> of other "under-the-hood" machinery would likely be needed.
>
> 3) AHEAD is already defined in Forth-94 to do the forward branch.
>
> I do not support this proposal.
>
> --Leon
>
>
>
>
>


--
==================================================
Elizabeth D. Rather   (US & Canada)   800-55-FORTH
FORTH Inc.                         +1 310.999.6784
5959 West Century Blvd. Suite 700
Los Angeles, CA 90045
http://www.forth.com

"Forth-based products and Services for real-time
applications since 1973."
==================================================

---

From: forth200x@yahoogroups.com [mailto:forth200x@yahoogroups.com] On Behalf Of Hugh Aguilar
Sent: Friday, October 09, 2009 11:10 AM
To: forth200x@yahoogroups.com
Subject: [forth200x] RfD: XT

Well, I'm going to try again with another RfD. I am only posting this here for now. After I get some feedback, I will possibly repost it over on c.l.f..

Author

-------

Hugh Aguilar

Problem

---------

There is a lack of support for programmers to write control structures similar to IF, BEGIN, etc..

This also includes a lack of support for quotations (ala Factor and Joy).

Current Practice

-----------------

Writing control structures, such a IF and BEGIN, are largely the province of the compiler writer, as doing so involves generating platform-specific code.

The old FIG implementations had 0BRANCH, and BRANCH, words that compiled zero-conditional branches and unconditional branches. These were used by IF and ELSE respectively, but could also be used by the programmer to implement custom control-structures. I think FIG also had a word for patching an existing branch with a new destination address, but I don't recall the details now.

Solution

--------

The solution is to provide a word that returns the current compilation address.

This will allow quotations to be implemented. The programmer can compile a branch forward (using ELSE and the orig of a fake IF), then compile his quotation, then patch the branch (using THEN) so that it will skip over the quotation at run-time. This is somewhat inefficient because we end up executing an unneeded unconditional branch, but this should have minimal effect on even the smallest microprocessor. The alternative would be to introduce a word such as DP! that would change the dictionary pointer and allow code to be compiled temporarily into memory obtained with ALLOCATE. This solution is a lot more complicated. What happens if the compilation crashes while DP is modified? Who is going to deallocate all of that heap memory? What about Harvard architecture machines in which you can't compile into the heap? It is easier to just compile the quotation into the memory where we are already compiling code, and suffer the minor performance hit of the branch instruction.

Proposal

---------

Introduce a new word:  XT ( -- xt )

-----------------------------------

This word will provide the current compilation address, such that it can later be given to EXECUTE.

Optionally introduce a new word:  BRANCH, ( -- orig )

--------------------------------------------------------

This word would be like ELSE except that it wouldn't expect to follow an IF and to patch that conditional branch. This word is optional because ELSE can be used if it is given an orig for a fake IF somewhere, but that is a gross kludge that we would be better off without.

I came to the conclusion that Hugh's proposal does not provide
anything that does not exist already, so I oppose it.

As to the "quotations" part, which has lead to the embedded
definitions discussion, Hugh should define what he means by a
quotation. Quotation is a term that is used in linkers, and was
used in the OTA system over a decade ago.

Mitch said:
> Specifying an embedded-definitions feature will require
> careful wording to accurately define the "save and
> restore the state of compilation" portion.  Which is not
> to say that it isn't do-able, but rather that, in my
> estimation, it is not a "piece of cake".

In addition we need to be very careful not to disable or
disenfranchise future implementation techniques. The indications
are that what Bernd needs is doable, but expanding what
Minos/Theseus needs to the general case requires more ports and
more applications before we can generalise.

Stephen
--
Stephen Pelc, stephen@...
MicroProcessor Engineering Ltd - More Real, Less Time
133 Hill Lane, Southampton SO15 5AF, England
tel: +44 (0)23 8063 1441, fax: +44 (0)23 8033 9691
web: http://www.mpeforth.com - free VFX Forth downloads