release 0.0.0.20 and thoughts

[Francois-Rene Rideau]

Dear joyous Tunesers,
   I hope you're having a nice summer (I have a special thought for those of
our friends in the south hemisphere).
   I am releasing the long delayed internal version 0.0.0.20, now that
the core of the i386 seems stable.


   So what is new in .20 is mostly the code in the i386 subproject:
the set of m4 macros is well tested; the boot loader is fully debugged
(hopefully) with a checksum; the "biosapi" allows the protected-mode
LLL to call real-mode routines and interrupts in "R86" *real* real-mode;
hardware interrupts are by default reflected into their usual real-mode
handlers, but pmode versions can be added without interfering with the
ability to call the BIOS (well, still, each device can be managed by only
one of the BIOS or Tunes). Reentrance problems currently prevent the V86
reflector to work, but this is not very important; we'll eventually
provide reentrance anyway to allow multithreading, and then we can reconsider
V86 mode, too.
   What is important is that we have a basis to program a 32-bit OS for the
i386 that can use the BIOS for IO while its own drivers are being developped.
Basically, we're at the stage one would be who'd just acquired a
well-documented CPU development board; just that instead of buying
some isolated specific hardware, we now have code for a wide-spread generic
platform. So now, real LLL coding can begin.


   I have received no help at all, but in a few available source files;
books about the ix86 and the PCs are crap; docs on the internet are better,
but do not tackle the specific problems of the kind of software we're writing.
The most useful sources of information were existing *-extender systems.
I think I will have to write a bibliography about where to find useful
information.
   As a result, the code yield is poor for so much work-time. But my time was
not lost for that: by munging with assembly, I have learnt (and crystallized)
a lot, not only about the lame PC hardware, but about programming in general.


   Firstly, doing low-level code is very difficult and annoying, because the
hardware is complex and not well documented, that is, it has evil semantics.
The only reasonable way to develop is at abstract over the hardware to
obtain objects with better, *cleaner semantics* (abstracting without cleaning
the semantics, as in C, is lame); else you get too many bugs. When things run
correctly, it is time to make them quicker, but not before. So I wrote
*lots* of macros, that made development possible. I also saw how no good
macro language exists. CPP is pure shit; I chose m4, but it has
evil reflectivity semantics (I had to systematically remove reflectivity
from my code, but for some trivial macros). Anyway, with as86 (a simple, but
robust assembler), and m4 (a powerful macro system), I could do much more
than anyone can do with lame DOS-based macro-assemblers, or ad-hoc C code.
I'm sure that a test for our language system will be that it be based upon a
clean, simple and robust low-level core, extended using a cleanly extensible
parser.

   The second point is that most of the hassle (and a great source of nasty
stupid bugs) with low-level code is what I call "calling conventions",
which is commonly solved in HLLs by "typing": 1) you have to give explicitly
implementation details of functions, and 2) even when you've successfully
defined your function, you must manually conform to these details when
using the function further in your code. Note that calling conventions
are not just immediate arguments, but all contextual dependencies and
all the background required for a function to work properly (e.g. such
pointers are of such "type", are not aliased; such structure is uniquely
referenced, such list is injective, those two functions are inverse one of
the other, etc). Actually, this is the more general problem of language
*expressivity*: no existing low-level languages can't abstract over calling
conventions; conversely no high-level language can't talk about low-level
details; and all existing general purpose languages have a deterministic type
system (when not a trivial one), and thus very little expressive.
   The work-around for using languages that lack abstraction power,
is that calling conventions should always be particularly well documented,
and great caution is taken when invoking a function; for that reason also,
people often choose arbitrary global conventions that reduce performance
greatly to reduce the amount of bugs. All that is stupid. A language just
*should* have enough abstraction power so that one could define arbitrary
conventions or meta-conventions, and let the computer do all the dirty
work (programmable calling conventions).
   The converse problem is just to define interfaces between low-level objects
and the higher-level system. This has to be done anyway while writing a
compiler and libraries; so instead of providing it only in a specific static
piece of software, we can and must do it in a generic, dynamic module.


   I will be satisfied of the HLL compiler only the day when I can achieve
better and more reliable code than I can currently do with hand-coded
assembly, and in much less time, while having a high-level prototype
immediately; and this can be done easily by programmably combining
higher-order code transformations on high-level code, and generic
meta-implementation operator that map high-level objects to low-level
implementations (which is the generalization of calling convention),
with a simply customizable syntax, so I can adapt the input tool to the
input data. *This* is the trend where we should go.


--    ,        	                                ,           _ v    ~  ^  --
-- Fare -- rideau@clipper.ens.fr -- Francois-Rene Rideau -- +)ang-Vu Ban --
--                                      '                   / .          --
Join the TUNES project for a computing system based on computing freedom !
		   TUNES is a Useful, Not Expedient System
WWW page at URL: "http://www.eleves.ens.fr:8080/home/rideau/Tunes/"