A Summary of Project so far [Re: Kernel LISP - how low down can it go?]

[Dwight Hughes]

| From: Patrick Logan <patrickl@servio.gemstone.com>
| 
| Is there a good summary of the entire project so far? E.g. how many
| people are interested in working on it, what are their areas of
| interest, etc.?
| 
| Building a "LispOS" from top to bottom leaves a *lot* of room for
| parallel development from the compiler/language people, to the
| utilities and applications people, to the Lisp-level run-time people,
| to the bit-head device people (if there are people that *want* to jump
| right into an experiment running Lisp down to the metal.

Here's a rather subjective summary of the project so far (perhaps
others might want to expand on this as they see fit):

The project is not quite 4 weeks old. We have gone through the primary
general brainstorming period (over 2,000 messages in the first 3 weeks
I believe) and are starting to actually do some research and apply some
practical thought to how exactly we need to begin the actual work.

There is a Duty Roster (actually a list of volunteers) which is being
compiled by Ahmed Hammad <cosc19z5@bayou.uh.edu>. By my count we should
have over 30 people that have volunteered for various parts, or for
anything - Ahmed, would you post another "Call for Volunteers" with an
updated list please?

We still need to define exactly what the separate parts of the whole
project are and to get some project/group coordinators for each
subproject that really know what the hell they are doing. They don't
necessarily have to have a lot of time to give to programming, but
guiding and mentoring will be vital as the work begins.

The fundamental concept behind the LispOS is to create, on modern
vanilla PC hardware, a true LispMachine - not a direct copy of course,
but a system in the same spirit using all the best ideas, concepts,
and system models we can beg, dig up, steal, or generate out of our
pointy heads. The LispVM group split off to deal with the issues
of bringing the LispOS, as much as is practical, to other OSs, and to
serve as a portable target for higher level code developed using
the LispOS. Common between these groups is, of course, the LispEnv,
LispOS Environment -- all the higher level goodies, including
the windowing interfaces, some utilities, mailers, .... Needless to
say, considerable overlap exists and will require considerable
cooperation.

ANSI Common Lisp will be the "standard" high level programming
language. For lower-level work we are just beginning to wrangle
what we will need and how precisely to get there. Eventually,
a layered implementation of ANSI Common Lisp seems a likely
development, with each layer actually being an intermediate
dialect with its own advantages and finer and finer levels
of control over the compiler and code generation. The CMU-CL
implementation will be the starting point for all this in the
LispOS. CMU-CL does not have threads - so we are free to roll
our own how we wish. The debugger has problems single-stepping
with x86 CPUs - so that will need to be figured out. However,
it has an awesome optimizing compiler, "Python", and has been
ported to several architectures - but always for flavors of Un*x.
Weaning it from its Un*x heritage will prove interesting but 
shouldn't be horrible -- CMU-CL tries to do everything it can
within its own environment and most of it is written in CL; it
even includes its own editor, "Hemlock".

The LispOS: will use a single logical address space, will (eventually)
be Lisp-to-the-metal, is generally defined as a completely reflective
object-oriented multi-threaded OS with garbage collection fundamentally
built-in, and will have an object-based file system (there is much
contention about how far this should go - for myself, just a plain old
OOFS, even without true transparency, would make me very happy, but I
have no axes to grind on this topic).

The intention is to make the LispOS the best LispOS it can be - low
level compatibility to any other OSs be damned. All functionality
from the metal to the user interface will be directly and easily
available and can be directly modified as desired (how security can be
integrated into this has been discussed somewhat, but no real answers;
it is agreed that security should be integral to the LispOS and should
be both high-level and fine-grained -- the mechanisms to accomplish 
this will have to be developed as we progress; the answers may lie in
the fundamentally reflective nature of the LispOS and the use of
appropriate MOPs - see Aspect Oriented programming). Threading
implementation has been discussed some, though not in any detail - an
early favorite is continuations as threads, though futures have also
been presented (my understanding is that these approaches don't mix well).

How to begin: the odds-on favorite is the Utah Flux OS Toolkit (several
of us are not-so-patiently waiting for their latest and greatest version,
due any-day-now). This will take care of the worst of the to-the-metal
grunge and allow us to use device drivers from NetBSD, FreeBSD, and Linux
to begin with. It also includes the boot-up code and various low-level
local and remote console debugging code, and other goodies. There is
another camp that thinks we should start with CMU-CL on a minimal Linux
or FreeBSD kernel and replace and tear away the monolithic kernel in
an ongoing fashion. At any rate, we will have to work in a cross-compile
fashion from Linux or FreeBSD to do the Flux OS Toolkit approach until
the LispOS is self hosting. We need not absolutely lock ourselves into
one or the other - experience will chose a winner.

The LispVM: will be rather synergistic to the LispOS. High level code
should be portable between the two, indeed most can be developed for
the LispEnv before either the LispOS or the LispVM is ready. It will
help bring the fruits of the LispOS to other OSs and should even be
embeddable in web browsers. Various approaches have been discussed.
Using and extending the JavaVM was brought up, but was met by enormous
disgust from most of us, for a lot of good reasons. Developing a
UniversalVM has also been hashed over, but seems to present too much
complexity and/or too many compromises. A leading candidate right
now is an eventual combination of a word-based VM with some of the
approaches used in Juice, but there are several other excellent
approaches being considered. In essence, the LispVM group will be
working top-down while the LispOS group works bottom-up. From
the LispOS side several people have put forth the idea of VMs as 
first-class objects.

For the user interface, CLIM will certainly be important, as will X
(CLX?). Beyond that, there has been no more detailed discussion.

There has been much discussion of running standard Un*x binaries
within the LispOS environment, but no definitive solutions. A couple
of people have put forth environments/address spaces as first-class
objects, for other purposes - a possible solution to the "alien"
binary problem also.

-- Dwight