The LispOS Project: a position paper, part 2

[Scott L. Burson]

This is the remainder of what I was planning to write on 5/25.  Again,
comments are invited.

-- Scott


III.  Approaches

As we have observed, there are a variety of approaches we could take to
building LispOS.  This section unabashedly pushes my own preference in that
regard, and also discusses some other possibilities.

My preference, as I have mentioned before, would be to take T, the Yale
dialect of Scheme, outfit it with a high-performance persistence mechanism
such as ZetaBase, and run it on top of Mach.  I think this is a good idea for
several reasons:

 -- Most important to me personally, it is the bare minimum necessary to
    support my research needs.  Sure, there are a lot of things I could
    imagine wanting to add to T -- many good language ideas have been floated
    on this list and elsewhere -- but none of them are *required* for what I
    want to do.

 -- This is a fairly well-defined project.  We could plan it out and get
    started rather directly, and have something running in a few months.

 -- T is a high-quality implementation with a good native-code compiler,
    called Orbit.  Importantly, the source code is available with only
    minimal restrictions (basically acknowledgement).

 -- It will be relatively easy, as far as I can see, to layer other important
    languages like Common Lisp, Java, etc. on top of T.  (There is already an
    RnRS Scheme layer in T.)

 -- I see running on top of Unix as a plus, at least in the short term.
    Others disagree, I know, but personally I have no desire to see the entire
    Unix software base, or even just the free portion thereof, replicated in
    Lisp.  I think we all have better things to do than that.  Besides, until
    we really get a handle on how to do things in a world of persistent
    objects, there is the very real danger that our designs will be
    contaminated by file-think.  We are so used to the notion that all
    persistent information is in the form of character strings that it will
    take some thought and experience to break out of that.

I do understand the attraction of having "Lisp down to the metal" (a goal the
Lisp Machines never quite reached, by the way, because of their use of
microcode).  But it's a massive amount of work to do well, and given that the
more traditional OS world is producing more and more customizable operating
systems (e.g. Mach and, now, the Hurd), it seems to me that the potential
returns are somewhat diminished.  I don't mean to discourage Kelly from doing
whatever it is he's doing, but I don't necessarily want to wait for him to
finish.

Speaking of Kelly, certainly his SilkOS represents another approach we could
take.  But he hasn't given many details about his project -- at least not
since I've been here -- so I can't form much of an opinion about it, and as I
say, I don't necessarily want to wait for it.  He seems like the sort who
prefers coding to talking, which I can appreciate, but in the present
circumstance I think a bit more talking might serve his cause better.

Anyway, beyond PToM (Persistent T on Mach) and SilkOS, there are other
approaches which I lump together in the category of research.  There have been
many interesting ideas floated, and it would be nice to see them pursued, but
one can easily imagine people spending a year just designing something.  Again
I don't want to discourage that -- in fact, I think PToM would make a good
testbed for many of those ideas -- but again I don't want to wait for the
result (I gather a number of people here feel similarly).  At the risk of
offending Fare', I would suggest that Tunes seems to be in this category.


IV.  Suggested Plan

So the plan I want to float at this point has two tracks (maybe SilkOS counts
as a third track).  The first track would be to go ahead and implement PToM
(suggestions for a better name are invited :-).  The second track would be to
think about, and begin designing, the "real" LispOS, whether that's SilkOS or
some evolution thereof, or something else.  I won't say more about the second
track here, not because I don't think it's interesting or valuable, but just
because it's not what I want to focus on.

So the steps, as I see them, toward PToM are as follows.

First, we need to select an initial hardware/OS platform.  My suggestion for
hardware would be the PowerMac.  Orbit can currently generate code for the
68000, MIPS, and SPARC architectures.  It would be much easier to port to the
PowerPC than to the x86 architecture.  Power Mac clones are priced
competitively with Intel boxes, and are very fast (a 200MHz 604e is as fast as
a 200MHz UltraSPARC).  MkLinux is already available on the PowerMac, and when
the CHRP machines come out (any month now :-) they will make a very good
target for a Hurd port.  (I.e. I'd think it would be easier, and more
satisfying, to port the Hurd to CHRP than to write an x86 back end for Orbit.)

MkLinux (a Mach-based Linux) seems the obvious choice for the initial base OS.
There may well be additional benefits from running on top of the Hurd, but
right now it's not entirely clear what those would be.  I suppose it's worth
asking the Hurd group whether they are currently contemplating a CHRP port.

The next step is to begin the T port.  The Orbit back end will have to be
modified for the target hardware, and also, probably, to integrate the
persistence mechanism.  That's a substantial but fairly well-defined effort.
Once that's done, we can cross-compile the T runtime and do the remaining
low-level integration with the persistence mechanism.  That will give us the
bare beginnings of a usable system.

However, it won't be time yet to start writing a lot of application code.
There are some important issues around the integration of persistence into the
T environment.  These have a substantial design component, and so are somewhat
ill-defined.  But perhaps we don't have to come up with the ultimate answers
for PToM.  The issues as I see them are:

 -- There are a variety of higher-level semantic issues around the integration
    of persistence into a programming language: the transaction model, the
    locking model, restarts, etc. etc.  All these things need to be worked out
    and the appropriate language primitives designed.

 -- One of the key kinds of structure that I, at least, want to store in the
    form of persistent objects is source code itself.  To me, the only LispOS
    application that should ever be written using the traditional textual
    representation of source code is the structure editor.  Once that's done,
    we will have a far more powerful code creation and maintenance capability
    than has heretofore existed.  I suppose I should write this up separately,
    as I've thought about it for years.  In any case, it's a big project; we
    can't expect to get it right immediately, but perhaps we can make
    something barely usable fairly quickly, and go from there.  (The CMU
    Gwydion project, a Dylan environment, was supposed to have something along
    these lines called HyperCode, but I never saw any details.  It's worth
    checking it out to see if anything ever came of this.)

 -- Although I don't want to put a lot of language design tasks on the
    critical path for getting something working, it just might be worth
    incorporating immutable types, as we have been discussing.  (I should
    say "additional" immutable types, since T, like all Lisps, already has
    immutable consed numbers such as bignums).  This, at the very least,
    should be thought about and decided.  The potential performance benefits,
    at least if ZetaBase is the persistence mechanism, are too substantial to
    dismiss without consideration.  Of course, it will likely be nontrivial to
    incorporate these into T; that has to be weighed against the potential
    benefits.

In parallel with the effort of porting T, I would suggest it would be
worthwhile to undertake to build a Common Lisp on top of T.  This effort might
make use of a lot of CMUCL code.  I would think it would not be too hard to
make Python output T instead of machine instructions.  (The type declaration
processing abilities of Python need not be discarded; T has builtins to
perform arithmetic on known fixnums etc. etc.)  Of the two tricky semantic
issues I am aware of involved in running CL on top of Scheme-like languages,
one is not a problem: in T 3.1, #F and () are the same object.  (The T manual
promised that they would be distinct in a future release, but there never was
a release past 3.1.)  The other issue is that the order of argument evaluation
is undefined in T (as in Scheme), while CL specifies left-to-right; this can
be handled either by having Python's T back end generate code that fixes
evaluation order, or by passing some kind of declaration to Orbit.

It would be nice if CLOS and T's object system could be integrated, but
they're different enough that it's not clear that will be possible.  On the
other hand T's system is rather rudimentary so maybe it can be accomodated in
CLOS.