Lisp Machines

[Antonio Leitao]

Someone asked about what's like working with a Lisp Machine.  I will
give my experience and some considerations for a future LispOS.


I worked a few years with Lisp Machines made by Texas Instruments. I
also work a few hours with a Smalltalk Machine made by Tektronix.  The
feeling was the same.

During those wonderful years, I had the chance to work on a daily
basis with the Explorer, Explorer II, and Explorer II+.

I will now describe these machines.

The Explorer II+ had both a large color display and a B&W display.  As
someone said, the keyboard was something special.  Very large, very
comfortable, with left and right control, meta, super and hyper keys,
dedicated keys for parenthesis, dedicated keys for undo, help, break,
abort, etc.  The B&W display was something marvelous.  Extremely
sharp, with a dedicated region for online documentation.  The mouse
was an optical one, very precise.  In a word, everything was
ergonomical.  I find today's "ergonomical" keyboards a joke when
compared with the Explorer's ones.

The computer was very big. It was the size of a small fridge and the
disks and fans made so much noise that was terrible to work near them.
Of course, we were supposed not to do that.  The connection from the
input/output devices to the computer was a quite long fiber-optics
cable that allowed us to put the machines in a room and work on
another rooms, at a fairly large distance.  The disks' size was 300MB
each (or maybe 150MB, I don't remember).  Each of our machines had 2
or 3 of these disks.  There was also a QIC-tape per machine.  All the
machines were connected by a 10MBit ethernet.

The first models of the Explorer had not much RAM (maybe 16 MB, can't
be sure) and the GC was slow and buggy.  The last model I worked with,
the Explorer II+ had 128 MB of RAM and an generational GC that, as
long as you did not need a full GC, was barely noticeable.

In those times, the configuration described above was not at all
"standard" in most computers, specially when you think that this was a
single-user machine.  When I talked with my friends that were working
on Macintoshes (the best one at the time was the Macintosh II) and
PC's (286? 386?) I felt like I was a "Top Gun". I was really driving a
very special machine.

Although the machines' configuration was amazing, what was really
special was the fabulous CPU and the software provided.

In what relates to the CPU, I'm not the best person to discuss this
(people from LMI, Symbolics, Texas, please, correct me if I'm wrong)
but the CPU provided a lot of help to speedup the usual Lisp
operations.  Just to mention a few, CDR-coding for compact list
representation (although there were some disadvantages with this--but
this is off-topic), hardware-assisted GC, type-checking in parallel
with optimistic execution of instructions, etc.


Now, let's see the software.

The first machines come with ZetaLisp and a very nice object-oriented
system called Flavors.  When CommonLisp come out, the machines were
updated to use it, but still with Flavors.  I never saw CLOS on them.
But more about this latter.

All the software was either microcode or CommonLisp (or ZetaLisp) with
Flavors.  But there were other compilers or interpreters on top of
this.  I vaguely remember a Prolog interpreter...

My main task at the time was to implement some AI algorithms, develop a
lot of code for managing schedules and, specially, to deal with all the
graphical interface of the application we were developing.  I must
say that it was a great experience.  I didn't know much about software
at the time (and most probably I still don't know much nowadays) but
I'm sure I become a different programmer after using those machines.

As someone pointed out before, you had all the source code on your
hands.  You could see and modify the actual code used by the machine,
compile it, and the machine would behave immediately according to your
modifications.  Your code was on exactly the same stand as the system
code.

We had a lot of tools to help us dealing with the system.  The editor
was Zmacs, written in ZetaLisp in an Object-Oriented-way.  Very
modifiable.  I had more productivity with it at that time than with
current Emacs versions.  There were an inspector, a debugger, a class
and instance browser, a processes browser, a stepper and a tracer, all
of them window-based.  You could switch from one to the other
instantaneously.  There were also a lot more tools, like telnet, mail,
etc, but I used this less often so I will not comment on them.

Again, I want to stress that you had all the code for this at your
finger-tips.  Really.  Is nothing like having the sources of Linux.
On the Explorers (and presumably on all the other Lisp Machines) when
you wanted to see what was going on, you just need to hit the break
key (or select a process from the processes inspector), look at the
window debugger, point at some function, edit it, read its
documentation, etc.  You could also modify it, compile it, restart the
computation from some point, etc.  You could also look at the
data-structures, inspect them, modify them, look at the class
hierarchy, include some extra mixins to a class definition, compile
them, restart, etc, etc.  The only thing I missed was a "computational
undo" that I heard that existed in Interlisp.  Of course, I think it
would be absolutely infeasible to have this on a system with the
complexity of a Lisp Machine, but "to just miss this" means a lot to
the value of the system.

An important point (to be discussed latter) was that the Explorers
used a traditional file-system, with directories, files, ownership,
etc.  The code was written to files, formatted according to the user's
intention.  These files were just ASCII files.  All the Lisp Machines
shared the same file system, just like in NFS. There was a versioning
system, like in VMS, where the filename had also an version number.
Since all the operations of the file system were written in Lisp, you
could modify the behavior of the file system in whatever ways would
suit your needs.  I remember a situation where we wanted to hide some
files from the defsystem module to allow us to modify the code without
disturbing the state of the system defined at same point. (You would
use RCS or CVS for this nowadays). It was just a change on the
function that returned the last version of a file.  Something that you
could accomplish in five minutes.

I must stress that it is extremely important to have the all the code
easily accessible for you to be able to do this.  I rarely read a
manual and that's because in Lisp the code is so self-documented that
it was preferable in most cases to just browse the code. The
disassembled code was also so understandable that the debugger showed it
for every function you were inspecting.

As a side-story, I remember we were using an very nice knowledge
representation tool (which I will not tell you the name for reasons
that will be apparent latter) and I found a bug in one function of
that tool.  The bug occurred because the function was recursive
(tail-recursive, actually) and we were using some very unusually deep
class hierarchies and the stack blowed up.  Note that there were no
hard-limits on the stack size (AFAIK), but the system warned you when
the stack reached some pre-defined limits.  You could just tell the
system to increase the stack size and continue the operation (really,
it was just a sort of yes-or-no question, either abort or continue
with a larger stack) but this was not very practical from the
standpoint of the user of our software, so I decided to correct the
bug.  Meta-Control-w (If I remember correctly) and there I was looking
at the marvelous window debugger, seeing the stack, the local
variables, the objects, everything that I wanted.  Note that the tool
we were using _didn't_ come with sources.  Of course, nothing
prevented us to look at the disassembled code, so that's what I did.
This disassembly was so clear, so documented, that I could generate a
copy of the original function within minutes.  Well, at least, it
compiled to exactly the same code.  With that copy, it was now very
simple to convert it to an iterative loop, compile it again, restart
it, and there it was, the knowledge representation tool resumed its
working without any problems.  It was like if the bug never
happened.

Well, the moral is this: I can't imagine this story happening in
current operating systems.

Nowadays, when the software that I use has a bug (be it X-windows,
Linux, Netscape, whatever) well, it just crashes and that's all.  Of
course, usually, I don't even understand the bug ("Bus error" is not
very informative), I don't even think about looking at the source
(usually, I don't have it and probably it is written in C or even
worse), much less to modify it (probably, even if I looked at the
source I would not understand it well enough to be able to modify it)
and, after all, I cannot restart my work where it was before the bug,
so what's the point?  It's much faster to start all over again and
hope that the bug will not show up this time.

Of course, there are some exceptions.  Emacs is one.  I'm constantly
using emacs.  I would say that 99% of my time is spend using Emacs.
When you have a bug in Emacs packages you can at least try to solve
the problem because Emacs contains some of the functionality of a
Lisp Machine.  You have all the source, you can modify and recompile
functions at any time, you have a limited but helpful debugger and
you have an excellent (really!) stepper.  Unfortunately, a big part of
Emacs is written in C and when there is a bug on that part, you're out
of luck.


Now, I would like to enter in some details about an hypothetical
LispOS.

Everybody would agree that the Lisp Machine way was a better way, but
is it really necessary these days?  We must remember that Lisp Machines
were used to run software that run for days and weeks (most AI
problems need huge amounts of processing) so, if a bug showed up after
a week of running time, you didn't want to start all over again.  You
really wanted to correct the bug and continue the computation from
some safe point.

Do we still need this feature?  I know I need (and I still use it with
Allegro CL), but I think that most of the people that use Windows (and
even Linux) do not need it.  I don't expect to see a secretary, or a
doctor, or a lawyer using Word and, upon discovering a bug, entering a
window-debugger to correct it.  Not even the final users of the
software that we developed for Lisp Machines would do this.  There is
a fundamental difference between a user and a programmer, IMHO.

>From what I saw in this mailing list, it looks like some of the
participants pretend that the Lisp OS could replace Windows or MacOS.
I think (based on what I just said) that for the large majority of
Windows or MacOS users a LispOS (similar to the one I just described)
isn't useful at all.  They will not be able to understand it, much
less take advantage of it.  These users just want to use Word or Excel
or CorelDraw or PowerPoint and they definitely do not want to look at
window-debuggers, much less to correct bugs.  After all, these people
payed for the software.

Even if we consider the Linux community, I suspect that most of the
users aren't really concerned about Lisp.  Most of the programmers I
know do not know enough about Lisp.  They almost always prefer to
code in C or C++. Of course, they could be convinced to look at Lisp,
but from the flame wars that we sometimes see between the Lisp field
and the C/C++ field, I don't believe it, at least in the near future.

As a result, I think that a LispOS would be great, but almost
meaningless in what accounts the number of users that can really take
advantage of it.

This does not mean that I'm against a LispOS. Quite the opposite.  I
can live almost exclusively with Emacs, Allegro, and LaTeX. Allegro,
obviously, is CommonLisp.  Emacs is also very close to CommonLisp.
LaTeX would be great if its extension language was something
Lispy-like, but I would be already happy if it was something different
from TeX. So, apart from LaTeX, I could already be using a
LispOS. Unfortunately, there are some rare occasions when I have to
use something else, like Acrobat, or Netscape or whatever.

It would be unrealistic to expect an implementation of Acrobat (or any
other piece of already existent software) for a LispOS, primarily
because there wouldn't be enough users that would justify such
implementation and secondly because the effort needed to convert an
application developed for Windows or MacOS or some sort of Unix to
something so radically different as a LispOS is tremendous.  But some
of us really need to use these tools.  As an example, since the
invention of Java there is a big demand for Java programmers.  Let's
suppose someone asked one of us to developed something in Java.  I
don't like Java, probably you don't like Java, but we all need to earn
some money, so we must code in Java for a while.  How much time could
we wait for a Java compiler for the Lisp Machine?  Or do you think we
should include the Java compiler as our first task to accomplish our
client's needs?


So, for a LispOS to be really useful, either it emulates some already
popular OS, or it runs within some already popular OS.

I suspect that for most Lisp purists, the first option is the best
one, but unfortunately is also the most difficult one.

But I think there are some advantages on the second option also.  We
actually have most of the work already done.  We already have Allegro
CL, or CMUCL or CLISP or RScheme or whatever Lisp implementation you
prefer running on top of most operating systems.

All these implementations lack a lot of the functionality of a
Lisp Machine but why don't we develop such functionality?

I remember that Allegro CL had a package called Composer that could
give us some of the feeling of a Lisp Machine.  Of course, we have to
pay for this, but can also develop a new one from scratch.  If we want
to have a Lisp Machine, why not start with one that runs within Linux,
for example, and implement all the functionality that we need?

I must say that I think this is _not_ an extremely difficult task, at
least when compared with something as hard as replacing an OS with a
new one.

I base my opinion in one of my last assignments on the company where
I worked: to convert all the graphical interface from Explorers to
Suns.

In the Sun Stations, we had Allegro and Lucid.  At beginning we
started with Lucid because it looked faster (on our tests) but in the
end Lucid went bankrupt so we changed to Allegro.  Both
implementations were for CommonLisp with PCL and they also had
CLX. Our system run originally on CommonLisp with Flavors and the very
non-standard Explorer windowing system.

Well, the good news are that I managed to create a layer that isolates
the underlying object system and another one that isolates the
underlying windowing system.  I managed to be able to run the same
code on the Explorers and the Suns.  This gave me the confidence to
think that with some more work it would be possible to translate all
the software from the Explorer environment to the Allegro/X
environment.  In other words, we could replicate a Lisp Machine within
a Sun.  Nowadays, I can say the same for Linux, of course.


I'm sure some of the participants in this mailing list will complain
that this isn't a genuine LispOS.  I know that my particular idea of
Lisp Machine doesn't use a persistent store, but neither did the
Explorer.  I know that it doesn't use a structured editor, but neither
did the Explorer.  I know that it doesn't transmit Lisp objects over
the network, but neither did the Explorer.  Do you think the Explorer
was not a Lisp Machine?


I still think this is a feasible task.  Most probably, we would need
to have access to the internals of the CommonLisp implementation
(specially for the window debugger and inspector), so this excludes
(in part only) a proprietary implementation such as Allegro CL or
LispWorks.  Maybe CMUCL would be a good starting point.  I don't know
enough about CMUCL to judge.

But most of the functionality already exists, at least in a primitive
form.  If you take a look at the Emacs-Allegro interface that comes
with Allegro CL, you will see a very primitive (and buggy)
"window-debugger", but that's the idea.  We just have to make it much
more powerful.  We would also need to convert Emacs to a truly
CommonLisp implementation.  This would need someone with deep
knowledge of Emacs internals. All the Elisp packages would also need
translation, but these are tasks that can be easily accomplished by
other CommonLisp programmers like myself.


If we could manage to implement this sort of Lisp Machine, then we
could start thinking about was is bad in our implementation that could
be better if we had the _right_ operating system.

Maybe then, we can start thinking about the _right_ operating system.
By then, we will know what we want and we will have the tools to
develop it.

Of course, this is just my opinion, anyway.

Sorry for the long message.

Best regards,

António Menezes Leitão.