The LispOS Project: a position paper, part 1

[Scott L. Burson]

Hi all,

This seems like a good time to toss in my thoughts about how this project
might proceed.

I. General Comments

First off, I would point out that the current state of uncertainty is only
natural.  There is the temptation to compare this project to Linux; but
building Linux, although a very large implementation effort, was relatively
straightforward in the sense that the goal was well-defined: duplicate Unix.
The kernel design and implementation strategies were well-known when Linus
started.  This meant that he could practically just sit down and start coding.
I don't mean to diminish his achievement; I'm just pointing out that we are
not in the same situation.  Oh, I suppose we could just try and duplicate the
LispM, but that doesn't strike even me, a die-hard LispM fanatic, as a useful
goal.

So I think we just need to accept that it may take some time for a consensus
to emerge on goals and strategy.  It certainly doesn't bother me that this
can't be pinned down in a month.  And even once we select a strategy, there
will be many details to attend to.  Better to invest some thought up front
than to get to the middle and discover it's not going to work the way we
thought.

II. Goals

As I listen to this conversation, I hear a number of ideas about what people
want LispOS for.  Let me see if I can enumerate the major ones:

 -- a highly flexible, mutable system, with as much source code in Lisp as
    possible so we can change practically anything we want to change

 -- something lots of people will want to use

 -- a basis for other work, i.e., something on which new kinds of programs can
    be written (this is my own desire; I don't know if anyone else here shares
    it)

 -- a testbed for various research ideas in language design, compilation,
    etc. etc.

Let me discuss the issue of popular acceptance.  I have seen someone on the
list voice the idea "if it's great, people will use it".  This is a common
sentiment among techies, but it's well known to be false.  The way it works,
instead, is this: for a new, incompatible technology to be accepted, there
must be a single, well-identified problem that it, and only it, solves.
Incremental improvements (e.g., in programmer productivity) don't do it; the
cost of converting existing structures is too high.  There has to be a "killer
app", an application of the technology where there's nothing that begins to
compete with it.

Once the technology establishes itself, *then* it can begin to spread.  People
who have started using it because they *had* to may discover that its virtues
extend beyond its initial sine-qua-non application.

Case in point: Java and Dylan.  I gather Dylan existed before Java (at least,
I heard of it some time earlier).  The two languages have some very similar
goals and design philosophies.  But Java is apparently going to make it in the
marketplace while Dylan appears to be dead.  Why?  Applets.  Gosling
identified and targeted an application for which no existing technology was
acceptable.  There was no such "killer app" for Dylan.

Now that people have started using Java for applets, they are discovering that
it is a very nice general-purpose language.  In particular, they are learning
that a safe, garbage-collected language is worth the overhead it entails.  In
the long run, I expect only a tiny fraction of the Java code in the world to
be applets.

See Geoffrey Moore's excellent book "Crossing the Chasm" for a fuller
exposition of these ideas (to which I owe my present understanding).

I'm going into this because some degree of popular acceptance has been voiced
as a goal on this list.  Most techies don't understand what that involves;
*they* use new technologies just because they're cool (ambiguity intentional),
and they assume, falsely, that everyone else is the same way.  Most techies
shun marketing because they don't understand it.  But my sense is, once it can
be explained to them as a design problem rather than something mysterious and
uncontrollable, they will find that it's actually a lot more fun to think
about than they might have guessed.  Of course, it does require paying
attention to the actual needs of potential users.

Another principle I would mention is that while a "killer app" is necessary,
it isn't, or may not be, sufficient.  The new technology must be adequately
easy for the intended users to learn and adapt to.  Had Java used Lisp syntax,
for instance, it might still be catching on but not with the same velocity.
In that case, chances are someone else would have come along with a C-syntax
variant, which would sooner or later have displaced the original.  (Don't get
me wrong.  I love Lisp syntax.  But to most people, it's unfamiliar.)

So perhaps the first question to ask is, Who is the intended audience for
LispOS?  This deserves some discussion.  I see a few groups who might be
served:

 -- students who want to see another view of the computing world than the
    C/Unix orthodoxy; possibly also, educators who want to expose their
    students to such a view (I'll call these "academic users")

 -- researchers in language and OS design and implementation who want to
    try out their ideas, and to share ideas with others in a concrete form
    ("CS researchers")

 -- professional programmers who want easier ways to build applications, and
    to build new kinds of applications

 -- AI researchers, like myself, who could never quite build the systems we
    wanted to build on the available platforms

 -- general users who want a system that functions more automatically and
    intelligently than current designs

Let me say a few words about each of these.

The academic community has already picked up on Scheme to some extent.  A
high-quality implementation with some important new features (like
persistence) would surely garner some acceptance.  Still, I don't know what
the potential market is here.  If I were a CS professor teaching Scheme, I
would almost certainly switch to Java immediately.  (Of course LispOS could
support Java -- more on that later.)

The CS researchers, who of course overlap with the academic community, are
probably the most important target group.  Indeed, most of the people on this
list appear to be in this category.  These are the people who will gravitate
to LispOS most readily, who will be most forgiving about its idiosyncrasies,
and who likely will get the most benefit out of it.  Object-oriented operating
systems are already an established research area, and LispOS will, I expect,
make a clear and substantial contribution to the field.

The professional programmers are a group I think we'd like to serve, but I
don't know if it's going to be possible to reach them.  Here the need for a
"killer app" is particularly pronounced, and I don't see offhand what that
application could be.  A high-quality persistent Java implementation will be
of interest to some, but of course there are likely to be commercial offerings
in that area fairly soon.  This deserves more thought.

As for AI researchers like me, we already need LispOS badly, but we're not a
large group (we don't even comprise the whole AI community; just those of us
who need to work with very large amounts of stored structure).

And finally, general users.  I have thought a little about what value LispOS
might provide an unsophisticated user.  This is just an idea, and it's a bit
of a long shot, but it does strike me that today's computer systems are still
far too hard to use.  People need them to be self-administrating to a much
greater extent; to build a detailed model of the user and his/her goals and
preferences; and particularly, to provide much more intelligent handling of
errors of various kinds.  This is something a fully integrated system like
LispOS could do far more easily than a present-day OS.  Again, this deserves
more thought, but I wanted to toss it out there.

[Whew!  I've already spent too much time on this and though I have a *lot*
more to say, think I will go ahead and distribute this, as I'm not sure when
I'll be able to write more.]

Comments solicited.

-- Scott