microkernels

[Francois-Rene Rideau]

Dear Prof Shapiro,

> [...]
Thanks a lot for your feedback about Mach, Chorus, and Windows NT.
You indeed helped me remove quite some silliness in my glossary entry.

> Second, the polemic stuff simply doesn't add any value.
I agree it should be well delimited from raw data,
but I think it does add value, or at least could, if enhanced.

> Given good information
> followed by *reasoned* explanations, people are quite good at drawing correct
> conclusions.
Certainly, and I will try to improve my style to make arguments clearer,
and to remove gratuitously offensive parts.
However, sometimes, it is useful to suggest an original point of view on
well-known data than trying to establish it in detail from collected data.
Imagination is as important a part of understanding as data.
And I think my point of view is original since I never saw it expressed
elsewhere in either academic papers, discussions with researchers,
or the internet, except by people I had previously convinced of it.

> Polemics are therefore interpreted by intelligent readers as a
> place where the author actually didn't have any facts that supported their
> opinion and so resorted to name-calling.  They conclude from this -- often
> correctly -- that the author doesn't know what they are talking about
> and should not be taken seriously.  In short, I think this kind of stuff
> damages your credibility.
I'm not sure I have much credibility to damage :(
However, I acknowledge you're right, and will try to improve. Thanks.
Also, even more intelligent people might try to see beyond the polemics,
taking them as just a bore-proof tone, and weigh the arguments behind them.
It mightn't be the habit within academia, but it sure is within usenet,
and I feel this page is nearer to the latter than to the former.

> Your comments about academics are equally inappropriate
> and more than a little offensive.
Well, I put the academics mostly on par with industrial developers;
however, I feel academics if anyone do have a responsibility
in detecting and fighting theoretical flaws in existing designs,
and that they failed to their duty at least on this topic
(that I happen to know a bit).
Being an academic and working in an industrial lab myself,
which lab once believed in microkernel design
and learnt its interest and limits the hard way (Chorus hacking),
I feel as entitled as anyone to make these remarks.
Of course, my statements do not constitute
an official academic contribution, either.

> I don't know of any academic who would laugh at a monolithic design
> proposal these days.
*These days*. Such was not the case in the late 1980's and early 1990's,
during the Microkernel craze, where it appears from reading OS design
literature that microkernels were "obviously" the Way To Go(TM),
and no one would fund a new OS project but microkernel based.
and this literature is still crippling the minds of people interested
in OS design.

Nowadays, the craze is waning, and the buzzword "microkernel"
has no more the hype value it used to have.
Today, "Object-Oriented" is still high, and waxing is "Java".

> EROS is a case in point.  It's a major departure in design,
> and people asked hard questions about that.
There are also questions I ask myself about EROS...
One is a low-level one, and has to do with committing change and IDE disks
(since the ATA specifications says little about buffer flushing,
as opposed to the SCSI specifications);
I'm particularly curious since the documentation only talk about IDE drivers;
this may be a silly question, but it still bothers me,
since the OS tries to be "Extremely Reliable".
Another one is higher-level has to do with
"what language/API are reliable applications to be written in?".
i.e. a system "kernel" is but a component of a system framework,
and is useless without the rest of the framework.

> It is not a microkernel, but this
> has never come up as a design issue one way or the other.
Good.

> The kernel
> architecture is structurally similar to a microkernel,
> and has some carefully
> thought out layering.  So does Linux.
I understand that as a confirmation to my arguments:
the important issue is the high-level structuration of the system,
and the concretization of the system structure into low-level barriers
is but overhead that proves useless at best, and otherwise harmful.

> Nobody at the University of Pennsylvania
> laughed at it, and several commercial efforts based on it are now underway.
I'm glad about it. I just hope for everyone's benefit
that "commercial" won't equate to "proprietary" as too often in the past.

> If you have an example of an academic who has laughed at monolithic designs
> when research on them has been proposed, make that known.
Well, the MINIX vs Linux "flamefest" of 1991 is such an example,
where Andrew Tanenbaum contended that a monolithic kernel was
an obsolete design that no academic would develop.
>From testimony of other people having hacked OSes at that time,
it looks like there was indeed a general academic consensus
for microkernel design and against monolithic design.
I guess I'd have to dig into old archives to make that into a strong point.
Now, I could return the burden of the proof, and ask for evidence
of research on monolithic kernels started between say 1988 and 1991,
or of papers infirming a general bias towards microkernel design.
Anyway, I'll make it clearer that this statement is but an opinion of mine.

> If you do NOT have several
> concrete examples, then your text is an undeserved slur on a lot of people,
> and
> it is a fundamentally dishonest thing to say.  You are certainly entitled to
> that opinion, but a "glossary" is a place for statements of fact, not for
> opinions.
You seem to have a particular view on what a glossary should be
that be restricted to purely technical, neutral, glossaries.
I invite you to read Ambrose Bierce's "The Devil's Dictionary"
	http://www.vestnett.no/cgi-bin/devil
(listed at the end of the Tunes glossary) for an otherwise designed glossary.
I do think technical glossaries are useful
(if you know of ones, I will be glad to list them, too).
When the Tunes glossary is moved to a database,
I'd like it to have a "strictly technical" mode
that would strip opinions and purely subjective definitions,
together with opinionated modes that include (possibly diverging) comments
by one or many different members (or non-members).
But in the meantime, I conceive this glossary
as a way to express my own opinions as well as as a technical document.

I reject the concept of neutrality of technology or technique,
or that of an aseptized science, anyway.
Only by being able to err can we be able to be right.
Of course, with the right to a strong opinion comes the responsibility
to make it as good as possible, and to be open-minded in amending it.

> I think it's worth noting that POSIX benchmarks are part of the problem, not
> part of the solution.
They are indeed part of the problem, although this problem is orthogonal
to that of microkernel vs monolithic kernel design:
they introduce a strong bias in the expected behavior of user programs;
but whatever the behavior is, two barrier crossings will take more time
than one barrier crossing, and a "multispace" system implementation
will have a runtime overhead not present in a monospace one.
Whatever benchmark you choose, and whatever microkernel-based system
you choose, you'll improve the benchmark by folding
the whole microkernel+servers into a one "monolithic" system,
removing unnecessary barrier crossings.

> One result is that most
> microkernel architects have been forced to work on getting lmbench numbers
> as a condition of publication rather than working
> on whatever made their system interesting.
This reminds me of a point I raised in my article
	Metaprogramming and Free Availability of Sources
	http://www.tunes.org/~fare/articles/ll99/index.en.html
about proprietary software leading to software partitioning
and the establishment of strong stable software patterns
resiliant to any global innovation or argument of utility,
because the only allowed improvements are ones that are purely local
and preserve the global software structure that is protected
by intellectual property barriers.

> That is, the research has been compromised in a way that means we
> don't really know what the current generation of microkernels can do,
> and we are really unlikely to find out.
I can but be sorry about that.
I will be the last one to defend POSIX and UNIXish architectures,
as you might know by browsing along the TUNES site.

> As an aside, an argument can be made
> that this sort of politics is essential to
> the process of research.  An argument *has* been made that for this reason
> science is primarily concerned not about fact but about orthodoxy, and is
> therefore indistinguishable from religion.
I would contend that such is the unhappy case in a world
where information is proprietary,
but that such it shouldn't be, since information should be free (of rights).

Although science cannot be devoid of political choices,
just like any human activity,
and cannot be devoid of opinions,
just like any human thought process,
its role however is to develop argumented choices and a posteriori opinions,
as opposed to arbitrary choices and a priori opinions.
The distinction between science and religion isn't in the static contents,
but in the dynamic process.
Proprietary information flaws the process.
[Ahem. This kind of arguments should go to cybernethics@tunes.org
rather than to tunes@tunes.org...]

> If your primary goal is to run UNIX, you will find
> that UNIX implementations are surprisingly good
> at running UNIX and everything
> else will tend to be less good at being UNIX.
> If this is your primary goal, a microkernel is inappropriate.
Indeed.

> In fact, after a certain point, adjusting a
> microkernel-based system to do well on POSIX benchmarks
> proves to be a good way
> to make it bad at doing microkernel stuff
> -- the stuff for which you built it in the first place.
Just what that stuff is, and how microkernel design helps about it,
is most likely what I fail to understand.
Whatever the programming model you choose to implement, UNIX or not,
a microkernel-based system will be a poor choice
as compared to a same-functionality monolithic-kernel-based system.

> If, on the other hand, modularity, security, or debuggability are important,
> microkernels or systems like EROS that encourage dekernelized design may be a
> good choice.
>From the little I read about EROS, it may be a good design;
but I contend that microkernels are definitely not a good choice
as far as modularity, security, or debuggability go.

> I have read about the problems that the HURD folks are having.
> These problems reflect poor architectural choices in the structure of HURD
> rather than any fundamental problem with microkernels.
I beg to differ. Their problems were directly due to having HIRDs of
Unix Replacing DAEMONs, i.e. a heavily asynchronous programming model,
for which no adapted debugging tools are known, manually implemented
in a low-level language, which makes every single low-level bug
a hell to track and debug. These guys followed the microkernel model
to the letter, and were bitten by bitter reality.

> Finally, I think your arguments about modularity are not really to the point.
> That is, you are correct that disciplined programmers
> can achieve modularity in
> large systems, and that this is not a compelling advantage of microkernel
> systems from a theoretical point of view.
> From a practical point of view, with
> a lot of experience writing and modifying large systems,
> I can also say that the devil is in the details,
> and that various Linux subsystems have "cheated" for years
> before sufficiently modular interfaces became available and migration to
> them slowly occurred. It can be argued that this is good, but it should be
> noted that the resulting evolution was more often accidental
> than intentional.
I will contend that the ability to "cheat" and to delay
decisions concerning system structure is precisely what make
"monolithic" systems so much superior to "microkernel" systems:
they don't force you into making early decisions at a time you just
don't know enough yet to make it right;
they allow you to reorganize things from a high-level point of view
without having low-level details of barrier interface getting in the way;
they allow you to integrate your objects in a one coherent system
instead of desintegrating your system into lots of servers the coherence
of which you have to manually ensure.
Of course, every evolution step will be "accidental".
The fact that evolution exists is not;
it is a direct consequence of the programming model.
[The case against "early optimization" has been largely developed
on comp.lang.lisp in great posts by Kent Pitman or Erik Naggum,
as the ultimate reason why C and C++ are bad as compared to LISP].
Time to re-read Alan Perlis' Epigrams...

> Also, your claims about the weak value of such modularity
> are most dependent on
> arguments about drivers, which are not by their nature isolatable.
> Anything
> with access to physical DMA is not terribly isolatable in real systems.
In as much as I understand your remark (not much), I disagree with it.
Modularity is a *high-level* property of the system source design,
and is independent from low-level binary failures that could possibly occur
due to buggy components.
An example of source-level modularity for device drivers is the Flux OSkit
(although this modularity comes at the price of runtime COM invocations
due to lack of infrastructural support for partial evaluation).

Maybe you're thinking of some kind of component-wise fault tolerance
within the system infrastructure, but I see no point in it,
all the less in a system where all the source is visible
and you don't fear "attacks" from within system components.
See again my remark with the only possible justification for microkernels
being third party proprietary black-box system components.

> The more critical concerns involve fault isolation *in applications*.
> HURD is not far enough along to really be able to exploit this,
> and without process persistence it will be hard to achieve.
I fail to see how HURD does any better than Linux in this respect;
note that "middleware" already exists to allow for process persistence
under Linux.

> Driver fault isolation is largely an
> illusion, because driver errors leave hardware in states that take whole
> machines down.  This is true whether the driver is in user-mode code or
> kernel-mode code.
Indeed. Again, I see that as backing my argument for modularity in the
system design being useful only at a high-level (source-level),
and utterly useless (and even harmful) at the low-level (binary).

>>  Now, as far as polemics go, what is your
>> take on microkernels? Maybe you have pointers
>> to articles that support or contradict some
>> of my opinions, and that I may link to?
>
> I gave some in my netnews posting.
I fear I missed that posting. Do you have a URL (possibly through deja.com)?
I admit one of the reasons why I put my opinions in the glossary,
besides the fact that I originally intended that glossary
as much as a manifesto as as a technical document,
is that I hate the ephemeral nature of netnews posts
as contenders of opinions you have and have to repeat every so often.
Moreover, a web document is something that can be incrementally improved upon,
just as I hope I'll be doing thanks to your feedback.


> My basic response is that you either have
> hard data supporting your position or you are simply shouting uninformed
> opinion.
I think that things are not black or white as you depict.
There are such things as established facts, but even among them,
there is no such thing as absolute certainty.
And there are very partially informed opinions,
but even among them, not all are useless.

My opinions may not be perfectly informed,
I still think there is something about them that should be said.
However, I do agree that it should be made clearer
what is not well-established about them;
and of course, I am most willing to make them better
in the light of more eventual information,
although my current arguments make me think it unnecessary
to gather more information to constitute my opinion.

> If you have the hard data, present it.  All of the commentary I have
> seen on your web site is anecdotal.  Much of it has been examined in the
> literature.
> Each of the problems you identify existed in some particular system
> or family of systems.
> Several present good arguments for why those individual
> systems made some poor choices.
> NONE of them, as far as I can tell, present a
> fundamental argument for why microkernels are unsound.
> All of the substantive
> issues you raise have been addressed in subsequent designs.
I've tried to synthetize a simple, useful, coherent, predictive
point of view, i.e. to understand the general phenomenon,
out of well-known experience on microkernels.
I'm sorry that I couldn't convince you,
and even more sorry that my attempt appeared to you
as a mere enumeration of uninformative factoids.
I fail to see how subsequent designs addressed
the fundamental question I raise about the abstraction inversion
that constitutes the very core of microkernel design.

> For example, you complain about the shrinkage of microkernels into interrupt
> dispatch + messaging.
If you understood that, then I definitely must rewrite my entry.

I do think that L3, L4, Lava, Fiasco or Xok are quite interesting designs,
and definitely improvements over Mach and "traditional" microkernel designs.
But I also think the included improvements are independent
from their being used as "microkernels"
versus their being used as inner parts of a monolithic kernel,
and that the first case (microkernel) will only lead to
the usual decrease in performance to be expected from microkernel design.

i.e. L4 et al. are interesting, but the microkernel aspect of them
is part of the problem set, not of the solution set.

BTW, why didn't IBM just free L4/Lava? is the issue now solved?

> Based on this, Lava ought to be very good at running Linux. [...]
> Now you may say that a 3% to 5% degradation is still a degradation.
Indeed, and I cannot imagine that adding new unjustified barrier crossings
could fail to introduce performance degradation. And of course

> I agree, but remember that the primary purpose of Lava isn't running Linux.
And what is the purpose of Lava? Allowing the implementation
of a few real-time threads? Or of specific memory-mapping mechanisms?
How does it make it better than directly hacking and enhancing
the Linux or {Free,Net,Open}BSD kernel? (see RT-Linux, etc.)
Certainly, Linux, BSDs, and other kernels have a lot of complexity
and idiosyncrasies that you have to live with when adding such functionality.
But such will be the case with any other same-functionality system,
even Lava-based.
And if you want to have a "simpler" system,
you can strip a kernel such as Linux or *BSD to just what you want,
or you can synthetize it with the Flux OS Kit or similar technology.

In any case, I fail to see what a microkernel brings,
except for some overhead and useless impedance to which to adapt.

> Also,
> remember that this is after an effort of only a small number of months, which
> argues that the modularity structure actually works very well.
Yes, but this is hardly an argument FOR a microkernel;
it shows that *Linux* is a usefully modular yet monolithic system.

> Personally, I don't think microkernels are better or worse.
> I think they are suited to a different set of problems.
I think microkernels are fundamentally flawed,
as a low-level attempt to solving a high-level problem,
namely modularity in system design.

I'm sorry for having dodged your requests for providing hard data
and less opinionated comments.
Give me a laboratory and fund such research, and I might <evil grin>,
although I wonder what kind of data would satisfy you (either way).
Meanwhile, all I can give is opinions constituted
during the copious free time left by my main PhD research:
the semantics of reflective concurrent systems.

Best regards,

[ "Faré" | VN: Уng-Vû Bân | Join the TUNES project!   http://www.tunes.org/  ]
[ FR: François-René Rideau | TUNES is a Useful, Nevertheless Expedient System ]
[ Reflection&Cybernethics  | Project for  a Free Reflective  Computing System ]
As long as software is not free, we'll have hardware compatibility,
hence bad, expensive, hardware that has decades-long obsolete design
		-- Faré