Reflecting on reflective computing.

[Francois-Rene Rideau]

Dear Happy Tunespeople,
   there's quite a lot of traffic on the list lately.
I'll try to give incremental answers wrt to what has already been said
(particularly by David Manifold or David Jeske --
is it there a David Conspiracy Lab?).

>: Chris Barry, 19981021
> Firstly, and most importantly, have you guys *really* questioned what
> our computers are for and why they are useful? You might make this
> really cool OS based on <insert novel ideas here> that uses <insert
> buzzwords> technology, what difference is say running a word
> processor on Tunes going to make to the average MS Word user or AOLer?

What difference is there to him if it's Windows or Linux or Tunes or OS/2,
or MacOS, or whatever below? What difference if it's a 8 bit or 32 bit or
244626 bit CPU below? What difference to him if it's written in COBOL,
8088 assembly, or C++? None that he can see, at least, not directly.
But does that mean that there is none that will affect him?
That would be ostrich-thinking: "what I can't see can't hurt me".

Please (re)read:
	http://www.tunes.org/doc/WhyNewOS/Part_1.html#Computing Systems

If you think that nothing new needs be done with computers, then indeed,
what's below doesn't matter, and we may as well stop any software development
activity altogether. But if progress means anything to you,
if you think that computer science and engineering is a field with a lot
of evolution ahead, then meta^n-programming tools do matter.

Another take at it:
given any specific enough topic, most people don't give a damn about it
(by very definition of specific). If this criterion was enough
to conclude that something doesn't matter, then, quite frankly,
nothing at all in the whole universe would matter. The fact that things
that matter are each taken in charge by only few people is called
division of labor, and is the single greatest reason
of the success of mankind.

Note to Tril: maybe we should turn it into a FAQ?


The whole purpose of Reflection in computing systems
is to allow people to control through dynamic negociation
what is explicit and what is implicit,
so they can focus on what they care about,
and not be annoyed by the irrelevant.

Reflection is a long-term win for the same reason that free-software is
a long-term win: because it allows incremental development.
Actually, reflection and free software are deeply related topics:
metaprogramming requires sources be available to transform; widely (wildly?)
modified sources require metaprogramming to ensure consistency.

As for why Linux (which I use) isn't enough of an OS,
well, simply put, it's just not reflective!
It won't ensure consistent global object naming;
it won't enforce any user-defined invariants;
it won't ensure fine-grained access to system components;
it won't allow seamless dynamic tayloring of code to deployed systems;
it won't permit reliable extraction of a functional subsystem for embedded use;
it won't support dynamic metaprogramming of the system; etc.
Ahem. Time that I finish that WhyNewOS article, after all!
Well, just re-read the Tunes feature list below, and
see that because Linux is not reflective,
those features, that it doesn't provide, can't be dynamically added to it.


As for buzzwords, I've undertaken a Glossary precisely to explain
that Tunes was using technical terms, not buzzwords.
Can you make a difference between the following two,
and tell which gives positive technical details, and which is hype?

1) a fully reflective architecture
with standard support for unification of system abstractions,
security based on formal proofs from explicit negociated axioms,
higher-order functions, self-extensible syntax, fine-grained composition,
distributed networking, orthogonally persistent storage,
fault-tolerant computation, version-aware identification,
decentralized (no-kernel) communication, dynamic code regeneration,
high-level models of encapsulation,
hardware-independent exchange of code, migratable actors,
yet (eventually) a highly-performant set of dynamic compilation tools (phew). 

and
2) a proven 32-bit cutting-edge state-of-the-art industrial-strength
Y2K-compliant zero-administration plug-and-play industry-standard
Java-enabled internet-ready multimedia personal-computer Operating System
that is even newer and faster yet compatible,
with a user-friendly object-oriented 3D graphical user interface,
amazing inter-application communication and plug-in capability,
an enhanced filesystem, full integration into Entreprise networks,
an exclusive way to deploy distributed components,
seamless network sharing of printers and files.


> What audience are you guys targeting Tunes for anyways?
First, ourselves, and the likes of us.
I firmly believe that individualism is a positive value,
that makes the whole world move.


> I can
> see how you guys have some goals to this end, but I don't see why you
> need to write a new OS to do it, instead of just writing algorithms atop
> of Linux
Why do you recommend Linux? Why not Windows? MacOS?
Let's admit you see a point in free software, and say that Linux ~= *BSD.
Why not (Free|Open)DOS? VSTa? MMURTL? Any of those free OSes listed in
	http://www.tunes.org/doc/Review/OSes.html
? Because Linux offers some infrastructure that the others don't.
Well, why do I want an OS, and not just an application?
Because I want some infrastructure that Linux (or any non-reflective OS)
won't *ever* be able to provide.
Certainly, Tunes could run as a closed world atop Linux;
that's precisely our plan with LLL/OTOP.
It will use Linux as much as DOS apps use DOS:
by bypassing and reimplementing as much of its services as doable,
keeping it only as a virtual device driving service provider.

Now, if you think you don't need anything that Tunes aims to do,
well, then, so good for you. We're not forcing anyone to join,
and certainly not recommending anyone to wait.


>: Chris Barry, 19981022, in response to Tril

> To make really cool OO system is certainly possible, but to have it
> incorporate *every* miracle feature you guys discuss I doubt is possible
> with current technologies.
I think that every single feature talked about has already been done
somewhere before, so is definitely possible with current (or past) technology.
What hasn't been done is a framework that allows
such independently developed features to be seamlessly integrated.

[duplication of effort]
> In the free software community this isn't such a big deal/thing, since
> there isn't really commercial competition.
There *is* competition, even in free software:
competition between features that may be implemented,
but only a few of which are, because resources lack.
Existing development frameworks, with their low-level interfaces,
force a lot of redundancy in efforts,
a lot of dissipation in irrelevant details,
a lot of stupid low-level incompatibilities,
and result in less features being coded.
[Hum; the above will make it on my article
on the Catallactics of Free Software]

> yes, I agree that OO is very important
Have you read my Glossary entry on OO?
I also have an incoming article *against OO*
(but I've begun it in french).

[Saving programmer time for better uses]
> [...] until the very processes
> themselves are made obsolete when the computer is intelligent enough to
> do it for us. I don't see Tunes as directly a step towards this end,
> because I think implementation of intelligent algorithms and the
> improvement of language and compiler technology itself through the
> implementation of these intelligent algorithms is the most important
> thing, and actually probably a nessecary first if you guys wish to have
> what you need to create Tunes.
>
Before you can have any automated intelligence at programming,
you need a metaprogramming framework. TUNES is precisely such framework.
And no, despite its being alone in the competition, LISP just isn't enough.
It only has ad-hoc reflection, and provides no way to control
what is explicit and what is implicit at a fine grain.
It doesn't provide a generic framework for arbitrary metaprogramming.
ML has perhaps better metaprogramming facilities, but no reflection at all.


[Automatically optimizing the system for known special cases]

> Do you mean optimizing in the sense of say special hardware cases
> (presence of MMX unit or deep pipeline), or software, like case N=13 for
> an algorithm or system=SysV? Or perhaps both? In any case if the system
> you guys are making has the intelligence to automatically optimizes
> programs and algorithms for all relevant special cases that the
> programmer would be interested in automatically, well, good luck.

Yes, that's what we mean, and more: we may optimize a group of programs
to run together, given the processor type and subtype, the size of the
various caches, the typical load, etc.
The technology to produce code for dedicated processors exists:
it's called a compiler (see -m options of pgcc).
The technology to automatically specialize code for a given case
already exists and is mature: it's called Partial Evaluation.
It has been shown that even with fairly simple technology
to determine the interesting special cases,
it already leads to great speed benefits.
I can very well imagine that integrating all these technologies
with the kind of dynamic global analysis made possible by reflection
would result in quite satisfying speed increases wrt what we have today,
***without the user or programmer having to even care about it***
(yet still *allowing* him to, if he so pleases);
human time can thus be spent on higher-level tasks,
that is, finding ways to achieve "AI".

Trivial application of automatic code specialization: fast boot.
Instead of fetching lots of software, running initialization code,
and swapping the result off, during 5 minutes,
just download a working copy of a preinitialized memory image of all
the system daemons. If prefetch active pages first, you can run the rest
of the download in parallel (or lazily), and achieve amazing boot times!


>: David Jeske, 19981022

> Tunes is following the trend
> of software abstraction to it's logical extreme. Hopefully, when
> enough people (on this list or elsewhere) have enough stuff figured
> out we can take a few steps back from the ideal and come up with an
> implementation.
Anyone, feel free to be the one who steps back.


> Tunes, more than anything else, is proof that there is some convergent
> goal out there just beyond the horizon. Many of the people on the
> Tunes list developed their ideas on their own, and while many of the
> discussions on this list are filled with nitpicking discussions over
> semantics, or approach, we each have a private version of the same
> vision stirring in our heads, knawing at us. Something we know is
> achievable, if we can just put the right pieces together.
>
> THAT is Tunes.
>
Thanks, David. [Rest of text cut: I have nothing to add!].


>: Chris Barry
> You must define intermediate goals where
> there is a clear path to what you want to achieve.
Well, *here*, I agree. Doesn't help me much, tho.

> And why would we use non-window based graphical interfaces?
Because windows just plain suck.
I only use full-screen or tiled "windows" in a one big virtual desktop,
together with the scwm pager to help move around and xdaliclock.


>: David Jeske

> Tunes (at least my idea of Tunes) seeks to change this, by no longer
> forcing the software and drivers to be coded 'for' one of these
> models. In Tunes, code your hardware-access code... let it live in
> user space, let it live in kernel space, let it run in a software
> simulation, it dosn't matter, and the code won't change. Those are
> run-time details which should not change the operation of the code.

Well put.

> If I could kill one thing in Linux, it would be the "standard C
> library". Don't even get me started on that. The standard UNIX/POSIX
> libc is the source of at least half of the 'implementation dependence'
> and 'application install problems' that exist on UNIX today.

Indeed! Due to C having no language support for modularity,
everything ends up in one huge monolithic library.
Libc authors had the choice between lots of object files
with complex mutual dependences to manage manually,
or just one big file, so chose the latter option.
Anyway, it real sucks.


>>: David Jeske 19981022
>: Chris Barry 19981022

>> where I can run out of disk space, pick
>> up a new drive, drop it on my machine, and have it say "10gb
>> available", no partitioning, no moving data around, nothing. Where I
>> can do all of the above without having to deal with 'installing' or
>> 'configuring' even a single 'application'.
>
> This isn't that impressive. You usually don't have to change any
> hardware settings anyways to just drop a new disk in. And like
> partitioning and formatting is *that* much of a PITA....
You forget: managing the filesystem, and deciding what hierarchies
to put on what disk. Manually migrating complete filesystems from
one disk to the other, stopping services until it's done. Manually
managing redundancy or critical data (or do it in the hardware with RAID,
which is very cost-inefficient, and won't let you take advantage of decreasing
disk price, since you have to buy all disks of same model at the same time).
etc, etc.

> Now if Tunes could write a driver for you, that would be something....
See how the use of a DSL and Partial Evaluation allowed the COMPOSE team
at IRISA to write X video drivers one tenth the size of the original source,
and as fast. Metaprogramming can sure ease writing of device drivers!


>> However, it's not even about this (to me anyhow). It's about removing
>> all the add-hoc libraries and headers, and manual installation and
>> configuration which is necessary to actually use new optimizations, or
>> build with new stuff.
> 
> So how does one do the equivalent with Tunes?

Allow both precise extensional definition and checking of interfaces,
and precise global intensional reference of modules,
which permits easier and more reliable interoperability.


>: David Jeske 19981023

> - provide information hiding for all entities. OO has [...]
Information hiding is EVIL. Noise removal is GOOD. "OO" is noise.


>: Chris Barry 19981023

> I was expecting to find a community working on a project in the sense I
> am accustomed to in the Free Software community, but you guys are not
> working on this in that sense, which threw me off initially but now
> everything has kinda sunk in.

Any project is what participants make it to be.
I sure share my (big) part of responsibility in it still being
but a Town Council(tm) of an OS project.
I hope that at least it's a Town Council where the stakes are high,
as compared to other ones that I don't dare name.

> Proving the correctness of a program is a problem that the world
> presents to intelligence.
So what? Programs require intelligence, yet they are possible,
and often largely automatable. Proofs are no different.
Allowing automation doesn't mean that people won't have to think and work;
it means that the low-level details will be taken care of, and they will
only have to focus on the real high-level conceptual design.

Anyway, proofs of correctness is just one of the many applications
to a reflective architecture. The central focus is this architecture.


> And what will be the consequences of bugs within Tunes?
Quite the same as in any other systems.
Only, metaprogramming allows
1) to simplify programs and remove all the potential bugs due to
 manually managing low-level aspects irrelevant to given program.
2) to use tunable type systems to statically check correctness of programs,
 given enough human meta^n-proof information.
3) to amplify bugs, and make them easier to detect


## Faré | VN: Уng-Vû Bân   | Join the TUNES project!  http://www.tunes.org/ ##
## FR: François-René Rideau |    TUNES is a Useful, Not Expedient System     ##
## Reflection&Cybernethics  | Project for a Free Reflective Computing System ##
Every program has at least one bug and can be shortened by at least one
instruction -- from which, by induction, one can deduce that every
program can be reduced to one instruction which doesn't work.