general comments

[Kyle Hayes]

I think that Mike and Jecel have hit the nail on the head.  Until there
is a clear concensus on what this group will do, all the technical talk
in the world is not going to come to anything.

In order to lay the ground for discussion, I want to bring up some
topics that I think need to be discussed.  These things come from the
recent work I did with distributed OO hard real-time systems and
languages.

What I would like to see in the MOOSE TUNES project goals:

- OO all the way down.  Polymorphism is your friend.  Whether it is done
  in some nifty language like Self or in a kind of kludge like X-Windows
  widgets, I don't really care.  My ego is not on the line about what
  language we use _as_ _long_ _as_ _it_ _does_ _not_ _constrain_ _us_!
  I would like to see garbage collection support deep into the language
  and system.  I have yet to see an argument against garbage collection
  that I can't refute with a few references to papers.  I also want to
  see support for persistence.  I think that persistent object stores
  can replace file systems eventually in a more flexible way.

- mechanism not policy.  Provide mechanism at the low level and let policy
  be decided by higher level entities. E.g. provide a way to switch tasks,
  but allow difference schedulers to be used.  Provide a way to send
  messages, but do not force a single way of using them.  This is the
  microkernel way, but can be applied to any level in the system.

- distribution, but don't go overboard.  Distribution is poorly understood
  at best.  After a couple of months of research I have found that there
  are few or none who can even approach distributed systems in any serious
  analytical way without making such drastic simplifications that their
  models are almost useless.  Distribution and parallelism are often
  highly related.

  Rather than trying to make the perfect distributed system on a first go,
  I would much rather see an approach that allows all kinds of distribution
  models to be tried.  This kind of goal can be met in large part by close
  adherence to the second goal above.  Allow systems to talk to each other,
  but do not enforce a single kind of interaction.  Go with the kinds of
  mechanisms that do not preclude the use of higher level protocols.  For
  instance, a lot can be built on a datagram model of distributed messaging.

  I want to see object migration possible under MOOSE TUNES.  This is a
  very powerful concept and should be looked at closely.  If we play our
  cards correctly with the object model and platform independence stuff,
  I think we can make it so that object migration does not need low level
  support and can be handled on the system or language level.  Provide
  a mechanism for object migration and it can be used by automatic load
  balancing code or directly by users themselves.  Let the higher levels
  handle policy.

- platform independence.  Make a reasonable attempt to avoid things that
  would make MOOSE TUNES incapable of running on most platforms.  If we
  use transputer messaging or x86 segments, we will not be able to easily
  port code to other platforms that don't have these "features".  I think
  that it is worth sacrificing a lot of performance in order to achieve
  a high level of hardware independence.  You can always performance tune
  things later.  First let's design with hardware independence in mind.
  For all that I can't stand Micro$oft, their HAL layer in NT is not a bad
  idea.  I think that the definition of such a layer would be a good thing
  to look at _after_ we have made some decisions about the other goals.

- support standards _where feasible_.  I don't expect to be able to take
  the ELF binaries off my Sparc and run them under TUNES tomorrow.
  However, it would be nice if TUNES machines could at least support 
  socket connections from the outside world, deal with some UNIX-style
  security issues etc.  It would be nice if I could have a TUNES system
  do some talking to a "standard" network.  I am not asking to be able
  to run my binaries, just that there is some support for working with
  such older systems.  If that means that we need to write UNIX/PC/MAC-side
  support routines to allow us to use TUNES from such a platform, that will
  just make TUNES more palatable.

- wheels were invented a long time ago, let's not to reinvent them.  Truly
  new ideas are surprisingly rare.  All the research I have done indicates
  that 99.9% of supposedly new ideas were written up several years ago
  and tested.  I will send a bibliography after this message.  I encourage
  people to read this stuff.  I always learn new things and see new problems.
  Some of the seemingly simple solutions I have seen posted here have
  hidden catches that you might only find after doing exhaustive testing.

  Don't get me wrong.  New ideas are very good and very necessary.  However,
  don't let your egos get on the line about this.  A lot of extremely bright
  men and women have struggled and are still struggling with the kinds of
  things we are talking about.  It is pure hubris and egomania to assume
  that our ideas are either new or perfect.  Maybe some are, but only one
  out of a thousand is truly new _and_ useful.  Often it is very hard to
  determine just how useful until a long time has passed.

  There is absolutely nothing wrong with using old designs.  Sometimes you
  need to call into question basic fundamentals, but make sure that there
  is a reason for it.  There are very sound reasons why airplane wings are
  not swept forward on most planes.  It can be done (X-29 to mention but
  one try), but not without a firm grasp of how to get around the problems.

  Let us try to build on proven ideas _where possible_.  If you start out
  with a solid foundation, you eliminate a lot of uncertainty and gain a
  lot of confidence.  If we have the choice between an idea that is well
  proven and understood and one that is promised to be better (faster,
  smaller, lighter, fewer calories etc.), let's use the proven one and
  test the new one.  There are so many places things can go wrong in a
  system with the kind of complexity we are talking about that it makes
  sense to go with an incremental approach.

  I would love to see a test platform that runs on top of UNIX or the Mac
  or Windows or some other proven system before people start trying to
  make pure MOOSE TUNES partitions on their harddisks.   Such a platform
  can also be a good aid in helping us define a HAL layer that can then
  be ported to all the hardware we could want.

  It is not a sin to build incrementally.  It is a sin to think that you
  can build it all at once and have it work.  I realize that a lot of
  people like to code.  Be aware that well controlled software projects
  spend less than 20% of the total project person-hours coding and that
  a good portion of that time is for testing.  The well run projects I
  have seen (few and far between) seem to have more like 10% of the time
  spent coding.  Such projects take a very small amount of time to reach
  their goals compared to ones that start hacking from the first day.  They
  are more adaptable to change.  They are easier to maintain. Etc. etc.

  If you have a idea for something technical, but it is about something
  we have not started discussing as a group, just jot it down somewhere
  and see if you can find other work that is similar or that will give
  you some data.  It would be a lot of help to the group if people would
  take the time to think through their ideas carefully and do this
  checking.  We can move much faster and we will certainly all learn a
  lot more.  If you find other work supporting your idea, it is much
  more likely that people will agree with it.  If you find that the
  evidence is against your idea, you can post that too and use it to
  argue against the idea if someone else proposes it.

  I can go on with the sermon for hours, but if you don't believe it from
  the beginning and work with some real guidelines like these in mind you
  will not be using your abilities to their full nor participating as much
  as you could in this project.  Everyone has a contribution to make and
  a new way of seeing things.  The best organizations are those in
  which people are encouraged to check their own ideas and look for new
  ones in others' work.  There are many thousands of theses published
  each year, surely there are few useful ideas in them.  This doesn't
  count the books and journal articles.  Who knows what is published in
  corporate journals...

You may see this sermon as being an example of such egotism on my part. 
I have had this stuff work for me and I know of places where it works.
I have read about it working in yet more places ("Managing the Software
Process" is a good place to start reading).  I have never heard about it
failing completely*.  I have had projects fail deadlines and even fail
completely due to disorganized approaches.  I have heard and read about other
projects with similar problems.  I think this gives me some justification
in making the points I made above.  It is not that such organization is
guaranteed to make projects work smoothly every time, it is that with
such ideas, project _tend_ to go more smoothly.  Without such ideas,
few projects do well.

* That is to say that I have never heard of projects that really used
  these things and failed due to that use.  Lots of projects use the
  most wonderful origanizational ideas and then fail for reasons entirely
  outside the scope of the project like having their funding taken away.

In short, what I would like to see for MOOSE TUNES are the following
goals:

- OO model with GC and persistence built in at a very low level,

- datagram-style message passing at a very low level,

- separate mechanism and policy from the first instant and at every level,

- distribution supported at a fairly high level (language or system),

- platform independence; I want to run heterogeneous MOOSE TUNES networks,

- work with standards where feasible,

- research, research, research; no new wheels please.

I think Mike has done a pretty good job of hammering a sort of initial
organizational charter into place.  I tend to have a bit of worry (like
some others apparently) that there are a lot of officers and few soldiers
in this army.  I think that the General Coordinator (was Secretary) is
a good position to keep, but that other coordinators should have several
projects each.  Each project should be internally self organizing.  The
coordinator for a project just looks over their progress from time to 
time and makes sure that they are going in more or less the right direction.
Each project might have a project leader to manage the day to day activities
but _only_ if there are enough people in the project to warrant such a 
position.  It would not be a very good use of our time to have a project
leader for project with less than, say, three people.  So, initially,
a General Coordinator and maybe one other project coordinator to handle
all the initial projects.  No more than that many permanent positions
until we have more than five or six simultaneous projects.

The point was made that all this voting on each and every issue will 
take all our time.  I tend to agree with this to an extent.  However,
if the General Coordinator is given the power to make decisions, then
the other people need to have the power to override that decision.
This may be where the 2/3 majority rule could be used.  Most decisions
need not be voted on and will be acceptable to most people.

Does anyone have any references to work on organizing such things?
Robert's Rules of Order?  Non-Profit Business Organization? (I am
stabbing in the dark here.)  After all, I have to apply the above ideas
to all aspects of our organization :-)

names:

I think we want to try to get some of the following "gee whiz" terms
in the name:

Network, Parallel, Object, Distribution, Heterogenous... Any more?

Best,
Kyle