CANDO
Francois-Rene Rideau
fare@Sit-Hyps
Sat, 13 May 1995 11:54:18 +0200 (MET DST)
Dear Tunespeople,
here is some announce I collected from comp.lang.forth.
I invited the author to join us, with the only objectioon that
specification in a formal system was much more secure, accurate and
doable, that is much more useful, than descriptions in english...
------>8------>8------>8------>8------>8------>8------>8------>8------>8------
From: mfiring@primenet.com (Martel Firing)
Newsgroups: comp.lang.forth
Subject: Old Dog's New Tricks
Date: 13 May 1995 21:34:18 GMT
Organization: Primenet Services for the Internet (602)395-1010
NNTP-Posting-Host: usr3.primenet.com
X-Newsreader: TIN [version 1.2 PL2]
CANDO - A Software Architecture for the Next Century
by
Martel Firing
This paper describes my ideas for a software architecture based on
observations, revelations and frustrations over 16 years of programming.
It began as an attempt to analyze why software development was so
difficult, unpredictable, and bug prone. My conclusion was that the
programmer was required to do too many things that the computer should
do, and the computer simply couldn't do the things only the programmer
could do. And in the end, I concluded that programmers need more
freedom, not a more rigid set of rules to create better software. And
the software itself needs to thrive on diversity, change and error
instead of choking up and quitting. So here are my ideas.
CANDO is a software structure designed to work in both large and
small systems, distributed and centralized hardware, and importantly,
able to adapt to changing requirements on-the-fly.
The guiding principal behind CANDO (which means Can Do!) is that
software should be built from competing independent modules advertising
their needs and capabilities. The purpose of the CANDO system is to
simply match needs with capabilities without regard for what these
actually are. The CANDO core does not, itself, generate any needs nor
have any other capabilities.
What's Lacking Now
Before discussing the conceptual details of CANDO, let's examine
some of the characteristics and failings of current software languages
and programming fashion.
1. The most striking observation about software is that it is designed
on a totalitarian style model, the 5 year Gosplan. Everything has to
be thought out in advance and every detail must be correct or the
whole edifice may crumble, and usually does at one time or another.
The massive software manuals for C++, Windows, etc. are proof of
this. In an attempt to enforce correctness, good design practice,
and standards, the "requirements" have gotten completely out of
hand.
2. A second feature of most software is that it is nearly impossible to
determine in advance how difficult it will be to develop and when
(or at what cost) it will be finished.
3. Most software is not designed to be re-usable, and if so designed, I
suspect that little of it is actually re-used in practice.
4. Many software projects become so large and complex that no single
person completely understands the whole thing. Documentation
becomes as large and daunting a task as writing and testing the
code.
5. And finally, all software has bugs. Most of the work in software
development involves testing, identification, and fixing of bugs.
Solutions have been proposed to solve some of these problems. The
most fashionable solution is OOPS, Object Oriented Programming Systems.
The advocates of the OOPS approach argue that their languages: a)
produce re-usable modules, b) allow better program design and team work,
c) reduce bugs, and d) produce more reliable/verifiable results.
OOPS does, undoubtedly, have some of the advantages claimed for it
and appears to be an improvement over prior art. The purpose of this
paper, however, is not to knock OOPS. But rather, to suggest that OOPS,
and languages that preceded it suffer from a singular, fundamental flaw
that will make them fail in the end.
The flaw behind all of the computer languages is that some
programmer must ultimately anticipate every conceivable requirement and
provide a "means" to handle it. And only one means at a time can be
used. Given the inevitable complexity of software, implementing this
"means" requires a great deal of knowledge about the overall system,
even in an OOPS environment. And if the "means" fails, the system can
fail, and usually does.
The flaw, then, is that the programmer has to know too much, has to
adhere to a rigid discipline, and has to produce code that always works.
These real life requirements are only realized consistently in our
dreams.
CANDO: The Free Enterprise Model
The CANDO system is based on what might be described as a
"free-enterprise" model. The core system matches requests for
information or services with advertised capabilities. Its job is simple
so it can be very fast. "Worker" modules post the capabilities ads and
make requests. The features of this model are:
1. A core system whose only purpose is to rapidly match "needs" with
"capabilities." The needs and capabilities of worker modules are
advertised on the core system's "bulletin board."
2. Each worker module posts an advertisement telling what it CAN DO,
i.e., what it supplies. When called upon to do a job, it posts a
"want ad," asking for something another module may be able to
supply.
3. The worker modules don't have to know anything about one-another, or
even if they exist. If a "want ad" goes un-answered, then it is the
programmer's or the user's job to either give it what it needs, or
produce a module that will do that automatically. The system itself
tells the programmer what modules are needed, and what each module
has to supply.
4. Consuming modules can judge whether or not their requests have been
satisfactorily met and will not supply the requested product until
and unless they first get what they need.
5. Modules can compete for business. A scoring system allows
successful, popular modules to remain close to the system and
migrates others to more distant points or eventually even to
oblivion. (Module usage could become a means for programmers to
receive royalties based on the utility of their work.)
The advantages of the CANDO architecture are:
1. It is by design adaptable to changing system requirements.
2. Modules are completely independent of one-another. Any language can
be used to implement any module so long as the basic communication
protocols and encapsulation are observed.
3. Worker modules can reside and simultaneously execute on multiple
processors.
4. The system should be bug tolerant. No system will ever be
completely bug-free.
5. CANDO lets each programmer be an artist, enforcing minimal
constraints on his productivity and creativity, yet gives him access
to all prior work accessible to the system.
There are several requirements for CANDO to work. First, data must
be described and encapsulated in a simple way that allows modules to
describe it, send it and receive it in a predictable manner. I propose
that all data be represented as strings with only the address, length
and type transmitted from one module to another. This needs to be very
simple and highly generalized.
Secondly, a means has to be devised for modules to tell one another
what they do or what they need. This is the most difficult problem.
I've worked on this for some time looking for a "computerish" solution
of formal words and phrases that could be used. My conclusion, at this
point, is that English would work best, because people (even
programmers?) understand it.
The programmer's job is to work out the matches and let the system
turn these matches into tokens that the system can then efficiently
manipulate. The system manages the facts while the programmers manage
the ambiguities.
An overall view of the CANDO system, then is like the input-output
matrix of a free-enterprise economic model. Suppliers are shown in rows
and consumers in columns. Most suppliers are also consumers and vice
versa. At the intersections of rows and columns you have potential
transactions, with the effects of these transactions rippling through
the structure as requests for services produce requirements for services
from others. When the system naturally generates requests for services
or information that don't already have a supplier, then the programmer
can create a new worker, and the process continues.
A Little More Philosophy
CANDO is modularized on the basis of an observation. If a computer
is represented as shown in the diagram, below, all of the capability
flows upward from the most primitive hardware functions. All
information flows downward from the real world toward the primitive
hardware level. This is information needed to perform a task. The
information desired by the user also flows "downward" to the hardware
and out to the CRT or the Printer. See below.
Real World Information: Payroll, Spreadsheet,
Text, Formula, Program.
--------------------------------------
all information - no capability
INFORMATION FLOWS DOWNWARD
+---------------------------+
| A program resides here, |
| feeding capability |
| up and information down. |
+---------------------------+
CAPABILITY FLOWS UP
all capability - no information
--------------------------------------
Hardware Capability: PUSH, POP, MOV, XCHG, IN, OUT etc.
CANDO worker modules ask for information when needed and advertise
capabilities. These modules can be employed at any level right down to
the assembly level. Lower level worker modules acquire information from
higher levels and capabilities from lower levels. However, although we
can talk about levels (of primitiveness), there is no actual layering.
The worker module that moves a cursor on the screen, for example, can
supply this service at the most primitive level, where it is most
efficient. There is no need to burrow down through multiple layers of a
structure to do a primitive function.
The significance of all this is that the programmer is typically
required to keep track of too many of these transactions and
capabilities in the context of an application. The software and
hardware are better suited to this task. What a computer language needs
is more "self-awareness." The language should tell the programmer what
is missing, based on what it has been told to do. The programmer should
not have to guess. The computer can't, by itself, decide what
capabilities are needed to accomplish a task, but it can do a good job
of keeping track of the capabilities that are available and letting the
programmer make that decision. Modern compiled languages do a miserable
job of this if they do it at all.
Language Structure
The CANDO worker modules each have a common structure that is part
of the language. The inputs (NEEDS) of one module are supplied by the
services (CAN DO) of another. A typical module looks like the
following:
WORKER MODULE
+-------------+
| I CAN DO X | This is the advertised capability
|-------------|
| I NEED Y | These generate requests when the
| I NEED Z | worker's capability is requested.
|-------------+
| CODE | Code that actually performs the task.
+-------------+
The worker module will only provide its service after its own
requests have been satisfied. In the example, Y and Z (NEEDs) can be
needs for data or capability. Data and capability are interchangeable
in the sense that a module must call on a (CAN DO) capability of some
other module to get information.
Simplicity is enforced on these modules because the order of
fulfilling a modules NEEDs is indeterminable. So one need can't depend
upon another. Interdependence or order-dependence of NEEDs suggests
that another, simpler module is required.
Data is supplied as a string with an address, and a length. The
user can ask: "WHAT IS z" to get type information about data string "z"
from its supplier. Another question: "WHERE IS z" was anticipated, but
it is probably better for the supplier to be capable of supplying its
data no matter where it is (on disk, on the network, in memory, in the
user's head, etc.), or for some other worker module to do this job.
Additional dialogues in addition to "WHAT IS" will probably be
needed to make it all work, but the philosophy is to keep these to the
absolute minimum required.
The Programming Environment
Programming in CANDO occurs at multiple levels. The highest level
consists of defining a high level task worker that usually makes certain
requests. As the module's requests are executed and no other module
supplies them, the programmer is notified. He may then may search for
descriptions of services that match the request of his module. Various
tools can be envisioned to make this search easier. When a candidate is
found, the system can tokenize the match and the new module can then be
tested. The important difference here is that a) the programmer can
work in English, and b) the system generates requests for missing
capabilities or information.
Lower level programming consists of creating new worker modules
containing execution code. This can be done without any prior knowledge
of the application(s) in which these capabilities may eventually be
used. The programmer supplies English language descriptions of the
worker's capability (its advertisement) and of its needs (requests for
information or capability). He also generates code that uses the
information requested to provide the capability advertised. The
language provides a framework for inter-module dialogues, data
verification, and entry/exit points for code execution.
Note that the execution of the worker module's code is not
scheduled by the CANDO core module, because execution depends on
fulfillment of the worker's own requests. Also, modules may reside on
multiple processors. Thus, execution scheduling is tied to the local
processor's time-slicing mechanism independently from the core module.
In effect, each worker module has its own independent thread of
execution.
Speed & Efficiency
In many respects this CANDO concept is similar to Smalltalk, a
language noted for neither speed nor efficient use of hardware
resources. Undoubtedly some of C++'s popularity comes from the
perception that the speed of C can be combined with the advantages of
OOPS. But it seems to me that both are compromised, and at great expense
to the programmer.
I don't think CANDO will ever be a speed demon, but it need not be
a slow poke either. The core mechanism for matching capabilities and
needs can be tightly coded and fast. The code internal to various
worker modules can also be tightly coded and fast. Most of the
performance penalty in software comes from the innermost levels of the
software that perform highly repetitive tasks, while the outer layers
make infrequent requests for system resources. Thus, a software system
that has tightly coded (fast) inner layers with very high level (slow)
outer layers may not be appreciably slower than a system coded entirely
in a fast language.
In the future, the greatest gains in performance will come not from
tight code, but from the use of multiple processors. This is because
much of the commercially produced software, although coded in C, is
seldom tight or fast. A system designed for multi processor execution
will be faster than one that is not.
Efficiency in the use of hardware resources is inevitably declining
as programs get bigger and do more. Languages that encourage multiple
use of standard modules will produce more compact applications as
applications become larger.
I suspect that CANDO can be implemented as a very compact system
except for the English language module descriptions, which can be kept
on disk. I also believe that the speed of performance for most
applications will be as good as any other comparably high level software
and better than most when multiple processors become the norm.
What's the Real Difference?
Although borrowing ideas from OOPS, Smalltalk, FORTH and lots of
other languages, CANDO departs from the crowd in several important
respects:
1. It introduces the idea that program components should be allowed to
compete with each other to perform a task, with the system selecting
more useful and efficient modules.
2. It permits the programmer to create and use modules based on English
language descriptions. The system, not the programmer, keeps track
of the tokens.
3. The System keeps track of the modules, their interconnections and
their descriptions. The programmer is notified when a requirement
is missing, rather than having to always anticipate the requirement.
4. CANDO worker modules can be created for use in applications not
anticipated by the programmer and can call on capabilities not yet
available to the system.
5. The concept of dedicated application programs essentially
disappears. Instead, the system is endowed with capabilities.
These dynamically configure themselves to solve problems presented
to the system. As the system grows, it can solve more problems.
The programmer becomes involved when a) the system can't solve a
problem because something is missing, or b) the programmer thinks of
a better, faster, or more efficient way to implement a module.
Implementing CANDO
CANDO can be implemented on today's hardware taking the
multi-processing future into consideration. I believe that OS-2's
multi-threaded multi-tasking mechanism is a suitable environment for
this.
It would be possible to define and implement CANDO entirely as a
kind of "communications protocol" similar to Microsoft's object
embedding, linking and other data sharing protocols. The advantage of
this would be speed of implementation -- if acceptance could be somehow
endorsed by a power that be. The disadvantage is that much of the
fundamental benefit of CANDO would never be realized.
The implementation I would favor is a bottom up approach starting
at the assembler level. A CANDO assembler would support higher level
structures. The ultimate implementation of CANDO should be in CANDO
itself. This would completely validate the concept and create a
platform from which the system could grow indefinitely.
Programming with CANDO will be completely interactive. There will
be no edit/compile/link/test cycle.
The actual syntax of the CANDO language will be sparse, with very
few reserved words. The system, in effect, generates and keeps track of
its own reserved words (tokens) invisibly to the programmer. The main
language components will be the core module and the encapsulation
components used to build worker modules, to implement inter-module
dialogues and to validate exchanged data and capabilities. Programming
tools will evolve and become part of the structure.
As an incentive to the growth of the system, I would advocate some
means for contributors to the system to earn royalties for successful
CANDO modules. This could be an integral part of the CANDO structure.
Users could have access to the work of thousands of independent
programmers via networks, with CANDO sorting out which modules are
useful and which are less so for a given set of applications. Competing
modules would survive or die based on the law of the jungle. Producers
of successful modules would be rewarded with growing royalties.
Unsolved Problems
There are many unsolved problems in the CANDO idea. A few spring to
mind.
A) How to let modules compete. How to score and select the best
modules for an application instance.
B) How to make the system bug-tolerant. I believe that this should be
a goal of CANDO. Some means must be devised to determine which of
competing modules is providing the desired result.
C) How should the user initate a process and interact as the solution
is being composed? Worker modules should handle this task, I
believe, not CANDO itself. But the open-endedness of a capability
based system makes this a real challenge for creative spirits. In
any case, it can evolve.
D) How to enable participants to receive compensation for their
efforts.
E) How to make CANDO simple enough that it will accommodate growth and
evolution.
F) How to get started.
Conclusion
As you have probably figured out by now, CANDO is not so much a
language as it is a new way of computing, in which the computer (and its
networks) is a repository of capabilities that are called upon to
perform tasks not necessarily guessed in advance by the programmers. A
program is a (possibly temporary) arrangement whereby these capabilities
are knitted together to perform some useful task. The task or tasks,
however, may change quite dynamically from day to day. As software is
improved, the improvements automatically appear and are used. As
software falls into disuse, it goes away. The overall system, however,
grows ever stronger and more capable.
This kind of dynamic, self-adaptive software is not possible in a
rigid centrally planned model. It requires new thinking: the idea that
a machine can compete with another machine - and survive or die based on
merit. And, as in economics, some chaos is helpful if not always
pretty. The "machines" we're talking about, after all, are the
encapsulated cleverness, skill and ego of programmers. Why not let them
compete?
The Role of Forth
CANDO could be implemented in any language and the design
encourages "worker" modules to be implemented in any language the
programmer prefers.
The reasons that this is posted to the FORTH newsgroup are:
A) I like Forth and use it myself.
B) Forth encourages simplicity, and the implementation of CANDO
requires simplicity.
C) A language-transparent applications architecture would be good for
the future of Forth.
Undoubtedly, some will ask: "Why hasen't Firing implemented CANDO
himself?" The answer is simple. I haven't thought it out completely and
don't see how to make a living on it. But I might do it anyway if
someone else doesn't do it first.
---------------------------------------------------------------------
Martel Firing
mfiring@primenet.com
---------------------------------------------------------------------
------>8------>8------>8------>8------>8------>8------>8------>8------>8------