From jiml@inconnect.com Wed, 22 Dec 1999 01:55:08 -0700 Date: Wed, 22 Dec 1999 01:55:08 -0700 From: Jim Little jiml@inconnect.com Subject: Prism Manifesto draft available Hi folks, It's been a while since I worked on Prism -- the holiday season and Ultima: Ascension have both taken their toll. :) But today I sat down and got some good stuff done... still pretty high level, but important nonetheless. I've published a draft of the "Prism Manifesto," which is a statement of the ideas and goals behind Prism. This is one step down from "The Will and the Word" (which has a new draft revision out as well) -- Will&Word discussed what it means to program; the Prism Manifesto takes those ideas and says, "Here's what's wrong with programming, and here's how the ideas from Will&Word can fix it." If you're at all interested in Prism (and I assume you are, or you wouldn't be on this list), take a look at this and let me know what you think. In particular, do the goals I have for each phase seem reasonable and worthwhile to you? Am I missing anything that fits in with the rest of the project? I'm completely open to change, so let me know. The Prism Manifesto may be found at: http://www.teleport.com/~sphere/documents/001c/draft/index.html The updated Will & Word essay may be found at: http://www.teleport.com/~sphere/documents/0006/draft/index.html Jim From jiml@inconnect.com Wed, 22 Dec 1999 01:58:49 -0700 Date: Wed, 22 Dec 1999 01:58:49 -0700 From: Jim Little jiml@inconnect.com Subject: URL bug: Prism Manifesto The URL for the Prism Manifesto is actually: http://www.teleport.com/~sphere/documents/001d/draft/index.html Sorry! Jim From jiml@inconnect.com Thu, 23 Dec 1999 14:19:48 -0700 Date: Thu, 23 Dec 1999 14:19:48 -0700 From: Jim Little jiml@inconnect.com Subject: Prism Meta-metamodel draft available Folks, I've made a draft of the Prism meta-metamodel available. This document describes the Prism metacode, which pretty much controls what Prism applications can do. The metacode itself hasn't changed any, but it's described more clearly, and it's no longer part of the Prism tool specification. The URL is: http://www.teleport.com/~sphere/documents/001c/draft/index.html Here's the first few paragraphs to whet your appetite: === Metacodes constrain the creation of metamodels, which in turn constrain the creation of codes, models, and even programs. Before a programmer can create a program, he must choose a code with which to model it. The choice of code can significantly affect how he creates his program. Similarly, before he can create a code, he must choose a metacode with which to metamodel the code. The metacode similarly impacts the creation of his code, and that code in turn impacts how he creates his program. Therefore, the choice of metacode should be a very careful one -- even more careful than the choice of code when creating a program. For most metamodels, though, the choice of metacode has been an ad-hoc one, driven by tradition and individual need. Most metamodels don't use a formal metacode. Instead, they use natural language and some form of grammar description notation, such as BNF. The resulting codes are usable, but the emphasis on BNF means that the semantic relationships of the code are lost in the noise of syntactic relationships. In addition, although all of the metamodel's metacodes are similar, there is no formal metacode tying them together. Therefore, it's very difficult for any one metaprogram to operate on more than one encoding. The Prism metacode is a formal metacode. It emphasizes simplicity, semantics over syntax, and ease of code definition. In line with Prism's goals, it also supports code interoperability by providing mechanisms for unambiguously determining a model's code and for models to nest arbitrary other models within them. === As always, comments and criticism are welcome. Jim From brandon@badmoon.com Wed, 22 Dec 1999 04:14:38 -0600 (CST) Date: Wed, 22 Dec 1999 04:14:38 -0600 (CST) From: Brandon brandon@badmoon.com Subject: Prism Manifesto draft available First let me say right on. At the most abstract level, we have exactly the same goals. Of course, we differ in the implementation. Here are my thoughts. By having to specify everything in terms of the atoms bit and list, you're basically requiring people to write a compiler for every language they want to use with Prism. Fortunately, you're "machine language" (Prism/Minimal) is entirely abstract. However, that's quite a pain and, I think, unnecessary. I think you should allow for arbitrary user-defined atoms. Let's say I have a mere two languages which I want to interoperate. And they're quite normal and in fact similar. They both have a while loop construct. I should be able to say hey, there's this atom called a while loop. Then write something which translates source code from both languages for while loops into this one construct. Whoosh, while loops are interoperable. Of course, I may come across a language which doesn't work quite the same. Perhap my while loop construct is almost right, but not quite. I should be able to define the while loop construct in terms of some new atoms, demote it from atom to construct, and everything still interoperable without changing my old code. Hm. Pretty sleepy. Let me know if this is at least coherent. But in general, keep up the good work. It's refreshing that someone's trying to do something fundamentally good for the world. From jiml@inconnect.com Wed, 29 Dec 1999 01:15:27 -700 Date: Wed, 29 Dec 1999 01:15:27 -700 From: James Little jiml@inconnect.com Subject: Prism Manifesto draft available Sorry for the late reply. I'm away from home for the holidays, and only recently got my email set up. > Here are my thoughts. By having to specify everything in terms > of the atoms bit and list, you're basically requiring people to > write a compiler for every language they want to use with Prism. > Fortunately, you're "machine language" (Prism/Minimal) is entirely > abstract. However, that's quite a pain and, I think, unnecessary. Just to make sure I understand, you mean a compiler that converts the textual model ("void main() {...}") into a Prism model ("_List{_Bit{FOO}...}")? Also known as a parser? If so, then that's certainly true. But how can it be avoided? > I think you should allow for arbitrary user-defined atoms. Let's > say I have a mere two languages which I want to interoperate. And > they're quite normal and in fact similar. They both have a while > loop construct. I should be able to say hey, there's this atom > called a while loop. Then write something which translates > source code from both languages for while loops into this one > construct. The Prism metacode already has user-defined atoms, although you have to turn your head and squint. ;) Atoms provide structure only. Meaning, such as, "this is a loop," is provided by codes. So rather than saying, "hey, there's this atom called a while loop," you say, "hey, there's this atom, and you interpret it with the 'Loop' code." Naturally, you'd have to create the "Loop" code and describe it with a metamodel, but users ARE allowed to do that. Once you create a Prism metamodel, you can reuse it in other metamodels, just as if it were an atom. (Technically, this is a feature of the Prism/GTL language, not the Prism metacode, but the metacode makes the language feature possible.) For an example of this sort of "user-defined atom," see the Text Document metamodel (http://www.teleport.com/~sphere/documents/0010/2/index.html). You'll see that it references the String metamodel just as if it were an atom, not a complete metamodel. > Hm. Pretty sleepy. Let me know if this is at least coherent. I think so. :) Let me know if I misunderstood you. > But in general, keep up the good work. It's refreshing that > someone's trying to do something fundamentally good for the world. This made my day. Thanks. Jim From jiml@inconnect.com Wed, 29 Dec 1999 22:21:12 -700 Date: Wed, 29 Dec 1999 22:21:12 -700 From: James Little jiml@inconnect.com Subject: Prism/CompactModel draft available I've made a draft of Prism/CompactModel available. Prism/CompactModel is a generic language for describing any Prism model, regardless of code. It's also -- as the name implies -- fairly compact. This is the language metaprograms will use to communicate models to each other. As such, it's a crucial part of the Prism Metaprogram Protocol, which I'll be working on next. Take a look and let me know what you think. I've included the language grammar below. You can find the full spec at: http://www.teleport.com/~sphere/documents/0017/draft/index.html Jim Here's the language grammar, in EBNF. The above-linked document has more information and an example. START := VERSION CODE_GUID MODEL VERSION := 'Prism/CompactModel v1.0.0a1:1' END_LINE END_LINE := ASCII_13 CODE_GUID := MODEL MODEL := ATOM ATOM := BIT | LIST BIT := 'B' ; "BAR" | 'F' ; "FOO" LIST := 'L' (ATOM)* END_VARIABLE END_VARIABLE := '.' From jiml@inconnect.com Thu, 30 Dec 1999 01:05:29 -700 Date: Thu, 30 Dec 1999 01:05:29 -700 From: James Little jiml@inconnect.com Subject: Prism Metaprogram Protocol draft available (I think my first announcement got eaten by my remote-mail program. Oops.) I've made a draft copy of the Prism Metaprogram Protocol available. This protocol allows metaprograms to interoperate, and is the primary deliverable for phase 1 of Prism. As such, it's kind of important. :) Take a look at it and let me know what you think -- I'm particularly interested in suggestions for metaprogram naming conventions. I don't like the ones I put down. The URL is: http://www.teleport.com/~sphere/documents/001e/draft/index.html Here's an excerpt: === The advantage of this approach is that metaprograms can be very small and focused, allowing difficult tasks, such as optomization, to be undertaken more easily. It also opens up the entire compilation pipeline (literally!) to competition. If someone comes up with a better x86 optomizer algorithm, she can create and release just the optomizer, rather than having to create a complete compiler around it, or trying to get her code integrated into an existing compiler. This approach also allows more metaprograms to be reused. Someone creating a Java compiler can create her own Java-to-x86 translator, but reuse the x86 optomizer, Linux executable wrapper, and machine code emitter. Similarly, someone creating a C source code analyzer can reuse the existing C parser. The above approach also gives the programmer more flexibility. Optomizers and debuggers can be swapped in and out of the pipeline at will. The semantic analyzer (linter) can be swapped for a more robust one, or taken out entirely. A trusted C optomizer can be kept, but the target machine completely changed. === Jim From idiot@slack.net Thu, 30 Dec 1999 12:50:58 -0500 Date: Thu, 30 Dec 1999 12:50:58 -0500 From: Matt Miller idiot@slack.net Subject: Prism Metaprogram Protocol draft available I promised some work on this by the end of this year or early next.. I'm about as close as you can get to either of those right now.. :) I'm going to scratch pad my notes below here which i may or maynot clean up before I sed to you: - There's a broken link in http://www.teleport.com/~sphere/documents/001c/draft/index.html to http://www.teleport.com/~sphere/documents/001d/1/index.html - In the same document: while I like the idea of abstracted values for bits of "foo" and "bar", does this abstraction make 'common sense' reasoning about what the semantic value of a bit is more difficult? Most programmers are used to reasoning out what the appropriate meanings of 0 and 1 are. Yes, they sometimes get them wrong, hence the value of the abstraction... Can I create 'header' files which define the meanings? For example: definition type values bootstrap with: _Bit bit (foo, bar) sign _Bit (+, -) Am I completely missing the point here? :) - We go thru a bit of work to get all of these GUIDs assigned, but then for our program names we use some 'commonly understood' meaning of what we are doing. What if I take 'c' to mean ANSI-standard 'c' and you take it to mean K&R 'c'? Would it be more profitable to have a lookup table which makes it easy for me to use GUIDin-GUIDout for my program names? This would allow the system to 'verify' that what you are emitting is what is expected by the next link in the chain. Am I misunderstanding the meaning of GUIDs? This is all i have for the moment. I am going to try to build some simple models and get a better grasp on the mechanics. From idiot@slack.net Thu, 30 Dec 1999 22:54:23 -0500 Date: Thu, 30 Dec 1999 22:54:23 -0500 From: Matt Miller idiot@slack.net Subject: Prism Metaprogram Protocol draft available On Thu, Dec 30, 1999 at 06:50:44PM +0000, James Little wrote: > > Nope, not at all. As for "headers," that's a good idea. None of the current generic Prism languages allow it, but there's no reason that couldn't be changed. However, I'm not sure how often generic languages will be used to describe models; I think it's more likely that code-specific languages (that don't mention _Bits or _Lists) will be used instead. Ok, I think it's time that I get a little more info on the ultimate project goal here. Do you just want this really cool way to write a C/Fortran/COBOL compiler? Or is the intent to create a full system with Prism as its foundation? I think you're already well on your way to the first bit. But from what I am hearing above it sounds like thats where the intent stops. Maybe that's just because thinking beyond that is just way too much at this point? At this point the project sounds a bit like a grander version of what Borland did back in the 80s... write compilers that emit a virtual machine code and then use the same backend to turn that into machine code. You've added some modularity and opened up the process, which is always better! > Nope, you're on target here, too. If you've got a better idea for the program names, I'm definitely ready to change the naming conventions. My feeling, though, is that a name like "c_x86" is a lot more understandable than "10000.13_10000.42", even though the names don't necessarily uniquely describe the codes. > Right-o. I agree that it is more understandable. What I was thinking is more along the lines of a 'registry' (shudder) inthe system which maps 10000.13 to 'c'... as I install a program with an unknown GUID, it adds itself into the table with some default value.. but I can reassign those values as I wish. As i think about it, this make a huge support nightmare.. if I call 10000.13 'c' and your prefer to call it 'baz'.. we'd end up with different program names.. So, no, I don't have a better naming convention.. :) Also, where do I store the version info? > However, the program names are a human convenience only; in the Prism/CompactModel language that the protocol uses, code GUID's are attached to models. Presumably metaprograms will check the code GUID of an input model and fail if its not correct. (Maybe I should make that a requirement.) Also, for humans, it's formally recommended in the protocol that the help screen for a metaprogram spell out exactly what code GUID's the metaprogram operates on. > > Do you think that's enough, or should we go a step further? I think making it a requirement that we do some kind of sanity checking on our input is a good idea.. maybe a simple output of 'GUID is, GUID expected was.' On the assignment of GUIDs.. I need some more info on this. I want to create a C syntax checker. One already exists.. We'll share a group, but have a different individual (sorry not near the web don't know if I got the right term) number, correct? Once the authority has assigned a group id, do individuals creating members of that group come back to the assigning authority for their numbers? Won't that get cumbersome? Could we make a 'true' individual/organizational id? So all MiniHard software has a trailing .666 or some such... My guid is a combination of the type of software and the author... is this even less workable? Sorry I'm still dwelling on some peripherial issues, I know. More later! Matt From jiml@inconnect.com Thu, 30 Dec 1999 23:56:40 -700 Date: Thu, 30 Dec 1999 23:56:40 -700 From: James Little jiml@inconnect.com Subject: Prism Metaprogram Protocol draft available [Matt wrote:] > On Thu, Dec 30, 1999 at 06:50:44PM +0000, James Little wrote: > > > > However, I'm not sure how often generic languages will be used > > to describe models; I think it's more likely that > > code-specific languages (that don't mention _Bits or _Lists) > > will be used instead. > > Ok, I think it's time that I get a little more info on the > ultimate project goal here. Do you just want this really cool > way to write a C/Fortran/COBOL compiler? Or is the intent to > create a full system with Prism as its foundation? That's a harder question to answer than you might think. :) Some background might help: Originally, my intent was to create a full system with Prism as its foundation. The result of that was the Prism Metaprogramming Tool (http://www.teleport.com/~sphere/documents/0009/3/index.html). The Prism tool was to be a complete programming environment built around the Prism metacode. (Although the metacode was called the Prism *meta-metamodel* then; that was before I recognized the importance of distinguishing metamodels and codes.) I actually have a complete, if bare-bones, working version of the Prism tool; you can download it at the above link. The Prism tool was a scriptable environment; it could interpret the Prism Control code (http://www.teleport.com/~sphere/documents/0012/3/index.html). Prism Control was a bare-bones code for manipulating Prism models. It understood the concepts of _Bits and _Lists and could move them around, branch on their contents, etc. However, like the Prism metacode, Prism Control was very abstract and minimalistic. It had built-in instructions for atom manipulation and very little else. It had a pretty good extension mechanism that would allow you to build your own "instructions," so theoretically you could add anything you wanted. One of the things Prism Control lacked was any understanding of integers. You may recall that, when I started this list, I was talking about building programs to do arithmetic on integers. Those programs were going to be used to "extend" Prism Control so that it would support integers. The idea was that Prism Control -- built by me, and relatively unchangable -- would provide a bare kernel that would be built up by users, and thus be completely changable. As I dived into this, creating integer arithmetic programs, I realized that it would take me *FOREVER* to get anything accomplished this way. The Prism Control code was a pain in the neck to use, and the Prism tool had very rudimentary debugging tools. I compared the amount of time it took me to build the existing Prism/Control parser, written in Perl, (about a day) to the amount of time it would take me to write a bare-bones version of the same in Prism Control, (several months at best) and decided it wasn't worth it. ("Prism/Control," the *text-based* language, was a code-specific language for "Prism Control," the *Prism* code. You can see it at http://www.teleport.com/~sphere/documents/0013/2/index.html. Yes, the choice of names is confusing and bad.) The result of this realization was a re-evaluation of Prism's strengths and weaknesses, which I posted to this list (http://www.tunes.org/list/prism/9911/msg00008.html). That was when I decided to create the Prism Metaprogram Protocol. However, the protocol is just the first step, as I describe in the Prism Manifesto (http://www.teleport.com/~sphere/documents/001d/draft/index.html). I still want the interoperable languages that the Prism tool was going to provide. But at this point, I'm not sure that a closed, my-way-or-the-highway (though open source) platform is the way to go. I like the open, toolkit-like approach used by the protocol better. I guess the short answer to your question is, "I don't know." The Prism protocol is exactly what you describe: a really cool way to write a compiler. But it's just a stepping stone to my ultimate goal: interoperable, interactive, and visual domain-specific languages. However, I don't know how I'll be accomplishing that goal. At one point, I *was* going to create a "full system with Prism as its foundation." Now, the Prism metacode still lies at the foundation of everything, but I'm not so sure that the "extensible monolith" approach is the best one. Working on the Prism protocol will help me answer these questions. In addition to being a cool way to write a compiler, its deeper purpose is to be a "shakedown cruise" for the Prism metacode, and a way to get some useful metamodels written and tested. That experience will give me a better understanding of how Prism's next goal, interoperable codes, will be achieved. Okay, you probably got a lot more than you bargained for in that answer. :) I hope it was helpful. > I think you're already well on your way to the first bit. But > from what I am hearing above it sounds like thats where the intent > stops. Maybe that's just because thinking beyond that is just > way too much at this point? Well, I have goals beyond this point, but not implementation plans. Also, my work will tend to focus on the immediate goal -- not because I'm not thinking of the future, but because I want to keep things simple. You can see Prism's goals in the Prism manifesto (http://www.teleport.com/~sphere/documents/001d/draft/index.html), but here's the list: 1) Create a way for metaprograms to interoperate (well on its way) 2) Create a way for codes to interoperate 3) Make codes interactive and visual, and highly domain specific. The manifesto describes the reasoning behind these goals in detail. [Back to the specifics of the Prism protocol] > Also, where do I store the version info? I'm not sure if we're on the same page with regards to metaprograms and GUID's... I'll address that a bit more below. The straight answer, though, is that every time a code changes, it gets a new GUID. There's no prescribed way for handling versioning. > I think making it a requirement that we do some kind of sanity > checking on our input is a good idea.. maybe a simple output of > 'GUID is, GUID expected was.' Yes, I agree. I'll make that change when I "officially" publish the protocol. > On the assignment of GUIDs.. > > I need some more info on this. I want to create a C syntax > checker. One already exists.. We'll share a group, but have a > different individual (sorry not near the web don't know if I got > the right term) number, correct? Once the authority has assigned > a group id, do individuals creating members of that group come > back to the assigning authority for their numbers? Won't that get > cumbersome? Well, first let me correct a misconception: at this time, only *codes* have GUID's. Metaprograms don't. So your C syntax checker wouldn't have a GUID. The C *code* would, but not the metaprogram. As for GUID allocation, I'm using the Sphere GUID infrastructure (http://www.teleport.com/~sphere/002c/2/index.html), which provides several ways to allocate GUID's, each with their own trade-off between convenience and reliability. Read the above document for all the details, but it boils down to this: A GUID consists of a "group ID" and an "individual ID." (I should probably come up with better names for those.) To create a GUID, you first select one of the group ID's available for the GUID application you're interested in. In this case, the GUID application is Prism GUID's, and the available group ID's are listed as an appendix to the Prism meta-metamodel (http://www.teleport.com/~sphere/documents/001c/draft/guid.html) The way you get your individual ID depends on the group ID that you select. Each group ID has its own allocation strategy. The strategies are: Anonymous: Individual ID's are allocated as needed, and can change over time. Useful in some situations, but not really useful for code GUID's. Personal-private: Individual ID's are allocated by the user, but then the thing (code, in this case) can't be published, because the GUID could conflict. Good for personal experimentation and development. Organizational-private: Same as personal-private, except the ID's are allocated by the user's organization. Ad-hoc: Pick any individual ID you want and hope it doesn't conflict. (This is the technique used for most GUID's. Take Perl package names, for example.) Algorithmic: Follow some pre-defined algorithm to create your individual ID. If it's a good algorithm, the ID won't conflict. Managed: Ask an allocation authority to give you an individual ID. Slow and bureaucratic, but safest of all the public options. So, you see, the "group ID" doesn't represent a group of related codes. Instead, it represents a particular way of allocating ID's. (The name "group ID" definitely needs to be changed. "Strategy ID" or something similar is more accurate.) > Could we make a 'true' individual/organizational id? So all > MiniHard software has a trailing .666 or some such... My guid is > a combination of the type of software and the author... is this > even less workable? If a particular organization was extremely prolific and needed their own "managed" group ID, that could be arranged. For example, Prism has it's own "managed" group ID (0) that is for Prism use only. But for most people, that's not likely to be necessary. Right now, I'm the "allocation authority" for Prism's public "managed" group ID (10000). Right now, I'm handling them manually. I figure that I'll set up a GUID server on the web somewhere when requests start taking up significant amounts of my time. At this point, I've never received a single GUID request, other than from myself, so I think I can handle it. :) If the workload became more than I could handle, then I'd pass the job on to someone else. That's not very likely at this point, though. > Sorry I'm still dwelling on some peripherial issues, I know. > More later! No problem at all. These infrastructure issues are important, and they often get neglected. I've done my best to address them, but it's good to have another point of view. Jim From jiml@inconnect.com Fri, 31 Dec 1999 01:48:03 -700 Date: Fri, 31 Dec 1999 01:48:03 -700 From: James Little jiml@inconnect.com Subject: Prism Metaprogram Protocol draft available [Laurent Martelli wrote:] > So if want to call a cat a `cat', and I want to call it a `dog', > it's allright because when we communicate, the systems won't use > our bindings, it will its internal OID, `123' for instance. So if > send you `dog', I'll write `dog', but my system will send your > system `123', and it will display `cat' to you. So we'll understant > each other even if we use different names to refer to the same > thing. Yep, that's a good system. We can't use it for metaprogram names, though, because those are just file names that don't change on the fly. I suppose the user can change them if he wants, but I doubt he will. :) However, the same principle applies: the code GUID is the "official" identifier for codes, and then people can come up with human-friendly names that map to GUID's however they wish. But in this case, there won't be any global "bindings" or dictionary to map names to GUID's, since the names are file names. Jim From martelli@iie.cnam.fr 31 Dec 1999 11:54:29 +0100 Date: 31 Dec 1999 11:54:29 +0100 From: Laurent Martelli martelli@iie.cnam.fr Subject: Prism Metaprogram Protocol draft available >>>>> "Jim" == James Little writes: Jim> [Laurent Martelli wrote:] >> So if want to call a cat a `cat', and I want to call it a `dog', >> it's allright because when we communicate, the systems won't use >> our bindings, it will its internal OID, `123' for instance. So if >> send you `dog', I'll write `dog', but my system will send your >> system `123', and it will display `cat' to you. So we'll >> understant each other even if we use different names to refer to >> the same thing. Jim> Yep, that's a good system. We can't use it for metaprogram Jim> names, though, because those are just file names that don't Jim> change on the fly. I suppose the user can change them if he Jim> wants, but I doubt he will. :) Of course, I never had any intention to call a `cat' a `dog'. But I might want to call it a `chat', in french. Most people would still use the same names for the same things, but not being bound to it is important. Jim> However, the same principle applies: the code GUID is the Jim> "official" identifier for codes, and then people can come up Jim> with human-friendly names that map to GUID's however they wish. Jim> But in this case, there won't be any global "bindings" or Jim> dictionary to map names to GUID's, since the names are file Jim> names. But in this case, couldn't the filename be the GUID ? -- Laurent Martelli martelli@iie.cnam.fr