The future of Slate's implementation

[Brian Rice]

Hello all,

I need to outline some things I've been thinking about with respect  
to how Slate is implemented, centered around the way that Pidgin  
works, and ways to migrate forward into something more flexible (like  
eventually using our optimizer framework). VM and FFI architectures  
also come into play.

In the immediate timeframe, main and alpha have new source code for  
the translator, which are cleaned up now and commented for the most  
part. The primary files to understand what Pidgin is capable of are  
here:

http://slate.tunes.org/repos/main/src/mobius/c/generator.slate
http://slate.tunes.org/repos/main/src/mobius/c/cr.slate

There are some TODO's in there, mostly cosmetic. The big changes  
won't fit into comments. Pidgin shares c/types.slate with the FFI,  
and it's basically an object-oriented description of C's type system.

In any case, the eventual goal of the compiler was to provide a  
"primitive mode" for the full Slate language. Lee was intent on this  
but didn't (see note below) seem to make any progress beyond  
requirements for it. See:

http://slate.tunes.org/repos/main/doc/mode_notes.txt

for his thoughts in the area. To summarize, Slate could be  
recursively embedded with primitive modes using a simple macro that  
annotated source trees. The mode of evaluation for the primitive mode  
sections would have been to use a translation system /akin/ to Pidgin  
but targeting the IR tree form (and some memory model starting with  
the optimizer types plus the MR perhaps) instead of C Syntax and  
types/memory semantics. This would be qualitatively different from  
the compiler front-end, which does exist and mostly works.

I think we can finish this by looking at Pidgin and improving it to  
the point that it can be abstracted from specific C issues, and then  
implement a similar translator for (a sub/super-set of) the same  
resulting dialect to the IR. There are a number of things necessary  
to accomplish this; I'll outline what I've thought of already below.

(1) We'd need to raise the level of abstraction of the type system so  
that the IR types were used and then translated to C types for the C  
translator. This mostly consists of abstracting over size of various  
types and mapping the ones for specific types to short/long/etc. If  
you read:

http://slate.tunes.org/repos/main/src/mobius/optimizer/types.slate

there are still a lot of 32/64-bit hardcoded assumptions that just  
make things a bit messy. I think parametrized/dependent types are the  
kind of direction it should move in, mostly to generate most of this  
code.

(2) Adjust Pidgin so that it is not so quirky. For example, code  
outside of method definition bodies is written in "actual" Pidgin,  
whereas code outside of it is processed in a mix of Slate-in-a-funny- 
namespace but minus some things you'd normally get in Slate. For  
example, -> and True and False don't mean the same thing there. So I  
need to sort out how that all works, even if it means making more  
verbose code. Ultimately, the idea is that we can compartmentalize  
code and semantics and descriptions into places where we just wrap it  
in a macro. For example, we do this with `pidginPrimitive in the  
libraries in src/mobius/vm/ext/ which define Slate methods which can  
be called from the image-side but are defined in the VM as wrapping C  
routines or other VM internal routines.

I've also thought about the merits of embedding more C code into the  
Pidgin sources, but this maybe makes re-compilation heavier and is  
probably best not delved into until we are free of source files per se.

(3) I'd like there to be (but there doesn't need to be) use of better  
control-flow idioms at the low-level for code that can handle it. For  
example, in Pidgin, we can translate the following into C syntax:
ifTrue: ifFalse: ifTrue:ifFalse: whileTrue: whileTrue whileFalse:  
whileFalse
loop
upTo:do: downTo:do: below:by:do: below:do: above:do: to:by:do:

This is because code has been explicitly written for the C  
SimpleGenerator to translate them. However, some of these methods are  
defined in src/lib/method.slate:

http://slate.tunes.org/repos/main/src/lib/method.slate

as merely calling the more "basic" methods with default arguments. So  
it occurs to me that there are a lot of opportunities to use higher- 
level methods without extra coding, by treating specific method  
definitions as translations. I say "specific" because I don't want to  
have to use an inference engine for this at first; I'll probably just  
gather up idioms I'd like to use and run one analysis on them to get  
the translation out of them. Eventually it might be better, and I  
have ideas about that, but they're not worth detailing until the  
basics work. Sufficed to say, that it'd be nice to have Array's #do:  
and relevant kin be available - anything that translates well enough  
to low level code with trivial re-interpretation.

As a note, src/mobius/optimizer/ir/generator.slate has similar but  
slightly outdated translations.

(4) The FFI needs to share more code with this combined C-and-IR  
system than it does now, which is just src/mobius/c/types.slate.  
Being able to generate plugins and FFI binding specs from the same  
source Pidgin code seems like it would be more usable and powerful  
than what we have now. The bootstrapper and so forth are pretty  
monolithic right now and only do one job, so it'll take time to  
figure out a more generic approach or at least how to re-use the  
Pidgin generator for a different task.

As a side-effect, it occurs to me that an FFI for the VM itself as  
well as some extension for bytecode-level stepping of some sort would  
be a productivity boost when talking about debugging and diagnostics.  
Basically we wouldn't need GDB for a lot of purposes in testing new  
changes and so forth. At first, I'd probably be the only user, but  
it'd be a lot more productive and maybe be useful to everybody once  
it had a decent setup.

A more important change might be that I'd translate the relatively  
simpler FFI plugins like the Socket and Posix and Time stuff into  
Pidgin and try those out for basic tests. It's certainly important  
for them to work well (and have good performance), and I think we can  
do this without compromising our goals if the above plans work.

A very simple FFI change would also be to adapt the use of CObject,  
which takes C structure/type definitions and makes faux Slate objects  
that get/set binary representations of them in Slate ByteArrays, to  
the new malloc()-memory VM module. This might have implications that  
I don't really foresee too well, but obviously we'd have to consider  
security even while the benefits of easily seeing at a Slate level  
what's going on "below" could be profound.

(5) As part of the longer-term goal of moving to an optimizer-based  
platform, I will try to move parts of the VM into malloc()-managed  
memory via our new VM module or something like that. This will remove  
compile-time-specified limits on resource usage and might make the  
migration less onerous on the VM design itself, via the CObject  
extension I mention above.

==========

This turned out to be really involved to explain, and rightly so.  
I'll record these thoughts on the wiki pretty soon and probably will  
work on these goals over the next few months. Our next release is a  
bigger priority, since we have a huge number of changes already since  
the last one, and progress needs to be evident on the website.

Anyway, have a Merry Christmas, or whatever you like to celebrate! :)

NOTE: I use the past tense about Lee because he seems to have lost  
the will to continue on this project, for his own reasons. I don't  
know how I could have affected this, but I did try to keep him  
around. It seems he qualitatively did not want to work on a project  
with a lot of collaboration and goals oriented towards a UI and non- 
OS-focussed systems code. I would like a Slate OS, but I don't  
consider it a primary focus. Or rather, I don't think it worth doing  
until the system is worth hosting on its own. One point of Slate is  
to flexibly accommodate such design decisions without leaving either  
side at a disadvantage, but for the core engineering progress, it  
seems that this didn't work out with him on a basic level. All I know  
is that he really seemed to hate C and the Unix paradigm, which I  
cannot fault him for, but wouldn't work towards separating us from it  
in increments. This made progress and consensus difficult. You'll  
have to ask him to know more, since I really don't understand his  
thought process; obviously I couldn't get him to communicate his  
concerns here, either.

--
-Brian
http://tunes.org/~water/brice.vcf

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : /archives/slate/attachments/20051222/68a14f40/PGP.pgp