POS

[BRIAN SPILSBURY]

>  > BRIAN SPILSBURY wrote:
>  > > The only point at which I might need to take persistence into account
>  > > is where I'm creating an object whose persistent characteristics are
>  > > not the default. And that is not terribly common, and certainly should
>  > > not be visible to the user.
>  > 
>  > Most of the apps I write have different persistence characteristics.
>  > They do a lot of work with data that should not be saved to disk and
>  > then return a little bit of data that SHOULD BE. So you're making your
>  > apps easier to write and making mine either harder (I must explicitly
>  > tag hundred of objects "not persistent") or slower.

Speed hit from completely asynch disk writes is negligible, and there's
almost always idle time in which do put that negligible bit.

> If I understand what's being proposed (which is admittedly a little
> difficult given that no one's formally submitted an API proposal), I
> don't think you'd have to do this.  I think, for example, you'd just
> have to make sure that some global symbol contains a list containing
> those little bits of data that you're refering to, and that all the
> rest of the data is created inside of (let ..) constructs so that it
> ultimately gets thrown away.
> 
> What you'd gain is that you don't have to write anything to explicitly
> write those little bits to disk, and you wouldn't have to write
> anything to explicitly read them - they'll always be in that global
> variable into which you put them.
> 
> If you want versioning of these little bits, you'd basically do the
> same as you'd do if you had a file system, but it'd be easier because
> you wouldn't have to try to munge version info into the file names.
> You could, for example, keep a global variable with an assoc list of
> (version-number little bits of data), or keep afew global hash tables
> around - little-bits-by-date, little-bits-by-version-number, etc.
> 
> It's not as if anything that would have been garbage collected will
> suddenly stick around and show up on the disk.  I think it's more like
> maintaining a workspace that survives between reboots - not much
> different from periodically running (dumplisp).  I guess it's similar
> to Smalltalk systems where the user can periodically hit the save
> button, and the entire system state is written to disk.

What I'm proposing is a little different. Everything everywhere
eventually gets to disk, but mostly in an asynchronous manner which means
while I'm making data persistent your threads can quite happily run around
ray-tracing.

My proposal is to simply have all of memory as cache for disk,
looking at memory in chunks as mapping to disk as chunks.

When I want to access a bit of address space that isn't cached in memory
at the moment I load it up disk - ok, that's swap for you. Nothing I can
do about it. Once its in memory I just work on it directly.

When I _write_ to this chunk of memory I mark it as dirty. Something 
similar to unix's fork's copy-on-write method would probably do
the trick bettern in h/w than in software.

All this timeyour threads a merrily whirring away, only pausing if we need
to load a block into memory - ie swapping in.

The _other_ direction doesn't necessarily involve blocking at all, unless
you want that block to necessarily be in synch with the drive at all
times, and you'd use that for your magical-important-transactional-database
implemented on to of this larger grain system.

Every so often you'll want to go though your list of dirty blocks, and
write them out to disk. To do this we first copy the block, mark the
block we copied as 'possibly clean'. Then we throw the block at the
disk writer, which on civilized systems should support DMA or something else
polite like that. When it finishes our write-dirty-blocks-to-disk-thread
checks to see if the original block is now dirty, if it is, it takes off the
'possibly clean' mark, and leaves it as dirty. If its not dirty at this
point the block becomes 'clean'.

Note that when we write the block to disk, we don't overwrite the original.
We put it somewhere new, and further loads will happen on the new block.

When we run out of memory we can start to throw away clean blocks.

Then we have (commit) function, which halts everything, flushes all
dirty blocks to disk, then removes the originals from the disk.

We have a (rewind) function halts everything, and deletes the copies from the
disk, then restarts.

A (shutdown) function would also be nice, halting everything, flushing all
dirty blocks, but not removing originals.

If you want more than one level of commital that's not a problem either.

This is what _I_ am proposing  as the simple, easy, and relatively
sensible approach to transparent persistence.

just saving globals is ok, if you want halt-and-persist, but tht will
indeed cut cpu along the way. Given that we're saving everything, we can do it 
whil e things are changing with fine grained atomic actions (like the 
copying a block of memory to another block of memory).

For the same reasons I would suggest that a proper thread-happy garbage
collector be used.

I can (shutdown) my machine at night, restart it in the morning. If something
bad happens I can (rewind) it, and at points I can (commit) it.

If I lose power I can attempt to repair the memory system, and see if I get
disjoint states that cause problems, but most of the time it should be ok, 
if not I can (rewind) and go on my way.

I can support synchronous writes to support things where I can't tolerate
even slightly disjoint data.

Criticism welcomed,

Brian