[gclist] Wiring up the Boehm GC to the Ruby interpreter

[Boehm, Hans]

Some comments/questions from someone who knows the collector but not Ruby:

1) The top priority here is identifying the cause of the leak.  Once there is a leak, the collector will have to do a lot more work tracing things that should be garbage, pause times will increase, etc.  Try building the collector with KEEP_BACK_PTRS and DBG_HDRS_ALL (or with --enable-full-debug), build the application with GC_DEBUG defined before gc.h is included, and so that it calls the all-caps versions (e.g. GC_MALLOC) of the GC_ routines, run your application for long enough that the heap has grown substantially as a result of the leak (say by a factor of 4), and the call GC_generate_random_backtrace() several times.  This should tell you why some typical objects are being retained.  If you waited long enough, the probability is high that the "random" object is one that was leaked.

When I looked at some corresponding issue with Python a long time ago, a big part of the problem seemed to be a few optimizations in the runtime that "knew about" Python's original memory management.  Any code that keeps its own free lists should be looked upon with some suspicion, since those free lists will be treated as live by the collector.  If those free lists point to other known garbage, that will also be retained.

But it's also possible that the collector will need some type information in a few places, e.g. when you're allocating arrays containing image data. 

2) Did you turn on incremental collection with GC_enable_incremental()?  Otherwise the GC_collect_a_little() calls will have no effect, other than possibly triggering a complete GC at that point.  On a Linux system (and on many others) turning on incremental collection will cause the heap to be write protected.  The collector will handle the resulting write faults, unless they occur inside a system call.  Thus you will need to be careful with, and preferably avoid, system calls that write directly to the garbage-collected heap.  (If the application is single-threaded, or nearly single-threaded, you can also wrap the system calls with some routines provided by the collector.  In some cases, it's also OK to have system calls write to objects allocated with GC_MALLOC_ATOMIC.  But this has turned out to be enough of an issue that I believe there are relatively few clients for the incremental collector.)

Hans

> -----Original Message-----
> From: Matthew Bloch [mailto:matthew@bytemark.co.uk]
> Sent: Friday, July 12, 2002 3:37 AM
> To: ruby-core@ruby-lang.org
> Cc: gclist@lists.iecc.com
> Subject: [gclist] Wiring up the Boehm GC to the Ruby interpreter
> 
> 
> Hi there;
> 
> I tried wiring up the Boehm GC (downloaded a couple of days 
> ago) to the 
> latest Ruby (from CVS) in an effort to smooth out the 
> interpter's lengthy 
> delays in an interactive application written in the language: 
> the only 
> changes I made to the Ruby source were:
> 
> * changed malloc/realloc/free to call GC_ equivalents like 
> this in ruby.h:
> 
>    #ifdef HAVE_BOEHM_GC
>    # include <gc/gc.h>
>    # undef malloc
>    # undef free
>    # undef realloc
>    # define malloc(x) GC_malloc(x)
>    # define free(x) GC_free(x)
>    # define realloc(x,y) GC_realloc(x,y)
>    #endif
> 
> * disabled Ruby signal handlers for SIGBUS/SIGSEGV;
> * disabled all the code in rb_gc() and replaced with 
> GC_collect_a_little();
> * GC.enable / .disable / .start methods changed to affect the 
> Boehm GC.
> 
> Only problem was-- it leaks!  Or at least the memory use from 
> my application 
> went up and up while the standard Ruby collector kept it 
> bounded at around 
> 10%.  I could tell it was doing *something* because the 
> memory shot up much 
> more quickly if I turned off collection.  Are there any 
> 'roots' in the Ruby 
> interpreter that an external GC should need to be told about? 
>  There were no 
> shared library dependencies other than the usual libc etc.  
> Can this *only* 
> be because there's memory Boehm doesn't know about, or are 
> there other 
> pitfalls to watch out for?
> 
> Also, I was a little disappointed not to see any difference 
> 'smoothness' of 
> garbage collection between the two GCs-- the Boehm GC still 
> paused the 
> application as often despite much fiddling with the global 
> variables that can 
> control the frequency & severity of incremental collcetion; 
> is a realistic 
> strategy to try to call GC_collect_a_little() every animation 
> frame, with the 
> time limit set to, say 10ms?  It didn't seem to stick to this 
> unfortunately 
> (though I realise this isn't a guarantee by the collector-- 
> will 10ms always 
> be too quick for it to do any useful work?).
> 
> Seeing as I'm fairly new to both the Ruby interpreter 
> internals and the Boehm 
> GC, I'd appreciate tips on using either.
> 
> thanks,
> 
> -- 
> Matthew Bloch                 Bytemark Computer Consulting
>                                 http://www.bytemark.co.uk/
>                                   tel. +44 (0) 8707 455026
>