[gclist] Memory leaks kill stock exchange

[Henry G. Baker]

[hbaker: from comp.risks]

Date: 	Wed, 16 Apr 1997 11:05:17 -0400
From: Dave Wortman <dw@pdp1.sys.toronto.edu>
Subject: Why Bre-X crashed the Toronto Stock Exchange

Previously undiscovered bugs in a legacy software system and record high
trading volumes in one stock are being blamed for crashes that stopped
trading on the Toronto Stock Exchange several times in the last few days.

The stock is question is Bre-X Minerals Ltd. Its price is in free fall due
to alleged misrepresentations concerning assay results for their large gold
discovery in Indonesia.

Although the Toronto Stock Exchange (TSE) is able to handle large trading
volumes (e.g. 23.2 billion shares/year), the frantic buying and selling of
Bre-X shares exceeded the normal volume for a single stock by a couple
orders of magnitude leading to memory and system congestion problems.

In "TSE speak", the active set of buy and sell orders for a particular stock
is a "book".  Under normal conditions a book contains 200 .. 300 orders.
The largest book size encountered before Bre-X was around 1600 orders.  The
Bre-X book was averaging 2,500 orders with peaks to around 4,500 orders.

The TSE trading system is estimated at 3,000,000 LOC (language unspecified).
The article describes the code as poorly documented and heavily modified.
The software is designed so that when it tries to execute a single order
(buy or sell transaction), it reads the entire book for the stock into main
memory (probably to match buy and sell orders).  Normally this isn't a
problem but the size of the Bre-X book caused memory contention/overload
problems that apparently crashed the system.  This problem was fixed by
adding more memory to the system.

Subsequent attempts to run the system exposed another bug in the legacy
software. There is apparently a memory leak that occurs when an order is
canceled.  The memory for the order (or possible the entire book) isn't
released properly and eventually the system strangles for lack to available
memory. The very high trading volume in Bre-X caused a much higher than
normal number of cancelled orders. The TSE is working on a solution to this
problem and as an interim measure is carefully controlling trading in Bre-X.

These problems hadn't surfaced in 20 years of operation because the order
books had never been big enough and the trading in one stock volatile enough
to trigger the memory problems.

  [Source: Abstracted from a very well written article by Geoffrey Rowan in
  the *Toronto Globe and Mail*, 12 Apr 1997]

-- 
Henry Baker
www/ftp directory URL:
ftp://ftp.netcom.com/pub/hb/hbaker/home.html