THE RELATIONSHIP BETWEEN RIDGE REGRESSION AND THE MINIMUM MEAN SQUARE ERROR ESTIMATOR OF CHIPMAN
| Date | 01 May 1974 |
| Author | R. W. BACON,J. A. HAUSMAN |
| DOI | http://doi.org/10.1111/j.1468-0084.1974.mp36002004.x |
| Published date | 01 May 1974 |
THE RELATIONSHIP BETWEEN RIDGE REGRESSION
AND THE MINIMUM MEAN SQUARE ERROR
ESTIMATOR OF CHIPMAN
By R. W. BACON and J. A. HAUSMAN
Recently several papers have been written on a new method of estimation
called 'ridge regression', which is designed to be used when the data are multi-
collinear. This method, which is of possible interest to econometricians, does
present some problems in its actual use and the purpose of this note is to discuss
these problems and to suggest a solution to the difficulties of using ridge regression.
This is achieved by showing that ridge regression (RR) is equivalent to a special
case of Chipman's minimum mean square error estimator (MMSE) and that the
latter approach does suggest a way to avoid the problems associated with ridge
regression. An illustration of both methods is given using a simple economic model
of import determination.
1. Ridge regression
The method of ridge regression has been developed by Hoerl and Kennard in a
series of papers [1, 2, 3, 4] with subsequent contributions by Marquardt [5],
Bannerjee and Carr [6] and Mayer and Willke [7]. The model considered is of the
form: Y=Xß+u, E(u)=O, E(uu')=a21 (1)
where X is a matrix of full rank. Hoerl and Kennard show that for ordinary least
squares (OLS) the expected distance between the true and estimated vectors of
parameters is given by:E(L2) E[(ß)'(ßß)] = cT2 trace (X'X)' (2)
Since the trace of a matrix is equal to the sum of its latent roots and the latent
roots of the inverse of a matrix are the inverses of the latent roots of the matrix
itself, it follows that if A1 . .. À, are the latent roots of (X'X) then the expected
distance is: E(L2)rr2 1/At (3)
Now if the data are highly collinear and as a result (X'X) is nearly singular then
the smallest latent root will be near zero and the expected distance will be very great.
That is, the sum of the variances of the OLS estimators will be very large; this is a
familiar phenomenon with multicollinear data. Hoerl and Kennard therefore sug-
gest augmenting the matrix (X'X) in such a way that the latent roots become
larger and then they base a new estimator on the augmented matrix. The esti-
mator they propose is: ß=(X'X+kI)-'X'Y (4)
where k is a small positive constant. It can be shown that the i'th latent root of
the matrix (X'X + kI) is (À1 + k) so that the trace of the matrix to be inverted is
115
To continue reading
Request your trial