Local Regression and Likelihood

Local Regression is a method for smoothing data. The smoothing problem assumes we have data (xi,Yi), and fits a smooth curve through a set of these data points. Smoothing problems are central to many branches of science. Early development of local regression, in the late 19th century and early 20th century, occured in many fields; most notably in econometric time series and acturial sciences. But the computational burden was formidable, and most early work treated only the case where the xi are equally spaced. This changed in the 1970's, with the increasing power and availability of computers. Since this time, Bell Labs researchers have been at the forefront of the development of local regression and other smoothing methods, and associated computational and visualization problems.

The Local Regression Model

The underlying model for local regression assumes that the observations can be written as the sum of a signal, or predictable, component, and a random error term:
Yi = m(xi) + ei
where m(xi) is the signal component to be estimated, and ei is the noise components. In traditional parametric regression models, one assumes that m(x) has a simple parametric form, such as a straight line, m(x) = a + bx, or a low order polynomial. The regression coefficients are then estimated; for example, by least squares.

In local regression, one does not assume the parametric model holds globally. Instead, the model is fitted within a sliding `smoothing window'. That is, for a fitting point x, we weight the data points according to their distance from x: observations xi close to x receive large weights, while observations far from x receive zero weights. We write this weight W(xi-x), and define a locally weighted sum-of-squares criterion:

RSS(x;a,b) = å W(xi-x) (Yi - (a+bxi))2
This is minimized over a and b. The resulting coefficients give the estimate of m(x) = a+bx for one value of x. When the smoothing window slides to a new value of x, the smoothing weights W(xi-x) are updated, and a new set of regression coefficients are found. Sliding the window continuously allows us to form an estimate of the entire curve m(x).

A Dataset.
Fix a smoothing window, fit a line by weighted least squares. This gives one point on the fitted curve. Slide the smoothing window across the dataset, to generate the local regression curve.

There are, of course, many ways in which to generalize the basic model, and these have formed the basis for much of our research. Examples include:

Software Development

Our research has been disseminated through three of the most widely used programs for local regression, each providing extensive development over their predecessors:
Lowess (Cleveland, 1979)
Implements the local regression method for a one dimensional predictor. Features include a robust reweighting algorithm to make the fit less susceptible to outliers. This algorithm has been adopted in numerous statistical packages, including R, S, S-Plus, ... Lowess also introduced an innovative computational scheme using linear interpolation to speed up the algorithm.
Loess (Cleveland and Devlin, 1988).
The major development was the extension to multivariate predictors. Other contributions beyond Lowess include the use of local quadratic regression; conditionally parametric fitting and new computational algorithms.
Locfit (Loader, 1999).
This software incorporates local likelihood, density estimation models and censored data models. It also uses more flexible schemes for varying the size of the local neighborhood.

Publications


[HOME][RESEARCH]

Catherine Loader
catherine@research.bell-labs.com
Copyright © 2000, Lucent Technologies.