Local Regression and Likelihood
Local Regression is a method for smoothing data. The smoothing problem
assumes we have data (xi,Yi),
and fits a smooth curve through a set of these data points.
Smoothing problems are central to many branches of science. Early
development of local regression, in the late 19th century and early
20th century, occured in many fields; most notably in econometric
time series and acturial sciences. But the computational burden was
formidable, and most early work treated only the case where the
xi are equally spaced.
This changed in the 1970's, with the increasing power and availability
of computers. Since this time, Bell Labs researchers have been at the
forefront of the development of local regression and other smoothing
methods, and associated computational and visualization problems.
The Local Regression Model
The underlying model for local regression assumes that the observations
can be written as the sum of a signal, or predictable, component, and
a random error term:
Yi = m(xi) + ei
where m(xi) is the signal component to be estimated,
and ei is the noise components. In traditional parametric
regression models, one assumes that m(x) has a simple
parametric form, such as a straight line, m(x) = a + bx,
or a low order polynomial. The regression coefficients are then estimated;
for example, by least squares.
In local regression, one does not assume the parametric model holds
globally. Instead, the model is fitted within a sliding `smoothing window'.
That is, for a fitting point x, we weight the data points according to their
distance from x: observations xi close to x receive large
weights, while observations far from x receive zero weights. We write this
weight W(xi-x), and define a locally weighted
sum-of-squares criterion:
RSS(x;a,b) = å
W(xi-x) (Yi - (a+bxi))2
This is minimized over a and b. The resulting coefficients give the
estimate of m(x) = a+bx for one value of x. When the smoothing
window slides to a new value of x, the smoothing weights W(xi-x)
are updated, and a new set of regression coefficients are found.
Sliding the window continuously allows us to form an estimate of the
entire curve m(x).
 |
 |
 |
| A Dataset. |
Fix a smoothing window, fit a line by weighted least squares. This
gives one point on the fitted curve. |
Slide the smoothing window across the dataset,
to generate the local regression curve. |
There are, of course, many ways in which to generalize the basic model,
and these have formed the basis for much of our research. Examples include:
- Multivariate predictor spaces (`surface smoothing') when the
xi are in two or more dimensions.
- Properties of the estimate: Bias, Variance, equivalent degrees of
freedom.
- Choice of the width of the smoothing window: how do we decide what
is `close' in defining W(xi-x)? Should the width be
fixed, or can better results be obtained by varying the width as
a function of x?
- Choice of local model: Using a higher order model (say local quadratic
instead of local linear) can improve the fit obtained, but is
more variable since it involves estimating more coefficients.
- Robustness: Since the method defined above is based on local least
squares, it is sensitive to outliers and long-tailed distributions.
Can robustness methods improve the estimate in this situation?
- Likelihood models; for example, when the Yi are 0-1 data;
or count data, and it is appropriate to replace least squares by
a likelihood criterion.
- Censored Data models, when the Yi may not be completely
observed.
- Density Estimation.
- Computational issues: can we develop computational schemes that
approximate the local regression curve (or surface) without
repeating the entire computation for each x.
- Statistical Inference: testing for the significance of structure
in the fitted curve; finding confidence bands for the true m(x).
Software Development
Our research has been disseminated through three of the most widely
used programs for local regression,
each providing extensive development
over their predecessors:
- Lowess (Cleveland, 1979)
- Implements the local regression method
for a one dimensional predictor. Features include a robust reweighting
algorithm to make the fit less susceptible to outliers. This
algorithm has been adopted in numerous statistical packages, including
R, S, S-Plus, ... Lowess also introduced an innovative computational
scheme using linear interpolation to speed up the algorithm.
- Loess (Cleveland and Devlin, 1988).
- The major development was
the extension to multivariate predictors. Other contributions
beyond Lowess include the use of local quadratic regression;
conditionally parametric fitting and new computational algorithms.
- Locfit (Loader, 1999).
- This software incorporates local likelihood,
density estimation models and censored data models. It also uses
more flexible schemes for varying the size of the local neighborhood.
Publications
- Cleveland, W. S. (1979).
Robust locally weighted regression and smoothing scatterplots.
Journal of the American Statistical Association
74, 829-836.
- Cleveland, W. S. (1981).
LOWESS: A Program for Smoothing Scatterplots by Robust Locally
Weighted Regression. W. S. Cleveland (1981).
The American Statistician 35, 35:54.
- Cleveland, W. S. and Devlin, S. J. (1988).
Locally weighted regression: An approach to regression analysis by
local fitting.
Journal of the American Statistical Association
83, 596-610.
- R. B. Cleveland, W. S. Cleveland, J. E. McRae, and I. Terpenning (1990).
STL: A Seasonal-Trend Decomposition Procedure Based on Loess
(with Discussion).
Journal of Official Statistics 6, 3-73.
- Cleveland, W. S. and Grosse, E. H. (1991).
Computational methods for local regression.
Statistics and Computing 1, 47-62.
- Cleveland, W. S; Grosse, E. H. and Shyu, W. M. (1992).
Local Regression Models.
In Statistical Models in S, ed. Chambers and Hastie. 309-376.
- Hastie, T. J. and Loader, C. R. (1993).
Local regression: Automatic kernel carpentry
(with discussion).
Statistical Science 8, 120-143.
- Loader, C. R. (1993).
Nonparametric Regression,
Confidence Bands and Bias Correction.
In Computing Science and Statistics: Proceedings of the 25th Symposium
on the Interface, 131-136.
- Cleveland, W. S. (1994).
Coplots, Nonparametric Regression
and Conditionally Parametric Fits.
In Anderson, T.W; Fang, K. T. and Olkin, I, editors, IMS Lecture Notes
-- Monograph Series 24. IMS, Hayward, Calofornia.
- Sun, J. and Loader, C. R. (1994).
Simultaneous Confidence Bands for Linear
Regression and Smoothing.
Annals of Statistics 22, 1328-1345.
- Loader, C. R. (1994).
Computing Nonparametric Function Estimates.
In Computing Science and Statistics: Proceedings of the 26th Symposium
on the Interface, 356-361.
- Cleveland, W. S. and Loader, C. R. (1996).
Smoothing by Local Regression: Principles and Methods.
In Hardle, W. and Schimek, M. G, editors, Statistical Theory
and Computational Aspects of Smoothing. Physica Verlag, Heidelberg.
- Loader, C. R. (1996).
Change Point Estimation Using Nonparametric
Regression. Annals of Statistics 24, 1667-1678.
- Loader, C. R. (1996).
Local Likelihood Density Estimation.
Annals of Statistics 24, 1602-1618.
- Loader, C. R. (1996).
Censored Local Likelihood.
- Loader, C. R. (1999).
Bandwidth Selection: Classical or Plug-In?
Annals of Statistics 27, 415-438.
- Loader, C. R. (1999).
Local Regression and Likelihood. Springer, New York.
[HOME][RESEARCH]