Created: Mon Mar 16 09:35:50 EST 1998

Chapter 13. The C version of LOCFIT

Section 13.1. Installation

Windows 95/NT

You should have the self-extracting zip file. Run this file (if it didn't automatically when downloading); the contents should be extracted to a C:\locfit folder. The locfit executable is C:\locfit\locfit.exe; the example data sets should be automatically installed in the LFData subfolder.

Unix

You should have the locfit source archive. Unpack this file (sh locfit.shar) in a new locfit directory. Then

% make locfit
at the unix prompt, and pray it works!

Some possible problems and solutions (thanks to users for reporting these; please keep the problem reports coming):

Section 13.2. Reading data into LOCFIT

The basic data structure in LOCFIT is a matrix, consisting of a number of named variables of the same length. The basic commands for creating data matrices are readfile, readdata and design. readfile reads an ASCII file; readdata reads previously saved locfit binary datafiles, and design generates variables using random number generators and other features. An example is:

locfit> readfile file=ethanol.dat NOx C E
locfit> design E2=E*E g=rnorm()
locfit> prdata
       NOx          C          E         E2          g 
  3.741000  12.000000   0.907000   0.822649  -1.320317 
  2.295000  12.000000   0.761000   0.579121  -2.436422 
  1.498000  12.000000   1.108000   1.227664  -0.863074 
...
locfit> summdata
Dataset contains 5 variables and 88 observations.
Variable names are:
  NOx
  C
  E
  E2
  g
Here, we have read a file ethanol.dat (the file= can be omitted, as long as the filename is given as the first argument), creating three variables: NOx, C and E. The design command then generates two new variables, E2=E*E, and g=rnorm(); the latter using a standard normal random number generator. We can then print the dataset (with prdata), or summarize the dataset (with summdata).

Another optional argument to readfile is arith. If arith=T, then each item in the input file is parsed by LOCFIT's arithmetic interpreter before being assigned to the data matrix. If the input file is numeric, this has little effect other than to slow things down slightly. One use of the arith argument is processing categorical input; for example

locfit> versicolor=0 virginica=1
locfit> iris=readfile iris.dat arith=T species seplen sepwid petlen petwid
Here, the input file contains a categorical variable, species that can't be handled by LOCFIT. By first assigning numeric values to the two species, and then specifying arith=T, the species variable will be created with 0-1 values.

The design command can also be used to start a new dataset, simply by giving a sample size n argument:

locfit> design n=50 x=runif() y=rnorm()

Variables can have names up to 9 characters long, consisting of almost any characters. It's best to stick to the alphanumeric characters -- naming a variable x+y would be quite legitimate, but might result in considerable confusion! Also, you can't create a variable called n in a design command.

For more details of legitimate expressions for generating variables in the design command, see `Arithmetic Expressions' below.

Permanent datasets are created using savedata and readdata.

locfit> readfile ethanol.dat NOx C E
locfit> savedata ethanol
reads the ethanol.dat file, and saves it (in a binary format) in the file LFData/ethanol.lfd. In the next LOCFIT session, restore the dataset with the command
locfit> readdata ethanol

In fact, it's rarely necessary to directly use the savedata and readdata commands. Datasets can also be saved by naming the readfile and design commands; for example,

locfit> ethanol=readfile ethanol.dat NOx C E
automatically saves the ethanol dataset. Also, many commands have an argument to specify a dataset name as an argument; for example,
locfit> summdata data=ethanol
is equivalent to
locfit> readdata ethanol
locfit> summdata

Section 13.3. Fitting with Locfit

The main purpose of LOCFIT is local fitting, using the locfit command. The simplest use is

locfit> locfit NOx~E data=ethanol
where the first argument NOx~E specifies a model formula, and data=ethanol provides a dataset (if omitted, the current dataset is assumed). The model formula is similar to that in the S version: NOx~E+C specifies a model with two predictors; ~E specifies a model with no response; e.g. for density estimation. One difference is that variables cannot be created in the formula. Thus log(NOx)~E is not valid; instead, the transformed variable must be created separately. Thus use
locfit> readdata ethanol
locfit> design logNOx=log(NOx)
locfit> locfit logNOx~E

A closely related command is relocfit. This differs from locfit only in that default values for parameters are not set; thus, values from the previous fit are retained. Thus

locfit> readdata ethanol
locfit> locfit NOx~E alpha=0.7
locfit> plotfit data=T
locfit> # decide smoothing parameter is too large, try smaller value
locfit> relocfit alpha=0.4
locfit> plotfit data=T

The locfit command has numerous optional arguments, mostly in 1-1 correspondence with the same arguments in the S version:

alpha=0.7
The smoothing parameter or bandwidth, specifying a nearest neighbor bandwidth covering the specified fraction of the data. Can also be given as a vector: alpha=0,0.5 for a fixed bandwidth h=0.5; alpha=0.7,0.5 for a mixed nearest neighbor and fixed bandwidth; alpha=0,0,2 for a locally adaptive bandwidth.
deg=2
The degree of local polynomial.
scale=[2.1,3.8]
Set relative scales for variables in multivariate fitting. scale=1 (the default) corresponds to no scaling; scale=0 uses the sample standard deviation for each variable.
weights= cens= base=
Variables to use as prior weights, censoring variable and baseline for the fitted mean.
family=gauss
Family to use for local likelihood fitting. Options include the regression families gauss (ordinary local regression); binomial; poisson; gamma; geom; and density/rate families density; rate and hazard.
kt=sph
Multivariate weight function to use: sph (spherically symmetric and full polynomial model); prod (product of one dimensional kernels and model without crossproduct terms).
kern=tcub
The weight function: tcub (tricube); rect (rectangular); epan (Epanechnikov); bisq (bisquare); trwt (Triweight); tria (triangular); gauss (Gaussian).
eval=tree
Evaluation structure. Available options are data (data points); grid (grid of points); kdtree (k-d tree, as used in loess); kdc (centers of k-d tree cells); tree (adaptive tree structure; the default); phull (triangulation); cross (cross-validated fit at data points). See evaluation structures (Chapter 11) for more details.
mg=10
For evaluation on a grid, the number of evaluation points (or points per side in the multidimensional case; may be a vector).
maxk=50
Assign space for the evaluation structure. For the adaptive structures tree and phull, it is often impossible to guess in advance the size of the structure. If a warning message about insufficient space is given, re-run the command with larger maxk (default: 50).
itype=default
Type of numerical integration to use in density estimation. Options are prod; hazard; mlin and mult. By default, the `best available' will be chosen.
mint=20
Number of points per dimension to use in the numerical integration rules in density estimation.
maxit=20
Maximum number of iterations in local likelihood optimization.
dc=F
Derivative adjustments, F for no adjustments; T for make adjustments.
cut=0.8
Refinement parameter for adaptively grown structures.
xlim=0,1
Sets boundaries of a compact support for density estimation. Default is no limits.
flim=0,1
Sets limits for the boundaries of the evaluation structure. Default is the range of the data. Not to be confused with xlim.

Like data matrices, the fits can be saved in binary format. The commands are savefit and readfit, to save and read the fit respectively. For example:

locfit> readdata ethanol
locfit> locfit NOx~E alpha=0.5
locfit> savefit ethanol
creates a file LFPlot/ethanol.fit. It's rarely neccessary to call readfit and savefit explicitly; the above commands could be accomplished through the single line
locfit> ethanol=locfit NOx~E alpha=0.5 data=ethanol
Likewise, most of the commands in the next section have an optional fit= argument, that can be used to read a fit.

Section 13.4. Output Commands

predict grid=20 deriv=0
Interpolate fitted values in the current evaluation structure. Exactly one of the options grid; data and file must be given. Options are
data
Predict at the data points.
grid=10
Predict on a grid, with 10 points per dimension. Bounds for the grid are determined by the limits in the evaluation structure.
fit=trim
Predict from the evaluation structure trim. Default: The current evaluation structure.
deriv=0,1
Predict a derivative rather than the function itself. The argument of the deriv command specifies the variables to differentiate with respect to, so for example, deriv=0,1 specifies the second order mixed derivative with respect to x0 and x1. The derivative order should not exceed the degree of local polynomial fitted.
Note: The predict command prints the predicted values on the screen or current output file. To save predictions as part of a dataset, use the fitted command.
fitted name=fit
residuals name=resid type=l
Generate fitted values and residuals for the current fit. If the optional name= argument is given, then the result is added to the current dataset. type specifies the type of residual; deviance (default); pearson raw; ldot.

The current dataset should have variables of the same names as the dataset used in the original fit. Usually, of course, it should still be the same dataset, but there is no check to see whether this is the case.

plotdata x y
plotfit fit=trim.tre m=50 type=w
Produce a plot. Depending on the format specified, this will either create a graphics window or a file lfplot containing ploting commands for reading into graphical software. Options include:
fit=trim.tre
Plot the evaluation structure saved in trim.tre. Default, the current evaluation structure.
m=50
Number of evaluation points, or size of grid for two predictors.
type=w
For two predictors, the type of plot to produce; either wireframe or contour (w and c are acceptable abbreviations). For fits with one predictor or scatterplots, type=p produces point plots; type=l produces line plots.
fmt=maple
Format for plot file. See Chapter 14 for details of available formats.
deriv=0
Derivative specification. See the predict command for details.
main=title
Overall title for the plot. Should not include spaces, since the words would then be interpreted as separate options.
xlab=xaxislabel ylab= zlab=
Labels for the axis.

prfit options
Prints fitted values and the like at the evaluation structure. The options specify what is printed, and the order in which it is printed. The default is to print the fitted values. Available options are
fhat
Fitted values, back-transformed for the current likelihood.
nlx
||l(x)||
infl
Influence values.
se
Standard errors s ||l(x)|| .
cband
Simultaneous confidence bands
h
bandwidth used at point.
fit=file
Use the fit saved with savefit.

summfit
Produces a summary of the current fit -- fitting variables, smoothing parameters etc.

Section 13.5. Some additional commands

outf filename a
Sets output file for results, default is standard output. Important output is sent to this file; working output will still be sent to terminal. If the a is specified, output will be appended to the file. Default is to overwrite.
for 1 10
A for loop, repeats following commands ten times in this example. Can only be used in a command file; not when the programs are run interactively. The current value of the for command can be accessed in arithmetic expressions, as a variable name i0 (or i0, i1, ... for nested for loops). The loop is terminated by a endfor command.
seed -(ThcjK2
Seed for the random number generator. A string of eight characters should be used.

Section 13.6. Arithmetic Expressions

Arithmetric expressions can be typed at the locfit prompt:

locfit> c1=3+4
       7 
locfit> c1*c1 sin(c1)
      49 0.656987 
In the first line, 3+4 is computed, and the result assigned to the constant c1. In the second, we compute c1*c1 and sin(c1), and leave the results unassigned.

The arithmetic interpreter is used for many numeric arguments to other commands. For example, we could use log(c1)/10 as a smoothing parameter:

> locfit NOx~E data=ethanol alpha=0,log(c1)/10

The arithmetic interpreter is also used to generate variables in design commands.

Constants can be named almost anything, although it's best to avoid arithmetic symbols and the like. Also, don't duplicate names from the current dataset.

There are a number of predefined constants, which point to components of the locfit structure: alpha; h; pen (the three components of the smoothing parameter); like (the likelihood at the fitted values); infl, vari (fitted degrees of freedom by the \tr(L) and \tr(LTL) definitions) and resv (estimate of the residual variance). You can assign to these constants like any other; obviously this should only be done with extreme care. More usefully, these constants can be used in computations; for example,

locfit> locfit NOx~E data=ethanol alpha=0.7
locfit> alpha vari gcv=88*(-2*like)/(88-infl)^2
     0.7  4.76494 0.151757 
Here, we fit a local quadratic model to the ethanol dataset. Then, the smoothing paramter, fitted degrees of freedom and GCV statistic are printed.

Recognized components of arithmetic expressions include:

In the design command, these can also be used:

Section 13.7. Building an Interface

For some users, the interface provided may not be satisfactory. A `custom' interface can access the LOCFIT code at several points. You will always want to include the header file local.h. Also, you will need to link with the object file locfit.o, and all the files listed in the OBJS line of the Makefile.

A simple interface to locfit for a one dimensional local regression is

#include "local.h"
struct design des;
struct tree lf;

void mylfcall(x,y,n) double *x, *y; int n; { lf.x[0] = x; lf.y = y; lf.w = lf.c = lf.base = NULL; lf.mi[MDIM] = 1; lf.mi[MN] = n; fitdefault(&lf); evaluator(&des,&lf,procv); }

Here, x and y are predictor and response variables, of length n. dum0 and dum1 should also be vectors of length n, filled with 0's and 1's respectively. These are used as `defaults' for the weights, censoring and base variables.

There are two parameter arrays on the tree structure: The integer array lf.mi and double precision array lf.dp. These store the smoothing parameters and numerous other options (the full list is given later). In the above example, we set lf.mi[MDIM] and lf.mi[MN] to be the dimension and sample size for the regression problem. Other components are set to default values by the routine fitdefault(&lf).

The final line in the above example, evaluator(&des,&lf,procv) is the call to locfit. The first two arguments, des and lf, are the design structure (used as work space) and the locfit tree structure. The third argument, procv is a vertex processing function; this will be described more later.

To set the data pointers:

lf.x[0],lf.x[1],...
should point to the first component of the predictor variables.
lf.y, lf.base, lf.c, lf.w
should point to the first components of the response, base, censoring and weights variables. Must point to legitimate variables, even when they aren't required in the models. Defaults should be a vector of 0's for lf.y, lf.base and lf.c, and a vector of 1's for lf.w. Also note that lf.c is a double precision pointer, even though it should usually contain only 0's and 1's.
lf.xlim, lf.flim, lf.sca
vectors for the x limits, fitting limits and scale variables; should be of length 2d, 2d and d respectively. xlim is ignored except for compactly supported density estimation. flim can be initialized to 0, then limits are chosen automatically. sca can be set to 0 to use sample standard deviations.
lf.tw=NULL; lf.ltw=0;
work space; will be assigned automatically.
lf.iw=NULL; lf.liw=0;
more work space.

The tree structure contains two parameter vectors texttt{lf.mi} and lf.dp for integer and double parameters respectively. The lengths should be allocated to LENM and LEND respectively. The two components lf.mi[MN] and lf.mi[MDIM] must always be set; other components can be set to default values by fitdefault(&lf). Other components that may want setting are

mi[MDEG], mi[MP]
Dimension d; degree of local polynomial and number of parameters in local polynomials (mi[MP] is computed automatically, it needn't be set).
mi[MKER]
Weight function: WRECT; WEPCH; WBISQ; WTCUB; WTRWT or WGAUS.
lf.mi[MKT]=KSPH
kernel type; KSPH (spherically symmetric); KPROD (product); KANG (angular); KLEF, KRIG for one-sided smooths.
mi[MIT]=IDEFA; lf.mi[MMINT]=20
integration type for density estimation (selects `best' method when set to IDEFA) and number of integration points when numerical rules are necessary.
mi[MMXIT]=20
maximum iterations for local likelihood.
mi[MEV]=ETREE
Evaluation structure; ETREE; EPHULL; EDATA; EGRID; EKDTR; EKDCE; ECROS; EPRES. For grid evaluation (EGRID) the integer vector lf.mg must also be set, giving the number of marginal points in each dimension. EPRES allows preset fitting points, but requires careful additional assignments (not yet documented).
mi[MTG]=TGAUS
Local likelihood family; TDEN; TRAT; THAZ; TGAUS; TLOGT; TPOIS; TGAMM; TGEOM.
mi[MDC]=0
Derivative adjustment.
mi[MK]=50
Increase this if there's warnings about insufficient space for evaluation structure.
dp[DALP]=0.7; dp[DFXH]=0; dp[DADP]=0
The smoothing parameters. DALP represents the nearest neighbor component; DFXH represents the fixed component; DADP is the penalty for adaptive smoothing.
dp[DCUT]=0.8
cut parameter for adaptive evaluation structures.

The third argument to evaluator, vfun, is a vertex processing function; it is called sequentially for each fitting point. The vertex processing function should have three arguments: the design structure des; the tree structure lf and an integer v, representing the index of the fitting point. Two vertex processing functions are provided with locfit: procv and procvraw. The second computes the bandwidth and fits, while the first also computes variance estimates. For specialized applications, custom vertex processing functions could be written; typically these would be a call to one of the existing functions, accompanied by some additional processing. An example is in the file simul.c; see the procvscb and scbmax functions in that file.

When evaluator returns, components lf.nv will be set to the number of evaluation points; lf.xev stores the evaluation points; tr.coef contains the fitted values and all coefficients of the local polynomials, tr.nlx the standard errors.

To interpolate the fit to additional points, use the intp(), intv(), intg() or intd() functions. e.g.

y = intp(&lf,x,PCOEF,NULL,0);
to interpolate at a single point, pointed to by x. PCOEF says to interpolate the coefficients (other choices are PNLX, PT0 and PBAND); the last two arguments can be used for derivative estimation.
intv(&lf,x,f,n,PCOEF,NULL,0);
to interpolate to a vector of n points; x[0]...x[d-1] point to the first component of each variable, and f is a vector of length n to contain the result.
intg(&lf,x,f,mg,PCOEF,NULL,0);
interpolates on a grid -- x[0]...x[d-1] point to the grid margins and mg is the length of the margins.
intd(&lf,lf.x,f,mg,PCOEF,NULL,0);
interpolates to data points -- like intv, but no checks when lf.mi[MEV]=EDATA.


Copyright © 1996,1997 Lucent Technologies, Bell Laboratories
Clive Loader, clive@bell-labs.com

[LOCFIT home page]