- TITLE OF RESEARCH PROJECT:
Scheduling Problem in Learning from Hints
- NAME OF ADVISOR:
Yaser S. Abu-Mostafa
- PROJECT DESCRIPTION:
Consider the problem of designing a system to
recognize characters, faces, or to predict the next day
in the stock market, or the weather. In these cases, since
the rules defining the problem may not be well-defined or
may change in time, explicit programming may not be applicable.
Given input-output examples of such a problem,
learning-from-examples can be used to find a solution.
Figure 1.
Generalization improves when additional hints are used.
When not only input-output examples, but also some rules
about the unknown function are available, these can be
incorporated in learning. Learning-from-hints
[1]
allows expressing rules about a
function as examples, and using a
learning-from-examples system to learn these rules.
Figure 1 shows the effect of hints on
generalization error, which improves when additional
hints are used.
Hints affect generalization. When more than one hint is
available, how much each should be taught becomes an issue.
The scheduling problem
[2]
is the problem
of how much to teach each hint.
The goal of learning is to achieve minimum generalization error.
Therefore, if the generalization error can be estimated as a
function of hint errors, the resulting estimate can be used
to solve the scheduling problem.
Figure 2.
Generalization error is closer to its noise estimate
than it is to training error.
In the case of invariance hints, such an estimate exists
[3] .
In this approach the generalization error
is approximated as the sum of mean and variance of a noise function,
which are computed in terms of hint errors. Figure 2
depicts the plot of the training, generalization
and estimate of generalization error, for a training session
using the function examples only. As seen in the figure,
the estimate of generalization error follows it better
than the training error, which is the only estimate of
generalization error in the absence of additional hints.
Unfortunately, the above estimate can only be used with invariance
hints, so our research continues to find estimates of
generalization error for any type of hints.
The experiments above were performed by a program (train)
written in C. The program can be used to train
feed-forward neural networks using different hints,
schedules, and optimization methods.
REFERENCES:
[1]
Yaser S. Abu-Mostafa (1995),
``Machines that Learn from Hints'',
Scientific American ,
Vol. 272, No. 4, pp.\ 64--69.
[2]
Zehra Cataltepe (1994),
"The Scheduling Problem in Learning from Hints ,
M.S. Thesis, California Institute of Techology,
Caltech-CS-TR-94-09.
[3]
Zehra Cataltepe, Yaser S. Abu-Mostafa (1993),
"Estimating Learning Performance Using Hints",
Proceedings of the 1993 Connectionist Models Summer School
M. Mozer et. al. (Eds.),
Lawrence Erlbaum Associates, Publishers,
Hillsdale, NJ. pp. 380--386.
For a postscript version of this page click
here.
Last modified: April 6, 1995
Copyright @ Zehra Cataltepe
zehra@cs.caltech.edu.