Massive Statistical Tracking
Rick Becker
(AT&T Labs) and
Diane Lambert
The Problem
There are many engineering and statistical tools, such as Kalman
filters and dynamic hierarchical models, for tracking a few series of
regularly sampled measurements. There are far fewer tools for
separately tracking millions of series of randomly occurring events,
and even fewer practical tools for real-time tracking when there
are millions of events per day, rates of
events vary by several orders of magnitude across individuals, the
timing of events (day-of-week and hour-of-day) varies dramatically
across individuals, and events have several characteristics, some of
which, like geographical location, have hundreds of possible unordered
values. The need for tracking millions of event
histories (or customers) occurs in credit card fraud detection,
targeted marketing, and website analysis, for
example.
The Solution
We develop new statistical summaries or signatures of event
histories that are useful for massive tracking. The summaries are short
(only 500 bytes in our application) but include information on customer
identification, transaction rate, timing patterns, duration
distribution, and geography. To save space, some variables,
such as geography, are treated hierarchically, so that
a site is nested within a region which is nested
within a time zone, for example. A summary for a new customer is
initialized automatically and then updated whenever the customer makes
a transaction. The updating can be viewed as quick, approximate
Bayesian dynamic modeling. In our testbed application, signatures are
maintained and updated in real-time for tens of millions of customers
who make millions of transactions per day and successfully used
to identify customers whose behavior has changed.
This work falls into an emerging area of statistical
methodology that is concerned with both statistical criteria and
fast computations for enormous databases.
[
Statistics Project Page |
Statistics Research
]