Lucent Technologies Logo

Massive Statistical Tracking


Rick Becker (AT&T Labs) and Diane Lambert


The Problem

There are many engineering and statistical tools, such as Kalman filters and dynamic hierarchical models, for tracking a few series of regularly sampled measurements. There are far fewer tools for separately tracking millions of series of randomly occurring events, and even fewer practical tools for real-time tracking when there are millions of events per day, rates of events vary by several orders of magnitude across individuals, the timing of events (day-of-week and hour-of-day) varies dramatically across individuals, and events have several characteristics, some of which, like geographical location, have hundreds of possible unordered values. The need for tracking millions of event histories (or customers) occurs in credit card fraud detection, targeted marketing, and website analysis, for example.

The Solution

We develop new statistical summaries or signatures of event histories that are useful for massive tracking. The summaries are short (only 500 bytes in our application) but include information on customer identification, transaction rate, timing patterns, duration distribution, and geography. To save space, some variables, such as geography, are treated hierarchically, so that a site is nested within a region which is nested within a time zone, for example. A summary for a new customer is initialized automatically and then updated whenever the customer makes a transaction. The updating can be viewed as quick, approximate Bayesian dynamic modeling. In our testbed application, signatures are maintained and updated in real-time for tens of millions of customers who make millions of transactions per day and successfully used to identify customers whose behavior has changed.

This work falls into an emerging area of statistical methodology that is concerned with both statistical criteria and fast computations for enormous databases.


[ Statistics Project Page | Statistics Research ]