The Analysis of Data


February 21, 1963

ABSTRACT

Engineers have long been interested in data analysis and now use tools developed by statisticians. Major challenges remain, including a complete remaking of attitudes and procedures, courtesy of the computer, and the development and foundation of a data analysis that recognizes the;realities, including the real existence of "wild shots".

Computers and data analysis are not well integrated anywhere, though a few places are making serious progress. The computer offers a revolution, not only in the practice of data analysis, but in its theory. General data analysis programs resemble compilers in the essential diversity of their uses and the necessary extensiveness of their output. Today we can have the necessary output for a reasonable price, but rarely do so.

There is a necessary antithesis between mathematics and its applications. Mathematical models should be wrong, should provide guidance, not certainty. It is better to be approximately right than to be exactly wrong. Some feel that mathematical statistics is concerned with finding optimum solutions to neatly and precisely posed problems -- and with nothing else. This is far from enough, even though, as the example of Eddington, Fisher, the mean error and the mean square error shows, optimization under falsely restrictive assumptions can, on occasion, be very helpful for a time. If we are to be prepared for the wild shots that occur in practice, we must seem to omit, overtly or implicitly, certain observations from our calculations. This can be done without unfairness or any approach to dubious procedures of rejection of data because it appears to be in error. The case of inference from a single sample to the center of a symmetrical population provides a single example. The body of the text is a revised version of a talk given at the Princeton University Conference on Mathematical Studies in Engineering, 26 October 1962.

MEMORANDUM FOR FILE

In discussing this subject, I shall avoid, almost completely, discussion of specific procedures of data analysis; talking instead about these general characteristics of the analysis of data which seem to be least well recognized, at least by engineers and mathematicians. And I shall try to be brief, and leave time for questions and debate.

Engineers have always been interested in the analysis of their data -- or, sometimes, of any data they could find. Once-upon a time, engineers were quite good at data analysis -- by contrast with most of their competitors. But new techniques, spawned to deal with biological and agricultural data, first surpassed the engineering techniques in incisiveness and then began to penetrate the practice of engineering. The best engineering practice in data analysis today -- employing, where appropriate, the new techniques that have grown up in engineering contexts; techniques such as response surface analysis in research and development, and evolutionary operation in production improvement -- is again good by contrast with many fields; particularly by contrast with poor engineering analysis of data.

Is there more to be done than to patiently and blandly polish up the developments of the past? Have we reason to be content with engineering analysis of data?

Certainly, we dare not be content until we have met, honestly and thoroughly, at least these three challenges, which loom large for engineering as a whole;

  1. adequate analysis of behavioral data, and adequate incorporation of the results into problems of engineering planning and design.

  2. a complete remaking of attitudes and procedures in data analysis, courtesy of the computer.

  3. development and foundation of a data analysis that recognizes the realities of data, including the realities of "wild shots."

To pinpoint the first challenge, which must be met if engineering is to deal adequately with the problems of a human world, would require more intimate contact than I possess with the classes of engineering problems in which it is so essential.

We have all heard much of the computer revolution -- during these two days we have heard something in the way of examples. How does it challenge the engineer's analysis of data?

In data analysis this revolution has barely begun. There is not, in the known world, a single programming computing system adapted to a broad field of data analysis. (Frank Yates' program for the analysis of surveys may be almost an exception.)

Indeed, I know of but three places in the Western World where the computer and data analysis are strongly coupled: Rothamsted, England; Murray Hill, New Jersey; and Westwood, California. I look toward Princeton's joining this group; I have hope for a few other places.

These are sorry statements, now that the number of large computers is far into the hundreds -- they represent a whole variety of failures: failures of statisticians to be data analysts, failures of computing groups to recognize data analysis as different from general computing, failures of data analysts to escape from the press of daily problems long enough to lead a major effort -- several man years -- in systems programming.

So much for what has been done for the computer -- what does the computer offer data analysis?

On the one hand, the computer has greatly increased the class of mathematical problems about data analysis that we can afford to tackle.

By making extensive numerical analysis cheap, it has done as much for mathematical questions about probability models, as for the questions of any other area.

And it has done much more than this in other areas by serving as a partner in the use of Monte Carlo methods, where statistical insight, probability theory, and all the devices developed for sampling human populations and physical collections -- and new computing power -- can combine to enhance each other's strengths.

If a problem is meaningful for data analysis, and has a simple numerical answer, we should be -- and, I believe, are now -- able to compute an adequate,, approximation to this answer.

The importance of thus breaking the closed-form-or-simple-series barrier cannot be overestimated. Yet, this opportunity, too, has been taken up in very few places.

It is easy to assign reasons which certainly do contribute to such a failure (but hard to regard them as justified):

  1. the importance of individual numerical results has been undervalued, in part, perhaps, because they have not been regarded as fit for Ph.D. theses, but even, perhaps, because getting to numerical answers to specific questions, especially by Monte Carlo, exposes the young statistician to problems too much like those he will have to face in the real world,

  2. such results have also been undervalued because it has not been recognized that the most that data analysis can expect from mathematics, algebraic or numerical, is guidance, not certainty,

  3. in academic statistics, contrary to many other fields of science, expenditure for computing has not become a commonplace.

So much for the development of theory -- what of the development of practice? Surely the impact bf the computer on how we can analyze data is even more important. What are the barriers here?

John von Neumann once said, in effect, "The only good Monte Carlo is a dead Monte Carlo..." The implication being that, once having done, adequately, one instance of a particular type of problem by Monte Carlo, one should be able to understand what analytic approximation would be good enough to treat that class of problems; thus one would first need Monte Carlo for problems of the next higher level of complication.

This is a reasonable parallel to something we recognize in most computing: If something is to be done more than a few times, we ought to understand the purposes the output serves, we ought to adapt the output to these purposes, and narrow it down accordingly. As a result, the repeated calculations that support engineering, physics, and chemistry in design, comparison, and theory rightly become data processing rather than data analysis.

In these sorts of data processing, the output can almost always be concentrated in nature, small in bulk. And most of the time this can also be true of the input.

Data analysis is far from data processing in these respects. Analysis is needed to tell one what the data is trying to say. Accordingly the output must be flexible enough to convey many different sorts of messages, especially unexpected ones. Until we are smart enough to build into our computer programs a tremendous capability for seeking, finding, assembling and building upon clues -- until we can include in them many functions still only dimly identifiable when carried out by data-sensitive human analysts -- and that time will not come very soon -- until then we can only provide flexibility by providing a wide variety of kinds of output, many given in varying detail, and presenting this output to a human analyst -- with the expectation that he will usually come back to the machine for another cycle. When we use a data-analysis program over and over, we are not doing problems of the same kind; sets of data with the same formal structure are quite different -- because they are about different things.

Far from being a parallel to a conventional production program, such as might be used again and again, with modified inputs, in engineering design or scientific theory, a reasonably general data-analysis program is far more like a compiler, each of whose uses is different, since each produces a different program. Diversity of use and volume of output are both shared by compilers and data-analyzors.

The need for volume of compiler output is clearly recognized, that for volume of data-analyzor output is not, although the most compelling reasons why volume is needed are easily stated:

  1. the costs of unnecessary return to the expensive central facilities of the computer should be held down, printing costs are small by comparison -- data-analysis problems need not come back too many times,

  2. skilled data analysts, are, and will remain, in shorter supply than machines of the 7090 class.

These facts need to be faced, and dealt with as best we can.

In England and on the Continent, the union of computing and data analysis suffers a severe disability. Substantial computers use low-capacity inputs and, even, typewriter outputs.

It is not unusual, in this country, to have printers producing from 500 to 1000 lines of more than 100 characters every minute. We have output devices which yield microfilms of point patterns or of curves. And we can obtain output from such devices in movie form if we wish. (The latter, for example, offers us our first chance to understand, as a matter of perception, what happens in really complex iterative processes of analysis.)

In this country, we have the equipment, only people stand in the way of adequate output for data analysis. When a group concerned with data analysis, and duly entitled to one shift per day on a 7090, feels that it has to appease operating management by buying the necessary output paper itself, and delivering it specially, we have allowed our feelings and shop philosophies to go a long way -- a long way too far. And there have been such cases. In conventional computing, massive output usually means lack of understanding, either of the problem or the program, and measures to control output volume, and thus force the user to think, are likely to be justified. In data analysis, massive output also means lack of understanding, but this, lack is only to be remedied by looking at the data via the computer output. Cutting down on data analysis output too vigorously is certain to throw out many babies per thousand gallons of bath water.

Today, computing speeds are high, output speeds are at least moderate, and much larger reasonably-accessible memories are just becoming available. The hardware is here. Our understanding of the analysis of data is far from complete, but we know far more than enough to begin to move from isolated programs to organized systems. Easy access, efficient computation, and adequately varied output can be combined as soon as we are prepared to make the effort.

After the computer, the deluge -- a deluge of much more effective analyses of data. But how long after?

We have heard much about the application of mathematics, especially during these two days, but have we heard anything about the necessary antithesis between mathematics and its applications! This is a topic that cannot, I feel, be discussed too much. The modern world is coming more and more to depend upon mathematics -- partnership is essential. It can only be effective if both parties can adopt the French attitude toward the two sexes: Vive la difference!"

Examples are easy to find. Mathematicians studied the problem of the least area to enclose a given volume long ago, and students repeat this in freshman calculus every year. A sphere, or under some restrictions a cylinder, is the answer. How many of you have seen a spherical milkbottle, or even a cylindrical one?

The application of mathematics consists -- in physics, in engineering, in psychology, in business -- and in data analysis -- of seeking guidance from answers to problems whose hypotheses do not apply to the situation at hand.

Such actions are, and ought to be, more or less repugnant to the mathematicians involved, who may have developed the techniques which led to the answer, or who may have carried out, or checked, the detailed steps from hypotheses to "answers."

Some can usefully and easily shift their gears, doing mathematics at one moment, applying it at another. Along with all others concerned, such fortunate people need to recognize the gulf between these two cultures, a gulf at least as important as the one stressed by Sir Charles Snow.

Failure to recognize the gulf and to accept it as proper, has had as great an impact on data analysis as on any other field I know. Some of my friends regard this impact as all bad. While it would be easy to seek here for a scapegoat, it would, I believe, be quite wrong to go so far. But the time to temper, adjust, modify, and generally weaken this impact is now here. We need urgently to "vive la difference." Data analysis is not alone in needing to assert that it is better to be approximately right than to be exactly wrong. But it must prefer approximate results based on more realistic assumptions to exact results based upon less realistic ones. It dare not, and ought not to dare, to ask mathematicians to do this; it must do it for itself.

Data analysis is not alone, too, in needing to attend to my favorite quotation from our dinner speaker of last evening, George Kimball:

"There is a further difficulty with the finding of 'best' solutions. All too frequently when a 'best' solution to a problem has been found, someone comes along and finds a still better solution simply by pointing out the existence of a hitherto unsuspected variable. In my experience, when a moderately good solution to a problem has been found, it is seldom worth while to spend much time trying to convert this into the 'best' solution. The time is much better spent in real research ... '
George E. Kimball, 1958, A critique of operations research, 48 J. Wash. Acad. Sci., 33-37 (especially P. 35).

There are many, perhaps some here present, who think that mathematical statistics is concerned with finding optimum solutions to neatly and precisely posed problems -- and with nothing else. This could be true, if mathematical statistics were to justify itself solely as a branch of pure mathematics, but if that becomes the only road of mathematical statistics, guidance for data analysis will have to come from other sources.

Let me not give the impression that optimization, even misleading optimization, is, necessarily bad. There must be consideration for time and place. No single man did more to establish data analysis, and experimental design, on the heights they hold today than Sir Ronald Fisher. His first widely recognized paper came from a dispute between Fisher and Eddington, the astronomer and cosmologist, with whom Fisher had been working -- a dispute about relative quality. Given a sample of observations, did one do better -- i.e., obtain a more precise result -- for the variability of the source by calculating the mean square error or the mean absolute error -- the mean magnitude without regard to sign of deviation. Eddington favored the latter for practical reasons, even after Fisher showed the former to be about 5/4ths as efficient when sampling from a normal or Gaussian distribution, from the standard bell-shaped curve. Who was right?

Fisher's mathematics have never been challenged; certain of the concepts he introduced in that paper have been key concepts of statistical theory ever since; and Fisher went on to be the first to bring mathematical power to the solution of the mathematical problems of statistics. (He did many other things as well.) Yet if we form our deviations by drawing from an exactly normal or Gaussian distribution and making a randomly chosen one per cent of these deviations three times as large as they were drawn -- one percent chosen without regard to their size -- the situation is turned upside down, the mean absolute error of the resulting samples is more stable than the mean square error, not less. This is equivalent to drawing from the result of mixing two bowls, each normally distributed, whose means are equal but whose spreads are in the ratio of 3 to 1 -- a mixture made up of 99% of the narrower distribution and 1% of the broader one. Only part, at most one-fifth, of the 1% from the broader bowl can be recognized as coming from something other than the narrow one. With no more than two recognizable "wild shots" per thousand, a seemingly trivial number, the relative merits of competing methods of analysis are entirely changed.

Some of my colleagues say, on the basis of decades of experience in first-class engineering organizations, that not 0.2%, not 0.5%, not 1%, not 2%, but 5 to 10% of all engineering measurements are recognizably wild shots. Even if this is only half true, or one-fifth true, it follows that engineering cannot be content, in the long run, with results based on assumptions of sampling from precisely normal distribution. Much of good has come from normal theory, enough to more than compensate for the misleading aspects of many, perhaps most, of the optimization results based on normality.

The Fisher paper was important for the development of statistics and for the development of Fisher as a statistician. The paper was, and has continued to be, valuable. Yet, so far as data analysis goes, Fisher was wrong and Eddington was right, Sometimes misleading optimization is all right for a time. But the time to do better must arrive. We must now learn to be approximately right, not exactly wrong.

In doing this we need to avoid the ill effects of It wild shots" without neglecting eitheir their possible importance, which is sometimes greater than that of all the "good values", or the useful information contained in "good values" that appear to be "wild shots". In doing this, we will be forced, either overtly or covertly, to appear to omit certain values. It is important to remember that appearances can be misleading. Let us consider a simple example: To use the middle one (in algebraic value) of 11 observations is far from throwing the other 10 away, much further, in fact, than the uninitiated can realize. (The stability of such a median is comparable with the stability of a mean of at least 7 observations.) To use the middle one routinely need have no connection with such dubious procedures as the censoring of observations just because their values are judged, without further evidence, to be bad. Used- routinely, the middle value allows all observations an equal chance to influence the result -- just as equal a chance as when the mean is used, though not so obviously equal.

Changes will be needed in the simplest of situations, even when assessing a location from a single sample of obervations. Thus, if observations x(j), for j = 1 to n, are to be analyzed as a sample, and M is a contemplated value for the population average, Student's t is found from

t = (x-bar - M)/(s/ sqrt(n))

where n(x-bar) = sum x(j), (n-l)s^2 + n(x-bar)^2 = sum x(j)^2. The use of Student's t, either for a test of significance or in setting confidence intervals, is optimum on precisely normal theory.

If we fear that our distributions will have longer tails, than the normal distribution, or that we are likely to be afflicted with wild shots, there are far better things to do than to use Student's t. One good compromise is to use "trimmed t", where

t_T = (x_T-M)/(s_W/sqrt(M))

where the values of x_T and s_W are calculated from modified x(j)'s as follows:

  1. Choose an integer, g, and set m = n-2g.

  2. Let x_T be the mean of all values x(j) except the g lowest and the g highest.

  3. Put v(j) = x(j) for the x(j) entering into x_T ; replace the g lowest x(j) by the value of the (g+l)st lowest x(j), replace the g highest x(j) by the value of the (g+l)st highest x(j), thus defining v(j) for all j.

  4. Let

    n (v-bar) = sum v(j)

    n (v-bar)^2 + (m-1) s_W^2 = sum v(j)^2

  5. Consider choosing g to make s_W/sqrt(m) as small as possible.

The problem is simple, -and occurs frequently, the change in technique is simple and easy to remember. But it is going to take the power of the computer to calculate the critical values of this new statistic to the accuracy, say 2 or 3 decimals, which will be needed to make it seem respectable. (This is, of course, more accuracy than is practically needed.) And it is going to take the power of the computer to produce evidence enough of the absolute and relative quality of this modified t to bring it into use in reasonable time. Now that we understand both how, and why, the modification is better, the main issue is a behavioral one -- how are we to influence the behavior of the many people who analyze data. Accordingly, this example contains, in one form or another, all three challenges.

The challenges are real, the means are on hand to meet them. Improvement of the techniques of analysis of engineering data awaits only sufficiently earnest efforts by sufficiently many earnest and able workers. Improvement in the actual analysis of engineering data will then await only adequate recognition, by engineers and their leaders alike, of what can then be done.