|
The Analysis of Data
February 21, 1963
In discussing this subject, I shall avoid, almost completely, discussion of specific procedures of data analysis; talking instead about these general characteristics of the analysis of data which seem to be least well recognized, at least by engineers and mathematicians. And I shall try to be brief, and leave time for questions and debate. Engineers have always been interested in the analysis of their data -- or, sometimes, of any data they could find. Once-upon a time, engineers were quite good at data analysis -- by contrast with most of their competitors. But new techniques, spawned to deal with biological and agricultural data, first surpassed the engineering techniques in incisiveness and then began to penetrate the practice of engineering. The best engineering practice in data analysis today -- employing, where appropriate, the new techniques that have grown up in engineering contexts; techniques such as response surface analysis in research and development, and evolutionary operation in production improvement -- is again good by contrast with many fields; particularly by contrast with poor engineering analysis of data. Is there more to be done than to patiently and blandly polish up the developments of the past? Have we reason to be content with engineering analysis of data? Certainly, we dare not be content until we have met, honestly and thoroughly, at least these three challenges, which loom large for engineering as a whole;
To pinpoint the first challenge, which must be met if engineering is to deal adequately with the problems of a human world, would require more intimate contact than I possess with the classes of engineering problems in which it is so essential.
We have all heard much of the computer revolution -- during these two days we have heard something in the way of examples. How does it challenge the engineer's analysis of data? In data analysis this revolution has barely begun. There is not, in the known world, a single programming computing system adapted to a broad field of data analysis. (Frank Yates' program for the analysis of surveys may be almost an exception.) Indeed, I know of but three places in the Western World where the computer and data analysis are strongly coupled: Rothamsted, England; Murray Hill, New Jersey; and Westwood, California. I look toward Princeton's joining this group; I have hope for a few other places. These are sorry statements, now that the number of large computers is far into the hundreds -- they represent a whole variety of failures: failures of statisticians to be data analysts, failures of computing groups to recognize data analysis as different from general computing, failures of data analysts to escape from the press of daily problems long enough to lead a major effort -- several man years -- in systems programming. So much for what has been done for the computer -- what does the computer offer data analysis? On the one hand, the computer has greatly increased the class of mathematical problems about data analysis that we can afford to tackle. By making extensive numerical analysis cheap, it has done as much for mathematical questions about probability models, as for the questions of any other area. And it has done much more than this in other areas by serving as a partner in the use of Monte Carlo methods, where statistical insight, probability theory, and all the devices developed for sampling human populations and physical collections -- and new computing power -- can combine to enhance each other's strengths. If a problem is meaningful for data analysis, and has a simple numerical answer, we should be -- and, I believe, are now -- able to compute an adequate,, approximation to this answer. The importance of thus breaking the closed-form-or-simple-series barrier cannot be overestimated. Yet, this opportunity, too, has been taken up in very few places. It is easy to assign reasons which certainly do contribute to such a failure (but hard to regard them as justified):
So much for the development of theory -- what of the development of practice? Surely the impact bf the computer on how we can analyze data is even more important. What are the barriers here? John von Neumann once said, in effect, "The only good Monte Carlo is a dead Monte Carlo..." The implication being that, once having done, adequately, one instance of a particular type of problem by Monte Carlo, one should be able to understand what analytic approximation would be good enough to treat that class of problems; thus one would first need Monte Carlo for problems of the next higher level of complication. This is a reasonable parallel to something we recognize in most computing: If something is to be done more than a few times, we ought to understand the purposes the output serves, we ought to adapt the output to these purposes, and narrow it down accordingly. As a result, the repeated calculations that support engineering, physics, and chemistry in design, comparison, and theory rightly become data processing rather than data analysis. In these sorts of data processing, the output can almost always be concentrated in nature, small in bulk. And most of the time this can also be true of the input. Data analysis is far from data processing in these respects. Analysis is needed to tell one what the data is trying to say. Accordingly the output must be flexible enough to convey many different sorts of messages, especially unexpected ones. Until we are smart enough to build into our computer programs a tremendous capability for seeking, finding, assembling and building upon clues -- until we can include in them many functions still only dimly identifiable when carried out by data-sensitive human analysts -- and that time will not come very soon -- until then we can only provide flexibility by providing a wide variety of kinds of output, many given in varying detail, and presenting this output to a human analyst -- with the expectation that he will usually come back to the machine for another cycle. When we use a data-analysis program over and over, we are not doing problems of the same kind; sets of data with the same formal structure are quite different -- because they are about different things. Far from being a parallel to a conventional production program, such as might be used again and again, with modified inputs, in engineering design or scientific theory, a reasonably general data-analysis program is far more like a compiler, each of whose uses is different, since each produces a different program. Diversity of use and volume of output are both shared by compilers and data-analyzors. The need for volume of compiler output is clearly recognized, that for volume of data-analyzor output is not, although the most compelling reasons why volume is needed are easily stated:
These facts need to be faced, and dealt with as best we can. In England and on the Continent, the union of computing and data analysis suffers a severe disability. Substantial computers use low-capacity inputs and, even, typewriter outputs. It is not unusual, in this country, to have printers producing from 500 to 1000 lines of more than 100 characters every minute. We have output devices which yield microfilms of point patterns or of curves. And we can obtain output from such devices in movie form if we wish. (The latter, for example, offers us our first chance to understand, as a matter of perception, what happens in really complex iterative processes of analysis.) In this country, we have the equipment, only people stand in the way of adequate output for data analysis. When a group concerned with data analysis, and duly entitled to one shift per day on a 7090, feels that it has to appease operating management by buying the necessary output paper itself, and delivering it specially, we have allowed our feelings and shop philosophies to go a long way -- a long way too far. And there have been such cases. In conventional computing, massive output usually means lack of understanding, either of the problem or the program, and measures to control output volume, and thus force the user to think, are likely to be justified. In data analysis, massive output also means lack of understanding, but this, lack is only to be remedied by looking at the data via the computer output. Cutting down on data analysis output too vigorously is certain to throw out many babies per thousand gallons of bath water. Today, computing speeds are high, output speeds are at least moderate, and much larger reasonably-accessible memories are just becoming available. The hardware is here. Our understanding of the analysis of data is far from complete, but we know far more than enough to begin to move from isolated programs to organized systems. Easy access, efficient computation, and adequately varied output can be combined as soon as we are prepared to make the effort. After the computer, the deluge -- a deluge of much more effective analyses of data. But how long after?
We have heard much about the application of mathematics, especially during these two days, but have we heard anything about the necessary antithesis between mathematics and its applications! This is a topic that cannot, I feel, be discussed too much. The modern world is coming more and more to depend upon mathematics -- partnership is essential. It can only be effective if both parties can adopt the French attitude toward the two sexes: Vive la difference!" Examples are easy to find. Mathematicians studied the problem of the least area to enclose a given volume long ago, and students repeat this in freshman calculus every year. A sphere, or under some restrictions a cylinder, is the answer. How many of you have seen a spherical milkbottle, or even a cylindrical one? The application of mathematics consists -- in physics, in engineering, in psychology, in business -- and in data analysis -- of seeking guidance from answers to problems whose hypotheses do not apply to the situation at hand. Such actions are, and ought to be, more or less repugnant to the mathematicians involved, who may have developed the techniques which led to the answer, or who may have carried out, or checked, the detailed steps from hypotheses to "answers." Some can usefully and easily shift their gears, doing mathematics at one moment, applying it at another. Along with all others concerned, such fortunate people need to recognize the gulf between these two cultures, a gulf at least as important as the one stressed by Sir Charles Snow. Failure to recognize the gulf and to accept it as proper, has had as great an impact on data analysis as on any other field I know. Some of my friends regard this impact as all bad. While it would be easy to seek here for a scapegoat, it would, I believe, be quite wrong to go so far. But the time to temper, adjust, modify, and generally weaken this impact is now here. We need urgently to "vive la difference." Data analysis is not alone in needing to assert that it is better to be approximately right than to be exactly wrong. But it must prefer approximate results based on more realistic assumptions to exact results based upon less realistic ones. It dare not, and ought not to dare, to ask mathematicians to do this; it must do it for itself. Data analysis is not alone, too, in needing to attend to my favorite quotation from our dinner speaker of last evening, George Kimball:
"There is a further difficulty with the finding of 'best' solutions. All too frequently when a 'best' solution to a problem has been found, someone comes along and finds a still better solution simply by pointing out the existence of a hitherto unsuspected variable. In my experience, when a moderately good solution to a problem has been found, it is seldom worth while to spend much time trying to convert this into the 'best' solution. The time is much better spent in real research ... 'George E. Kimball, 1958, A critique of operations research, 48 J. Wash. Acad. Sci., 33-37 (especially P. 35). There are many, perhaps some here present, who think that mathematical statistics is concerned with finding optimum solutions to neatly and precisely posed problems -- and with nothing else. This could be true, if mathematical statistics were to justify itself solely as a branch of pure mathematics, but if that becomes the only road of mathematical statistics, guidance for data analysis will have to come from other sources. Let me not give the impression that optimization, even misleading optimization, is, necessarily bad. There must be consideration for time and place. No single man did more to establish data analysis, and experimental design, on the heights they hold today than Sir Ronald Fisher. His first widely recognized paper came from a dispute between Fisher and Eddington, the astronomer and cosmologist, with whom Fisher had been working -- a dispute about relative quality. Given a sample of observations, did one do better -- i.e., obtain a more precise result -- for the variability of the source by calculating the mean square error or the mean absolute error -- the mean magnitude without regard to sign of deviation. Eddington favored the latter for practical reasons, even after Fisher showed the former to be about 5/4ths as efficient when sampling from a normal or Gaussian distribution, from the standard bell-shaped curve. Who was right? Fisher's mathematics have never been challenged; certain of the concepts he introduced in that paper have been key concepts of statistical theory ever since; and Fisher went on to be the first to bring mathematical power to the solution of the mathematical problems of statistics. (He did many other things as well.) Yet if we form our deviations by drawing from an exactly normal or Gaussian distribution and making a randomly chosen one per cent of these deviations three times as large as they were drawn -- one percent chosen without regard to their size -- the situation is turned upside down, the mean absolute error of the resulting samples is more stable than the mean square error, not less. This is equivalent to drawing from the result of mixing two bowls, each normally distributed, whose means are equal but whose spreads are in the ratio of 3 to 1 -- a mixture made up of 99% of the narrower distribution and 1% of the broader one. Only part, at most one-fifth, of the 1% from the broader bowl can be recognized as coming from something other than the narrow one. With no more than two recognizable "wild shots" per thousand, a seemingly trivial number, the relative merits of competing methods of analysis are entirely changed. Some of my colleagues say, on the basis of decades of experience in first-class engineering organizations, that not 0.2%, not 0.5%, not 1%, not 2%, but 5 to 10% of all engineering measurements are recognizably wild shots. Even if this is only half true, or one-fifth true, it follows that engineering cannot be content, in the long run, with results based on assumptions of sampling from precisely normal distribution. Much of good has come from normal theory, enough to more than compensate for the misleading aspects of many, perhaps most, of the optimization results based on normality. The Fisher paper was important for the development of statistics and for the development of Fisher as a statistician. The paper was, and has continued to be, valuable. Yet, so far as data analysis goes, Fisher was wrong and Eddington was right, Sometimes misleading optimization is all right for a time. But the time to do better must arrive. We must now learn to be approximately right, not exactly wrong. In doing this we need to avoid the ill effects of It wild shots" without neglecting eitheir their possible importance, which is sometimes greater than that of all the "good values", or the useful information contained in "good values" that appear to be "wild shots". In doing this, we will be forced, either overtly or covertly, to appear to omit certain values. It is important to remember that appearances can be misleading. Let us consider a simple example: To use the middle one (in algebraic value) of 11 observations is far from throwing the other 10 away, much further, in fact, than the uninitiated can realize. (The stability of such a median is comparable with the stability of a mean of at least 7 observations.) To use the middle one routinely need have no connection with such dubious procedures as the censoring of observations just because their values are judged, without further evidence, to be bad. Used- routinely, the middle value allows all observations an equal chance to influence the result -- just as equal a chance as when the mean is used, though not so obviously equal. Changes will be needed in the simplest of situations, even when assessing a location from a single sample of obervations. Thus, if observations x(j), for j = 1 to n, are to be analyzed as a sample, and M is a contemplated value for the population average, Student's t is found from
where n(x-bar) = sum x(j), (n-l)s^2 + n(x-bar)^2 = sum x(j)^2. The use of Student's t, either for a test of significance or in setting confidence intervals, is optimum on precisely normal theory. If we fear that our distributions will have longer tails, than the normal distribution, or that we are likely to be afflicted with wild shots, there are far better things to do than to use Student's t. One good compromise is to use "trimmed t", where
where the values of x_T and s_W are calculated from modified x(j)'s as follows:
The problem is simple, -and occurs frequently, the change in technique is simple and easy to remember. But it is going to take the power of the computer to calculate the critical values of this new statistic to the accuracy, say 2 or 3 decimals, which will be needed to make it seem respectable. (This is, of course, more accuracy than is practically needed.) And it is going to take the power of the computer to produce evidence enough of the absolute and relative quality of this modified t to bring it into use in reasonable time. Now that we understand both how, and why, the modification is better, the main issue is a behavioral one -- how are we to influence the behavior of the many people who analyze data. Accordingly, this example contains, in one form or another, all three challenges. The challenges are real, the means are on hand to meet them. Improvement of the techniques of analysis of engineering data awaits only sufficiently earnest efforts by sufficiently many earnest and able workers. Improvement in the actual analysis of engineering data will then await only adequate recognition, by engineers and their leaders alike, of what can then be done.
|