This article appears in
IEEE MultiMedia, Vol. 7, No. 2, April-June 2000.
The IEEE has copies of a PDF version.

168 Beethovens/Second: An optical film soundtrack as suggested by John Cage. The image is by Rubin and Hansen, inspired by Cage's writings.

The audiences would be the artists
and their life would be the arts.

By Mark Hansen and Ben Rubin

To most of us, the transmission of information across the Internet is entirely abstract and intangible. But in 1995, artist Natalie Jeremijenko quite literally tapped into the bit stream coursing through the local network at Xerox PARC, transforming the flow of data into a physically perceptible phenomenon. Her installation, titled "Dangling String," consists of a long plastic fiber that is attached to an electric motor and mounted on the ceiling. Each bit traveling across a stretch of nearby Ethernet cable triggers the motor. During periods of heavy network activity, the string whirls rapidly; but when traffic is light, the motor twitches sporadically, almost undetectably. Jeremijenko's elegant and accessible piece represents an important step in the quest by artists to reveal the mostly invisible realm of electronic data, a world that is increasingly astonishing in its size and nature.

Today, almost every aspect of our lives can be "rendered" digitally. Advances in data collection technologies have made commonplace continuous, high-resolution measurements of our physical environment (weather patterns, seismic events, ecological indicators). Equally open to observation are our routine movements through and interactions with our physical surroundings (automobile and air traffic, large-scale land use). In computer-mediated settings, our activities either depend crucially on or consist entirely of complex digital data (financial transactions, accesses to global information systems, Web site and Internet usage). As a reflection of the diversity and variety of the systems under study, these data-based descriptions of our daily lives tend to be massive in size, dynamic in character, and repleat with rich structures.

The advent of these enormous repositories of digital information presents us with an interesting challenge: How can we represent and interpret such complex, abstract and socially important data? In a new collaboration, Ear to the Ground, we have begun an exploration into ways of creating experiential encounters with otherwise abstract data streams, especially through sound [1]. As part of our initial efforts, we have gathered several examples of what we will term "data art," works produced by artists, designers and engineers that challenge our traditional notions of information display. These pieces each have an aesthetic function, but because of their reliance on data, they also serve as a basis for forming inferences about the phenomena they describe.

The phrase we borrowed as the title for this article, "The audiences would be the artists and their life would be the arts," is from a comment made in 1966 by pianist Glenn Gould, who was speculating on the nature of art in the electronic age. Gould imagined the possibilities of involving the audience, and information about them, in the process of creating and performing works of art. What makes this statement compelling now is the fact that so much information is readily available about the public, Gould's audience/artists.

Music and Mathematics

Musicians, as well as visual artists, have a long history of creating shapes and compositions from observations of the natural world. In some cases, these works offer us a new way to experience physical phenomena. While many composers have reveled in nature's regularity, say in the cycles of planetary orbits, others have attempted to capture the scale and force of nature. Avant-garde composer John Cage, for example, randomly generated compositional elements to imitate the "character" of nature. In the 1950s, composer Iannis Xenakis applied the term "Musique Stochastique" to music characterized by "masses of sounds"; systems consisting of so many separate elements that we can only hear the effect of the "whole," but the contributions of any single individual cannot be determined precisely [2]. Xenakis motivates this idea by describing the sonic phenomena of a "crowd of dozens or hundreds of thousands of people" -- each one an individual, listening, reacting, and producing sound, propagating chants and cheers in graceful, chaotic waves [3].

While Xenakis's image was inspired by the mid-century political demonstrations around him, groups of this size gather daily on large Web sites like Yahoo! or CNN.com. To what extent is it possible to represent these voiceless, virtual crowds through sound? How do we make perceptible their activities, their movements? And in so doing, what new insights do we form about the nature of Web traffic? This is the starting point of our project, Ear to the ground. While we have focused initially on Internet usage, our broad goal is to "sonify," or make audible, digital transcripts of human activity. As a collaboration between a statistician (Hansen) and an artist (Rubin), we are looking to create sound representations that further the arts of data analysis, discovery, and expression.

In the 1980s and 90s, Xenakis became interested in a more mathematically-derived form of composition to produce what he thought were forms of classical perfection. In 1991, he created a computer program that generates a complete musical piece by pure algorithmic computation. Xenakis is not alone in his use of music or sound as a means of experiencing concepts in pure mathematics. Daniel Cummerow, for example, has several compositions based on the digits of pi and other well known sequences [4]. Lars Kinderman's MusiNum (the Music of the Numbers) is an example of "fractal music"; a number-theoretic algorithm generates an entire musical score, a "self-similar" series, from a single integer [5]. Applying this idea, Kinderman's Web site will construct a four-part musical composition based on the separate digits of the IP address of a visitor's computer.

Data Art

In considering our Ear to the Ground project, we have come to understand an important distinction between sonic compositions that are mathematically derived and those that are data-driven. The former amount to transformations of mathematical structures, created or selected by the composer, into the sensory domain of sound. Data-driven compositions, on the other hand, take streams of measurements as their basis. Both math-derived and data-derived music have in common the need for sonification methods, the direct mechanism by which the numbers, whether algorithmically generated or empirically measured, are rendered audible. A primary goal in our Ear to the Ground effort is to establish methods that preserve and reveal the inherent structure and dynamics of the data. The selection of the data -- deciding what to measure and how to measure it -- becomes a central issue in creating data art. With Ear to the Ground, we are looking initially to data on human behavior as expressed in patterns of Internet usage -- data that will potentially have the same kind of rich, dynamic patterns that Xenakis found so compelling in his crowd imagery.

The Life Music compositions of John Dunn (an artist) and Mary Anne Clark (a biologist) are another example of data art [6]. The subject of their "sonification" experiments is the basic component of all life, DNA. The Human Genome Project (HGP) has now "sequenced" roughly 65% of the 3 billion chemical base pairs that make up human DNA, and at the current rate, a complete picture of our 100,000 genes will be available by 2003. The HGP poses a number of ethical, social and legal questions that are already filtering into public forums. By giving a voice to strands of protein in human DNA, the compositions by Dunn and Clark are a concrete point of contact with this rich data set. Their goal is to convey information about the primary amino acid sequence (mapped to pitch), as well as the folding pattern (mapped to separate instruments). Life Music stands as an example of music being directly inspired by the audience, a literal and detailed fulfillment of Gould's statement.

In as much as it retains "fidelity" to data streams, the concept of data art has useful precedents in the scientific community as well. Recently, Andrew Michael of the US Geological Survey "played" the seismograms recorded from earthquakes in the Los Angeles area [7]. His audio representations allow the listener to discriminate between earthquakes based on their distance from the siesmogram. A number of other features about earthquakes can be derived from their sound. Michael's work can be seen as an extension of earlier ideas presented by Speeth in 1961, who listened to seismograms in the hope of detecting differences between earthquakes and underground bomb blasts [8]. These examples barely scratch the surface of scientific applications of sound to help investigate data. An excellent survey of this area is available through the ICAD (International Community for Auditory Display) Web site [9].

"A revelation of sound even where we don't expect that it exists"

The quote above by John Cage (taken from [10]) reflects his insatiable desire to hear everything -- even things that are not intrinsically audible. In 1938, John Cage became fascinated by the new technology of optical sound reproduction (transforming sound into light modulations in order to record it photographically onto motion picture film) that had revolutionized the motion picture industry. Cage pushed the idea of sound-as-light further, observing in his Future of Music: Credo that "Any design repeated often enough on a soundtrack is audible. Two hundred and eighty circles per second on a soundtrack will produce one sound, whereas a portrait of Beethoven repeated fifty times per second on a soundtrack will have not only a different pitch but a different sound quality" [11]. Cage understood that the new technology had created a way not only of recording sound and playing it back, but also of transforming other kinds of information — images, for example — into sound.

Carrying the idea in a different direction, Cage went on to imagine ways to amplify and make audible the molecular and atomic vibrations of a static, inanimate object, such as an ashtray. Experienced in this way, how would the sound of an ashtray differ from the sound of a piece of wood, or a stone, or a grapefruit? Could sound somehow help us fathom the true nature of each of these things? Consider again the seismic data of Speeth and Michael. While the events themselves (earthquakes) have an intrinsic sound, the data recorded from these events do not. Creating sound from these data offer us a new experience. If, in our sonic rendering, we preserve the character of the data, then we can draw conclusions about different types of events.

By why stop at sound? We rely on many different sensory inputs for collecting information from the natural world besides sight and sound. Hiroshi Ishii, who founded the Tangible Media Group at the MIT Media Lab, is exploring how physical phenomena can be used to construct new modalities or representations for experiencing data [12]. In a project called Tangible Bits, Ishii and his collaborators follow Jeremijenko's lead, using physical kinetic motion, streams of air, and other "media" to construct ambient data displays [13]. In one project, the group linked multiple streams of data to large, rotating pinwheels that spin faster or slower depending on fluctuations in the data sources. Perceptible in one's peripheral vision, or merely by the sensation of the gentle breezes it stirs, the Pinwheel apparatus is an effective means of communicating a non-specific sense of activity and change.

Listening Post

As we become more adept at observing, measuring, and recording both our physical and virtual movements, we can expect to see a proliferation of large, complex data sets. These new digital records of human activity often force us to consider difficult societal questions. From the current debate over privacy on the Internet, to ethical concerns over the use of genetic information, we recognize that the now simple act of compiling data has serious political implications. To further complicate matters, the scale and structure of these data make them difficult to comprehend in the large, relegating activities like data analysis to a select few. Even then, experiencing and appreciating what the data have to say is a difficult task, stretching current methods and tools to their limits.

The notion of data art offers to the public a means of engaging these often abstract measurements. In our Ear to the Ground collaboration, for example, we plan on establishing "listening posts," points in the physical world or on the Web where people can tap in and listen to the pulse of global information systems. In the same way, Dunn and Clark provide us a unique impression the human genome, a view that in their words offers "empirical proof of the esthetic patterning of nature's deep structure" [14]. Taken farther, pieces of data art also serve the seasoned analyst, highlighting features of the data that might have been difficult to ascertain otherwise. Ishii's ambient displays, for example, allow for passive monitoring of often remote data sources, a novel approach to statistical process control.

Throughout our discussion, we have stressed the need for data art to provide an impression that is in some sense true to the character of the source. By interacting with these pieces, we inevitably engage in some form of data analysis, albeit informal. We form questions, draw conclusions, and make inferences based on an experience of the data that might involve many senses at once. How do we gauge the effectiveness of these new displays? When are they trivial or misleading and when are they likely to spawn new insights? This is certainly new territory for both statistics and the arts. Our success in making the most of these new ideas will depend on collaborative efforts like Ear to the ground, projects that combine empirical science with artful media.

References

  1. Ear to the Ground, Hansen, M. and Rubin, B., /stat/ear

  2. 20th Century Music, Emory University, http://www.emory.edu/MUSIC/ARNOLD/xenakis.html

  3. Xenakis, I. 1992. Formalized music: thought and mathematics in composition. Rev. ed. Pendragon Press, Stuyvesant, NY.

  4. The Sound of Mathematics, Cummerow, D., http://www.geocities.com/Vienna/9349/

  5. MusiNum -- The Music of the Numbers, Kinderman, L. http://www.forwiss.uni-erlangen.de/~kinderma/musinum/musinum.html

  6. Life Music, Dunn, J. and Clark, M., href=http://www.algoart.com/dnamusic

  7. The Music of Earthquakes - Waveforms of Sound and Seismology, Michael, A., http://quake.wr.usgs.gov/more/listen/

  8. Speeth, S. D. 1961. "Seismometer Sounds." J. Acous. Soc. Am. 33, 909--916.

  9. International Community for Auditory Display, http://www.icad.org

  10. Kahn, D. 1999. Noise, water, meat: a history of sound in the arts. MIT Press, Cambridge.

  11. Cage, J. 1961. Silence. Weslyan University Press, Hanover.

  12. Tangible Media Group, MIT Media Lab, http://tangible.www.media.mit.edu/groups/tangible/

  13. Ambient Fixtures, Tangible Media Group, MIT Media Lab, http://tangible.www.media.mit.edu/groups/tangible/projects/Ambient_Fixtures/Ambient_Fixtures.html

  14. Dunn, J. and Clark, M. 1997. Life Music: The Sonification of Proteins. Leonardo on-line . http://mitpress.mit.edu/e-journals/Leonardo/isast/articles/lifemusic.html