Brooklyn Academy of Music: Arts in Multimedia Project

EAR TO THE GROUND:
Hearing movement and change in the world


Story Idea 1: Collaborations between Science and Art.

  • Goal: The creation of tools for (identifying|highlighting|experiencing) dynamic (patterns|characteristics) of data.

    • Advances in data collection technologies has led to the creation of huge datasets (Internet traffic, global transportation networks, the telephone system)
    • Often, these datasets record our comings and goings, in both the electronic and the physical worlds;
    • Traditional approaches to experiencing these data break down because of their size and complexity;

  • Why?

    • We need new tools to experience large, dynamic data sets.
    • As these data often involve human activity, there is a social component to experiencing the data.

  • Why sound?

    • Sound can more easily express dynamic characteristics in data.
    • Sound is an intrinsically ambient medium that we routinely use for information gathering. We collect information about the physical world through sound; the same can be true of the digital world.
      (examples: car engine, wind direction, building occupancy)
    • Sound is immersive, you don't have to be looking at a screen to get it.

  • What is the end product?

    1. We can't really speculate on future ``products'' in a formal sense. This is forward-looking research.
    2. The math center and the statistics research group in particular has a long history of contributing to a corporate ``bottom line'' through really fundamental research.
    3. Statistics is about data, about learning from data. The telecommunications industry has produced some of the largest, most complex data sets around.
    4. Anything we can do to enhance our experience of a data set, making it richer or more expressive, offers the possibility of forming new insights. This was true of the fundamental work on visualization that came out of our group, and I am confident the same will be true of this work.
    5. Having said that, there are naturally application areas that are helping to inform our work. For example, we have already considered Web site traffic; we will get data on network operations either in the context of a network operations center or observations of traffic patterns across the Internet. We have also plan to give the Java virtual machine a sound, the classes and methods.
    6. This is, of course, forward-looking research. we have a lot to learn in terms of what makes an informative audio representation; what are the rules that govern how we perceive patterns in music or simply sound? what are the principles that govern how best to amplify the important features in data?
    7. Of course it is still early, but i think this work can be as important as the early studies on data visualization that took place at the labs.

  • What?

    • Passive monitoring of complex systems:

      Network operations center collects messages from various points in the network. Understading the patterns in these messages is difficult. An audio representation can help controllers understand the natural fluctuations in these data and more immediately alert them to problems.
    • Uncover patterns in web site usage:

      • We can help web masters and content providers better ascertain the characteristics of their traffic. A rich audio experience can provide clues to behaviors that are difficult to uncover through other methods.
      • Very few of us realize that there are often tens of thousands of people all reading material from the same web site.

    • Provide users of these new technologies with more insight into how they function and how others use them.

      • Realize your contribution to the traffic?
      • See your impact on the web site?

  • Why would artists and scientists collaborate?

    What we are really talking about is a new way to communicate information. Part of this must involve design and aesthetic concerns; part of it will require new mathematical models and methods.

    • Why involve a statistician?

      • Statistics is about data. It is about learning from data, extracting useful information from data.
      • Models or mathematical representations for data are at the heart of statistics. Models give us a way to identify important characteristics in a data set, and can offer clues as to how we might amplify them, draw them out, in an audio representation. Sometimes, it might be important to simply listen to the information left over in a data set after we have removed the effects of a model.
      • Many music theorists suggest that our experience of music has much to do with our expectation of what will come next, our prediction of the next note or phrase. Utilizing these kinds of ``predictive inferences'' is an intrinsically statistical exercise.

    • Why involve an artist?

      • An artist is not only interested in understanding or factually communicating data, but also conveying aspects of the wonder and amazement that these data exist at all; this is really a provocation, it is about surprising people; it is about creating a concrete point of contact with the data itself, so that the data become one step less a total abstraction to people. It is putting people face to face with this technological mirror
      • The audio must be aesthetic for people to fully grasp the character of the data being represented;
      • Data visualization has benefitted tremendously from good design principles that have essentially evolved from a basic understanding about how people consume graphical displays.

  • Future work

    Web sites, NOC, architectural applications

  • Preliminary experiments

    • Like many corporate web sites, www.lucent.com is an information source. The content is arranged heirarchically within the site. Top level areas describe the various business units within Lucent; and as you dig deeper into each, you find product information, etc.
    • Our initial audio representation expresses the depth that visitors are reaching into the site.
    • Accesses to high-level information contributed to overall background drones; one tone per area we considered.
    • More detailed information is captured by pulses that grow faster and louder in proportion to the number of people at that depth.

  • Statistics Research: Comments

    1. Statistics is an empirical science. It is a mathematical discipline, we build probabilistic models of the world.
    2. As a field, statistics is enriched by the applications we consider. Each new problem provides us with new insights about how to learn from data. The field advances by generalizing the lessons from particular applications, by constructing new tools that can be used on other problems. The newest challenges come from large complex problems that involve massive amounts of data. Hot application areas include data and telecommunication networks.
    3. So good statistics research is a balance between practical applications, theoretical investigations and methodological advances...tool building.
    4. ... and that's why i think bell labs is such a great place. the telecommunications industry has historically been an excellent source of challenging problems and complex data sets. Collaborations between different areas in the labs are encouraged. A lot of really good statistics research depends on borrowing tools from other branches of mathematics, computer science, signal processing...even information and coding theory.
    5. ... and in fact, this project is in some sense a good example of that. the use of sound seems to offer us a new way to experience complex, dynamic data. An experience that is difficult to achieve in large systems through visualization alone.
    6. Alternative: Statistics is a rigorous science with a heavy and healthy empirical component. As a mathematical field, it is based on the modern probabilitistic, rather than the conventional deterministic, thinking; It is enriched by the problems we solve. These problems historically have come from gambling, insurance, official census, genetics, industry quality control etc and now come from fields as diverse as finance, telecommunications, public policy, sociology, public health, and human genome project. It is actually very hard to find a field where statistics is not useful. Anyone with a good statistical training is a hot commodity in the market place and will continue to be that way for a long time to come.