S-Net:  An Internet Traffic Measurement and Analysis System

Packet header collection can result in a very large database because packets come across Internet wires continuously at high speed. Even if the total throughput on a wire is small, the data grow and grow, so the database eventually gets large if collection continues. The success of analyzing Internet traffic data depends heavily on an ability to intensively analyze the traffic database in great detail.

To cope with very large traffic databases, we developed S-Net a traffic measurement and analysis system that begins with packet header collection on network wires, and ends with data analysis on a cluster of linux PCs running S, a language and system for organizing, visualizing, and analyzing data.

S-Net has worked quite well. Because the PCs and switches can be inexpensive and linux is free, the cluster has a low overall cost. The cluster architecture scales readily; in our case, PCs and disks have been added and replaced incrementally as our database has grown. The S flow objects vary according to the specific analysis tasks; each is designed to enhance computational performance and to make the S commands that carry out the analysis as simple as possible. S is well suited to the task of analyzing Internet traffic data; its elegant design, which won it the ACM Software System Award for 1999, allows very rapid development of new tools.

Flow Processing Tools:

To convert from raw packet header files obtained by tcpdump to flow summary files and S/Splus objects, we developed a set of tools based on shell and Perl scripts.

 

Home   Papers   Visualization   S-Net