Distributed Computing with Data

A long term project that we are working on in Bell Labs involve communication between different statistical tools to share capabilities and allow concurrent processing. These two goals are achieved with a single mechanism but are quite distinct. The first allows simple reuse of (existing) software. Essentially, we are providing a piping mechanism between applications that extends the UNIX pipe idea by allowing multi-directional and dynamic connections. The second goal allows us to harness multiple processors to improve performance of simulations and analysis of large datasets, etc.

There are three different levels at which we might be able to communicate:

  1. process to process involving lower-level languages (C/C++ and Java)
  2. high level language to high level language - e.g. evaluating commands in one language from the "command line" of another. For example, in S we might evaluate the commands in Matlab to produce a plot. This is sufficient for high performance computing.
  3. direct invocation of methods for objects in high level languages. In this way, one can write code in one's favorite language and publish objects so that others can invoke methods on them supplying both their own local data or other remote objects. This is the goal which will allow users of different statistical environments to work as if they were one. In this way, the data and methods can be distributed across machines and languages and could be available across the web with little or no changes to the development or user interaction.

Recent Work and Information

More recent information is available here including our talk at Interface98.

(Note - this page is about to undergo surgery and will probably be re-rooted to here.)

Previous Work

The work extends/replaces the S/Java interface developed previously. This uses the S object ASCII format to communicate data and objects between clients. We are separating the content of the message from the communication mechanism to allow different scenarios to be easily integrated. Additionally, we allow other message types to be transmitted in some manner, agreed upon by the peers involved in the communication.

Two different approaches we are investigating for the communication mechanism are

  1. Corba
  2. PVM

PVM

Essentially, PVM is a lower level communication mechanism that removes the primitive socket and shared memory details of Inter-process communication (IPC) from us but does not constrain us in any way since it offers few other mechanisms. Corba on the other hand is more structured interface to communication that offers many more services at the potential expense of added complexity and some minimal constraints on how we approach matters. Our choice of system to employ will be strongly influenced by the facility for other developers to interface to our work, adding clients to the distributed environment.
Duncan Temple Lang<duncan@research.bell-labs.com>
Last modified: Sun May 17 17:57:32 EDT 1998