Distributed Computing with Data
A long term project that we are working on in Bell Labs
involve communication between different statistical
tools to share capabilities and allow concurrent processing.
These two goals are achieved with a single mechanism
but are quite distinct.
The first allows simple reuse of (existing)
software.
Essentially, we are providing a piping mechanism
between applications that extends the UNIX pipe idea
by allowing multi-directional
and dynamic connections.
The second goal allows us to harness multiple processors
to improve performance of simulations and analysis of large
datasets, etc.
There are three different levels at which we might be able to
communicate:
- process to process involving lower-level languages (C/C++ and
Java)
- high level language to high level language - e.g.
evaluating commands in one language from the "command
line" of another. For example, in S we might evaluate
the commands in Matlab to produce a plot.
This is sufficient for high performance computing.
- direct invocation of methods for objects in high level
languages.
In this way, one can write code in one's favorite language
and publish objects so that others can invoke methods on them
supplying both their own local data or other remote objects.
This is the goal which will allow users of
different statistical environments to work as if they were one.
In this way, the data and methods can be distributed across
machines and languages and could be available across the web
with little or no changes to the development or user
interaction.
Recent Work and Information
More recent information is available
here
including our talk at
Interface98.
(Note - this page is about to undergo surgery
and will probably be re-rooted to
here.)
Previous Work
The work extends/replaces the
S/Java interface
developed previously. This uses the S object ASCII format to
communicate data and objects between clients.
We are separating the content of the message from the communication
mechanism to allow different scenarios to be easily integrated.
Additionally, we allow other message types to be transmitted
in some manner, agreed upon by the peers involved in the
communication.
Two different approaches we are investigating
for the communication mechanism are
- Corba
- PVM
PVM
Essentially, PVM is a lower level communication mechanism that
removes the primitive socket and shared memory details
of Inter-process communication (IPC) from us but does not
constrain us in any way since it offers few other mechanisms.
Corba on the other hand is more structured interface to communication
that offers many more services at the potential expense of
added complexity and some minimal constraints on how we approach
matters.
Our choice of system to employ will be
strongly influenced by the facility for other developers
to interface to our work, adding clients to the distributed environment.
Duncan Temple Lang<duncan@research.bell-labs.com>
Last modified: Sun May 17 17:57:32 EDT 1998