An S to Database Inteface

Many of the projects we are involved with here in Bell Labs involve data that are maintained in SQL-compliant databases. Traditionally, tools for statistical analysis have required that the data be initially presented to them in flat ASCII files or some other processed version of the original data source. We are attempting to expand our horizons and allow direct and dynamic communication with databases from S and perhaps other statistical tools. The intent is to reduce the user's managerial responsiblities and complexities by requiring only one version of the data to be stored. This becomes an important issue in a dynamic system when the data collection process is continuous and asynchronous. Also, in the spirit of our distributed computing project, the connection between the statistical tool and the database allows for certain computations (e.g. subsetting) to be performed in the database. This has the potential to allow the statistical tools to work with larger data sets than previously possible. While this can also be achieved by having the return values from the database arrive as a stream so that the receiver does not hold the entire object at a single time. Instead, the processing is performed element-wise. The approach of performing some computations in the remotely, however, is a simpler approach requiring less modifications to existing algorithms.

Java and the JDBC package

There are at least two obvious ways to implement this functionality for a language such as S. The first is to use a C interface from within S (dynamically loaded as a chapter) which provides the low-level SQL functionality by communicating directly with one or more databases. The second approach uses a separate process as an interface between the S level tasks and the database(s). We have choosen to investigate the second of these approaches. The reasons for this include:
  1. JDBC provides a simple interface to a range of SQL databases each of which would require minor context changes in a C level inteface.
  2. The interface is easily created and maintained as much of the code exists and requires simple connections to be established.

Implementation

Our proposed implementation exploits the general distributed computing environment and an extraordinarily general way to provide access to existing code in both S and Java with little or no additional code. Really! The way we do this is to use each of the following:
Java Class Reflectance
This allows us to generate the IDL describing existing Java classes (i.e. the JDBC classes), identifying the different methods and attributes. This can be inserted directly into the interface repository and from there stubs and skeletons generated. This is implemented by a single application that is similar to the java2idl/Caffeine application from Visigenic
CORBA's Dynamic Skeleton Interface
This allows us to provide a single generic Java class which provides the functionality of an arbitrary Java object to CORBA clients with no additional code. The class converts CORBA requests into method invocations on the associated object by identifying the Java method and converting the arguments. Java Reflectance is again used to determine the method and construct the call. Essentially we are using the generic class as a primitive Java interpreter. (Note that the same mechanism allows us to provide remote access to S (and generally interpreted languages such as XLisp-stat, etc.) functions and objects acting as CORBA servers.)
CORBA's Dynamic Invocation Interface
This is used on the S side to convert calls to functions in S on a remote database object into corresponding CORBA remote method invocations. (e.g. S > plot(x,a,b,c) expands to a remote call to x.plot(a,b,c).) This and interrogation of the available IDL classes in the CORBA interface repository allows us to generate S methods for the remote methods on CORBA object references.
These three components allow us to utilize general remote mechanisms in Java and S via the CORBA infrastructure and require us to create only the Java application that creates the required remote objects/servers.

The basic idea is to interface to the simple but general parts of JDBC and allow arbitrary string commands be sent from a client to the Java inteface client (see figure above).


Duncan Temple Lang<duncan@research.bell-labs.com>
Last modified: Fri May 29 19:22:58 EDT 1998