next up previous contents
Next: 12.1 Pathname protocol encoding Up: CM The SML/NJ Compilation Previous: 11.3.3 Locally declared suffixes   Contents


12. Parallel and distributed compilation

To speed up recompilation of large projects with many ML source files, CM can exploit parallelism that is inherent in the dependency graph. Currently, the only kind of operating system for which this is implemented is Unix ( OPSYS_UNIX), where separate processes are used. From there, one can distribute the work across a network of machines by taking advantage of the network file system and the ``rsh'' facility.

To perform parallel compilations, one must attach compile servers to the currently running CM session. This is done using function CM.Server.start, which has the following type:


  structure Server : sig
      type server
      val start : { name: string,
                    cmd: string * string list,
                    pathtrans: (string -> string) option,
                    pref: int } -> server option
  end

Here, name is an arbitrary string that is used by CM when issuing diagnostic messages concerning the server12 and cmd is a value suitable as argument to Unix.execute.

The program to be specified by cmd should be another instance of CM--running in ``slave mode''. To start CM in slave mode, start sml with a single command-line argument of @CMslave. For example, if you have installed in /path/to/smlnj/bin/sml, then a server process on the local machine could be started by


  CM.Server.start { name = "A", pathtrans = NONE, pref = 0,
                    cmd = ("/path/to/smlnj/bin/sml",
                           ["@CMslave"]) };

To run a process on a remote machine, e.g., ``thatmachine'', as compute server, one can use ``rsh''.13 Unfortunately, at the moment it is necessary to specify the full path to ``rsh'' because Unix.execute (and therefore CM.Server.start) does not perform a PATH search. The remote machine must share the file system with the local machine, for example via NFS.


  CM.Server.start { name = "thatmachine",
                    pathtrans = NONE, pref = 0,
                    cmd = ("/usr/ucb/rsh",
                           ["thatmachine",
                            "/path/to/smlnj/bin/sml",
                            "@CMslave"]) };

You can start as many servers as you want, but they all must have different names. If you attach any servers at all, then you should attach at least two (unless you want to attach one that runs on a machine vastly more powerful than your local one). Local servers make sense on multi-CPU machines: start as many servers as there are CPUs. Parallel make is most effective on multiprocessor machines because network latencies can have a severely limiting effect on what can be gained in the distributed case. (Be careful, though. Since there is no memory-sharing to speak of between separate instances of sml, you should be sure to check that your machine has enough main memory.)

If servers on machines of different power are attached, one can give some preference to faster ones by setting the pref value higher. (But since the pref value is consulted only in the rare case that more than one server is idle, this will rarely lead to vastly better throughput.) All attached servers must use the same architecture-OS combination as the controlling machine.

In parallel mode, the master process itself normally does not compile anything. Therefore, if you want to utilize the master's CPU for compilation, you should start a compile server on the same machine that the master runs on (even if it is a uniprocessor machine).

The pathtrans argument is used when connecting to a machine with a different file-system layout. For local servers, it can safely be left at NONE. The ``path transformation'' function is used to translate local path names to their remote counterparts. This can be a bit tricky to get right, especially if the machines use automounters or similar devices. The pathtrans functions consumes and produces names in CM's internal ``protocol encoding'' (see Section 12.1).

Once servers have been attached, one can invoke functions like CM.recomp, CM.make, and CM.stabilize. They should work the way the always do, but during compilation they will take advantage of parallelism.

When CM is interrupted using Control-C (or such), one will sometimes experience a certain delay if servers are currently attached and busy. This is because the interrupt-handling code will wait for the servers to finish what they are currently doing and bring them back to an ``idle'' state first.



Subsections
next up previous contents
Next: 12.1 Pathname protocol encoding Up: CM The SML/NJ Compilation Previous: 11.3.3 Locally declared suffixes   Contents
Matthias Blume
2001-07-19