Literary Programming [1] for the S [2] language using XML


Duncan Temple Lang

Table of Contents

  1. Motivating Literate Programming for S language
  2. Using XML
    1. Code chunks
    2. Code fragments for differences between SPlus & R
  3. Drawbacks
  4. Points.
  5. Issues & Notes
  6. Examples
  7. C and S language code
    1. Source'ing the function into S language
    2. Compiling the C routine
    3. Putting it all together
    4. Notes
  8. Bibliography
  9. Footnotes

Appendices

  1. XML/XSL-related URLs
  2. Using Xalan-C++
  3. Using Xalan-Java
  4. Perl-related XSL URLs
  5. Auxillary files

In this document we discuss some ideas for using XML to both define S language functions and expressions and simultaneously document them, providing examples, sample output, etc. Generally, XML provides an extensible mechanism for providing rich semantic information for both code and documentation, making it a powerful literate programming format. S's capability of reading XML make it easy to process the code in traditional and new ways, and also transform the documentation into a variety of different formats. Some of the drawbacks are also discussed. [3]

1. Motivating Literate Programming for S language

There are several difficulties associated with documenting S source code.

2. Using XML

When writing documents using TeX and LaTeX, many of us attempt to provide semantic information rather than appearance information by defining macros to specify the content type. For example, rather than using \textbf to render a file, we use a macro \file. This allows us to change how we render a "file" object by changing just the macro and also allows readers of the raw document to understand was meant.

XML is currently the "language of choice" for specifying structure in a document or data. One of the defining factors that make XML so useful is that it has become a de-factor standard.

Valid documents as defined by a DTD mean that one can guarantee that a document conforms to a particular, well-defined structure. While such requirements are not essential for documents mean to be read by a human, it is important when software is used to "interpret" the contents of a document.

One can think of using XML for document creation as writing in HTML, but specifying the intent with a collection of macros. Then the processing is done by an external engine, and in particular, can be done in different ways by many different engines, maximizing the use of the document. In a sense, the document is the model or data in the Model-View-Controller paradigm so often cited in software development literature. The different rendered versions, be they code for the computer or typeset documents, are views derived from the model. And the tools that manipulate the document by either extracting segments or modifying the content and structure are the controllers.

The fact that we can use different tools for creating multiple views from the same document is one of the key benefits of XML. The data in the document is application-neutral. It does not tie us to TeX, MS Word or any other text processing system. Indeed, we can use XSL to ``massage'' a document into a form appropriate for any text processor/typesetting system.

1. Code chunks

One of the primary features of the existing literate programming tools is the ability to break the overall code structure (e.g. a routine) into different chunks and to discuss these separately in the document but group them together correctly when compiling. For example, in noweb, one might specify an S language function as follows


 bar <-
 function(x, y) {
   <<Initialize Variables>>
   <<Iterate>>
   <<Cleanup results>>
 }

and then define the different chunks Initialize Variables, Iterate, etc. in a different part of the document.

The purpose of using chunks and chunk references is to leave the author free to document the code in the most suitable fashion for the reader, rather than being a slave to the format expected by the application that will read the code (e.g. the compiler, Perl, etc.). An additional side-effect is that one can re-use chunks in different places to effectively in-line code while maintaining a single version. In other words, it yields a more powerful macro facility than the regular C pre-processor.

To use XML for literate programming, we will need to support the same mechanism of having chunks and chunk references. We will call these fragments and fragment references to distinguish them from noweb and other LP tools (and also to give them a slightly nicer term.)

We declare a code fragment using the fragment tag. As with the code tag, we can specify a target language. The fragment then behaves just like a code element and should contain the code for that fragment.


<function lang="S">
xml2tex <- 
function(node, mappings=.XMLTexMappings) {
   n <- 10
   x <- <fragmentRef id="frag1" />
   print(x+10)
   x
}
</function>


<fragment id="frag1">
 seq(1,n)
 <fragmentRef id="frag2" />
</fragment>

<fragment id="frag2">
 + 10
</fragment>

We can process the code chunks ourself in S language or alternatively construct XSL sheets to perform the link resolution. The S language version is implemented in xmlSource() currently distributed as an example in the XML package.

Using XSL rules to do this is attractive as it is language neutral approach and will work for C, Perl, Python, S language, etc. Such XSL rules are slightly more complex than the standard transformations one uses in XSL. This reason for this is that we must look for the fragment referenced in a fragmentRef element and its i is not known when write the XSL rules, but is a fragment-specific value. We might be tempted to try to find the relevant fragment definition using the XPath [5] mechanism something along the lines of

 <xsl:template match="fragmentRef">
    <xsl:apply-templates select="//fragment[@id=@id]" />
 </xsl:template>

The idea here is that we want to use the id of the fragmentRef node being processed to find the fragment element with the corresponding tag. (We can also specify a language value.) This however will not work and in some XSLT processors will induce an infinite loop.

What we really need is a different approach. Take the example above. In the first fragmentRef, we know the name of the fragment for which we are looking and it is frag1. So, we could write an XSL template to resolve this as

 <xsl:template match="fragmentRef">
    <xsl:apply-templates select="//fragment[@id='frag1']" />
 </xsl:template>

Obviously, to be able to process more than just one single fragment we need more of these templates. And specifically, we need to be able to generate them dynamically as we encounter each fragmentRef in the document and have the XSLT respond accordingly.

We can create this dynamic template effect by using variables in xsl:template rules. The following rule stores the value of the id attribute of the fragmentRef in the local variable named A and then uses that in the value of the select attribute of the xsl:apply-templates

<xsl:template match="fragmentRef">
  <xsl:variable name="A"><xsl:value-of select="@id"/></xsl:variable>
  <xsl:apply-templates select="//fragment[@id=$A]" mode="resolve" />
</xsl:template>

2. Code fragments for differences between SPlus & R

Here we examine how we can use XML to handle the case where we have common S language code for both SPlus and R and have minor differences between them. For example, we might define as follows

     foo <-
     function(x) {
        ....
       <fragmentRef id="removeElement" />
        ....
     }

where the ... code is valid for both dialects but the <fragment> element identifies dialect-specific code. We might identify these in the document as

  <fragment id="removeElement" lang="SPlus">
    # SPlus-specific code
    v = get(".MyPersistentData", frame=0)
     v$name = "a new name"
    assign(".MyPersistentData", v, frame=0)
  </fragment>

and

  <fragment id="removeElement" lang="R">
     # R-specific code using closures
    v$name("a new name")
  </fragment>

The XML source facility in SPlus would know to resolve fragment reference by finding the reference by its id and then determining that there were two matching fragments. It would then determine which one to use by looking at the value of the lang attribute. Obviously, it would match the SPlus fragment whereas R would use the one intended for it.

3. Drawbacks

  • Verbosity
  • XML is very verbose and the plethora of markup details (tags and attributes) can obscure the content of a document when viewed in raw format. For this reason, the need for tools to view and manipulate XML documents is essential. However, editing modes such as PSGML and other tools, both graphical and textual, do make it feasible to operate on XML documents. One can manipulate a document in S and then save the result back to a file. Given the extensive support for XML in languages such as Perl, Python, Java, C, etc. one is seldom at a loss for tools with which one can manipulate and modify XML documents. The difficulty is learning about these tools.
  • Editor Code Modes
  • Those of us that use Emacs or other editors that provide specialized functionality for editing code in a given language are hesitant to give up these convenient, time-saving tools. Integrating this style of preparing code and documentation with existing tools is an important challenge to using XML for these purposes. Two things rapidly come to mind. Firstly, integrating with "modes" such as ESS, etc. would allow that tool to extract S code from an XML document and to send it to the S process. One could navigate the XML document using the PSGML-mode style bindings to be able to jump to different XML elements. Secondly, noweb and cweb illustrate a mechanism for switching modes within a document based on the content of a region. mmm and other modes will hopefully support this relatively easily and allow us to go further than noweb and cweb modes in having support for modes for more than two languages.
  • Spell Checking
  • XML provides a way to validate structure, but obviously cannot determine whether the content of each element is valid or not. Spell checkers, etc. are needed for this. Those who create HTML documents using regular editors know only too well how aggrevating its when a spell checker that prompts you for the spelling of H1, href, etc. A patch for ispell allows it to handle HTML. The same patch should be easily extended to XML.
  • Extra Processing Steps
  • Since code and documentation stored in XML cannot be used directly, one must process it into a an appropriate form before using it. It obviously takes time to perform the processing and this increases the edit-compile-test cycle of the development process. There are several XSLT processors, and some of these are faster than others. Xalan is written in C++ and others are written in Perl. These approaches yield transformations very quickly and do not add any considerable time to the overall compilation. Transformation or filtering speed becomes even less of an issue when one considers the capabilities by Xalan to "compile" XSL files into C/C++ code.

    XSLT processors developed in Java may not be suitable. This not because the Java interpreter and language are slow, but because the startup time of the virtual machine is lengthy.

    Also, while we are claiming it is feasible to develop filters that massage the content of an XML document into different forms, it is still does take some small quantity of time. Many people will decide that they do not have the time to spend learning this new tool and transitioning to it. This has long been the argument against literate programming and even learning new applications and languages. This is, of course, a legitimate view, but frequently it is legitimate at a moment in time, but less frequently over time. In other words, one may not be able to afford the time now. However, when one cumulates the different nows, it would have been more effective to spend some time and amortize the learning over different projects. I believe that this is especially true when documenting S functions. The tendency for the documentation and code to diverge is very real and natural. The utility of the code is increased disproportionately to the existence of any documentation that allows you, the author, to understand anything about the function when you return to it in weeks, months or even years. Using XML provides a mechanism for creating searchable libraries or archives of S code.

    4. Points.

    Language-specific fragments. Need id= and lang= but want the id to be unique?
  • Consistency
  • Keeps the code and the documentation closer together allowing one to document more dynamically rather than change, re-source, edit .Rd file.
  • Pre-processing Macros
  • Allows one to do pre-processing macros (i.e. entities).
  • Partial Source
  • One can read the entire file and then only ``evaluate'' or instantiate specific functions.
  • Efficient Partial Parsing
  • One can use the SAX parser to do the partial sourcing on very large files, if this becomes an issue.
  • Two Escape Mechanisms
  • One can use entities or the CDATA construct to escape text from the XML parser.
  • Scoping of Fragments
  • One can potentially define fragments within other fragments and effectively hide them from top-level chunks.

    5. Issues & Notes

  • Line numbers
  • Need to be able to get the line numbers in code associated with each code chunk so that we can use it in debugging. Can we get this from libxml, XSL, and other parsers or translators?
  • Escaping Characters
  • When escaping code from being parsed as XML (e.g. < in C or S language code), we can use the CDATA construct or leave it as regular, parsed XML and use entities.
  • XLink
  • Resolving references to other chunks, including in other files. Using the nano-hhtp.
  • XPath
  • We may want to add XPath facilities in S language for identifying node sets via matching rules specified in the XPath facilities.
  • Within an XSL file, one can include other XSL files. In this way, we can maintain and use XSL files separately to handle things such as S language, C and Perl code S language, documentation, articles, etc. We ensure consistency across output by using these different files cooperatively in different contexts.

    6. Examples

    Here is how we ca read an XML document into R.
    d <- xmlTreeParse("/home/duncan/Projects/org/omegahat/XML/RS/data/literate.xml")  
    
    eval(parse(text=d$doc[[3]][[1]][[2]][[2]] ))  
    
    Now, what if that chunk refers to other chunks.

    A simple way to read and source a file in the SSource that does not contain fragment references is
    simpleSource <- function(file) {
      r <- xmlRoot(xmlTreeParse(file))
      for(i in r[names(r) == "function"]) {
        name <- xmlValue(i[["name"]][[1]])   
        def <- xmlValue(i[["def"]][[1]])
        assign(name, eval(parse(text=def)), envir=globalenv())
      }
    }
    

    7. C and S language code

    In this section, we illusrate with an example how we can mix related S language and C in a single document for the purpose of documenting and yet separate these for the purpose of compliling and executing the code.

    For the purpose of this document, it is not important the reader understand the particulars of the example. The issue is how we define both an S function and a C routine and separate these. In fact, the example allows one to query the value for a particular process limit. The idea is that we have an S function that maps the name of the resource to an integer corresponding to the internal identifiers for the different limit types. This value is then passed to the C routine and the result returned. This value consists of a soft and hard limit.

    The idea is that we decide to write an S function that gives us access to querying the value of a limit. We start by defining such a function adding it to a file, say limit.xml. We start simply, adding some documentation and then the function definition.
    
    This function is used to get the current value of a
    resource limit for the process.
    One can ask  
    <ul>
    <li> how big a core file can be created
      by the process
    </li>
    <li> how many CPU seconds the process can consume,
    </li>
    <li> how large the data segment can become,
    </li>
    <li> the maximum number of child processes
    </li>
    <li> and so on</li>
    </ul>
    
    <p/>
    The user specifies which resource is of interest by giving
    its name. The function that translates this to an internal identifier.
    
    
    Now, we add the definition of the function to this file, just after the text describing what we are doing. We identify it as a function and indicate that it can be understood by any S dialect by specifying the lang attribute as %S;.
    
    <function lang="S">
     processLimit <-
     function(name)  {
      ids <- getProcessLimits()
      which <- match(name, names(ids))
      if(is.na(which))
        stop(paste("Unknown process limit", name))
    
      vals <- .Call("RS_getLimit", ids[which])
      names(vals) <- c("soft", "hard")
      vals
     }
    </function>
    
    
    Next, we define the C code associated with S language function. This is given something like the following. Again, we add some documentation and the define the routine.
    
    
    The basic idea here is that we call the <croutine>getrlimit</croutine>
    with the specified resource identifier and then put the elements of the
    resulting <cstruct>rlimit</cstruct> structure into a numeric vector of
    length 2.  We handle the case of an infinite limit by mapping this to
    R's symbolic constant representing &infty;
    
    <code lang="C">
    < ![ CDATA [
    #include <sys/time.h>
    #include <sys/resource.h>
    
    USER_OBJECT_
    RS_getLimit(USER_OBJECT_ resourceId)
    {
     struct rlimit limit;
     int res;
     
      res = INTEGER_DATA(resourceId)[0];
    
      getrlimit(res, &limit);
      ans = NEW_NUMERIC(2);
      NUMERIC_DATA(ans)[0] = limit.rlim_cur;
      NUMERIC_DATA(ans)[1] = limit.rlim_cur != RLIM_INFINITY ?  limit.rlim_cur : R_PosInf;
    
     return(ans);
    }
    ] ] > <!-- End CDATA enclosing the C code. -->
    </code>
    
    
    Note that we escape the contents of the code so that the &, <, and > are escaped correctly.

    1. Source'ing the function into S language

    Having defined the S language function and C code to implement the functionality to get the process limit, we need to be able to utilize it. So we need to be able to read the function into S and evaluate the assignment of the function definition. We will show how to do this using the basic XML parsing tools. Higher level functions can be easily built from these to simplify the process.
    
     d <- xmlRoot(xmlTreeParse("limit.xml"))
     funs <- xmlApply(d, function(x) if(xmlName(x) == "function" &&
                                           xmlAttrs(x)["lang"] =="S")
                                       x
                                     else
                                       NULL
              )
    
      apply(funs, function(x, env) {
                       eval(parse(xmlValue(x)), envir=env)
                  }, env = environment())
    
    

    2. Compiling the C routine

    Next we want to get the C code from the document and create a .c file. We have to extract the contents of the code tag and put these into the C file. There are a variety of ways to do this. We might use an XSL stylesheet such as
    
    <xsl>
    <xsl:template match="code[@lang='C']">
      <xsl:value-of select="." />
    </xsl:template>  
    </xsl>
    
    
    This poses a slight difficulty in that the result of an XSL translation must be a valid XML document, and is therefore not a legal C file. We can remove the
    
       <?xml .. ?>
    
    
    element at the top of the document using Perl. However, we are still left with the issue of ensuring that the special characters such as <, > and & are not escaped as entity references, but expanded to their values. Again, we can do this in a Perl script. Assuming we know all possible entities, we can expand them at this point.

    As one might expect, this is a common enough problem that XSL provides a solution. One can specify the output type for the result of the XSL transformation to be something other than XML. The frequently supported ones are HTML and text. Additionally, we can tell the XSL processor not to emit the XML declaration. See croutines.xsl and http://www.ibiblio.org/xml/books/bible/updates/14.html. The command line arguments to Xalan also allow one to control these issues without changing the XSL file.

    Alternatively, we can write a special C/C++ application that process the XML file and expands entities, resolves fragment links, etc.

    3. Putting it all together

    We can now put all of this together in an automatic fashion that is very similar to the way that we develop code using standard (non-literate) approaches. We start by writing make rules to control the creation of the different input files. The two files derived from the literate source (limit.xml) are limit.c and limit.S. Both of these use the code extraction XSL file croutine.xsl. We specify the language for which the code should be extracted using the -param supported by Xalan.

    The remaining rules are as they would be in the absence of literate programming. They simply compile and link the C code into a shared library that R can load.
    
    limit.c: limit.xml croutine.xsl
    	($(XSLT_PROCESSOR) -in limit.xml -xsl croutine.xsl > $@)
    
    limit.S: limit.xml croutine.xsl
    	($(XSLT_PROCESSOR) -in limit.xml -xsl croutine.xsl -param language "'S'" > $@)
    
    limit.o: limit.c
    	$(CC) -I${OMEGA_HOME}/include/Corba -I${R_HOME}/include -D_R_=1 -DUSE_R=1  -c $<
    
    limit.so: limit.o
    	R CMD SHLIB -o $@ limit.o
    
    examples:
    	($(XSLT_PROCESSOR) -in limit.xml -xsl examples.xsl > $@)
    
    
    Now, we can use the rules to create and compile the different files. The following commands apply these make rules.
     make limit.so
     make limit.S
     make examples
     R
    
    Now, we run R and source the S language functions, load the C routines and execute the examples.
     source("limit.S")
     dyn.load("limit.so")
     source("examples")
    

    4. Notes

    1. Rather than having generic code chunks, we can mark these up to be C routine, S function, code snippet, etc. This would facilitate extraction and improve semantic meaning over the regular web setup.
    2. We could try using entities for referring to chunks/fragments.

    1. To Escape or not

    It is often more convenient to include code sections within a CDATA construct so that it is passed verbatim to the processor. This allows us to write it as we would write outside of XML so that it can be used directly with the system which will process it, e.g. R, SPlus, C compiler, Perl, etc. However, we can also mark it up. By treating it as verbatim text, we cannot mark it up to pretty print it or understand it in any way at the XML level. This is probably the most practical approach. However, the work we have started regarding writing S language objects in XML for peristent storage and exchange shows that one can markup internal objects using XML tags and then we can format it in HTML, TeX, etc. and produce nicer output.

    For example, consider a simple function
    
     f <-
     function(x,y)  {
       par(mfrow=c(2, 2))
       plot(x, y)
       hist(x)
       hist(y)
       plot(y, x)
     }
    
    
    Now, the xmlWrite() function and methods in S language would (currently) write this out as
    
    <function>
    <args length="2">
    <arg name="x">
    </arg>
    <arg name="y">
    </arg>
    </args>
    <body length="5">
    <call>
    <symbol>
    par 
    </symbol>
    <call>
    <symbol>
    c 
    </symbol>
    <numeric>
    2 
    </numeric>
    <numeric>
    2 
    </numeric>
    </call>
    </call>
    <call>
    <symbol>
    plot 
    </symbol>
    <symbol>
    x 
    </symbol>
    <symbol>
    y 
    </symbol>
    </call>
    <call>
    <symbol>
    hist 
    </symbol>
    <symbol>
    x 
    </symbol>
    </call>
    <call>
    <symbol>
    hist 
    </symbol>
    <symbol>
    y 
    </symbol>
    </call>
    <call>
    <symbol>
    plot 
    </symbol>
    <symbol>
    y 
    </symbol>
    <symbol>
    x 
    </symbol>
    </call>
    </body>
    </function>
    
    
    Now, we can then mark this up to display in a convenient fashion with color coding of the different language elements, links to the help pages for the function names that are called, and so on.

    2. Editing XML

    There are a variety of different tools and applications for editing XML. I tend to use PSGML mode for emacs. This provides a variety of different convenience facilities. It parses the DTD associated with the document (i.e. in the DOCTYPE element) and then is capable of determining the set of valid sub-elements and attributes within the current element. This allows it to provide automatic completion of tags and attributes. Additionally, PSGML mode allows for displaying different parts of the XML tree to simplify visualiztion of the content. For example, it allows folding and hiding of sections and provides an outline mode.

    It would be convenient if PSGML mode could display the value of an entity in the display to the user, but store it internally in the model of the document as an entity. This would allow us to see the <- symbol but have it represented as &sgets;

    Also, if font lock could be adapted to the particulars of the document being edited, one could see the different elements and the structure more clearly.

    One of the most important facility that we need in an editor for use with literate programming for S is the ability to support multiple modes. In other words, we want to be able to edit documentation using HTML mode, S code using S-mode facilities with the associated key bindings and syntax-highlighting, C code in c-mode; and Perl, Python etc. using the different modes for each of these.

    Bibliography

    1. http://www.oasis-open.org/cover/xmlLitProg.html#litProgXML
    2. Xalan
    3. The XSL companion, Neil Bradley, 2000
    4. XML Elements of Style, Simon St. Laurent, 2000

    Appendix A. XML/XSL-related URLs

    Appendix B. Using Xalan-C++

    setenv LD_LIBRARY_PATH /tmp/xml-xalan/c/lib
    /tmp/xml-xalan/c/bin/testXSLT -in literateS.xml -xsl literateS.xsl > foo.html 
    

    One can specify values for parameters to be used in the stylesheet. For example, one can control whether line numbers are used when formatting S functions via the Sfunction.xsl stylesheet. One defines a parameter with a default value in the stylesheet. Users of that stylesheet can then supply a value for the parameter via the -param flag. This takes a pair of values, one after the other on the command line. The first is the name of the parameter. The second is the value. If this is a string, it should be quoted. testXSLT -in Sfunction.xml -xsl Sfunction.xsl -param useLineNumbers '"true"'

    Appendix C. Using Xalan-Java

    To process an XML document with a given XSL style using the Java version of xalan, use the following.
    setenv CLASSPATH /usr/local/src/xalan-j_2_0_D01/bin/xalan.jar:/usr/local/src/xalan-j_2_0_D01/bin/xerces.jar
    
    java org.apache.xalan.xslt.Process -in data/literate.xml -xsl data/literate.xsl
    

    Appendix D. Perl-related XSL URLs

    Appendix E. Auxillary files

  • literateS.xsl
  • The XSL rules for converting this file to HTML.
  • literateS.xml
  • The XML file from which the HTML is generated.
  • literateS.html
  • The result of applying literateS.xsl to literateS.xml.
  • fragment.xsl
  • XSL rules for extracting code containing fragment references.
  • fragment.xml
  • XML file containing code with fragment references.
  • Sfunction.xsl
  • Initial version of an XSL file for displaying S objects as written in XML format by writeXML() .
  • Sfunction.xml
  • An XML file containing the output of an R function object marked up with XML tags for the different language elements.
  • croutine.xsl
  • XSL file for extracting C code elements from an XML file so they can be put in a source file and compiled.
  • limit.xml
  • Contains the description and code for some C code
  • limit.c
  • The C file created by processing the limit.xml file.
  • GNUmakefile
  • The makefile for creating the different files. One has to edit it to specify the correct settings for the testXSLT
  • literateS.tar.gz
  • A collection of these files for those who want to experiment with them.

    Footnotes

    1. The ideas here are more commonly known as "literate programming", but we use a slightly less emmotive term.
    2. As in most of our documents, we use the term S to refer to the S language and refer specifically to the different implementations of this language as SPlus and R.
    3. This document can be converted to HTML by applying the XSL file literateS.xsl to literateS.xsl.
    4. by surrounding the irrelevant functions within an
      if(F) {...}
      clause.
    5. The language used to specify nodes in the DOM via the "regular expression"-like language used in the select atributes of the different xls tags.


    Document generated by
       export LD_LIBRARY_PATH=/tmp/xml-xalan/c/lib
       /tmp/xml-xalan/c/bin/testXSLT -in literateS.xml -xsl literateS.xsl > literateS.html
     
    or
       CLASSPATH /usr/local/src/xalan-j_2_0_D01/bin/xalan.jar:/usr/local/src/xalan-j_2_0_D01/bin/xerces.jar
       java org.apache.xalan.xslt.Process -in literateS.xml -xsl literateS.xsl > literateS.html
     

    Copyright the Omega Project for Statistical Computing, 2000