In this document we discuss some ideas for using XML to both define S language functions and expressions and simultaneously document them, providing examples, sample output, etc. Generally, XML provides an extensible mechanism for providing rich semantic information for both code and documentation, making it a powerful literate programming format. S's capability of reading XML make it easy to process the code in traditional and new ways, and also transform the documentation into a variety of different formats. Some of the drawbacks are also discussed. [3]
source() operates on an entire file,
reading and processing all of its contents.
This is a rather blunt mechanism when maintaining
multiple functions in a single file. One cannot reload
just one function from that file without updating the others
or modifying the file
[4]
For example, one might be in the process of developing
two functions in the same file but want to read the latest
definition of only one. (The other may be in an intermediate state
of development in which it does not actually work and one wants to
use the existing version within the R session.)
With XML, we can read an entire file without evaluating its
contents but determining its semantic information. Then, we can
extract just the relevant sections and so exclude those that we do
not want.
#ifdef statements in C.
Others have developed Perl
scripts to handle the differences across the different S language
dialects. The pre-processor provides a mechanism for C.
The marriage of literate programming which uses chunks
in a "macro substitution"-like manner and XML allows us to
do this relatively easily.
Specifically, we may write a segement of code that is
shared between S language and R and in it include code
that cannot be shared. Rather than actually including
the code itself, we can simply refer to it by name.
A processing system that inserts the correct code
depending on the dialect for which it is being processed
will allow one to more readily support minor, but important,
differences between systems. See code
fragments below.
Literate programming for any language is a good idea. It is difficult for people to say that literate programming is a bad idea, but the drawbacks of added processing, additional tools and complexity are valid complaints. We hope to minimize these drawbacks and make it easy enough to use in practice so that the benefits outweigh the drawbacks.
XML is currently the "language of choice" for specifying structure in a document or data. One of the defining factors that make XML so useful is that it has become a de-factor standard.
Valid documents as defined by a DTD mean that one can guarantee that a document conforms to a particular, well-defined structure. While such requirements are not essential for documents mean to be read by a human, it is important when software is used to "interpret" the contents of a document.
One can think of using XML for document creation as writing in HTML, but specifying the intent with a collection of macros. Then the processing is done by an external engine, and in particular, can be done in different ways by many different engines, maximizing the use of the document. In a sense, the document is the model or data in the Model-View-Controller paradigm so often cited in software development literature. The different rendered versions, be they code for the computer or typeset documents, are views derived from the model. And the tools that manipulate the document by either extracting segments or modifying the content and structure are the controllers.
The fact that we can use different tools for creating multiple views from the same document is one of the key benefits of XML. The data in the document is application-neutral. It does not tie us to TeX, MS Word or any other text processing system. Indeed, we can use XSL to ``massage'' a document into a form appropriate for any text processor/typesetting system.
if(x < 1)
x <- 10
How can the parser recognize the < and the <- as being
S code and the start of an XML element identifier.
The answer is simply that it cannot. Hence, we must tell it so.
XML provides two simple and consistent ways to escape text from being interpreted as XML:
bar <-
function(x, y) {
<<Initialize Variables>>
<<Iterate>>
<<Cleanup results>>
}
and then define the different chunks
Initialize Variables,
Iterate, etc.
in a different part of the document.
The purpose of using chunks and chunk references is to leave the author free to document the code in the most suitable fashion for the reader, rather than being a slave to the format expected by the application that will read the code (e.g. the compiler, Perl, etc.). An additional side-effect is that one can re-use chunks in different places to effectively in-line code while maintaining a single version. In other words, it yields a more powerful macro facility than the regular C pre-processor.
To use XML for literate programming, we will need to support the same mechanism of having chunks and chunk references. We will call these fragments and fragment references to distinguish them from noweb and other LP tools (and also to give them a slightly nicer term.)
We declare a code fragment using the fragment tag. As with the code tag, we can specify a target language. The fragment then behaves just like a code element and should contain the code for that fragment.
<function lang="S">
xml2tex <-
function(node, mappings=.XMLTexMappings) {
n <- 10
x <- <fragmentRef id="frag1" />
print(x+10)
x
}
</function>
<fragment id="frag1"> seq(1,n) <fragmentRef id="frag2" /> </fragment> <fragment id="frag2"> + 10 </fragment>
We can process the code chunks ourself in S language or alternatively construct XSL sheets to perform the link resolution. The S language version is implemented in
xmlSource() currently distributed as
an example in the XML
package.
Using XSL rules to do this is attractive as it is language neutral approach and will work for C, Perl, Python, S language, etc. Such XSL rules are slightly more complex than the standard transformations one uses in XSL. This reason for this is that we must look for the fragment referenced in a fragmentRef element and its i is not known when write the XSL rules, but is a fragment-specific value. We might be tempted to try to find the relevant fragment definition using the XPath [5] mechanism something along the lines of
<xsl:template match="fragmentRef">
<xsl:apply-templates select="//fragment[@id=@id]" />
</xsl:template>
The idea here is that we want to use the
id
of the fragmentRef node being processed to find the
fragment element with the corresponding tag. (We can
also specify a language value.) This however will not work and in some
XSLT processors will induce an infinite loop.
What we really need is a different approach. Take the example above. In the first fragmentRef, we know the name of the fragment for which we are looking and it is
frag1. So, we could write an XSL template to resolve this
as
<xsl:template match="fragmentRef">
<xsl:apply-templates select="//fragment[@id='frag1']" />
</xsl:template>
Obviously, to be able to process more than just one single
fragment we need more of these templates. And
specifically, we need to be able to generate them dynamically as we
encounter each fragmentRef in the document and have the
XSLT respond accordingly.
We can create this dynamic template effect by using variables in xsl:template rules. The following rule stores the value of the id attribute of the fragmentRef in the local variable named A and then uses that in the value of the select attribute of the xsl:apply-templates
<xsl:template match="fragmentRef"> <xsl:variable name="A"><xsl:value-of select="@id"/></xsl:variable> <xsl:apply-templates select="//fragment[@id=$A]" mode="resolve" /> </xsl:template>
foo <-
function(x) {
....
<fragmentRef id="removeElement" />
....
}
where the ... code is valid for both dialects
but the <fragment> element identifies
dialect-specific code.
We might identify these in the document as
<fragment id="removeElement" lang="SPlus">
# SPlus-specific code
v = get(".MyPersistentData", frame=0)
v$name = "a new name"
assign(".MyPersistentData", v, frame=0)
</fragment>
and
<fragment id="removeElement" lang="R">
# R-specific code using closures
v$name("a new name")
</fragment>
The XML source facility in SPlus would know to resolve fragment
reference by finding the reference by its
id
and then determining that there were two
matching fragments. It would then determine which one to use by
looking at the value of the
lang
attribute.
Obviously, it would match the SPlus fragment whereas R would use
the one intended for it.
XSLT processors developed in Java may not be suitable. This not because the Java interpreter and language are slow, but because the startup time of the virtual machine is lengthy.
Also, while we are claiming it is feasible to develop filters that massage the content of an XML document into different forms, it is still does take some small quantity of time. Many people will decide that they do not have the time to spend learning this new tool and transitioning to it. This has long been the argument against literate programming and even learning new applications and languages. This is, of course, a legitimate view, but frequently it is legitimate at a moment in time, but less frequently over time. In other words, one may not be able to afford the time now. However, when one cumulates the different nows, it would have been more effective to spend some time and amortize the learning over different projects. I believe that this is especially true when documenting S functions. The tendency for the documentation and code to diverge is very real and natural. The utility of the code is increased disproportionately to the existence of any documentation that allows you, the author, to understand anything about the function when you return to it in weeks, months or even years. Using XML provides a mechanism for creating searchable libraries or archives of S code.
d <- xmlTreeParse("/home/duncan/Projects/org/omegahat/XML/RS/data/literate.xml")
eval(parse(text=d$doc[[3]][[1]][[2]][[2]] ))
Now, what if that chunk refers to other chunks.
A simple way to read and source a file in the SSource that does not contain fragment references is
simpleSource <- function(file) {
r <- xmlRoot(xmlTreeParse(file))
for(i in r[names(r) == "function"]) {
name <- xmlValue(i[["name"]][[1]])
def <- xmlValue(i[["def"]][[1]])
assign(name, eval(parse(text=def)), envir=globalenv())
}
}
For the purpose of this document, it is not important the reader understand the particulars of the example. The issue is how we define both an S function and a C routine and separate these. In fact, the example allows one to query the value for a particular process limit. The idea is that we have an S function that maps the name of the resource to an integer corresponding to the internal identifiers for the different limit types. This value is then passed to the C routine and the result returned. This value consists of a soft and hard limit.
The idea is that we decide to write an S function that gives us access to querying the value of a limit. We start by defining such a function adding it to a file, say limit.xml. We start simply, adding some documentation and then the function definition.
This function is used to get the current value of a resource limit for the process. One can ask <ul> <li> how big a core file can be created by the process </li> <li> how many CPU seconds the process can consume, </li> <li> how large the data segment can become, </li> <li> the maximum number of child processes </li> <li> and so on</li> </ul> <p/> The user specifies which resource is of interest by giving its name. The function that translates this to an internal identifier.Now, we add the definition of the function to this file, just after the text describing what we are doing. We identify it as a function and indicate that it can be understood by any S dialect by specifying the lang attribute as %S;.
<function lang="S">
processLimit <-
function(name) {
ids <- getProcessLimits()
which <- match(name, names(ids))
if(is.na(which))
stop(paste("Unknown process limit", name))
vals <- .Call("RS_getLimit", ids[which])
names(vals) <- c("soft", "hard")
vals
}
</function>
Next, we define the C code associated with S language function.
This is given something like the following.
Again, we add some documentation and the define the routine.
The basic idea here is that we call the <croutine>getrlimit</croutine>
with the specified resource identifier and then put the elements of the
resulting <cstruct>rlimit</cstruct> structure into a numeric vector of
length 2. We handle the case of an infinite limit by mapping this to
R's symbolic constant representing &infty;
<code lang="C">
< ![ CDATA [
#include <sys/time.h>
#include <sys/resource.h>
USER_OBJECT_
RS_getLimit(USER_OBJECT_ resourceId)
{
struct rlimit limit;
int res;
res = INTEGER_DATA(resourceId)[0];
getrlimit(res, &limit);
ans = NEW_NUMERIC(2);
NUMERIC_DATA(ans)[0] = limit.rlim_cur;
NUMERIC_DATA(ans)[1] = limit.rlim_cur != RLIM_INFINITY ? limit.rlim_cur : R_PosInf;
return(ans);
}
] ] > <!-- End CDATA enclosing the C code. -->
</code>
Note that we escape the contents of the code
so that the &, <, and > are escaped correctly.
d <- xmlRoot(xmlTreeParse("limit.xml"))
funs <- xmlApply(d, function(x) if(xmlName(x) == "function" &&
xmlAttrs(x)["lang"] =="S")
x
else
NULL
)
apply(funs, function(x, env) {
eval(parse(xmlValue(x)), envir=env)
}, env = environment())
<xsl> <xsl:template match="code[@lang='C']"> <xsl:value-of select="." /> </xsl:template> </xsl>This poses a slight difficulty in that the result of an XSL translation must be a valid XML document, and is therefore not a legal C file. We can remove the
<?xml .. ?>element at the top of the document using Perl. However, we are still left with the issue of ensuring that the special characters such as <, > and & are not escaped as entity references, but expanded to their values. Again, we can do this in a Perl script. Assuming we know all possible entities, we can expand them at this point.
As one might expect, this is a common enough problem that XSL provides a solution. One can specify the output type for the result of the XSL transformation to be something other than XML. The frequently supported ones are HTML and text. Additionally, we can tell the XSL processor not to emit the XML declaration. See croutines.xsl and http://www.ibiblio.org/xml/books/bible/updates/14.html. The command line arguments to Xalan also allow one to control these issues without changing the XSL file.
Alternatively, we can write a special C/C++ application that process the XML file and expands entities, resolves fragment links, etc.
The remaining rules are as they would be in the absence of literate programming. They simply compile and link the C code into a shared library that R can load.
limit.c: limit.xml croutine.xsl
($(XSLT_PROCESSOR) -in limit.xml -xsl croutine.xsl > $@)
limit.S: limit.xml croutine.xsl
($(XSLT_PROCESSOR) -in limit.xml -xsl croutine.xsl -param language "'S'" > $@)
limit.o: limit.c
$(CC) -I${OMEGA_HOME}/include/Corba -I${R_HOME}/include -D_R_=1 -DUSE_R=1 -c $<
limit.so: limit.o
R CMD SHLIB -o $@ limit.o
examples:
($(XSLT_PROCESSOR) -in limit.xml -xsl examples.xsl > $@)
Now, we can use the rules to create and compile the different
files. The following commands apply these make rules.
make limit.so make limit.S make examples RNow, we run R and source the S language functions, load the C routines and execute the examples.
source("limit.S")
dyn.load("limit.so")
source("examples")
For example, consider a simple function
f <-
function(x,y) {
par(mfrow=c(2, 2))
plot(x, y)
hist(x)
hist(y)
plot(y, x)
}
Now, the xmlWrite()
function and methods in S language would (currently)
write this out as
<function> <args length="2"> <arg name="x"> </arg> <arg name="y"> </arg> </args> <body length="5"> <call> <symbol> par </symbol> <call> <symbol> c </symbol> <numeric> 2 </numeric> <numeric> 2 </numeric> </call> </call> <call> <symbol> plot </symbol> <symbol> x </symbol> <symbol> y </symbol> </call> <call> <symbol> hist </symbol> <symbol> x </symbol> </call> <call> <symbol> hist </symbol> <symbol> y </symbol> </call> <call> <symbol> plot </symbol> <symbol> y </symbol> <symbol> x </symbol> </call> </body> </function>Now, we can then mark this up to display in a convenient fashion with color coding of the different language elements, links to the help pages for the function names that are called, and so on.
It would be convenient if PSGML mode could display the value of an entity in the display to the user, but store it internally in the model of the document as an entity. This would allow us to see the <- symbol but have it represented as &sgets;
Also, if font lock could be adapted to the particulars of the document being edited, one could see the different elements and the structure more clearly.
One of the most important facility that we need in an editor for use with literate programming for S is the ability to support multiple modes. In other words, we want to be able to edit documentation using HTML mode, S code using S-mode facilities with the associated key bindings and syntax-highlighting, C code in c-mode; and Perl, Python etc. using the different modes for each of these.
setenv LD_LIBRARY_PATH /tmp/xml-xalan/c/lib /tmp/xml-xalan/c/bin/testXSLT -in literateS.xml -xsl literateS.xsl > foo.html
One can specify values for parameters to be used in the stylesheet. For example, one can control whether line numbers are used when formatting S functions via the Sfunction.xsl stylesheet. One defines a parameter with a default value in the stylesheet. Users of that stylesheet can then supply a value for the parameter via the -param flag. This takes a pair of values, one after the other on the command line. The first is the name of the parameter. The second is the value. If this is a string, it should be quoted. testXSLT -in Sfunction.xml -xsl Sfunction.xsl -param useLineNumbers '"true"'
setenv CLASSPATH /usr/local/src/xalan-j_2_0_D01/bin/xalan.jar:/usr/local/src/xalan-j_2_0_D01/bin/xerces.jar java org.apache.xalan.xslt.Process -in data/literate.xml -xsl data/literate.xsl
XML-QL writeXML() .
The ideas here are more commonly known as "literate
programming", but we use a slightly less emmotive term.
As in most of our documents, we use the term S to refer to the S language
and refer specifically to the different implementations of this language
as SPlus and R.
This document can be converted to HTML by
applying the XSL file literateS.xsl to literateS.xsl.
by surrounding the irrelevant
functions within an if(F) {...}
clause.
The language used to specify nodes in the DOM via the
"regular expression"-like language used in the
select
atributes of the different
xls tags.
export LD_LIBRARY_PATH=/tmp/xml-xalan/c/lib /tmp/xml-xalan/c/bin/testXSLT -in literateS.xml -xsl literateS.xsl > literateS.htmlor
CLASSPATH /usr/local/src/xalan-j_2_0_D01/bin/xalan.jar:/usr/local/src/xalan-j_2_0_D01/bin/xerces.jar java org.apache.xalan.xslt.Process -in literateS.xml -xsl literateS.xsl > literateS.html