<?xml version="1.0"?>

<!-- Copyright the Omegahat Project for Statistical Computing, 2000 -->
<!-- Author: Duncan Temple Lang -->

<!DOCTYPE article [
  <!ENTITY R   'R'>
  <!ENTITY S   'S language'>
  <!ENTITY SPlus     "SPlus">
  <!ENTITY sgets "&lt;-">
  <!ENTITY ESS       "ESS">
  <!ENTITY TeX       "TeX">
  <!ENTITY LaTeX     "LaTeX">
  <!ENTITY MSWord      "MS Word">
  <!ENTITY C         "C">
  <!ENTITY Cplusplus         "C++">
  <!ENTITY Java      "Java">
  <!ENTITY Python    "Python">
  <!ENTITY Perl      "Perl">

  <!ENTITY lt        "<">
  <!ENTITY gt        ">">
  <!ENTITY OmegaCopyRight        "Copyright the Omega Project for  Statistical Computing, 2000">
]>



<article xmlns:bib="http://www.bibliography.org">
<title>Literary Programming<footnote>The ideas here are more commonly known as "literate
programming", but we use a slightly less emmotive term.</footnote> 
 for the <a href="http://cm.bell-labs.com/stat/S">S</a> 
<footnote>As in most of our documents, we use the term S to refer to the S language
and refer specifically to the different implementations of this language
as &SPlus; and &R;.</footnote>
language using XML</title>
<author><a href="http://cm.bell-labs.com/stat/duncan">Duncan Temple Lang</a></author>

<abstract>
In this document we discuss some ideas for using XML to both define S
language functions and expressions and simultaneously document them,
providing examples, sample output, etc. Generally, XML provides an
extensible mechanism for providing rich semantic information 
for both code and documentation, making it a powerful literate
programming format. <a href="http://www.omegahat.org/RSXML">S's
capability of reading XML</a> make it easy to process 
the code in traditional and new ways,  and also transform
the documentation into a variety of different formats.
Some of the drawbacks are also discussed.
</abstract>

<footnote>
This document can be converted to HTML by
applying the XSL file <url>literateS.xsl</url> to <url>literateS.xsl</url>.
</footnote>
<section>
<title>Motivating Literate Programming for &S;</title>
There are several difficulties associated with documenting S source
code. 
<itemize>
 <item> Code and help pages are developed separately and the latter
   typically lags behind the former. Experimental changes to the code
   don't warrant changes to documentation at the same time
   as they may not be permanent, remove focus from the development,
   etc.   However, when changes become permanent, one must return 
   to the documentation to bring it up to date.
   One must remember which documents to update and manually 
   determine which arguments have been removed and added.
   Providing programming tools to support this validation or
   verification of the documentation and the S object it is
   documenting would be valuable.
 </item>
 
 <item> Help pages are focussed on documenting particular functions and are not
   suited to documenting the overview of a collection of functions.
   The help pages give the local picture and do not provide a way for
   documenting the big picture about how to use the package as a whole.
 </item>
 <item>
   Both the &R; and &SPlus; help page format support an examples 
   section. However, all the examples are globbed together and it is
   only by convention that one knows where one example ends and
   another starts. 
   A clearer approach would be to have an examples section 
   which contained individual example elements. 
   Another difficulty is that  the examples must consist of legal S code and 
   so descriptions of the examples can be included only via
   comments in the language. Similarly, there is no support for
   showing sample output that the user can expect, etc. 
   So an example should have description, code and output sections.
 </item>
 <item>
   The <sfunction>source</sfunction> operates on an entire file, 
   reading and processing all of its contents.
   This is a rather blunt mechanism  when maintaining
   multiple functions in a single file. One cannot reload
   just one function from that file without updating the others
   or modifying the file<footnote>by surrounding the irrelevant
  functions within an <code lang="S">if(F) {...}</code>
  clause.</footnote>
   For example, one might be in the process of developing
   two functions in the same file but want to read the latest
   definition of only one. (The other may be in an intermediate state
   of development in which it does not actually work and one wants to 
   use the existing version within the &R; session.)
   With XML, we can read an entire file without evaluating its
   contents but determining its semantic information. Then, we can
   extract just the relevant sections and so exclude those that we do
   not want.
 </item>
 <item>
   Many of us write code in many languages. For example, we may
   implement functionality in S by writing a particular S function and
   an associated C routine which that function calls.  With the
   improved support for inter-system interfaces (e.g. Perl, Java,
   Python, SQL, XML, IDL, Tcl/Tk, etc.), the number of languages we
   use to implement particular functionality can become significant.
   Again, as with the documentation and code for S functions, it is
   convenient to keep these related functions in the same file
   or at least have a mechanism to link them.
 </item>
  <item> 
   Similar to having multiple languages within the same source file,
   many of us frequently find ourselves writing S code that is to be
   used in both &SPlus; and &R; and we would like to share as much of
   the code as possible between the two systems. The same applies
   with  <code>#ifdef</code> statements in &C;.
   <a href="?Brian and Bill">Others</a> have developed &Perl;
   scripts to handle the differences across the different &S;
   dialects. The pre-processor provides a mechanism for &C;.
   The marriage of literate programming which uses chunks
   in a "macro substitution"-like manner and XML allows us to 
   do this relatively easily.
   Specifically, we may write a segement of code that is 
   shared between &S; and &R; and in it include code 
   that cannot be shared. Rather than actually including
   the code itself, we can simply refer to it by name.
   A processing system that inserts the correct code 
   depending on the dialect for which it is being processed 
   will allow one to more readily support minor, but important,
   differences between systems. See <a href="#fragments">code
   fragments</a> below.
  </item>
  <item>
    In this document, we are focussing more on managing &S; code
    in a package or a user's library of useful functions.
    However, several approaches to using XML and literate markup 
    are being investigated for document data analysis and &S;
    sessions. The idea is that one runs an S session and each the 
    commands and their output are 
  

  </item>
  <item>
    Literate programming is valuable because it does encourage one
    to document code, explaining approaches in more detail than
    traditional comments. Whether anyone else ever reads the program
    document, it is likely that if you return to your code, you will
    benefit from having documented it more thoroughly.
    LP also allows one to write for the reader and not the
    computer/compiler. Additionally, it provides natural places for
    specifying auxillary or tangential comments and code that are not 
    processed in the "compilation" of the  program  but perhaps
    document other approaches or experiments.
<p/> 
    Literate programming for any language is a good idea.  It is
    difficult for people to say that literate programming is a bad
    idea, but the drawbacks of added processing, additional tools and
    complexity are valid complaints.  We hope to minimize these
    drawbacks and make it easy enough to use in practice so that the
    benefits outweigh the drawbacks.
  </item>
</itemize>
</section>


<section>
<title>Using XML</title> 

When writing documents using &TeX; and &LaTeX;, many of us attempt to
provide semantic information rather than appearance information by
defining macros to specify the content type.  For example, rather than
using <texCmd>textbf</texCmd> to render a file, we use a macro <texCmd>file</texCmd>.  This allows us
to change how we render a "file" object by changing just the macro
and also allows readers of the raw document to understand was meant.
<p/>
XML is currently the "language of choice" for specifying structure in
a document or data.  One of the defining factors that make XML so
useful is that it has become a <i>de-factor</i> standard.

<p/>
Valid documents as defined by a DTD mean that one can guarantee that a
document conforms to a particular, well-defined structure.  While such
requirements are not essential for documents mean to be read by a
human, it is important when software is used to "interpret" the
contents of a document.

<p/> 

One can think of using XML for document creation as writing in
HTML, but specifying the intent with a collection of macros. Then the
processing is done by an external engine, and in particular, can be
done in different ways by many different engines, maximizing the use
of the document.  In a sense, the document is the <i>model</i> or data
in the Model-View-Controller<cite id="MVC" /> paradigm so often cited
in software development literature.  The different rendered versions,
be they code for the computer or typeset documents, are <i>views</i>
derived from the model. And the tools that manipulate the document by
either extracting segments or modifying the content and structure are
the <i>controllers</i>.

<p/>

The fact that we can use different tools for creating multiple views
from the same document is one of the key benefits of XML. The data in
the document is application-neutral. It does not tie us to &TeX;,
&MSWord; or any other text processing system. 
Indeed, we can use XSL to ``massage'' a document into a form
appropriate for any text processor/typesetting system.

<itemize>
 <item>
   One can define pre-processor macro values to include or exclude certain
   segments or elements of a function definition depending on external
   variables. This is akin to the <croutine>assert</croutine> in &C;.
 </item>
 <item>
   XML is a <i>de-facto</i> standard and is supported by most
   popular programming languages.
 </item>
 <item>
   With any documentation system that can render code, the need
   to mix markup symbols and code symbols arises and the need to
   escape the latter so that the rendering engine treats it specially
   is vital.  For example, consider displaying a $ symbol in &TeX;.
   One must escape it so that &TeX; does not treat it as the initiator
   of an expression that is to be processed in  math mode.
   The general difficulty is a result of mixing two languages: the
   markup language and the language of the content, computer code in
   this case. Using XML for literate programming is no exception to
   this rule. Consider the following annotated S code segment.
    <quote>
      If x is non-positive, we set its value to <math>10</math>.
     <code><![CDATA[
         if(x < 1)
           x <- 10
       ]]></code>
    </quote>
    How can the parser recognize the &lt; and the &sgets; as being
   S code and the start of an XML element identifier.
   The answer is simply that it cannot. Hence, we must tell it so.
   <p/>
   XML provides two simple and consistent ways to escape text from
   being interpreted as XML:
   <description>
    <item><label>Entities</label>
    </item>
    <item><label>Verbatim sections</label>
      One can escape entire sections of a document
      using the &lt;![&lt;CDATA[ .... ]]&gt;
    </item>
   </description>
 </item>
</itemize>

<subsection>
<title>Code chunks</title>

One of the primary features of the existing literate
programming tools is the ability to break the
overall code structure (e.g. a routine) into different
chunks and to discuss these separately in the document
but group them together correctly when compiling.
For example, in <a href="">noweb</a>,  one 
might specify an &S; function as
follows
<verb>
<a name="nowebChunks"/>
<![CDATA[
 bar <-
 function(x, y) {
   <<Initialize Variables>>
   <<Iterate>>
   <<Cleanup results>>
 }
]]>
</verb>
and then define the different chunks
<nowebChunk>Initialize Variables</nowebChunk>,
<nowebChunk>Iterate</nowebChunk>, etc.
in a different part of the document.

<p/>

The purpose of using chunks and chunk references is to leave the author
free to document the code in the most suitable fashion for the reader,
rather than being a slave to the format expected by the application that
will read the code (e.g. the compiler, &Perl;, etc.).  An additional
side-effect is that one can re-use chunks in different places to
effectively <i>in-line</i> code while maintaining a single version.
In other words, it yields a more powerful macro facility than the
regular &C; pre-processor.

<p />
To use XML for literate programming, we will need to support
the same mechanism of having chunks and chunk references.
We will call these <i>fragments</i> and <i>fragment references</i>
to distinguish them from noweb and other LP tools (and also to give them
a slightly nicer term.)

<p />

We declare a code fragment using the <xmlTag>fragment</xmlTag> tag. As
with the <xmlTag>code</xmlTag> tag, we can specify a target language.
The fragment then behaves just like a code element and should contain
the code for that fragment.

<p />
<a name="SfragmentExample"/>
<code lang="S">
<![CDATA[
<function lang="S">
xml2tex <- 
function(node, mappings=.XMLTexMappings) {
   n <- 10
   x <- <fragmentRef id="frag1" />
   print(x+10)
   x
}
</function>
]]>
</code>

<code lang="S">
<![CDATA[
<fragment id="frag1">
 seq(1,n)
 <fragmentRef id="frag2" />
</fragment>

<fragment id="frag2">
 + 10
</fragment>
]]>
</code>



<p /> We can process the code chunks ourself in &S; or alternatively
construct XSL sheets to perform the link resolution.  The &S; version is
implemented in <sfunction>xmlSource</sfunction> currently distributed as
an example in the <a href="http://www.omegahat.org/RSXML">XML
package</a>.

<p/>

Using XSL rules to do this is attractive as it is language neutral
approach and will work for &C;, &Perl;, &Python;, &S;, etc.  Such XSL
rules are slightly more complex than the standard transformations one
uses in XSL. This reason for this is that we must look for the fragment
referenced in a <xmlTag>fragmentRef</xmlTag> element and its
<xmlAttrName>i</xmlAttrName> is not known when write the XSL rules, but
is a <xmlTag>fragment</xmlTag>-specific value. We might be tempted to
try to find the relevant <xmlTag>fragment</xmlTag> definition using
the XPath<footnote>The language used to specify nodes in the DOM via the
"regular expression"-like language used in the
<xmlAttrName>select</xmlAttrName> atributes of the different
<xmlNamespace>xls</xmlNamespace> tags.</footnote> mechanism something
along the lines of 
<verb>
<![CDATA[
 <xsl:template match="fragmentRef">
    <xsl:apply-templates select="//fragment[@id=@id]" />
 </xsl:template>
]]>
</verb>
The idea here is that we want to use the <xmlAttrName>id</xmlAttrName>
of the <xmlTag>fragmentRef</xmlTag> node being processed to find the
<xmlTag>fragment</xmlTag> element with the corresponding tag.  (We can
also specify a language value.)  This however will not work and in some
XSLT processors will induce an infinite loop.
<p/>

What we really need is a different approach. Take the example <a
href="#SfragmentExample">above</a>.  In the first
<xmlTag>fragmentRef</xmlTag>, we know the name of the
<xmlTag>fragment</xmlTag> for which we are looking and it is
<code>frag1</code>. So, we could write an XSL template to resolve this
as
<verb>
<![CDATA[
 <xsl:template match="fragmentRef">
    <xsl:apply-templates select="//fragment[@id='frag1']" />
 </xsl:template>
]]>
</verb>
Obviously, to be able to process more than just one single
<xmlTag>fragment</xmlTag> we need more of these templates. And
specifically, we need to be able to generate them dynamically as we
encounter each <xmlTag>fragmentRef</xmlTag> in the document and have the
XSLT respond accordingly.

<p/> 

We can create this dynamic template effect by using variables in <xmlTag
namespace="xsl">template</xmlTag> rules.  The following rule stores the
value of the <xmlAttrName>id</xmlAttrName> attribute of the
<xmlTag>fragmentRef</xmlTag> in the local variable named
<xmlVariable>A</xmlVariable> and then uses that in the value of the
<xmlAttrName>select</xmlAttrName> attribute of the 
<xmlTag namespace='xsl'>apply-templates</xmlTag>
<verb>
<![CDATA[
<xsl:template match="fragmentRef">
  <xsl:variable name="A"><xsl:value-of select="@id"/></xsl:variable>
  <xsl:apply-templates select="//fragment[@id=$A]" mode="resolve" />
</xsl:template>
]]>
</verb>

<comment>
<subsection>
<quote><i>This is arcane.</i></quote>
It would be <i>marginally</i> convenient to be able to use the <a href="#nowebChunks">noweb
approach</a> to naming chunks rather specifying the identifier
via the <xmlAttrName>id</xmlAttrName> attribute.
We can in fact do this using XSL rules, again in a dynamic manner.
Let's take the <a href="#nowebChunks">noweb example above</a>, and  modify
and expand it for use in XML.
<verb>
<a name="nowebChunks"/>
<![CDATA[
 bar <-
 function(x, y) {
   <Initialize-Variables>
   <Iterate>
   <Cleanup-results>
 }

 <Initialize-Variables>
  print("initialiazing")
 </Initialize-Variables>

 <Iterate>
   print("performing the iterations")
 </Iterate>

 <Cleanup-results>
   print("cleaning up results")
 </Cleanup-results>
]]>
</verb>

Now, to resolve these chunks, we have to write an XSL rule
that matches any chunk via the <code lang="XSL">match="*"</code>
attribute of the <xmlTag namespace="xsl">template</xmlTag>
element.
<code lang="XSL">
<![CDATA[
 <xsl:template match="*">
     <!-- Get the name of the tag -->
  <xsl:variable name="A"><xsl:value-of select="local-name(self)"/></xsl:variable>
     <!-- See if there is a top-level tag defined with that name -->
  <xsl:if test="element-available($A)">
    <xsl:apply-templates select="$A" />
  <xsl:if>
 </xsl:template>
]]>
</code>

</subsection>
</comment>

</subsection>

<subsection>
<title><a name="fragments">Code fragments for differences between
&SPlus; &amp; &R;</a></title>
Here we examine how we can use XML to handle 
the case where we have common &S; code for
both  &SPlus; and &R; and have minor differences
between them.
For example,  we might define as follows
<code lang="S">
<![CDATA[
     foo <-
     function(x) {
        ....
       <fragmentRef id="removeElement" />
        ....
     }
]]>
</code> 
where the <code>...</code> code is valid for both dialects
but the &lt;fragment&gt; element identifies
dialect-specific code.
We might identify these in the document as
<code lang="SPlus">
<![CDATA[
  <fragment id="removeElement" lang="SPlus">
    # SPlus-specific code
    v = get(".MyPersistentData", frame=0)
     v$name = "a new name"
    assign(".MyPersistentData", v, frame=0)
  </fragment>
]]>
</code>
and
<code lang="R">
<![CDATA[
  <fragment id="removeElement" lang="R">
     # R-specific code using closures
    v$name("a new name")
  </fragment>
]]>
</code>

The XML source facility in &SPlus; would know to resolve fragment
reference by finding the reference by its
<xmlAttrName>id</xmlAttrName> and then determining that there were two
matching fragments. It would then determine which one to use by
looking at the value of the <xmlAttrName>lang</xmlAttrName> attribute.
Obviously, it would match the &SPlus; fragment whereas &R; would use
the one intended for it.
</subsection>
</section>


<section>
<title>Drawbacks</title>

<description>
<item><label>Verbosity</label>
XML is very verbose and the plethora of markup details (tags and
attributes) can obscure the content of a document when viewed in raw
format. For this reason, the need for tools to view and manipulate XML
documents is essential.  However, editing modes such as <a
href="">PSGML</a> and other tools, both graphical and textual, do make
it feasible to operate on XML documents. One can manipulate a document
in S and then save the result back to a file. Given the extensive
support for XML in languages such as &Perl;, &Python;, &Java;, &C;,
etc. one is seldom at a loss for tools with which one can manipulate
and modify XML documents. The difficulty is learning about these
tools.
</item>
<item><label>Editor Code Modes</label> 

Those of us that use Emacs or other editors that provide specialized
functionality for editing code in a given language are hesitant to
give up these convenient, time-saving tools.  Integrating this style
of preparing code and documentation with existing tools is an
important challenge to using XML for these purposes.  Two things
rapidly come to mind.  Firstly, integrating with "modes" such as
&ESS;, etc. would allow that tool to extract S code from an XML
document and to send it to the S process. One could navigate the XML
document using the <a href="">PSGML-mode</a> style bindings to be able
to jump to different XML elements.

Secondly, <a href="">noweb</a> and <a href="">cweb</a> illustrate a
mechanism for switching modes within a document based on the content
of a region. <a href="">mmm</a> and other modes will hopefully support
this relatively easily and allow us to go further than noweb and cweb
modes in having support for modes for more than two languages.
</item>


<item><label>Spell Checking</label> 

XML provides a way to validate
structure, but obviously cannot determine whether the content of each
element is valid or not.  Spell checkers, etc. are needed for this.
Those who create HTML documents using regular editors know only too
well how aggrevating its when a spell checker that prompts you for the
spelling of H1, href, etc.  A <a href="">patch</a> for ispell allows
it to handle HTML. The same patch should be easily extended to XML.

</item>

<item><label>Extra Processing Steps</label> Since code and
documentation stored in XML cannot be used directly, one must process
it into a an appropriate form before using it.  It obviously takes
time to perform the processing and this increases the
edit-compile-test cycle of the development process.  There are several
XSLT processors, and some of these are faster than others. <a
href="">Xalan</a> is written in &Cplusplus; and others are written in
Perl. These approaches yield transformations very quickly and do not
add any considerable time to the overall compilation.
Transformation or filtering speed  becomes even less of an issue
when one considers the capabilities by <a href="">Xalan</a>
to "compile" XSL files into &C;/&Cplusplus; code.
<p/>

XSLT processors developed in Java may not be suitable. This not
because the Java interpreter and language are slow, but because the
startup time of the virtual machine is lengthy.

<p /> Also, while we are claiming it is feasible to develop filters
that massage the content of an XML document into different forms, it
is still does take some small quantity of time.  Many people will
decide that they do not have the time to spend learning this new tool
and transitioning to it. This has long been the argument against
literate programming and even learning new applications and languages.
This is, of course, a legitimate view, but frequently it is legitimate
at <i>a moment in time</i>, but less frequently <i>over time</i>.  In
other words, one may not be able to afford the time now.  However,
when one cumulates the different <i>nows</i>, it would have been more
effective to spend some time and amortize the learning over different
projects. I believe that this is especially true when documenting S
functions. The tendency for the documentation and code to diverge is
very real and natural. The utility of the code is increased
disproportionately to the existence of any documentation that allows
you, the author, to understand anything about the function when you
return to it in weeks, months or even years.  Using XML provides a
mechanism for creating searchable libraries or archives of S code.


</item>

</description>
</section>

<section>
<title>Points.</title>
Language-specific fragments. Need id= and lang=
but want the id to be unique?
</section>

<description>
<li><label>Consistency</label>
Keeps the code and the documentation closer together
allowing one to document more dynamically
rather than change, re-source, edit .Rd file.
</li>
<li><label>Pre-processing Macros</label> 
Allows one to do pre-processing macros (i.e. entities).
</li>
<li><label>Partial Source</label>
One can  read the entire file and then 
only ``evaluate'' or instantiate specific
functions.
</li>
<li><label>Efficient Partial Parsing</label>
One can use the SAX parser to do the partial
sourcing on very large files, if this becomes an issue.
</li>
<li><label>Two Escape Mechanisms</label>
One can use entities or the CDATA construct
to escape text from the XML parser.
</li>
<li><label>Scoping of Fragments</label>
One can potentially define fragments within other fragments
and effectively hide them from top-level chunks.
</li>
</description>




<section>
<title>Issues &amp; Notes</title>
<description>
<li><label>Line numbers</label>

Need to be able to get the line numbers in code associated with each
code chunk so that we can use it in debugging.  Can we get this from
libxml, XSL, and other parsers or translators?

</li>
<li><label>Escaping Characters</label>

When escaping code from being parsed as XML (e.g. &lt; in &C; or &S;
code), we can use the CDATA construct or leave it as regular, parsed XML
and use entities.

</li>
<li><label>XLink</label>
 Resolving references to other chunks, including in other
files.
Using the nano-hhtp.
</li>
<li><label>XPath</label>
We may want to add XPath facilities in &S; for 
identifying node sets via matching rules specified in the XPath
facilities.
</li>
<li>
Within an XSL file, one can include other XSL files.  In this way, we
can maintain and use XSL files separately to handle things such as &S;,
&C; and &Perl; code &S;, documentation, articles, etc. We ensure
consistency across output by using these different files cooperatively
in different contexts.
</li>
</description>

</section>



<section><title>Examples</title>

Here is how we ca read an XML document into R.
<code lang="S">
d &sgets; xmlTreeParse("/home/duncan/Projects/org/omegahat/XML/RS/data/literate.xml")  

eval(parse(text=d$doc[[3]][[1]][[2]][[2]] ))  
</code>

Now, what if that chunk refers to other chunks.  

<p/>

A simple way to read and source a file in the SSource
that does not contain fragment references is

<code lang="S">
simpleSource &sgets; function(file) {
  r &sgets; xmlRoot(xmlTreeParse(file))
  for(i in r[names(r) == "function"]) {
    name &sgets; xmlValue(i[["name"]][[1]])   
    def &sgets; xmlValue(i[["def"]][[1]])
    assign(name, eval(parse(text=def)), envir=globalenv())
  }
}
</code>
</section>


<section>
<title>&C; and &S; code</title>
In this section, we illusrate with an example how we can
mix related &S; and &C; in a single document for the purpose of
documenting and yet separate these for the purpose of compliling and
executing the code.

<p/>

For the purpose of this document, it is not important the reader understand the
particulars of the example. The issue is how we define both an S function and a C routine
and separate these.  In fact, the example allows one to query the value for a particular
process limit.  The idea is that we have an S function that maps the name of the resource
to an integer corresponding to the internal identifiers for the different limit types.
This value is then passed to the C routine and the result
returned. This value consists of a soft and hard limit.

<p/>
The idea is that we decide to write an S function
that gives us access to querying the value of a limit.
We start by defining such a function adding it to a file,
say <file>limit.xml</file>.
We start simply, adding some documentation and then the function definition.

<verb>
<![CDATA[
This function is used to get the current value of a
resource limit for the process.
One can ask  
<ul>
<li> how big a core file can be created
  by the process
</li>
<li> how many CPU seconds the process can consume,
</li>
<li> how large the data segment can become,
</li>
<li> the maximum number of child processes
</li>
<li> and so on</li>
</ul>

<p/>
The user specifies which resource is of interest by giving
its name. The function that translates this to an internal identifier.
]]>
</verb>

Now, we add the definition of the function to this file, just after the text describing
what we are doing.  We identify it as a function and indicate that it can be understood by
any S dialect by specifying the <xmlAttrName>lang</xmlAttrName> attribute as %S;.


<verb>
<![CDATA[
<function lang="S">
 processLimit <-
 function(name)  {
  ids <- getProcessLimits()
  which <- match(name, names(ids))
  if(is.na(which))
    stop(paste("Unknown process limit", name))

  vals <- .Call("RS_getLimit", ids[which])
  names(vals) <- c("soft", "hard")
  vals
 }
</function>
]]>
</verb>

Next, we define the C code associated with &S; function.
This is given something like the following.
Again, we add some documentation and the define the routine.
<verb>
<![CDATA[

The basic idea here is that we call the <croutine>getrlimit</croutine>
with the specified resource identifier and then put the elements of the
resulting <cstruct>rlimit</cstruct> structure into a numeric vector of
length 2.  We handle the case of an infinite limit by mapping this to
R's symbolic constant representing &infty;

<code lang="C">
< ![ CDATA [
#include <sys/time.h>
#include <sys/resource.h>

USER_OBJECT_
RS_getLimit(USER_OBJECT_ resourceId)
{
 struct rlimit limit;
 int res;
 
  res = INTEGER_DATA(resourceId)[0];

  getrlimit(res, &limit);
  ans = NEW_NUMERIC(2);
  NUMERIC_DATA(ans)[0] = limit.rlim_cur;
  NUMERIC_DATA(ans)[1] = limit.rlim_cur != RLIM_INFINITY ?  limit.rlim_cur : R_PosInf;

 return(ans);
}
] ] > <!-- End CDATA enclosing the C code. -->
</code>
]]>
</verb>

Note that we escape the contents of the <xmlTag>code</xmlTag>
so that the &amp;, &lt;, and &gt; are escaped correctly.

<subsection>
<title>Source'ing the function into &S;</title>

Having defined the &S; function and &C; code to implement the
functionality to get the process limit, we need to be able to utilize
it. So we need to be able to read the function into S and evaluate the
assignment of the function definition.  We will show how to do this
using the basic XML parsing tools.  Higher level functions can be easily
built from these to simplify the process.

<verb>
<![CDATA[
 d <- xmlRoot(xmlTreeParse("limit.xml"))
 funs <- xmlApply(d, function(x) if(xmlName(x) == "function" &&
                                       xmlAttrs(x)["lang"] =="S")
                                   x
                                 else
                                   NULL
          )

  apply(funs, function(x, env) {
                   eval(parse(xmlValue(x)), envir=env)
              }, env = environment())
]]>
</verb>
</subsection>

<subsection>
<title>Compiling the &C; routine</title> Next we want to get the C code
from the document and create a <ext>c</ext> file.  We have to extract
the contents of the <xmlTag>code</xmlTag> tag and put these into the C
file.
There are a variety of ways to do this. We might use
an XSL stylesheet such as
<verb>
<![CDATA[
<xsl>
<xsl:template match="code[@lang='C']">
  <xsl:value-of select="." />
</xsl:template>  
</xsl>
]]>
</verb>
This poses a slight difficulty in that the result of an XSL translation
must be a valid XML document, and is therefore not a legal &C; file.
We can remove the 
<verb>
<![CDATA[
   <?xml .. ?>
]]>
</verb>
element at the top of the document using Perl. However, we are still
left with the issue of ensuring that the special characters such as
&lt;, &gt; and &amp; are not escaped as entity references, but expanded
to their values.  Again, we can do this in a Perl script. Assuming we
know all possible entities, we can expand them at this point.
<p />

As one might expect, this is a common enough problem that XSL provides a
solution. One can specify the output type for the result of the XSL
transformation to be something other than XML.  The frequently supported
ones are <b>HTML</b> and <b>text</b>. Additionally, we can tell the XSL
processor not to emit the XML declaration.  See
<file>croutines.xsl</file> and
<url>http://www.ibiblio.org/xml/books/bible/updates/14.html</url>.  The
<a href="http://xml.apache.org/xalan-c/commandline.html">command line
arguments to Xalan</a> also allow one to control these issues without
changing the XSL file.


<p/>
Alternatively, we can write a special &C;/&Cplusplus; application that process the XML
file and expands entities, resolves fragment links, etc.

</subsection>


<subsection>
<title>Putting it all together</title>
We can now put all of this together in an automatic
fashion that is very similar to the way that we develop
code using standard (non-literate) approaches.

<P/>

We start by writing make rules to control the creation of the
different input files.  The two files derived from the literate source
(<url>limit.xml</url>) are <url>limit.c</url> and <url>limit.S</url>.
Both of these use the code extraction XSL file
<url>croutine.xsl</url>. We specify the language for which the code
should be extracted using the <flag>-param</flag> supported by 
<a href="http://xml.apache.org/xalan-c">Xalan</a>.

<p/>

The remaining rules are as they would be in the absence of literate
programming. They simply compile and link the C code into a shared
library that R can load.

<verb>
<![CDATA[
limit.c: limit.xml croutine.xsl
	($(XSLT_PROCESSOR) -in limit.xml -xsl croutine.xsl > $@)

limit.S: limit.xml croutine.xsl
	($(XSLT_PROCESSOR) -in limit.xml -xsl croutine.xsl -param language "'S'" > $@)

limit.o: limit.c
	$(CC) -I${OMEGA_HOME}/include/Corba -I${R_HOME}/include -D_R_=1 -DUSE_R=1  -c $<

limit.so: limit.o
	R CMD SHLIB -o $@ limit.o

examples:
	($(XSLT_PROCESSOR) -in limit.xml -xsl examples.xsl > $@)
]]>
</verb>

Now, we can use the rules to create and compile the different
files. The following commands apply these make rules.
<code lang="csh">
 make limit.so
 make limit.S
 make examples
 R
</code>

Now, we run &R; and source the
&S; functions, load the &C; routines and execute the examples.
<code lang="S">
 source("limit.S")
 dyn.load("limit.so")
 source("examples")
</code>
</subsection>


<subsection>
<title>Notes</title>
<ol>
<li>Rather than having generic code chunks, we can mark these up to be C
routine, S function, code snippet, etc.  This would facilitate
extraction and improve semantic meaning over the regular web setup.
</li>
<li>
 We could try using entities for referring to chunks/fragments.
</li>
</ol>
</subsection>

</section>

<subsection>
<title>To Escape or not</title>

It is often more convenient to include code sections within a
<xmlTag>CDATA</xmlTag> construct so that it is passed verbatim to the
processor.  This allows us to write it as we would write outside of XML
so that it can be used directly with the system which will process it,
e.g. &R;, &SPlus;, &C; compiler, &Perl;, etc.  However, we can also mark
it up. By treating it as verbatim text, we cannot mark it up to pretty
print it or understand it in any way at the XML level. This is probably
the most practical approach. However, the work we have started 
regarding writing &S; objects in XML for peristent storage and exchange
shows that one can markup internal objects using XML tags and then we
can format it in HTML, &TeX;, etc. and produce nicer output.

<p/>

For example, consider a simple function
<verb>
<![CDATA[
 f <-
 function(x,y)  {
   par(mfrow=c(2, 2))
   plot(x, y)
   hist(x)
   hist(y)
   plot(y, x)
 }
]]>
</verb>

Now, the <sfunction>xmlWrite</sfunction>
function and methods in &S; would (currently)
write this out as
<verb>
<![CDATA[
<function>
<args length="2">
<arg name="x">
</arg>
<arg name="y">
</arg>
</args>
<body length="5">
<call>
<symbol>
par 
</symbol>
<call>
<symbol>
c 
</symbol>
<numeric>
2 
</numeric>
<numeric>
2 
</numeric>
</call>
</call>
<call>
<symbol>
plot 
</symbol>
<symbol>
x 
</symbol>
<symbol>
y 
</symbol>
</call>
<call>
<symbol>
hist 
</symbol>
<symbol>
x 
</symbol>
</call>
<call>
<symbol>
hist 
</symbol>
<symbol>
y 
</symbol>
</call>
<call>
<symbol>
plot 
</symbol>
<symbol>
y 
</symbol>
<symbol>
x 
</symbol>
</call>
</body>
</function>
]]>
</verb>
Now, we can then mark this up to display in a convenient fashion with
color coding of the different language elements, links to the help pages
for the function names that are called, and so on.



</subsection>

<subsection>
<title>Editing XML</title>

There are a variety of different tools and applications for editing
XML.  I tend to use <a href="">PSGML</a> mode for emacs.  This
provides a variety of different convenience facilities.  It parses the
DTD associated with the document (i.e. in the <b>DOCTYPE</b> element)
and then is capable of determining the set of valid sub-elements and
attributes within the current element. This allows it to provide
automatic completion of tags and attributes.

Additionally, PSGML mode allows for displaying different
parts of the XML tree to simplify visualiztion of the content.
For example, it allows folding and hiding of sections
and provides an outline mode.

<p />

It would be convenient if PSGML mode could display the value of an
entity in the display to the user, but store it internally in the
model of the document as an entity.  This would allow us to see the
&sgets; symbol but have it represented as &amp;sgets;

<p />

Also, if font lock could be adapted to the particulars of the document
being edited, one could see the different elements and the structure
more clearly.

<p />
One of the most important facility that we need in an editor for use
with literate programming for S is the ability to support
multiple modes. In other words, we want to be able to edit
documentation using HTML mode, S code using S-mode facilities
with the associated key bindings and syntax-highlighting,
C code in c-mode; and  &Perl;, &Python; etc. using the different modes
for each of these.

<p/>

</subsection>

<bibliography>
<bibitem>
<url>http://www.oasis-open.org/cover/xmlLitProg.html#litProgXML</url>
</bibitem>
<bibitem>
<a href="http://xml.apache.org/xalan-c">Xalan</a>
</bibitem>
<bib:book>
<bib:title>The XSL companion</bib:title>
<bib:author>Neil Bradley</bib:author>
<bib:year>2000</bib:year>
<bib:publisher>Addison-Wesley</bib:publisher>
</bib:book>
<bib:book>
<bib:author>Simon St. Laurent</bib:author>
<bib:title>XML Elements of Style</bib:title>
<bib:year>2000</bib:year>
<bib:publisher>McGraw Hill</bib:publisher>
</bib:book>
</bibliography>

<appendix>
<title>XML/XSL-related URLs</title>
<ul>
<li> <url>http://www.xmlsoftware.com</url> </li>
<li> <url>http://www.w3.org/Style/XSL/</url> </li>
<li> <url>http://xml.apache.org/xalan-c/</url></li>
<li> <url>http://www.zvon.org/xxl/XSLTreference/Output/index.html</url></li>
</ul>
</appendix>

<appendix>
<title>Using Xalan-C++</title>
<code lang="csh">
setenv LD_LIBRARY_PATH /tmp/xml-xalan/c/lib
/tmp/xml-xalan/c/bin/testXSLT -in literateS.xml -xsl literateS.xsl > foo.html 
</code>

<p/>
One can specify values for parameters to be used in the stylesheet.
For example, one can control whether line numbers are used  when 
formatting S functions via the <url>Sfunction.xsl</url> stylesheet.
One defines a parameter with a default value in the stylesheet.
Users of that stylesheet can then supply a value for the parameter
via the <flag>-param</flag> flag.
This takes a pair of values, one after the other on the command line.
The first is the name of the parameter. The second is the value.
If this is a string, it should be quoted.
<example> 
  testXSLT -in Sfunction.xml -xsl Sfunction.xsl -param useLineNumbers '"true"'
</example>
</appendix>

<appendix>
<title>Using Xalan-Java</title>
To process an XML document with a given XSL style using the
Java version of xalan, use the following.
<code lang="csh">
setenv CLASSPATH /usr/local/src/xalan-j_2_0_D01/bin/xalan.jar:/usr/local/src/xalan-j_2_0_D01/bin/xerces.jar

java org.apache.xalan.xslt.Process -in data/literate.xml -xsl data/literate.xsl
</code>
</appendix>

<appendix>
<title>Perl-related XSL URLs</title>
<ul>
<li><a href="http://bitsko.slc.ut.us/libxml-perl/">libxml-perl</a> </li>
<li><a href="http://expat.sourceforge.net">Expat</a> </li>
<li> A variety of different Perl modules from 
      <a href="http://www.cpan.org/modules/01modules.index.html">CPAN</a>:
  <ul>
   <li> <code>XML-QL</code> </li>
  </ul>
</li>
</ul>
</appendix>

<appendix name="aux">
<title>Auxillary files</title>
<description>
<li><label><url>literateS.xsl</url></label>
 The XSL rules for converting this file to HTML.
</li>
<li><label><url>literateS.xml</url></label>
 The XML file from which the HTML is generated.
</li>
<li><label><url>literateS.html</url></label>
 The result of applying <url>literateS.xsl</url> to <url>literateS.xml</url>.
</li>
<li><label><url>fragment.xsl</url></label>
XSL rules for extracting code containing <xmlTag>fragment</xmlTag> references.
</li>
<li><label><url>fragment.xml</url></label>
XML file containing code with fragment references.
</li>
<li><label><url>Sfunction.xsl</url></label>
 Initial version of an XSL file for displaying S objects
as written in XML format by <sfunction>writeXML</sfunction>.
</li>
<li><label><url>Sfunction.xml</url></label>
 An XML file containing the output of an R function object
 marked up with XML tags for the different language elements.
</li>
<li><label><url>croutine.xsl</url></label>
 XSL file for extracting &C; code elements from an XML file so they can be
 put in a source file and compiled.
</li>
<li><label><url>limit.xml</url></label>
Contains the description and code for some C code
</li>
<li><label><url>limit.c</url></label>
 The &C; file created by processing the <url>limit.xml</url> file.
</li>
<li><label><url>GNUmakefile</url></label>
 The makefile for creating the different files.
 One has to edit it to specify the correct settings for the testXSLT
</li>
<li><label><url>literateS.tar.gz</url></label>
A collection of these files for those who want to 
experiment with them.
</li>
</description>
</appendix>
<copyright>
  &OmegaCopyRight;
</copyright>
</article>
