Introduction

It is frequently useful to be able to batch process many documents at a time. This example describes a useful pattern to perform batch processing. It can be adapted to do more sophisticated processing such as HTML to XHTML batch conversion of a website.

Specification

This example batch process will recursively find all system.xml files in a given directory. For each file found we will count the number of elements it contains. The results will be presented in an HTML table - this assumes that this process will be started by a web-browser request - it could equally be initiated through the command-line transport, email or any other available transport.

Overview

Below is a dpml batch process. It contains detailed comments which will allow you to adapt it to your needs. Here we'll summarize the 3 stages and suggest ways that this pattern can be modified.

Prepare resource list

Any batch process requires a list of resources to process. In the example we use the fls accessor to provide a document which lists the system.xml files below a root directory. You can change the root and filter to suit your system. Alternatively you could supply a hand crafted source document with the URI's of the files you want to process as elements. Note since NetKernel provides a URI resolver infrastructure you can source resources from anywhere not just the local filesystem so, for example, you could use this pattern as the basis for a web-bot...

Iterate over resource list

The example process iterates over the resource list generated by fls. Each URI element in the source document is used as the target resource in the inner batch process. In our example we perform an xquery operation to count the elements. We've used an XML process by way of example, you could do anything at all here, your resource could even be non-XML. A useful example is to use the XHTMLTidy to batch convert html files to valid XHTML files.

In our example we are not changing the target resource. You could as easily write the results of your process back to the target resource URI. Though take care since this is permanent and unrecoverable. It's usually a good idea to first execute a test process which simply logs the result to make sure your process is working as your expected. Once everything works you can add the URI writeback.

Show results

In our example we accummulate the results of each process in a variable. The result could be used for more extensive reporting including any exceptions that might have occured. To keep our example simple we've only provided basic exception processing. Finally the results are styled and presented as an HTML table. You could write them to a file or use them as the start of another process...

Executing this process

To execute this dpml process we need to create a host module.

1. Use the new module wizard to create and install a new module - choose the default settings ensuring your module supports dpml. Make sure you choose to import the module into the Front end fulcrum - this will make it available on localhost port 8080 by default. Your new module will be located in <install>/modules/your_module_name/.
2. The example process uses the fls accessor supplied from the ext_sys module and the xquery accessor supplied from ext_xquery. These modules must be imported into your module by adding the following two imports into the mapping section of your module.xml definition located in the root directory of your module.
   <mapping> ...Existing Imports...
     <import>
       <uri>urn:org:ten60:netkernel:ext:sys</uri>
     </import>
     <import>
       <uri>urn:org:ten60:netkernel:ext:xquery</uri>
     </import>
   </mapping>
   . You must now do a cold restart to pick up the module changes.
3. Finally copy the batch process listing below to a file batch.idoc in the resources/ directory of your module. You should edit the fls instruction to point to a different root directory and, if you wish, change the regex filter to match different file names.
4. You can start the process by requesting the URI with a web-browser http://localhost:8080/batch.idoc.