Soaplab2: Configuration Guide

Soaplab2 is configurable to the extent that it is sometimes annoying

Configuration sources How does Soaplab find its configuration file, or files? What is the format of property files?
Configuration properties
Logging

There are several moments and places when and how to configure Soaplab2:

Building Soaplab2
How to set various properties used when you build Soaplab2 (the first time after download), and when you deploy new services. These activities are usually done using Java tool Ant which uses property file build.properties. There is a template for this file build.properties.template. What Ant tasks are available and what properties can be used to customize the tasks is a topic of a How to build document.
Service metadata
There are lists of available services (analyses) and each service has its own metadata. They both can be considered as configuration options. But they are topic of other documents (partly the Developers Guide, and mainly the Metadata Guide).
Run-time configuration
How to configure Soaplab2 services for running and serving clients. This document is exactly about this type of configuration.

Configuration sources

The main format of Soaplab2 configuration are properties which are nothing else than the name/value pairs. They can be defined in several ways, and depending on it they have different precedence.

Java System properties have the highest precedence. They can be specified on the command-line with the -D option(s). They are good for testing but less suitable when services are run from a servlet engine, like Apache Tomcat. If you still wish to use them, see the Tomcat documentation about how to set CATALINA_OPTS environment variable.

Normally, Soaplab2 configuration properties are taken from a configuration file. There can be more configuration files - and if they are, the later ones have higher precedence (but still lower than the Java System properties).

How does Soaplab find its configuration file, or files?

Soaplab2 tries first to find the property soaplab.configuration. Its value should contain the name, or names, of the property files containing the rest of Soaplab2 configuration.

However, this is obviously a Catch-22 situation: how can you find a property pointing to a file if such property may be in that file? Well, you can't. Unless this one property is defined by a Java System property. Therefore, this property may be the only good candidate to define by the System property - even within Tomcat.

But there is still another way: If Soaplab cannot find the property soaplab.configuration, it tries to find a property file with a default name soaplab.properties. The file will be searched for automatically in the following locations:

in the current directory,
in the user home,
in directories defined in the CLASSPATH

If the property soaplab.configuration was found and its value is a file name with an absolute path, it is taken as defined and it does not look further.

In the Soaplab2 distribution (what you have when you CVS check-out Soaplab2), there is a template for the default property file src/etc/config/soaplab.properties.template. During building (compiling sources), Soaplab2 copies this file into build/classes/soaplab.properties - which is a directory on the CLASSPATH, and which is a default name, therefore, the property file is found.

You can change this behaviour, and let Ant copy your own property file, by specifying a build-time property my.soaplab.properties (the build-time properties are used by Ant and were mentioned briefly at the beginning of this document) when running the Ant's config task (the file name 'testing.soplab.properties' in the example is a fictive file name, you will use your own name):

ant -Dmy.soaplab.properties=testing.soaplab.properties config

The property soaplab.configuration can contain more property file name, separated by commas. For example, a plug-in project BioinfoGRID starts its command-line Java programs with this line:

exec java -cp $CLASSPATH -Dsoaplab.configuration=bioinfogrid.properties,soaplab.properties "$@"

It lists two property files, bioinfogrid.properties and soaplab.properties, both of them without an absolute path, so Soaplab2 will search for them in the places mentioned above (current, home, CLASSPATH).

What is the format of property files?

Simply speaking, it follows the same syntax as required by java.util.Property.load() method. The gory details can be found (for example) here. For example, here are few, randomly picked-up, properties:

base.dir = /home/senger/soaplab2
jobs.cleaning.interval = 1000
results.dir = RESULTS

However, Soaplab2 goes further. It allows:

To have properties with multiple values, either as a comma-separated list in one property, or by using the same property name twice. For example:
```
applist = /mydir/metadata/DefaultApplications.xml
applist = /mydir/metadata/generated/OtherApplications.xml
```
Of course, this feature is useful only when the Soaplab2 code expects to get more than one value. It is usually mentioned in the description of the individual properties whether it does or does not.
If a property is named include and the value of that property is the name of a file on the disk, that file will be included into the configuration. For example:
```
# properties of the third-party plug-ins
include = bioinfogrid.properties
```
Variable interpolation: the expressions like ${token} are automatically expanded when the configuration file is loaded. The expressions can contains names of other properties from the same configuration file, or of Java System properties. For example (taken from the BioinfoGRID plug-in):
```
grid.vo = biomed
grid.proxy.dir = ${java.io.tmpdir}/proxies
grid.proxy.file = x509${grid.vo}_${user.name}
```
The property java.io.tmpdir and user.name are Java System properties.
Boolean property with an empty value means true. For example these two properties are both considered set to true:
```
ignore.heartbeat.events = true
ignore.timeprogress.events
```
Prefixed properties: Almost any property can be prefixed by a service name or a class name. Soaplab2 tries to find and use first a property prefixed with a service name (using the current service the Soaplab2 code is just dealing with), then - if not found - to find and use a property prefixed with the full class name of the class where is the property being read, and finally - if it is still not found - to use a simple property name. For example:
```
results.dir = RESULTS
classic.helloworld.result.dir = HELLOWORLDS
```
It means that all services will store there results in the RESULTS directory except for service classic.helloworld whose results will go to the HELLOWORLDS directory.
Special characters: If you need a special character in a property like a line feed, a tabulation or an unicode character, you can specify it with the same escaped notation used for Java Strings. The list separator (comma by default), can also be escaped. For example:
```
key = This \n string \t contains \, escaped \\ characters \u0020
```
And finally, automatic reloading: properties file will be reloaded when its contents has been changed. Again, this feature can be useful only if the Soaplab2 code also re-reads the properties - but it often does so.

Configuration properties

All properties used by Soaplab2, together with short descriptions are listed in the class Config - looked there for fields started with PROP_ if this chapter has a missing property. Which may happen with brand new properties.

Small note about the default values: After installation or deployment, the Soaplab2 uses a configuration file that was created from a template src/etc/config/soaplab.properties.template. The values there are considered the default values in the descriptions below. If you, however, change this configuration file and remove some properties completely, Soaplab2 may still have internally coded and used some default values. Those values are listed in the class Config. Usually, just to avoid uncertainty, it is better to put always some values (at least for the properties defining path and file names) in your configuration file.

working.dir

Its value contains a directory name where the inputs and outputs of the running external process will be stored in.

By default, there will be created a _R_ sub-directory in the base directory, and within this there will be another sub-directory SANDBOX as the working directory. The base directory depends on how the Soaplab2 service is invoked: it can be either Soaplab2 main project directory, or (after services being deployed) a <tomcat-home>/temp directory.

For example, this is a part of a Soaplab2 configuration file used by deployed services in Tomcat, showing basic directories used by Soaplab2:

base.dir = \${catalina.home}/webapps/soaplab2

metadata.dir = \${catalina.home}/webapps/soaplab2/metadata
applist = ${metadata.dir}/OtherApplications.xml
applist = ${metadata.dir}/GowlabApplications.xml
applist = ${metadata.dir}/EBIApplications.xml

runtime.dir = \${catalina.home}/temp/_R_
working.dir = ${runtime.dir}/SANDBOX
results.dir = ${runtime.dir}/RESULTS

results.dir

Its value is an absolute path to a directory where Soaplab2 keeps results in the local file system.

Default value is RESULTS, created in the same place as the SANDBOX shown in working.dir property above.

metadata.dir

Its value is a name of a directory containing service metadata.

The importance of this property depends also on the contents of the service lists (given by the applist property). There are links to the individual service metadata files. If these links are absolute paths the metadata.dir property is not used. Otherwise (which is the usual and default way), its value is put as a prefix to those links.

.metadata.file

This is only a part of a property name. Its value is a name of a file containing metadata for a particular service.

The property name must be prefixed by the service name whose metadata this file contains. For example, for service edit.seqret, the full property name should be edit.seqret.metadata.file.

There is no default value.

applist

A filename containing a list of services. There is no default value. For example:

base.dir = /home/senger/soaplab2
metadata.dir = ${base.dir}/metadata/generated
applist = ${metadata.dir}/OtherApplications.xml
applist = ${metadata.dir}/GowlabApplications.xml
applist = ${metadata.dir}/EBIApplications.xml
applist = ${metadata.dir}/EMBOSSApplications.xml

jobs.cleaning.interval

Its value (an integer, number of milliseconds) indicates how often to scan a list of completed jobs in order to remove them from the memory (not to destroy them; their results will still be available). This property can help keeping the Soaplab2 memory footprint lower. Default value is 60000 (1 minute).

jobs.timeout

Its value (an integer, number of milliseconds) indicates how long after a job is started it should be terminated if it is still running. This property lets Soaplab terminate hanging or long running jobs automatically. You can prefix this property with service name to have different timeout values for different services, such as classic.helloworld.jobs.timeout=5. Checks for timeout are performed every jobs.cleaning.interval.

Default value is 864000000 (10 days).

jobs.waitfor.timeout

Its value (an integer, number of milliseconds) indicates how long after a waitfor call made should the call be returned if the job is still running after that period.

Default value is 864000000 (10 days).

services.cleaning.interval

Its value (an integer, number of milliseconds) indicates how often to scan a list of instantiated services in order to remove them from the memory (they can be still later re-instantiated if a request comes). Again, it helps to low the memory footprint. Default value is 300000 (5 minutes).

ignore.heartbeat.events

Its boolean true value indicates that event listeners should ignore Heartbeat Progress events (because there may be too many of them). A bit more about Soaplab2 events can be found in the developers guide.

Default value is false.

event.log.level

The Soaplab2 default event listener records the events into a log (usually) file. This property indicates a logging level under which the events are logged. The accepted values are (in case-insensitive way): trace, debug, info, warn, error, fatal, and off (off means no logging at all).

Default value is debug.

env

This is not a full property name, just a prefix. It should be followed by a dot and a name of an environment variable that will be set before invoking a command-line program. The property value becomes the value of this environment variable.

Be aware that playing with environment variable may be system-dependent and therefore not too portable. But in many occasions it may be very helpful (and can replace a need to write various script wrappers around the command-line programs that would set environment).

Also, be aware, that this does not work for the environment variable PATH. In order to let Soaplab2 to find the command-line program, use rather property addtopath.dir.

An example:

env.EMBOSS_HOME = /usr/local/emboss
env.EMBOSS_DATA = /usr/local/emboss/share/EMBOSS/data

addtopath.dir

Its value contains one or more directory names (separated by commas) that are searched for an external (command-line) program that has to be invoked. It is a replacement of setting the usual environment variable PATH (which would not work from Java).

The default value is the directory defined in another property scripts.dir. Therefore, that is a good place to put your scripts.

scripts.dir

This property contains a directory name with scripts that are supposed to be called by Soaplab services. So why not to add this directory name directly into addtopath.dir property? The reasons is that Soaplab2 treats directory defined here slightly more: Unless the Soaplab2 server is running on the Windows operating system (where this treatment is not needed), Soaplab2 tries to change file permissions of all files in this directory to become executable (by calling an external program chmod). This is needed because the original file permissions are gone when deployed from a war file (which is a usual way how the Soaplab2 services are deployed).

The default value depends on the protocol used. When deploying to Tomcat, it is directory webapps/soaplab2/WEB-INF/run. When running using the local protocol, it is the run directory in the Soaplab2 project directory (from where the scripts are copied to webapps/soaplab2/WEB-INF/run during deployment anyway).

accept.any.exitcode

Its boolean true value indicates that any exit code returned by the invoked external program is treated as a good one. Otherwise only zero exit code is acceptable as a good one.

Default value is false.

results.url.ignore

This (and the following three other properties) deal with the results shown by reference - by sending a URL instead the result itselves.

The true value of this property disables creation of such reference results. Default value is false, i.e. reference results are enabled. Unless, of course, they are permanently disabled already in their metadata (look for the make_url attribute in the metadata guide).

results.url.target.dir

Its value is a directory (usually with the full path) where Soaplab2 will be copying results that should be viewed (accessed) via protocols such as HTTP or FTP. This property is a full path to the same directory where the property results.url points to as a URL.

Default value depends on the context: if a Soaplab2 service is running in a servlet container (such as Tomcat - which is a usual case), a directory results is created inside Soaplab2 web application. Otherwise, the directory given by property base.dir and results is used.

If you change this directory and have then problems with results containing foreign characters, in other words if you need to be sure that the results are correctly sent with the UTF-8 character encoding, you need to specify the new directory name also in src/etc/config/*/web.xml.template.

results.url

Its value is a URL (without the last file name) where the Soaplab's results are accessible from. This URL points to the same place as specified in the property results.url.target.dir.

Default value is http://<tomcat.host>:<tomcat.port>/soaplab2/results.

tomcat.host

Its value is a host name where your Tomcat is waiting for Soaplab2 services requests. It is used to create default results.dir (in case that the results.dir is not given).

Default value is localhost.

tomcat.port

Its integer value is a port number where your Tomcat is listening for Soaplab2 services requests. It is used to create default results.dir (in case that the results.dir is not given).

Default value is 8080.

lsf.enabled

Use this property if you have LSF installed on your server machine and you prefer your Soaplab jobs submitted to LSF queues rather than executed on your Soaplab server machine.

Default value is false.

You can enable, or disable, LSF for different services as in the second example below.

                   lsf.enabled = true
classic.helloworld.lsf.enabled = false

lsf.queuename

Use this property to specify queue name for LSF jobs

You can specify different queues for different services as in the second example below.

                   lsf.queuename = queueforsoaplab
classic.helloworld.lsf.queuename = queueforfastreturningjobs

lsf.arch

Use this property to specify architecture to submit LSF jobs on.

lsf.project

Use this property to specify project name for LSF jobs.

Logging

Soaplab2 uses log4j logging, together with Apache Commons logging wrapper. This logging system is reasonably well documented. The only topic we should cover here is how Soaplab2 finds the log4j configuration.

There is a template containing a good starting point for your logging configuration in src/etc/config/log4j.properties.template.

When building Soaplab2, Ant copies this file (after making some substitutions inside, regarding the current directory) into build/classes directory - which is on the CLASSPATH so it could be discovered in the run-time. If you wish another file to be used, create a property my.log4j.properties pointing to your file of choice. Ant recognizes this property and copies your file.

More to it, Ant also puts a well-known property log4j.configuration into the run scripts (the run scripts are command-line clients created in the build/run directory). This property points to the same build/classes/log4j.properties file . For example:

exec java -cp $CLASSPATH "-Dlog4j.configuration=file:/home/senger/soaplab2/build/classes/log4j.properties" "$@"

Unless you change it in the log4j.properties, Soaplab2 will be using two log files, the soaplab2.log and soaplab2-others.log. The logged records will have the following format (the characters with the percent sign are used in the logging ConversionPattern property):

%d{ISO8601}             %-4r [%t]   %-5p  %c{1}      %x- %m%n
2006-11-24 14:38:49,447 296  [main] INFO  testing_echo - Service instance created.
(1)                     (2)  (3)    (4)   (5)        (6) (7)(8)

Where:

(%d) - Current date in ISO8601 format
(%r) - Number of milliseconds elapsed from program start to logging event
(%t) - Name of the thread that generated the logging event
(%p) - Level (priority) of the logging event
(%c) - Place (usually a class name) where the logging event happened
(%x) - not used yet - (kind of a user session)
(%m) - The message to be logged
(%n) - Newline

Last modified: Mon Oct 11 16:14:02 2010