Build (from CVS) Guide    

This guide describes what can be done after you check out the full Soaplab module from the CVS. All tasks described here are invoked using the Java's wonderful tool Ant (which is part of the CVS Soaplab module - you do not need to download anything else).

Usually, the Ant tasks are meant for developers who make changes in the source code. It is true also for Soaplab. But additionally to these tasks, the Soaplab module includes also tasks for creating and deploying Soaplab-based web services. These task are well suited for testing new Web services.

As explained already in the README file you can use either build.sh (or build.bat for Windoze), or build-dev.sh (or build-dev.bat). Because usually you use the latter, the examples below use the -dev form.

Just remember that

  • You can use build.sh (or build.bat) script anytime, but
  • You must use it the first time (because it gathers all third-party libraries needed for compilation). Otherwise
  • You may use build-dev.sh (or build-dev.bat) which is slightly faster because it does not attempt to fetch libraries from the Internet.

Why not just to type ant as usual? Because it would invoke your own Ant - which may be of a different version than the Ant used by Soaplab, and because it does not have one or two additional tasks that Soaplab's Ant has. Be aware that from the same reason the building scripts do not use the environment variable ANT_HOME even if it is set.
Be also aware that some tasks (especially those generating metadata for new services) use Perl scripts and that they would work almost certainly only under Unix. They may work with Cygwin under Windoze but I have not tried it.

You may also influence the building by setting various properties. The easiest way is to create file build.properties either directly in the soaplab directory, or in your home directory. You can use provided build.properties.template as a template.

This build.properties.template file has details what properties can be set and what they mean. Usually, however, the only properties you may need to set, are: 

catalina.home = /where/is/your/tomcat
jdbc.passwd = what is your password to local MySQL database
and very recommended one (if you have installed jikes on your machine): 
build.compiler = jikes
General rules about properties used by Ant are:
  • If you need something that is outside of the Soaplab CVS local copy you need to specify a property for it (e.g. the location of your Tomcat is definitely outside of your Soaplab directory).

  • If you wish to change default Ant behavior you may need to define your own property (e.g. names of services you wish to generate metadata for can be defined by a property aa - and if not used then Ant will take all services defined in src/etc/acd/applab directory). (By the way, the Ant default behavior is in build.xml file but its description is beyond the scope of this guide.).

 Soaplab CVS   What can be built

And finally, here is what can be done:

  • Usual tasks
    How to compile everything, how to generate documentation and jar files...

  • Distribution tasks
    How to create various distributions...

  • Unusual tasks
    How to prepare metadata and deploy web services...

  • And the rest...
    There is always something that does not fit anywhere else...

  Usual tasks

The normal build tasks are:

./build-dev.sh compile
Compile all source code (default).

./build-dev.sh docs
Generate all documentation.
All, except for Perl modules documentation that is already part of the CVS because it does not change much. You can find it in the docs/Bio subdirectory.

./build-dev.sh alljars
Create jar files with Soaplab core and plugin classes. The plug-in classes are used by some services - and you may add your own. This is more described below.

./build-dev.sh clean
Remove most of the generated files. You can also remove all third-party libraries by using task cleanall.

./build-dev.sh all
Clean, compile, generate docs and make main distributions (what kinds of distributions are available is explained in the next section).

The summary is that after checking out the Soaplab CVS module you should gather all third-party libraries and compile everything by: 

./build.sh
And that would be enough to invoke Soaplab's command-line clients (whose scripts are in the run subdirectory). At this moment, however, these clients can be used only to access Soaplab services running elsewhere. Your own services have not yet been deployed. Before going to explain how to do that let's talk briefly about how to build various distributions.

  Distribution tasks

You need these tasks if you wish to pack Soaplab files into a tarred package that can be used independently on the Soaplab CVS module. One reason for doing it is to have a separate space where you can put all your Soaplab services and use them on a production level - without being mixed with the latest changes that may happen in the CVS module itself.

The binary distribution has also a nice feature that you can add your and third-party services to it as they come - you do not need to create and install your binary distribution again and again from the scratch.

Binary distribution

The binary distribution contains all necessary files, plus a small number of testing services (both AppLab-based and Gowlab-based). It will be created in the doc/dist directory by typing: 
   ./build-dev.sh dist-binary
Be aware this tasks uses Perl in order to generate metadata for the testing services.

By default, it includes the following AppLab services: 

   helloworld
   dot
and the following Gowlab services: 
   emblsrs
   helloworld
   medlinesrs
   yeastgrid
   echo
You can change this by setting different names in the space separated lists in the properties aa (for AppLab services) and/or ga (for Gowlab services). The corresponding ACD files for such services must be found in the src/etc/acd/applab and/or src/etc/acd/gowlab.

For example, assuming you have created src/etc/acd/applab/bigdeal.acd, src/etc/acd/applab/biggerdeal.acd, and src/etc/acd/gowlab/favorite.acd files, you can create a binary distribution just with these three services by typing: 

   ./build-dev.sh '-Daa=bigdeal biggerdeal' -Dga=favorite dist-binary

Once you have a binary distribution, you can move it somewhere, unpack it, run the installation script (which is included in the binary distribution) and deploy included services to your Tomcat engine. The details how to install and use the binary distribution are here.

Note that instead of packing it yourself you can also download a ready binary distribution from the main Soaplab distribution site. So when should I use my own binary distribution? When you have made some changes in Soaplab, or when you have applied patches provided by someone else - but the official binary distribution is still not available. Obviously this task is meant primarily for the developers of the Soaplab core that are involved in new releases.

A service set distribution

One of the advantages of Soaplab is its uniformity how to access different tools and web resources. This uniformity allows to keep separately the Soaplab core that does not change often, and metadata (and other supporting files) for individual services. Because the metadata are site-independent (meaning that they do not, or should not, contain any site-specific locations or paths) they can be shared and re-used by other service providers, too.

Note, however, that this is about sharing metadata, not the services themselves. For Gowlab-based services, it is sufficient, but for the AppLab-based services you need to have also the command-line analysis tools that will be invoked by Soaplab/AppLab and that will do the real job.

This task can create such site-independent package.

A service set distribution is a package containing files related to a set of services (such as services metadata and additional plug-in classes written particularly for these services) - but without the Soaplab core files. This allows to have relatively small packages - that are independent on the latest changes in Soaplab core classes - and that can be shared by Soaplab services providers. For already existing packages check the contrib subdirectory of the Soaplab main distribution site. If you wish to put there your own service sets please contact me.

In order to use such service set you need to have installed somewhere a Soaplab binary distribution, and then you merge the service set with that binary distribution. You can add many service sets into one binary distribution.

Well, the expression "into one binary distribution" is not precise. You are not merging a service set into a tarred binary distribution, but you are merging into directories where the binary distribution had been unpacked and installed.
Of course, in order to create a service set you need to have some services. Which means you need to create first some ACD files describing new services. This image explains the relationship between binary and CVS distributions:

Now let's have a look how to invoke this task and how to tell which services should be included in the resulting package.

The basic typing is: 

   ./build-dev.sh dist-services
You can tune what services will be included in the package, how to name them, etc., by setting various properties before invoking this task. As always with Ant, you can do it directly on the command line, using -Dname=value syntax, or you can first put the properties into build.properties file where Ant reads them from. The more important properties are:

services.dist.name
A package name. This name will become a part of the resulting file name, a directory name of the generated API documentation, and a part of the name for a jar file with derived services.

aa
A space-separated list of AppLab-based service names that should be included in the resulting package. The names are the names of the corresponding ACD files (without the extension .acd). The ACD files are expected to be in src/etc/acd/applab directory.

ga
A space-separated list of Gowlab-based service names that should be included in the resulting package. Again the names are the names of the corresponding ACD files (without the extension .acd). The ACD files are expected to be in src/etc/acd/gowlab directory.

services.endpoint
A base URL (without service name) that will be used in the generated WSDLs as the endpoint for included services. Use this, for example, if your Tomcat is hidden behind a proxy server.

This is a bit strange option - the shared services should not have any site-specific URLs, names, or paths attached. But do not worry, the deployment of these distributed services has an option to generate WSDLs again with a different endpoint. I keep this option here from historical reasons perhaps.

The created package will contain:

  • XML file for each service (service metadata)
  • XML file Applications.xml listing all AppLab-based services
  • XML file Gowlab.xml listing all Gowlab-based services
  • A jar file with classes for derived services
  • A jar file with plugins classes for those Gowlab-based services that need them
  • XSLT stylesheets for those Gowlab-based services that need them
  • API for all derived services
  • WSDL files for both normal and derived services
  • ACD files for included services (this is not really needed because the ACD files are not used in run-time, but they are here because they may be used as templates for other, related or similar, services)

If you are interested in more details about the contents and structure of the created package check the description in script utils/add-services.pl.

Having such package, you can go to the next step which is to merge new services with the existing ones. This is done, as shown in the picture above, in the place where you have installed the binary distribution. Go to the Binary distribution Guide to find how to do it. You will find there that it is done by invoking a single script that does everything for you: 

   cd .../analysis-interfaces
   ws/add-services <file-with-set-of-new-services>

Distribution of EMBOSS services

A slightly special case of a service set is the set of all EMBOSS applications. Because of the EMBOSS's prominent place in Soaplab project there is a specialized Ant task for it: 
   ./build-dev.sh emboss-dist-services
This task requires the emboss.home property - pointing to a directory where your local EMBOSS is installed (and where this task gets all the ACD files for EMBOSS tools). If not set the default value /usr/local/emboss will be used (which may not be what you wish).

Once you create a package with EMBOSS services, however, everything is the same as with other packages. You deploy it again using ws/add-services in the binary distribution.

Only-clients distribution

The result of this distribution is a set of files needed for invoking Soaplab services. The details how to use it (which is quite straightforward) are in the accompanied (included) README file.

This is the only distribution that has been tested to work also under Windoze.
Type: 
   ./build-dev.sh dist-clients
The result will appear in the docs/dist directory.

  Slightly unusual tasks

The tasks in this group need to cooperate with your Tomcat engine. They will create services metadata, generate services supporting classes, compile them, and deploy services. I have used it only with Tomcat and Axis - but it may work with other servlet engines as well.

Whenever Tomcat gets new classes (or deployed services) it must be told about it. How to tell, it depends on the Tomcat version: in all versions restarting the Tomcat works fine. Therefore, some tasks will ask you to restart your Tomcat. For versions 5 and up, you may use, however, a less drastic way, because there is a Tomcat Manager interface to help. So if you use such manager, just do what should be done with the manager when a task asks for restarting.

It is generally wise to know what is happening "under the hood" when deploying services. Here are few tips:

  • Soaplab services are of two kinds - normal ones and derived ones. All normal ones use the same classes (packed together in soaplab.jar and moved to the Tomcat by the task populate, see below). So when you add a new service and you intent to call it as a "normal" service you really do not need to move any new classes to Tomcat, so perhaps there is no need to restart it.

    However, any derived service is represented by its own class (generated one, you do not need to write it). This class must be moved to Tomcat (it is done by the task populate-derived but you seldom call this task on its own, it is usually called by other, deploying tasks), and Tomcat must be restarted.

  • Deploying a service means that a new entry, representing this service, is stored in the Tomcat/Axis configuration file server-config.wsdd (you do not need to change this file manually, it is done by the tasks using an admin client - if you are interested in details, explore build.xml file). I am not sure about this but I think that Tomcat needs to be restarted when this configuration file is updated.

  • Remember that the AppLab-based services are available through a running AppLab server (as explained in a Binary distribution Guide). And whenever you restart AppLab server you need to restart also the Tomcat (but there is no need to restart AppLab server when you restart Tomcat). Therefore, some tasks will tell you to start an AppLab server.

  • For the Gowlab-based services, it is easier. Unless you want to use derived services (see a tip above) you do not need to restart Tomcat at all. Once a Gowlab "list" service has been deployed (done once, and it needs a restart), the Soaplab itself checks if there is a newer file metadata/Gowlab.xml, and if yes, it reloads all necessary metadata.

  • And finally, when anything goes wrong, cannot be found, or whatever, just restart Tomcat - and in many cases it helps.

Now, let's examine how to add new services. Note that the tasks described here are just doing what is explained in details in a the Binary distribution Guide.

Create a new ACD file (or files) in src/etc/acd/applab or in src/etc/acd/gowlab, and type: 

   ./build-dev.sh gowlab
   ./build-dev.sh applab
   ./build-dev.sh api-derived
The first invocation creates and deploys all Gowlab services, the second does the same for the AppLab services, and the last one creates an API for all derived services.

If you wish to limit what ACD files (e.g. what services) to deploy, use properties aa and ga. For example: 

   ./build-dev.sh -Dga=newgowlabservice gowlab
   ./build-dev.sh -Daa=newapplabservice applab
You will notice that the tasks above invoke a lot of sub-tasks. Sometimes it may be useful to call some of them independently: 

   ./build-dev.sh gen-applab
   ./build-dev.sh gen-gowlab
   ./build-dev.sh gen-all
These tasks generate XML metadata from the ACD files. It is useful to use it when you are in the process of creating ACD files and you want to be sure that they are syntactically correct before proceeding on.

  And few remaining tasks...

In the previous section you have learned how to create everything for a set of services, including derived services and their deployment to a Tomcat engine. But sometimes you need just to update API documentation of the derived services, but not to touch their deployment. Use create-derived task. For example, for a set of AppLab and Gowlab services that are always part of any Soaplab distribution you can invoke: 

   ./build-dev.sh  -Daa='helloworld dot' \
                   -Dga='emblsrs helloworld medlinesrs yeastgrid echo' \
                   create-derived
The Soaplab can use a mySQL database for keeping results of the analysis tools. The details how to set up such database are described in a Binary distribution Guide. The same can be achieved, however, by the create-database task: 
   ./build-dev.sh create-database
The task will first warn you that all current data in an existing local database will be removed if you continue, and it will give you a chance to abort the task.

The documentation for Perl Soaplab client can be re-generated by typing: 

   ./build-dev.sh perldoc
Martin Senger
Last modified: Thu Mar 3 21:29:01 2005