Chapter 9: A Distributed Grid Application

Running a complete workflow

This is an example of how a complete workflow can be constructed using a script. The goal is to show how a certain job can be divided into pieces, sent to multiple other nodes for execution and the results be returned and reassembled.

In this particular example we are taking a 24 minute full quality DV format movie, splitting it up and sending it to two different machines that represent a PBS cluster and a SGE cluster for re-encoding. The results come back and are re-assembled into a complete mpeg4 version of the movie.

In theory the process as executed on our three node grid should take less time than it does to simply execute a similar re-encoding operation on a single machine. Whether it actually does or not depends on many factors, however, the point of this example is to show how to build a useable compute grid.

A simple Python script "encodeWorkflow" can be downloaded and used to drive a complete workflow. After downloading this file be sure to run the chmod 755 encodeWorkflow command so that the script is executable.

Using the script the movie will be sliced into 12 equal pieces, each two minutes in length. The slicing will be done on the local client machine (nodeA). As each two minute piece is sliced off it will be sent out "onto the grid" to be re-encoded on a remote resource. The resource chosen (either the PBS or the SGE "cluster") will be random (a more sophisticated grid could choose based on the cluster load, but that is beyond the scope of this tutorial). The re-encoded slices will be sent back to the client node and when all the pieces are in place they will be recombined (on nodeA) to form the entire re-encoded movie.

Run the command "encodeWorkflow --help" for the usage message. In short you will need to provide the path to the input movie, the name of the host where the PBS GRAM WS jobmanager runs, and the name of the host where the SGE GRAM WS jobmanager runs (the script assumes that container is running on the default port along with the globus-gridftp-server).

[jane@nodeA ~]$ ./encodeWorkflow --input=./movie.dv --output=./movie.mp4 --PBShost=nodeb.ps.univa.com --SGEhost=nodec.ps.univa.com

Running the command above will produce output detailing events in the workflow. The details will depend on how fast your machines run and how fast their I/O is. Do not be surprised if most of the grid jobs are completed before nodeA is finished slicing the movie, an example of an obvious decrease in the wall clock time needed to complete the workflow, because of the use of grid resources. This will be evident towards the end of the output. Lower number slices will report that Grid encoding has completed almost immediately as these operations were farmed out and returned during the lengthy nodeA slicing operation.

When the workflow completes, the result should be a movie.mp4 file which is a mpeg4 version of the full quality DV format movie movie.dv. Note that although Quicktime does indeed play mpeg4 format files it may not play this one due to the newer header format. Instead use mplayer or VLC media player if you wish to see that this workflow does indeed produce a useable result.

The intermediate files can be removed by running the command again and using the single '-c' flag:

[jane@nodeA ~]$ ./encodeWorkflow -c