Stuart Martin, Senior Software Developer for the Globus Alliance at Argonne National Laboratory, is responsible for the ongoing development, distribution and use of the Globus Resource Allocation Management technology (GRAM) in the Globus Toolkit. The Globus Consortium Journal recently caught up with Martin to learn more about recent GRAM developments.

GCJ: When and why did Globus kick off the GRAM effort?

Martin: We kicked off the GRAM effort in April of '97. If you're going to share computational resources on the Grid, you need a mechanism for requesting remote system resources for the execution of the jobs. So with the Grid Resource Allocation manager, we cooked up a schema, a job definition -- and invented our own language to communicate the job requirements to the resources. GRAM is really there for the applications, to start and run Grid services. After the release of GT4 in April, now we use an XML schema to specify those requirements. The latest version, WS GRAM, is a WSRF service. This is the first version of GRAM to incorporate WSRF.

GCJ: Was that the biggest change to this version of GRAM?

Martin: Changing the guts of the GRAM service from OSGI to WSRF was considerable. But at the same time we wanted to make other improvements. The biggest sore spot for users in previous versions of GRAM was scalability.

Previously, GRAM had a problem with overloading the head node when too many job requests hit the GRAM service. First the GRAM service did not throttle/control itself and second each individual job would query the batch scheduler which caused a small, but continuous load. "Death by 1,000 cuts."

In GT3, we created a new internal GRAM service to poll the scheduler once for all jobs. Every individual job would be notified by this new internal GRAM service to find out the state of their job. But here, you’d have the delay of a process polling, and if they're polling and assembling lots and lots of data, each poll takes a lot of time and causes system load. And so in the end, that didn't scale well either and caused delays in latency.

So in WS GRAM, we now sit on top of the scheduler's log file, and we have one process reading the log file and injecting events back into our GT4 container into the WS GRAM service. This circumvents the typical method that scheduler provide for access to job state information, but it has proven to scale and provide low latency to job state changes.

GCJ: So WS GRAM has lower latency than previous versions of GRAM?

Martin: Lower latency than GT3 WS GRAM, slightly higher then pre-WS GRAM, but it is much more scalable than either. If you have thousands of jobs submitted into your scheduler, and only ten are running at once, then the thousands pending should not cause added load from the GRAM service. In past versions those pending jobs did cause added load. This would affect performance in a Grid. So it was a pretty significant problem that we fixed in this WS version of GRAM.

So basically we've been trying to just make it so that we are not a bottleneck. We just are this nice thin interface on top of a site's batch scheduler. And so that's what our focus has been with GRAM development, and I think we've come pretty far with WS GRAM... to where we can scale, and to where we can throttle so we don't fail over. It's now reliable and robust enough to handle demanding loads in a Grid.

GRAM cannot scale indefinitely, so what we are striving for is that the client will get a reasonable response in order to understand the situation of the GRAM service and the underlying batch scheduler it is interfacing with. For example, GRAM may start to reject jobs if it knows it cannot manage any more at a given time.

The ideal GRAM scenario is one in which it adds little to no overhead compared to an underlying batch system so the user can submit as many jobs to GRAM as they can to the underlying scheduler. Today we've reached the point where a WS GRAM service has successfully managed up 32,000 jobs submitted to Condor at one time.

WS GRAM manages states of the jobs too -- in a Grid you need to account for resources coming online and going offline. So you want the client to be able to go away but then come back and connect back up and control it. Within WS GRAM, we maintain some state about each job, and that way if the host machine does go down, then we can reread our state and recover quickly.

GCJ: Is GRAM only used for remote job scheduling, across administrative domains? Or if you had just a single contained grid within an organization, would there be a use for GRAM?

Martin: Even if you were just trying to schedule jobs on your own Grid, you might have reason to use PBS to schedule jobs on one resource pool, and use Condor to another resource pool so you could use WS GRAM as the common standard for both to schedule the jobs.

GCJ: What are some examples of GRAM use in production environments?

GRAM use in production envrionments has included:

• In 2005, ATLAS ran more than 500k GRAM jobs on GRID3.
• Remote access (through GRAM) of fusion applications at Princeton Plasma Physics Laboratory (PPPL) proved so successful that they had to buy another cluster due to the additional job load. 2700 runs of the fusion application TRANSP was performed over a one year period.
• SCEC plans to run production grade simulations of about 50,000 ruptures per earthquake site on TeraGrid, going up to about 100,000 GRAM jobs.

Currently, WS GRAM in GT 4.0.x is currently being evaluated by many organizations. Martin expects WS GRAM to be used in production in the near future and Pre WS GRAM will be deprecated after organizations have migrated.

close window