GCJ: Tell us all about the Metrics project.

Liming: The long-term vision for the Metrics project is to be an open source effort to evaluate the impact of the Globus Toolkit on the grid community and on the broader computing community in general. Where it could be going in the future, of course, is also always on our minds.

More immediately, we're using largely automatic reporting methods for collecting data and performing statistical analysis to gauge the use of Globus software. In other words, how much is the Globus Toolkit being used, and what can we tell about the impact based solely on usage? Recently, we've been focusing on developing self-reporting code so that Globus reports its usage. We combine that with the existing logging capabilities of our infrastructure services - for example, the Bugzilla system, the website, the e-mail lists that we run for the Globus community. The larger goal of the project is to allow the community to participate in making sense of what all this data is telling us about Globus's usage and impact.

GCJ: When was this project started?

Liming: The protoproject was proposed in May of this year, and then it was formally launched by the Globus Incubator project on May 29th.

But actually the groundwork was laid much longer ago than that. In April 2004 we received a small NSF grant to add instrumentation to the Globus GridFTP code that would allow us to directly measure GridFTP usage.

As one of the major sponsors of the development of Globus software, the National Science Foundation is very interested to know, quantitatively, how broadly the software they've been funding is being used, and, to the degree possible, who was using it, and for what end. As they continue to invest in the development of the software, they have to be able to justify to Congress and to the American public that this money is well spent.

So with a modest grant we were able to add this usage monitoring capability to the most popular services in the Toolkit, starting with GridFTP and the Web services GRAM service, the reliable file transfer (RFT) service, the replica location service (RLS), and the Web services containers in C and Java.

When the project was being envisioned and executed, one of our major concerns was that we avoid compromising our user base's privacy. So the scope of our inquiry has been carefully limited to the operations of the software itself - when it starts, quits, or performs a specific action. Personal information like who is running the software or where the data is transferred is not reported.

We're developing a formal usage policy to govern what we do with this data, and it will definitely include privacy provisions.

GCJ: I noticed on your site, on dev.globus, that there were two metrics you're measuring - quantity and quality. Can you talk a little about each of those?

Liming: Sure. When you're doing impact analysis or studying the appropriateness of technology to a particular user base, there's always a choice between quantitative and qualitative analysis of information.

Quantitatively, we might ask how much has the software been used? Or, how many times has a particular piece of code been run? How many different systems has it been run on? How much has it been used when it's running? Anything we can measure directly, either through self-reporting software, or by other means, is fair game in quantitative analysis. In our case, we're using the direct measurement as much as possible. Quantitative analysis is relatively easy to do, but the information it provides requires a lot of interpretation and isn't especially well suited to answering broad questions like whether the software is useful or not.

Qualitative analysis would usually encompass methods like surveying or interviewing users. Then there are methods of performing qualitative analysis on that kind of subjective data to generate sound conclusions. Qualitative analysis is difficult to do well, but the results are usually directly applicable to questions about value, strategic importance, or future directions.

Right now the Metrics protoproject is focused entirely on quantitative methods. And the reason for that is basically because we don't have the skills in the current team to do solid qualitative data collection or analysis. So we're going to stick to the quantitative stuff until we have more participation in the protoproject on the qualitative side.

GCJ: How are you delivering all this quantitative data and analysis?

Liming: Well, for some time the protoproject has been producing monthly reports that summarize the data collected from the sources I mentioned previously. Because we are still formulating policies to guide the handling of the data, we've been cautious in circulating those reports publicly.

One of the biggest challenges thus far has been balancing our desire to be open and collaborative in our approach with the outstanding concerns we have regarding the privacy of our users and the accuracy of the data itself. The reports are indeed publicly available in the Bugzilla system, but we haven't advertised that much.

Regarding the accuracy of our data, we're quite confident that some of the data-like the Web site usage statistics-are very accurate. But our usage reports from, GridFTP servers, for example, may not be as reliable. When we designed the usage reporting code, we opted for a less-intrusive reporting system that has a very low impact on systems and networks. The reporting system uses an unreliable transmission protocol for sending these reports, which means that we may not be receiving all of the reports that are being sent. So we've been hesitant to make high-profile claims about our usage data. We certainly don't want to sell ourselves short.

In the meantime we've thought hard about how to correct that uncertainty, and a couple of proposals have been circulated, but it's really a difficult problem. For the moment, at least we know something about the lower bounds of usage and the minimum number of users, and so on.

GCJ: Metrics sounds a bit like Netcraft, an organization which tracks usage of systems. They started reporting on Apache usage, and they've since expanded to other applications.

Liming: Yeah, that is very similar to what we're trying to do.

Of course, our motivations are unlike Netcraft's. We have a commitment from the National Science Foundation for roughly five years to do continuing development and support for the Globus Toolkit. And this Metrics protoproject benefits that work, so some of that funding is going towards work on the Metrics protoproject.

Our arrangement with NSF allows us to take a few risks in our analysis, as long as we are clear about what's certain and what isn't. Like I said, direct measurements lead to straightforward analysis, but the results of the analysis are harder to interpret and use in decision-making. Qualitative analysis is harder to conduct, but the results are likely to be more useful. Roughly midway through the project we hope to start asking some qualitative questions of people and getting some deeper answers. Many companies would not risk marketing the kind of analysis we're doing now, although some certainly have.

GCJ: Right. You know, Apache used Netcraft's numbers almost as a marketing tool to demonstrate their growth and to promote the adoption of Apache. It seems to me if the Metrics protoproject is a success, it might have a similar effect on Globus.

Liming: That's one of the reasons we're doing this. We want to demonstrate the growth in use and adoption of these tools, and indicate what it means.

But we're hopeful our data will also provide the Globus development community with data that they can use to help identify issues in the software that need to be addressed. For example, our GridFTP server reports how many parallel streams are used for transfers, how many stripes are used for transfers, and how many unique nodes are participating in each end of the transfer. And if we see, for instance, that nobody is using the parallelism or that everybody seems to be using a particular setting that we know is not optimal, that can tell us that we need to better document the software or change the default configuration. Ideally, then, the Metrics protoproject will help the community use the software more optimally and help us improve and streamline components like the installer.

GCJ: I see. What should the Globus community expect to see from you in the upcoming months?

Liming: Well, we just started including significantly more of the usage data in those monthly reports. Our July report includes usage data for the WS GRAM and RFT services, which had not been included before. Another milestone will be the addition of the GridFTP data.

And then sometime over the next six months, as we continue to gain confidence in the data and our analysis, we will make our reports more readily available on our website.

close window