GCJ: Give us an abbreviated history of Infiniband ...

Eiref: Infiniband is a server interconnect technology that was conceived back in the late 90's. At that time, there were lots of proprietary interconnect technologies from all of the server vendors - Giganet, Quadrix, Hyperfabric, etc. They were all interesting, but they were all proprietary - and all had lots of software problems and were only available from one vendor.

There were several competing industry consortiums around all of the server vendors, and they eventually merged together to form Infiniband. Infiniband ended up being an industry standard based largely on support from Intel and Microsoft. It's gone through several levels of evolution - we're in the third generation of Infiniband chips right now.

There are other technologies for interconnect, of course. The traditional way that enterprises hook stuff together is Ethernet. Infiniband uses a point-to-point architecture, and is really the best interconnect for server clustering. If you want to learn more about how it relates to Ethernet and Myrinet, you might want to check out this Cisco white paper on "Understanding Server Interconnect Technology."

GCJ: Cisco's acquisition of Topspin was big news a few months ago.

Eiref: Cisco loves this technology not because it's an alternative to Ethernet - because it actually creates interesting new possibilities for how servers are connected together, which really lends to some company directions in virtualization products and capabilities. In your typical large enterprise, you're still going to have your big Cisco routers and Andiamo storage switches, but you will also now have a set of products which connects to both of those, and serves as the interface of the servers themselves.

Infiniband is really a story about servers and how servers are connected to each other. It's the fastest standards-based technology for server clustering.

GCJ: What does that mean in the context of Grid computing?

Eiref: One issue that Infiniband tackles that's important to Grids and clusters is latency. Infiniband is fundamentally written around a low latency architecture called RDMA (remote direct memory access). It's a whole bunch of software protocols that allow nodes in a fabric that talk directly without having to go through the TCP stack or the kernel, which introduces a lot of latency and drags the processor down.

And Infiniband not only offers low latency, it also offers very high throughput. With Infiniband today, we're shipping up to 30 gig interconnect. Most people would say 'I don't need 30 gigs - what am I going to put on that pipe?' But as enterprise moves to commodity, 'scale-out' environments where they're buying Intel boxes instead of SMPs, high throughput becomes extremely important. You have to connect these servers so they all talk together really fast and behave like a bigger system with crossbar switches and stuff built inside, and so they can do direct memory access and sharing very quickly as well.

In a large Grid-like environment, the last thing you want to have to do is connect every server in your Grid to its own LAN, its own WAN, its own storage fabric, its own IPC fabric. Particularly as the server gets smaller and smaller and faster and faster, the last thing you want to do is have ten different connections in the back of the box. It's all about consolidating all of the pipes into a fabric. Ethernet is going to be here forever and will get faster and faster, and the same for fiber channel - but you don't want every node in your data center to necessarily have its own fiber channel connections. Fiber channel connections are expensive - a couple thousand bucks for an adapter card, another thousand bucks for a switch port.

Let's say you have a mid range application in the datacenter and want everything to be redundant. Now you need two of everything. Instead of one NIC [network interface card], you've got to have two. Instead of one proprietary interconnected to IPC [interprocess communication], you've got to have two. And then you have to have management on top of that - so you're talking 8 expensive connections going into really cheap servers. So with Infiniband, you can allow all of these things to talk to each other over one big pipe. That's very interesting for Grids. You want all of the systems to be able to run over very high performance levels and different types of applications. Infiniband allows you to consolidate lots of connections, which allows you to eliminate a lot of complexity.

The ultimate example of this is in the blade space. Many of the blade vendors are shipping blades with Infiniband built right into it. In a blade environment, you fundamentally can't have on a blade all sorts of PCI [peripheral component interconnect] slots. The whole idea of a blade is to compress everything down into a tiny blade and you plug it into a chassis with one back plane, and everything has to go in that one back plane. You can't have lots of different connections on that back plane. Whether it's LAN traffic, storage traffic or IPC communication between those blades. Infiniband has emerged as the way to connect blades together.

GCJ: What does management look like in Infiniband environments?

Eiref: The way we've engineered everything, it looks to the outside world just like a standard server. So if you've got some standard management tool - the server looks just like any other server in the datacenter. It looks it's got an Ethernet port. It looks like it's got a fiber channel port. You can manage it with any of the tools out there. We do also have a product called Vframe, which is our management capability that allows you to program the way in which CPUs, LUNs [logical unit numbers] and virtual LANs are managed.

Think about the three core resources in the datacenter - 1-CPUs; 2-storage LUNS, if you're using SAN; and 3- virtual LANS coming off your Cisco routers and switches. So you need to connect the three of them together. Vframe allows you to tell the fabric which storage images to boot which CPU off of, on the fly. Let's say you have a bunch of different apps and images in your storage arrays ... you can say "boot this cpu off this Red Hat image, and boot the other off that other type of image." So what you've got now - you've dissolved the old fashioned mainframe recipe world. It's core components - cpu's, storage and network - and now you're re-building out of commodity components by connecting them all together through a single fabric. And now you're applying on top of that intelligence that allows you to control who does what inside of that fabric.

close window