We sat down this week with John Powers, CEO of Digipede Technologies in Oakland, California. As a Microsoft Gold Certified partner working with several groups in Redmond, Digipede offers the first commercial grid computing solution built on .NET. I asked John about how the company’s offerings fit with Microsoft’s in traditional HPC and beyond.

Q. When people talk about HPC on Windows, your name always seems to come up. Can you tell us a bit about the Digipede Network?

A. The Digipede Network™ is a distributed computing solution that delivers dramatically improved performance for real-world business applications. The software scales from a handful of computing nodes to thousands. Built entirely on the .Microsoft NET platform, the Digipede Network is radically easier to buy, install, learn, and use than other grid computing solutions.

If a user has an existing command-line process they need to distribute and run on their network, the Digipede Workbench provides a simple, graphical user interface to make that task easy. Instead of requiring any shell scripting to move applications and data around, the Workbench makes it simple for a non-developer to get into grid, quickly and easily.

Of course, the Digipede Network can also work with applications that are already installed on compute resources—again, using the powerful, flexible Digipede Workbench.

Lastly, the Digipede Framework Software Development Kit (SDK) makes it straightforward for developers to create grid-enabled applications themselves.

As for HPC, we see applications there for sure, but my partners and I come out of the enterprise software business. At our previous firms, we used to develop big enterprise applications, and we ran into scalability issues all the time. We developed application-specific solutions for those issues, but we wished for a more general-purpose set of technologies for building distributed applications. At Digipede, we’ve built the technologies we always wanted. Our experience – and our focus – is in bringing the benefits of distributed computing to mainstream enterprise applications. That’s where most of our customers are today.

Q. Digipede is the first commercial grid computing software for .NET; why did you choose .NET?

A. While .NET is great technology, this was a market decision as much as a technology decision -- I like markets that are gigantic and lightly contested. There are plenty of distributed computing options on the Linux platform; most clusters, and most grid computing projects, are dominated by vendors selling solutions on top of Linux. That’s fine – that just leaves more room for us. Windows is the dominant operating system on the planet, period. Yet the options for distributed computing on the Windows platform are quite limited.

Our enterprise customers tell us that they run Windows for a broad variety of applications, and they don’t want to convert to Linux just to access the computing power of clusters or grids. They want to scale out their applications where they run today – which is on Windows.

And let’s be clear that when we say .NET, it’s true that the Digipede Network is built entirely on .NET, but we can distribute applications built with any technology that runs on Windows 2000 or later – including .NET, COM, Java, or for that matter, standalone executables developed in Fortran or C.

When we talk to customers, they tell us, very clearly:

- Most developers use Visual Studio to build applications. It’s the tool preferred by most developers of enterprise applications, both internal developers and ISVs.

- Most computers run Microsoft Windows. For a large class of customers, the added complexity of maintaining a cross-platform distributed computing system is not worth the extra 10 or 20 percent of compute resources they could capture. It is far more important for these customers to support a heterogeneous combination of dedicated and shared resources running Windows (e.g. cluster nodes, desktops, and departmental servers) than multiple operating systems.

But our focus on the Microsoft platform goes beyond Microsoft’s present technology and market share. Microsoft is a terrific partner, and their partner ecosystem is just unbelievable. As Microsoft Gold Certified partners, we’ve received marketing support, access to their product managers, insights into their product roadmap, and numerous other benefits. We just exhibited at the global launch of SQL Server 2005, Visual Studio 2005, and BizTalk Server 2006 in San Francisco, where we met with numerous potential partners – ISVs, OEMs, system builders, resellers, integrators – and we’re committed to playing an important role in the Microsoft distributed computing ecosystem.

Q. So that’s the enterprise story; what about scientific and technical computing, the traditional HPC market?

In traditional HPC, in academic and scientific computing, the story is different, of course. Most applications do NOT run on the Windows platform. But even here, we’re hearing a lot of compelling stories about why customers want a Windows option. They are tired of multiple Linux distros, of chasing down the right version of the right libraries to run with the right distro, and just the general level of tweaking that goes on to get an application running properly on a Linux cluster.

That’s still the minority. There are scientists who have really gotten into the whole Linux culture, and are happy to climb the relatively steep learning curve, to compile their own OS and libraries to tune their application just so. But there are also a lot of scientists who tell us this is not what excites them, and that they are eager to run with a single, reliable platform. Microsoft has a lot to do to establish itself as that platform, but the door is open, and the Microsoft HPC group is doing a great job in starting that process. We see a lot of opportunities to work with that group.

Q. You mentioned that your customers are mostly outside of traditional HPC; can you give some examples of how they are applying your software?

A. Sure. We’ve got customers running simulations, statistical analysis, heavy text processing, and media transformation. We have fund managers doing heavy financial engineering, we have electric utility analysts running complex system operations models, we have software developers researching better algorithms for email filters. The breadth of applications is incredible, and growing week by week.

We’ve also got customers using the Digipede Network to deploy “Software as a Service” or SaaS offerings. For example, there’s a company in San Diego that runs events – conferences, parties, and the like – and they’ve built a portal through which their event coordinators, customers, and vendors can share information. That site does a lot of report production, and churning out nicely formatted reports turns out to be more compute-intensive than you might think. Given the nature of their business, they get bursts of traffic around the time of each event (and they run thousands of events each year), so they need to support many simultaneous users. They used the Digipede Network to scale out this application, and increased the number of simultaneous users by a factor of ten with no change in quality of service.

The nice thing about this example is that there are so many similar possibilities. Think of a media company with a library of broadcast quality video that wants to make selections from that library available on cell phones or other hand-held video players. The configuration I just described above would allow them to do that on demand – without requiring the video files to be converted and stored in multiple formats on the company’s own storage systems.

Q. You mentioned Software as a Service. Can you tell us some more about that?

Really, this is just the beginning for Software as a Service. We see Microsoft’s increasing emphasis in this area as a huge opportunity for us. There are three infrastructure components you need to deliver really interesting next-generation services:

• A flexible, scalable, high-performance database. Microsoft has just delivered on that, with SQL Server 2005.

• A simple, powerful way to build and deploy Web services. Microsoft has just delivered on that, with .NET 2.0 and the huge productivity improvements in Visual Studio 2005.

• A simple, flexible way to allocate your computing power to your service. That’s what Digipede brings to the table. These are the components on which the most innovative and useful next-generation services will be built. And by integrating with Visual Studio 2005 and SQL Server 2005, and building entirely on .NET, we’re positioned to grow as Microsoft’s presence in this space grows.

Clearly these opportunities extend into the HPC arena as well, into scientific and technical computing. As organizations collaborate more, there are more opportunities to share expertise and resources. Creating a protein folding service, or a BLAST service, or astronomy services like those Jim Gray at Microsoft Research is working on – all kinds of new science will be possible with the right infrastructure behind new services.

Q. Microsoft’s efforts in HPC are centered on the Compute Cluster Solution being demonstrated this week at Supercomputing in Seattle. How does the Digipede Network compare to Windows Server CCS?

We’ve been in touch with Microsoft’s HPC team from the beginning, and we’ve shared our roadmap with them. By design, we’ve taken a complementary approach to theirs. The HPC team has been very clear about their focus: They’re going to make Windows compute clusters a viable choice for high-performance computing. To do that, they’ve been working on a Windows-tuned MPI stack, working with the Visual Studio team on support for parallel debugging, working with the 64-bit OS team on support for fast interconnects, and building a job scheduler to support all that. They’re getting support from compiler developers, from developers of numerical libraries. And they’re working with ISVs in scientific and technical computing, getting them to support the Windows platform.

As experienced Microsoft partners, we know how to take advantage of what Microsoft provides, and to build new capabilities on top of that. So we’re building on top of the tools provided by the HPC team -- and the SQL Server team, the Visual Studio team, the Windows Server 2003 team, the .NET Framework team, and so on.

The Digipede Network can live alongside CCS on each compute node, or it will interoperate with the CCS Job Scheduler to launch jobs on a CCS cluster. (I say “will” because they’re not shipping yet; we’ve got the most recent CCS beta, and can show you what I mean, but we will officially support that interface when CCS ships next year.)

So – we can take advantage of their Job Scheduler without recreating those capabilities. We don’t need to develop and tune an MPI stack; we can use theirs. For applications that require support for fast interconnects, Microsoft got that covered for us. Instead, we build new capabilities on top of their platform, and add the ability to run applications on shared resources (not just dedicated cluster nodes), the ability to run distributed applications on 32-bit systems (not just 64-bit), the ability to harvest resources from 32- or 64-bit desktops, servers, or cluster nodes, and so on.

We also offer a very “enterprise-developer-friendly” programming model, with the ability to instantiate objects locally and execute them remotely without worrying about any of the distributed computing plumbing. While there is a lot of MPI talent in the scientific and technical computing community, that talent just isn’t there in the enterprise computing community. So we’ve spent a lot of effort on our Digipede Framework SDK, through which programmers can grid-enable their applications without learning new programming techniques.

As a result, we’re targeting different types of applications than the Microsoft HPC team. They’re looking at shared memory MPI jobs, tightly coupled jobs with lots of inter-processor communication, things like finite element analysis and computational fluid dynamics. We’re looking at loosely coupled jobs, parameter sweeps, Monte Carlo analysis, other types of scenario analysis, complex searches, and other problems that are more naturally parallel.

But to me the most exciting opportunities are with customers that have a need for a hybrid solution, what we’ve called an “extended cluster.” In those scenarios, the ability to add shared desktops and servers to a dedicated CCS cluster dynamically really tips the economics for the customer away from a Linux cluster in favor of a Microsoft / Digipede solution. We’ll have some specific stories about that soon.

Q. Grid computing, distributed computing, cycle harvesting – these ideas have been around for a while; why are you having success when others have not? Is it just low price?

As we have talked to customers in this market, the feedback we’ve received about grid computing has been remarkably consistent. Price per se is NOT a barrier to adoption. The number one barrier to adoption is complexity.

With that complexity has come a very high threshold for the adoption of grid solutions. When you need consultants, and long feasibility studies, and longer implementation projects – just to get in the game – well, that’s a high threshold for most customers to cross. As a result, grid customers tend to be very large organizations, and they tend to spend a lot of time and money dealing with complex installation, configuration, integration, training and support issues.

Our goal is to eliminate complexity wherever possible, and to provide a much lower entry point. As you weed out complexity, you open this market to a much wider range of customers. That’s how the benefits of grid computing will really start to show up – when they reach small to medium businesses, and individual departments within larger organizations. That’s where the energy is, that’s where the innovation is, that’s where the greatest needs are,

Our pricing is part of that strategy. When you can buy great grid computing software for under a thousand dollars, when you can get it installed and running without spending a dollar on consultants, then it’s much easier to get started. You don’t have to cross this high threshold of cost and complexity – you can get started today and see the benefits for yourself, in your own organization, and then add more power as you need it.

When you talk with folks in small and medium businesses, and in small and medium labs and departments within much larger organizations, you find out – they buy products that solve problems. They are not going to do big consulting projects to study how they might change their whole IT infrastructure to improve application performance. They’re going to buy something they can use right away, without a lot of fuss.

We don’t see our product pricing as dramatically lower than other solutions out there; our price per node is fairly similar to other offerings. The real difference is the lower entry point; the simplicity of implementation makes it possible for us to support a five node sale. Our competition can’t say that.

Q. At the same time, five nodes don’t deliver the performance improvements of 5,000. Who can gain real benefits from these smaller systems?

A. Well, I have no objection to selling a 5,000 node system. But we’re seeing customers benefit from 5, 20, 50-node systems in many different fields. If your most important analytic application runs for an hour on a single machine, and we can cut that time to 12-15 minutes with a 5-node implementation of the Digipede Network Team Edition, that’s a big deal. That means an analyst can get four or five times as many model runs done in a day as before, which changes the way he or she works every day.

And then it’s time to add more nodes, which is just what we see happening. I’ll give you an example. One of our customers is a company called InBoxer. They make an advanced spam filtering product, and they do a lot of statistical modeling – their classification of email as spam or not spam gets better the more they calibrate it by exposing it to huge amounts of email. Their statistical models were running all night on a single processor. They got the Digipede Network, installed the Digipede Agents on four or five machines, and saw an immediate impact; their jobs were cut down to less than two hours. Their CTO was so impressed that he went shopping – he bought several additional servers, deployed additional Digipede Agents on those servers, and they’re down to about 40 minutes on those same jobs. That’s huge for them – it completely changes the way they interact with their data and their models, and the pace at which they can improve and grow their business. And we made exactly zero trips out to help them with this – they got everything up and running themselves.

And on slightly larger projects, the pattern is similar; we’re working with a big fund manager who is implementing a 50-processor system to replace some home-brew scripting they did themselves. They get better performance, ease of maintenance – but the biggest benefit they identified is ease of expansion. As they add new applications, add new users, add new assets to their portfolio, their processing requirements grow, and they can add more agents incrementally as needed.

Q. When you say “add more applications,” is that usually done using the Digipede Framework SDK you mentioned earlier?

A. Absolutely. This is a really important area for us, and if I were to point to any single point of differentiation that has gotten customers most excited, this would be the one. The whole Digipede team comes from the enterprise software industry, and we’re keenly aware of the needs of developers. We realize that application developers know their applications well, and want to focus on the functionality their applications deliver – and not on the nuts and bolts of distributed computing.

The Digipede Framework provides developers with the easiest path available for grid-enabling their applications. It fits right into Visual Studio (.NET and 2005) – and that means all the documentation is accessible the same way the MSDN documentation is, and that IntelliSense makes the Digipede Framework classes as easy-to-work-with as all of the .NET Framework. In short, we allow developers to work the way they’re used to working, so developing a distributed application is no more difficult than developing any other kind of application.

The Digipede Framework API is remarkably simple for the powerful functionality it enables. For example, developers can designate the classes in their applications that they would like distributed on the Digipede Network. At run time, each object of that class that they instantiate is automatically migrated by the Digipede Network to a machine ready to execute the work; a method is called by the Digipede Agent on that machine. After the work is completed, the objects are re-serialized and streamed back. Nearly all of the developer’s code is unchanged; the object-oriented nature of .NET programming is preserved completely. The developer no longer needs to worry about the difficult parts of distributed computing: how and where to move the appropriate EXEs and DLLs, determining which computers are on the network and available for work, and guaranteeing execution by monitoring processes. All of those chores are taken care of by the Digipede Network so the developer can concentrate on his or her own application.

Q. Do you have anything to add before we wrap up?

A. I want to wish you well with WindowsHPC.org – it’s a great site, and it fills a real market need. I thank you for the opportunity to discuss our role in the Windows HPC community and beyond. I look forward to working with you and Microsoft to build that community.

http://www.digipede.net