Saturday, February 26, 2011

Financial sustainability for UCAN and R&E networks

[ I am pleased to see that UCAN has established a committee to look at the long term financial sustainability of UCAN ( . The deliberations and output of this committee will not only benefit UCAN but R&E networks around the world.
One of the big challenges facing UCAN, as well as universities and R&E networks is the huge indebtedness and near bankruptcy of most governments. The glory days of large dollops of funding from government for R&E networking or basic research are over.  All public sector institutions from universities, libraries to hospitals are going to be seriously affected by budget cutbacks and layoffs. So finding new money to pay for membership in a new service such as UCAN, no matter how meritorious, is going to be particularly tough.

I have blogged about this many times in the past. I think this is a great opportunity for R&E networks to not only help these  institutions reduce their overall costs, but also provide a new funding source for R&E networks themselves.   Originally I had expected (hoped) that carbon reduction would be the major vehicle for achieving this with the much anticipated national cap and trade system in the US.  Although cap and trade may be effectively dead, reducing energy costs from IT at a these institutions still has merit and can still save millions if not tens of millions of dollars. Money that can be redirected to critical research and education activities as well as underwriting the costs of the R&E network itself.

In a recent study in the Netherlands of over 9 universities it was found that IT represents 20-40% of all electrical consumption at these universities ( Surprisingly it is the smaller institutions where IT represents the biggest energy consumption of close to 40%.  For larger institutions, at least those without large data centers, IT energy consumption is closer to the 20% mark. At the larger institutions there are many large pieces of research equipment that draw significant power and as a result lowers the relative consumption represented by IT.  These numbers correspond to data I have seen from other studies in the UK, USA and Canada.

R&E networks can also help institutions in reduce their overhead costs in managing their internal networks and services as well.  The complexity of managing networks and services of a campus network has grown exponentially. Many smaller institutions in particular, don’t have the skill sets or resources for managing complex infrastructure and supporting new services such as DNSSEC, IPv6, Eduroam, liightpaths, etc. eScience research is especially affected by this growing complexity in the campus network. An excellent presentation by Eli Dart presented at the recent Joint Techs clearly demonstrates the scope of this problem (  Most university IT folk are largely focused on the problems of maintaining the campus IP network and its related services.  Few have the skills or knowledge of the needs of high end eScience applications. Many campus networks have multiple choke points, packet loss problems and mis-configuration issues. As result many researchers who want to adopt eScience practices end up being frustrated and don’t move as quickly as they wish into the exciting world of eScience.

I believe R&E networks can play a significant role in not only reducing the energy costs of IT infrastructure at a typical university or UCAN member, but the same processes to reduce energy costs can also help reduce the high overhead costs of managing a campus IT infrastructure and more importantly remove many of the road blocks prohibiting the faster adoption of eScience practices and procedures at university or research institute. While we cant hope to eliminate the entire cost energy cost of IT, a small fraction in savings can be enabled  by the R&E network which would represent  thousands if not hundreds of thousands of dollars for a typical university or UCAN connected institution.  And all this can be done without requiring a single new dollar of additional investment from the connected institution.

One of the first and possibly largest challenge of achieving significant energy savings is to get the IT department to talk to the facilities people on campus.  Energy issues have typically been in the bailiwick of the facilities department. These departments, in most cases, have implemented various energy reduction strategies such as replacing lightbulbs, installing smart building management systems, and raising or lowering (depending on your location) the average temperature in a building.  However facilities people are generally stymied when it comes to the issue of reducing energy consumption of IT equipment.  There have been a few initial efforts by IT departments to reduce energy costs through more efficient data centers, virtualization and encouraging users to turn off computers. But these efforts have had marginal impact to date.  The biggest problem is that IT folks rarely  pay for the cost of power, space or cooling. This is unlikely to change since IT energy consumption is made up of thousands of diverse and independent devices scattered throughout the campus. Unfortunately there are no large, relatively simple campus wide solutions, such as the equivalent to changing to CFP light bulbs, that will be equally effective for IT.

To help IT departments understand the requirements of the facilities people, R&E networks I believe need to partner with what are called Energy Service Companies (ESCOs). Many universities, schools, libraries and hospitals already contract with ESCOs to reduce their energy costs. The usual arrangement is for the ESCO to invest in a variety of energy saving practices such as installing solar panels, outfitting buildings with insulation, deploying building management systems, etc. This is all usually done at no cost to the client. Instead the institution agrees to split the energy savings with the ESCO over a contractual period of several years.  The institution benefits in reducing its energy costs, without having to find additional money to underwrite the capital costs of such energy improvements.  And of course the ESCO makes money by sharing in the energy savings.

By partnering with an ESCO, a R&E network gets access to investment capital that can be used to help a university reduce IT energy consumption. More importantly the ESCO can talk the language of the facilities people in terms of contractual relationships and delivering real value in terms of cost savings as well as being a facilitator of the financial transactions between the facilities department, the IT department and the R&E network.  As I mentioned previously passing on the true costs of IT within a university will be extremely difficult for both political and technical reasons.  Doing it externally through an ESCO and a R&E network partner may be a lot easier.

The easiest energy saving that R&E network can do for an institution is the relocation of the various network servers and routers off the campus to some central location. Essentially the campus becomes a LAN extension to a central location where the R&E network can deploy virtual and or real servers and negotiate for low cost (if not  100%  renewable) power.  High speed optical network with multiple CWDM or DWDM wavelengths is essential for such a strategy,  as any congestion, will significantly affect LAN performance of many network services such as DHCP, DNSSEC, border routing etc. Effectively the campus DMZ and border router is moved off campus to a central location physically managed by the R&E network (Policy, configuration management, access control is still done by university IT staff). As most traffic at an institution is now to and from the outside world, and not within the campus, as in the old days, this arrangement starts to make a lot of sense.  It also allows the R&E network to aggregate many servers and provide the necessary high level support many network applications and services require.

Relocating networks servers and routers  also will help many eScience applications as now their traffic can be offloaded the campus IP network to a dedicated lightpath to the R&E network centrally managed facility. R&E network staff then can work with the eScience researcher to interconnect and fine tune their facility to work with global eScience infrastructure such as GLIF, LHCONE, XD, PRACE, etc. A great example of this type of arrangement is called the “Science DMZ” which is outlined in Eli Dart’s presentation mentioned previously.

But how do we motivate university staff, facilities departments and ESCOs to move in this direction?  Very simply – by making public state IP address a surrogate for energy consumption.  Most servers generally need static public IP address in order to be accessible to users on or off campus.   Charging users for public static IP addresses if their server is not located in a data center or off campus will be a simple way of using IP addresses as a surrogate for energy hogs.  The number of addresses assigned to a particular machine is also a rough guide to its size and energy consumption.  These charges can be applied by the R&E network to the university as part of its membership fee.  But if the university relocates and virtualizes the server off campus to a centrally managed location, then it can be relieved of the costs of the static IP address and the R&E network in partnership with facilities department and ESCO split up the energy savings.

Many institutions are plagued with the problem of what are called “server huggers” and/or “closet servers”.  Not are these devices hidden energy hogs in many cases they are breaking local electrical/fire safety codes.  In a recent study done by University of  British Columbia they found that about 15% of the university electrical consumption was directly due to these server huggers. To motivate the owners to move these servers off the campus, or at least into a campus data center, charging for static public IP addresses may be a simpler mechanism than trying to allocate the electrical, space and cooling costs from the facilities department.

Similarly dynamic IP address assignment as part of DHCP can be used as a motivator to get campus users to turn off computers and other devices.  Those users who send a note to the IT department noting that they are shutting down their computer (which also can be done automatically as part of the shut down processes) can be rewarded with automatic dynamic address when they turn up the computer in the morning. Those who don’t turn off their computer will have their IP address expire at midnight if there is no activity from that given computer. In the morning they would be forced to manually request an IP address.

These are just some early ideas of how R&E networks can help universities save money and also underwrite the costs of the R&E network itself.  Some of them may not work, but I am sure with a little thought readers of this blog can find many other innovative ways to reduce energy consumption and relocate precious dollars to critical research and education activities. The bottom line is that we desperately need to change our thinking of R&E networks  from looking for handouts from governments to be agents of change that truly benefit universities, schools and other institutions in reducing costs  and enabling further acceleration of eScience.

Green Internet Consultant. Practical solutions to reducing GHG emissions such as free broadband and electric highways.

twitter:  BillStArnaud
skype:    Pocketpro