[IBM has put together a good list of grid projects around the world -- BSA]
http://www.ibm.com/developerworks/grid/library/gr-gridorgs/index.html
Edna Nerona (edna@legacystudios.biz), Consultant, Consultant
26 Jun 2007
Previously, we gave you a "Recommended reading list for grid developers" and "A starter set of open source tools for grid developers." Now we've compiled a list of some of the production projects and organizations that are shaping the future of grid computing. This article provides a comprehensive list of current projects in such diverse areas as cancer research, astronomy, and physics, just to name a few. We also cover tool kits, security, and data management. These were taken from a variety of online sources to introduce programmers, administrators, and new users to specific information and projects related to using, deploying, and developing grid infrastructure.
Production grid organizations
This track will introduce you to production grid organizations, the problems they are solving, and how they're influencing grid technologies. Production grid deployments fall into various categories of grids: general-purpose grids, scientific and community grids, nationwide grids, regional grids, and university grids.
National and international general-purpose grids
Distributed European Infrastructure for Supercomputing Applications
As a consortium of leading national supercomputing centers, the Distributed European Infrastructure for Supercomputing Applications (DEISA) deploys and operates a secure production-quality distributed supercomputing environment. By enhancing and reinforcing European capabilities in high-performance computing, the research infrastructure facilitates scientific discoveries across a variety of science and technology fields. DEISA uses a deep integration of existing national high-end platforms, with a dedicated network and support by innovative system and grid software.
DutchGrid
Established in 2000, DutchGrid has many successful integrated efforts and initiatives that span a range of scientific collaborations. As an open platform for academic and research grid computing, DutchGrid provides globally recognized identity certificates to grid users in the Netherlands. The DutchGrid CA is fully project-neutral. Any not-for-profit researcher and academic user can obtain personal and server or host certificates for use with grid applications.
Enabling Grids for E-science
The Enabling Grids for E-science (EGEE) project brings together scientists and engineers from more than 90 institutions in 32 countries worldwide to provide a seamless grid infrastructure for e-science that is available to scientists. The EGEE grid consists of more than 30,000 CPUs available to users 24 hours a day, seven days a week, in addition to about 5 petabytes (5 million gigabytes) of storage, and maintains 30,000 concurrent jobs on average. Having such resources available changes the way scientific research takes place. EGEE is a four-year project funded by the European Commission.
Grid5000
The purpose of the Grid5000 project is a highly reconfigurable, controllable, and monitorable experimental grid platform grid researchers can use as a testbed for experiments in all the software layers between the network protocols and up to applications. Grid5000 brings together nine sites geographically distributed in France, featuring 5,000 CPUs. These areas include Bordeaux, Grenoble, Lille, Lyon, Nancy, Orsay, Rennes, Sophia-Antipolis, and Toulouse.
LA Grid
Pronounced "lah grid," the LA Grid is the first-ever comprehensive computing grid to connect faculty, students, and researchers from institutions across the United States, Latin America, and Spain to collaborate on complex industry applications for business and societal needs in the context of health. In addition to universities, LA Grid has partnered with industries worldwide, enhancing innovations in many areas, including healthcare, life sciences and hurricane disaster, life sciences, and disaster mitigation.
Open Science Grid
The Open Science Grid (OSG) is a distributed computing infrastructure for scientific research. The OSG consortium's unique alliance of universities, national laboratories, scientific collaborations, and software developers brings petascale computing and storage resources into a shared uniform cyberinfrastructure.
TeraGrid
TeraGrid is an open scientific discovery infrastructure funded by the National Science Foundation. Combining leadership-class resources at nine partner sites, TeraGrid creates an integrated, persistent computational resource. Interconnected via a high-speed gigabits-per-second dedicated national network, TeraGrid provides more than 150 teraflops of computing power and nearly 2 petabytes of rotating storage, numerous scientific data collections, specialized data analysis tools, scientific gateways, and user portals to simplify access to valuable resources, and visualization resources.
Scientific and community grids
AstroGrid
AstroGrid is an open source project built to create a working Virtual Observatory (VO) for U.K. and international astronomers. Funded by the U.K. government, AstroGrid works closely with other VO projects worldwide through the International Virtual Observatory Alliance (IVOA). As a leading member of this community, AstroGrid provides internationally recognized interface standards that are emerging to promote scientific integration of astronomical data and processing resources worldwide.
cancer Biomedical Informatics Grid
The cancer Biomedical Informatics Grid (caBIG) is a voluntary network or grid connecting individuals and institutions to enable the sharing of data and tools, creating a worldwide source of cancer research. The goal is to speed the delivery of innovative approaches for the prevention and treatment of cancer. The infrastructure and tools created by caBIG also have broad utility outside the cancer community. caBIG is being developed under the leadership of the National Cancer Institute's Center for Bioinformatics.
International Virtual Data Grid Laboratory
The International Virtual Data Grid Laboratory (iVDGL) is a global data grid that will serve forefront experiments in physics and astronomy. Its computing, storage, and networking resources in the United States, Europe, Asia, and South America provide a unique laboratory that will test and validate grid technologies at international and global scales. Sites in Europe and the United States will be linked by a multigigabit-per-second transatlantic link funded by the European DataTAG project.
World Community Grid
World Community Grid's mission is to create the world's largest public computing grid to tackle projects that benefit humanity. The success of the World Community Grid depends upon individuals collectively contributing their unused computer time to change the world for the better. World Community Grid is making technology available only to public and not-for-profit organizations to use in humanitarian research that might otherwise not be completed due to the high cost of the computer infrastructure required in the absence of a public grid.
Worldwide Large Hadron Collider Computing Grid
The Worldwide Large Hadron Collider (LHC) Computing Grid is designed to handle the unprecedented quantities of data that will be produced by experiments at CERN's LHC from 2007 onward. The computational requirements of the experiments that will operate at the LHC are enormous. Some 12-14 petabytes of data will be generated each year, the equivalent of more than 20 million CDs. Analyzing this data will require the equivalent of 70,000 of today's fastest PCs. The LHC Computing Grid will meet these needs by deploying a worldwide computational grid, integrating the resources of scientific computing centers spread across Europe, the United States, and Asia into a global virtual computing service.
U.S. regional grids
Northwest Indiana Computational Grid
Northwest Indiana Computational Grid (NWICG) is a partnership of researchers and educators from Purdue University-West Lafayette, Purdue University-Calumet, and the University of Notre Dame. With a focus on national science and research initiatives, NWICG creates cyberinfrastructure that supports the solution of breakthrough-level problems and enabling continuing world-class advances in the underlying technologies of high-performance computing. They are developing a scalable, high-speed, high-bandwidth, science-driven computational grid for Northwest Indiana across the three universities in collaboration with the Department of Energy's Argonne National Laboratories.
SURAGrid
Southeastern Universities Research Association (SURA) is a consortium of organizations collaborating and combining resources to help bring grid technology to the level of seamless, shared infrastructure. The SURAgrid focuses on direct access to a rich set of distributed capabilities for participating research and education communities. SURAgrid promotes the development of contributed resources, project-specific tools and environments, highly specialized access, and gateways to national and international cyberinfrastructure.
Texas Internet Grid for Research and Education
The mission of the Texas Internet Grid for Research and Education (TIGRE) project is to create a computational grid that brings together computing systems, storage systems, and databases, visualization laboratories and displays, and even instruments and sensors across Texas. By enhancing the computational capabilities for Texas researchers in academia, government, and industry by integrating massive computing power, TIGRE hopes to aid in the advancement of biomedicine, energy and the environment, aerospace, materials science, agriculture, and information technology.
Open source grid projects
These grid projects cover a diverse set of areas, ranging from grid infrastructure toolkits, middleware toolkits, data tools, security, and more. The following represent some fast moving projection grid projects and tools. Visit these sites often to keep up to date on how they are leading the progress in grid technology.
Grid infrastructure projects
Open source grid infrastructure projects that can help you set up your own grid.
Globus Toolkit
Open source software developed by the Globus Alliance. The Globus Alliance is an international collaboration that conducts R&D to create fundamental grid technologies. The toolkit includes software for security, information infrastructure, resource management, data management, communication, fault detection, and portability.
Berkeley Open Infrastructure for Network Computing
Berkeley Open Infrastructure for Network Computing (BOINC) is a software platform for projects, like distributed.net and SETI@home, that use millions of volunteer computers as a parallel supercomputer. Source code is available for the platform, and interested C++ developers are encouraged to help develop the platform code. BOINC is currently supported on Windows®, Linux®, UNIX®, and Mac OS X. CPU platform requirements may vary among project clients using BOINC.
Uniform Interface to Computing Resources
Uniform Interface to Computing Resources (UNICORE) offers a ready-to-run grid system, including client and server software. UNICORE makes distributed computing and data resources available in a seamless and secure way in intranets and the Internet. The UNICORE design focuses several core principles: seamless access to heterogeneous environments, security, site autonomy, a powerful GUI clients that provides ease of use, and quick start bundles that allow for simple installation.
Grid middleware projects
The following projects have successfully provided U.S. and international projects with the advanced tools to easily access numerous grid functionalities, such as computation, visualization, and storage resources. You can interact with various grids or have one customized to work with your own grid.
gLite
gLite is the next generation middleware for grid computing, born from the collaborative efforts of more than 80 people in 12 academic and industrial research centers as part of the EGEE Project. gLite provides a bleeding-edge best-of-breed framework for building grid applications tapping into the power of distributed computing and storage resources across the Internet.
National Research Grid Initiative
The National Research Grid Initiative (NAREGI), in Japan, focuses on the research and development of grid middleware so that a large-scale computing environment can be implemented for widely distributed, advanced research and education.
Ninf-G
Ninf is a Japanese project developing programming middleware which enables users to access various resources, such as hardware, software, and scientific data on the grid with an easy-to-use interface. Ninf-G is open source software that supports development and execution of grid-enabled applications using Grid Remote Procedure Call (GridRPC) on distributed computing resources.
NorduGrid
NorduGrid middleware, also known as Advanced Resource Connector (ARC), is an open source software solution distributed under the GPL license, enabling production-quality computational and data grids. ARC provides a reliable implementation of the fundamental grid services, such as information services, resource discovery and monitoring, job submission and management, brokering and data management, and resource management. Most of these services are provided through the security layer of the GSI. The middleware builds upon standard open source solutions like OpenLDAP, OpenSSL, SASL and Globus Toolkit (GT) libraries.
OGSA-DAI
The OGSA-DAI project focuses on the development of middleware to assist with the access and integration of data from separate sources through the grid. The project works closely with the Globus, OMII-Europe, NextGRID, SIMDAT, and BEinGRID, ensuring that the OGSA-DAI software works in a variety of grid environments.
ProActive
ProActive is the Java™ grid middleware library (with open source code under LGPL license) for parallel, distributed, and multithreaded computing. With a reduced set of simple primitives, ProActive provides a comprehensive API to simplify the programming of grid computing applications, distributed on LAN, on clusters of workstations, or on Internet grids.
Security projects
To protect the vital infrastructure and information, security is a constant evolving requirement of grid computing. These projects represent some of the cutting-edge security standards and implementations of grid security solutions.
GridShib
An NSF-funded project between NCSA and the University of Chicago, GridShib integrates federated authorization infrastructure (Shibboleth) with grid technology (the Globus Toolkit) to provide attribute-based authorization for distributed scientific communities.
Grid User Management System
The Grid User Management System (GUMS) is a Grid Identity Mapping Service. Identity mapping is necessary when a site's resources do not use grid credentials natively, but instead use a different mechanism to identify users, such as UNIX accounts or Kerberos principals.
PRIvilege Management and Authorization
PRIvilege Management and Authorization (PRIMA) is a system that provides enhanced grid security. PRIMA is a comprehensive grid security model and system. In PRIMA, a privilege is a platform-independent, self-contained representation of a fine-grain right. PRIMA achieves platform independence of privileges by externalizing fine-grain access rights to resource objects from the resource's internal representation.
Resource management and scheduling
An essential component of grids is to manage and schedule jobs across resources. These projects demonstrate a few strategies.
Community Scheduler Framework
Community Scheduler Framework (CSF) is an open source implementation of an OGSA-based metascheduler. It supports the emerging WS-Agreement specification and the Globus Toolkit's GRAM service. CSF fills in gaps in the existing resource management picture and is integrated with Platform LSF and Platform Multicluster. The CSF open source project is included in the Globus Toolkit V4.0 release.
Special Priority and Urgent Computing Environment
High-performance modeling and simulation are playing a driving role in decision-making and prediction. For time-critical emergency support applications, such as severe weather prediction, flood modeling, and influenza modeling, late results can be useless. A specialized infrastructure is needed to provide computing resources quickly, automatically, and reliably. Special Priority and Urgent Computing Environment (SPRUCE) is a system to support urgent or event-driven computing on traditional supercomputers and distributed grids.
Grid resource monitoring
Monitoring resources and applications is key to the success of grids. Through an easy-to-use interface, these sophisticated tools help users gather, catalog, and monitor various types of resources. Moreover, systems administrators are also able to monitor the health of their grids. These evolving grid projects list a few of the open source options.
GridCat
GridCat is a high-level grid cataloging system using status dots on geographic maps, as well as a catalog. The maps help debug site troubles. The catalog contains information on site readiness with many other valuable information per site to help job submission and job scheduling for application users and grid scheduler developers. GridCat tries to present the grid-site at its simplest status representation.
Gridscape II
Gridscape II is a customized portal component that can be used on its own or plugged in to compliment existing grid portals. Gridscape II manages the gathering of information from arbitrary heterogeneous and distributed sources and presents them together seamlessly within a single interface. It leverages the Google Maps API in order to provide a highly interactive user interface. Gridscape II is simple and easy to use, providing a solution to those who don't wish to invest heavily in developing their own monitoring portals from scratch, and also for those who want something easy to customize.
Storage and data management
From open source high-performance file systems to seamless access of data from heterogeneous environments, the following projects bring together and optimize a variety of storage and data management solutions. This track emphasizes storing, managing, and moving data across resources and connecting data resources over a network.
Lustre
The Lustre File System, a high-performance open source file system from Cluster File Systems Inc., is a distributed file system that eliminates the performance, availability, and scalability problems present in many traditional distributed file systems. Lustre is a highly modular next-generation storage architecture that combines established open standards, the Linux operating system, and innovative protocols into a reliable, network-neutral data storage and retrieval solution. Providing high I/O throughput in clusters and shared-data environments, Lustre also provides independence from the location of data on the physical storage, protection from single points of failure, and fast recovery from cluster reconfiguration and server or network outages.
NeST
NeST is a software network storage device providing secured storage allocation for a specific time period. The size and duration of allocation units or lots are negotiated between NeST and the user or application. These lots can also be expanded in size, extended in time and/or subdivided into a hierarchy. Plus, NeST offers access control lists for lot and file access. NeST offers multiple protocol interfaces, including its internal Chirp, HTTP and GSI-FTP.
SAMGrid
SAMGrid is a general data handling system designed to be a key device for experiments with large (petabyte-size) data sets and widely distributed production and analysis facilities. The components now in production provide a versatile set of services for data transfer, data storage, and process bookkeeping on distributed systems.
UberFTP
Building upon the technologies of GridFTP, UberFTP is the first interactive GridFTP-enabled FTP client. The basic GridFTP client is not interactive and allows only one file transfer at a time. UberFTP provides interactive tools that work much like the popular NCFTP tool. It supports GSI authentication, parallel data channels, and third-party transfers.
Back to top
Conclusion
Grid computing is one of the most exciting technologies that are having powerful effects on the way we solve complex problems and share diverse resources. In addition to cancer and physics, it also has great influence on security and authentication, discovery, monitoring, information services, data management, resource management, and scheduling.