Tag Archives: GridWay

Interoperating grid infrastructures with the GridWay metascheduler

This paper describes the GridWay metascheduler and exposes its latest and future developments, mainly related to interoperability and interoperation. GridWay enables large-scale, reliable, and efficient sharing of computing resources over grid middleware. To favor interoperability, it shows a modular architecture based on drivers, which access middleware services for resource discovery and monitoring, job execution and management, and file transfer. This paper presents two new execution drivers for Basic Execution Service (BES) and Computing Resource Execution and Management (CREAM) services and introduces a remote BES interface for GridWay. This interface allows users to access GridWay’s job metascheduling capabilities, using the BES implementation of GridSAM. Thus, GridWay now provides to end users more possibilities of interoperability and interoperation.

More information in the article:

Ismael Marín Carrión, Eduardo Huedo and Ignacio M. Llorente: Interoperating grid infrastructures with the GridWay metascheduler, Concurrency and Computation: Practice and Experience, Volume 27, Issue 9, June 2015, Pages 2278-2290, ISSN 1532-0634, http://dx.doi.org/10.1002/cpe.2971.

GWpilot: Enabling multi-level scheduling in distributed infrastructures with GridWay and pilot jobs

Current systems based on pilot jobs are not exploiting all the scheduling advantages that the technique offers, or they lack compatibility or adaptability. To overcome the limitations or drawbacks in existing approaches, this study presents a different general-purpose pilot system, GWpilot. This system provides individual users or institutions with a more easy-to-use, easy-to-install, scalable, extendable, flexible and adjustable framework to efficiently run legacy applications. The framework is based on the GridWay meta-scheduler and incorporates the powerful features of this system, such as standard interfaces, fair-share policies, ranking, migration, accounting and compatibility with diverse infrastructures. GWpilot goes beyond establishing simple network overlays to overcome the waiting times in remote queues or to improve the reliability in task production. It properly tackles the characterisation problem in current infrastructures, allowing users to arbitrarily incorporate customised monitoring of resources and their running applications into the system. This functionality allows the new framework to implement innovative scheduling algorithms that accomplish the computational needs of a wide range of calculations faster and more efficiently. The system can also be easily stacked under other software layers, such as self-schedulers. The advanced techniques included by default in the framework result in significant performance improvements even when very short tasks are scheduled.

More information in the article:

A.J. Rubio-Montero, E. Huedo, F. Castejón, R. Mayo-García, GWpilot: Enabling multi-level scheduling in distributed infrastructures with GridWay and pilot jobs, Future Generation Computer Systems, Volume 45, April 2015, Pages 25-52, ISSN 0167-739X, http://dx.doi.org/10.1016/j.future.2014.10.003.

GridWay 5.10 is now available!

Dear GridWay users,

A new stable release of the GridWay Metascheduler is available for download. It comes with improved drivers for CREAM and BES,the possibility to perform a FHS-compliant installation, changes in the scheduler to enqueue jobs, and other minor features and bugfixes. You can find more information in the release notes and the documentation.

Again, these developments have been funded by the IGE project. Packages for the GridWay core and the different drivers will be included in IGE 2.1.

Try it and send us your feedback!

The GridWay team.

Request for Requirements from EU Globus Users

Dear GridWay users,

The Initiative for Globus in Europe (IGE) is an EU project aimed at meeting the needs of the European Globus Community. As such, we are very interested in hearing about any requirements that you may have regarding Globus technologies, so we can consider them in our future development cycles. We’re here to help you, so if you need something from Globus, this is your chance to let us know!

If you’re aware of others using Globus (perhaps you know they have requirements?), please feel free to forward this message accordingly.

These requirements may originate within your projects or within your wider research communities, and might include (but are not limited to) requests for new features, enhancements of existing features or requests for bug fixes.

If you have any requirements for Globus feel free to register them through our Community Requirements page at http://www.ige-project.eu/hub/rt, or alternatively, email support@ige-project.eu. In either case we’ll get back to you.

We look forward to hearing from you!

Best Regards,

The Initiative for Globus in Europe

GridWay 5.8 released!

A new stable release of the GridWay Metascheduler is available for download.

This release comes with a new installation procedure, improvements in job polling, logging with syslog, forwarding of resource requirements to LRMSs, automatic disposal of jobs, a new execution driver for CREAM (Computing Resource Execution And Management) as used in gLite, a technology preview of an execution driver for OGSA-BES (Open Grid Services Architecture-Basic Execution Service) and, as always, some bug fixing. You can find more information in the release notes.

This stable release is based on a previous development one, 5.7, which has been thoroughly tested. It has been funded by the IGE project (Initiative for Globus in Europe), which aims to support and enhance the provision of Globus Grid software to European e-Infrastructures, like EGI (European Grid Infrastructure). Packages for the GridWay core and its GT5 drivers are included in the IGE Tech Preview.

IGE logo

Enjoy it! Your feedback is more than welcome.

The GridWay team.

GridWay 5.7 released!

A new development release of the GridWay Metascheduler is available for download.

This release comes with a new installation procedure, improvements in job state polling and some bug fixing. It also includes a technology preview of new execution drivers for CREAM (Computing Resource Execution And Management), used in gLite, and OGSA-BES (Open Grid Services Architecture-Basic Execution Service), the OGF specification for remote job management. Find more information in the release notes.

This development campaign has been partially funded by the IGE project (Initiative for Globus in Europe), started in October 2010, which aims to support and enhance the provision of Globus Grid software to European e-Infrastructures, like EGI (European Grid Infrastructure).

IGE logo

Your feedback on the functionality included in this release is highly appreciated, as well as any idea you could have about missing functionality.

The GridWay team.

GridWay, Globus and DRMAA at the European Space Astronomy Center

The fourth edition of the  ESAC GRID Workshop has just finished. Like in previous editions, the agenda included presentations about the state-of-art in the main components to build a Grid infrastructure. Although this time our contributions have been mostly related to the OpenNebula VM Manager for dynamic scaling computing clusters to Amazon EC2, we visited the new SciGrid infrastructure that is planned to be used for data processing in two ESA missions:

Hershel (in development), the largest space telescope of its kind when launched, will collect long-wavelength infrared radiation from some of the coolest and most distant objects in the Universe. Herschel will investigate the history of how stars and galaxies formed and to study how they continue to form in our own and other galaxies.

XMM-Newton‘s, the biggest scientific satellite ever built in Europe, mirrors were amongst the most powerful ever developed in the world, and with its sensitive cameras it will see much more than any previous X-ray satellite. Among its achievements, X-rays from accretion onto black holes, properties of exploding stars, nature of exotic matter, observations of GRB…

SciGrid at ESAC uses GridWay as metascheduler, Globus as grid middleware and SGE as local job manager. The Grid interface is the JAVA DRMAA binding implemented by GridWay. SciGrid will be used in the Herschel Pipeline Data Processing and in the XMM-Newton Science Analysis System.

Ignacio Martín Llorente

Images courtesy ESA

Newly released Globus Toolkit 4.2 includes GridWay 5.4!

Globus Toolkit logo

Recently, Charles Bacon announced, on behalf of the Globus Toolkit development team, the release of Globus Toolkit 4.2, containing an upgrade to the web services specifications used by the toolkit as well as new features in all services.

Starting from Globus Toolkit 4.0.5, GridWay 5.2 was included as a contribution for the GT4.0 final distribution, but contributions are not supported by Globus Toolkit and have very limited documentation. The new Globus Toolkit 4.2 includes a new stable release, GridWay 5.4, as a true Globus component, well documented and fully integrated in Globus installation, building and testing procedures.

This new stable release of GridWay is the result of the previous development release, GridWay 5.3, which was released on December 2007 and has been thoroughly tested and documented since then. In few days, GridWay 5.4 will be also available from GridWay’s web page.

RELEVANT LINKS

Grid Interoperability and Interoperation

The high expectations raised by grid computing have favored the development and deployment of a growing number of grid infrastructures and middlewares. However, the interaction between these grids is still limited, so reducing the potential large-scale application of grid technology, in spite of efforts made by grid community. In this sense, the Open Grid Forum (OGF) is developing open standards for grid software interoperability, while the OGF’s Grid Interoperation Now Community Group (GIN-CG) is coordinating a set of interoperation efforts among production grids. It is therefore clear that, according to OGF (as Laurence Field explains in his article entitled “Getting Grids to work together: interoperation is key to sharing”), there is a big difference between these two terms:

  • Interoperability is the native ability of grids and grid technologies to interact directly via common open standards.
  • Interoperation is a set of techniques to get production grid infrastructures to work together in the short term.

Since most common open standards to provide grid interoperability are still being defined and only a few have been consolidated, grid interoperation techniques, like adapters and gateways, are needed. An adapter is, according to different dictionaries of computer terms, “a device that allows one system to connect to and work with another”. On the other hand, a gateway is conceptually similar to an adapter, but it is implemented as an independent service, acting as a bridge between two systems. The main drawback of adapters is that grid middleware or tools must be modified to insert the adapters. Gateways can be accessed without changes on grid middleware or tools, but they can become a single point of failure or a scalability bottleneck.

GridWay provides support for some of the few established standards like DRMAA, JSDL or WSRF to achieve interoperability but, in the meanwhile, it also provides components to allow interoperation, like Middleware Access Drivers (MADs) acting as adapters for different grid services, and the GridGateWay, which is a WSRF GRAM service encapsulating an instance of GridWay, thus providing a gateway for resource management services.

GridWay 4.0.2, coinciding with the release of Globus Toolkit 4 and its new WS GRAM service, introduced an architecture for the execution manager module based on a MAD (Middleware Access Driver) to interface several grid execution services, like pre-WS GRAM and WS GRAM, even simultaneously. That architecture was presented in the paper entitled “A modular meta-scheduling architecture for interfacing with pre-WS and WS Grid resource management services” (E. Huedo, R. S. Montero and I. M. Llorente). GridWay 5.0 took advantage of this modular architecture to implement an information manager module with a MAD to interface several grid information services, and a transfer manager module with a MAD to interface several grid data services. Moreover, the scheduling process was decoupled from the dispatch manager through the use of an external and selectable scheduler module.

GridWay components

The resulting architecture, which is shown above, provides direct interoperation between different middleware stacks. In fact, we demonstrated at OGF22 the interoperation of three important grid infrastructures, namely EGEE (gLite-based), TeraGrid and OSG (both Globus-based), being coordinately used through a single GridWay instance by means of the appropriate adapters. To set an example, the application was written using the DRMAA OGF standard. GridWay documentation provides a lot of information on how to integrate GridWay in the main middleware stacks, like gLite, pre-WS and WS Globus, or ARC, and provides information on how to develop new drivers for other middlewares.

OGF22 interoperation demo

Regarding the GridGateWay, it is being used for provisioning resources from several infrastructures. For example, the German Astronomy Community Grid (GACG or AstroGrid-D) uses a GridGateWay as a central resource broker, providing metascheduling functionality to Globus-based submission tools (e.g. for workflow execution) without modification. GridAustralia also uses a GridGateWay as a WSRF interface for its central GridWay Metascheduler instance, allowing reliable, remote job submission.

 

 

 

Astrogrid-D metascheduling architecture
Picture by AstroGrid-D

More information about the GridGateWay component is provided in its web page, as well as in this blog entry, which shows how to build Utility Computing infrastructures with this Globus-based gateway technology.

Eduardo Huedo