Category Archives: Grid

Interoperating grid infrastructures with the GridWay metascheduler

This paper describes the GridWay metascheduler and exposes its latest and future developments, mainly related to interoperability and interoperation. GridWay enables large-scale, reliable, and efficient sharing of computing resources over grid middleware. To favor interoperability, it shows a modular architecture based on drivers, which access middleware services for resource discovery and monitoring, job execution and management, and file transfer. This paper presents two new execution drivers for Basic Execution Service (BES) and Computing Resource Execution and Management (CREAM) services and introduces a remote BES interface for GridWay. This interface allows users to access GridWay’s job metascheduling capabilities, using the BES implementation of GridSAM. Thus, GridWay now provides to end users more possibilities of interoperability and interoperation.

More information in the article:

Ismael Marín Carrión, Eduardo Huedo and Ignacio M. Llorente: Interoperating grid infrastructures with the GridWay metascheduler, Concurrency and Computation: Practice and Experience, Volume 27, Issue 9, June 2015, Pages 2278-2290, ISSN 1532-0634, http://dx.doi.org/10.1002/cpe.2971.

GWpilot: Enabling multi-level scheduling in distributed infrastructures with GridWay and pilot jobs

Current systems based on pilot jobs are not exploiting all the scheduling advantages that the technique offers, or they lack compatibility or adaptability. To overcome the limitations or drawbacks in existing approaches, this study presents a different general-purpose pilot system, GWpilot. This system provides individual users or institutions with a more easy-to-use, easy-to-install, scalable, extendable, flexible and adjustable framework to efficiently run legacy applications. The framework is based on the GridWay meta-scheduler and incorporates the powerful features of this system, such as standard interfaces, fair-share policies, ranking, migration, accounting and compatibility with diverse infrastructures. GWpilot goes beyond establishing simple network overlays to overcome the waiting times in remote queues or to improve the reliability in task production. It properly tackles the characterisation problem in current infrastructures, allowing users to arbitrarily incorporate customised monitoring of resources and their running applications into the system. This functionality allows the new framework to implement innovative scheduling algorithms that accomplish the computational needs of a wide range of calculations faster and more efficiently. The system can also be easily stacked under other software layers, such as self-schedulers. The advanced techniques included by default in the framework result in significant performance improvements even when very short tasks are scheduled.

More information in the article:

A.J. Rubio-Montero, E. Huedo, F. Castejón, R. Mayo-García, GWpilot: Enabling multi-level scheduling in distributed infrastructures with GridWay and pilot jobs, Future Generation Computer Systems, Volume 45, April 2015, Pages 25-52, ISSN 0167-739X, http://dx.doi.org/10.1016/j.future.2014.10.003.

Distributed scheduling and data sharing in late-binding overlays

Pull-based late-binding overlays are used in some of today’s largest computational grids. Job agents are submitted to resources with the duty of retrieving real workload from a central queue at runtime. This helps overcome the problems of these complex environments: heterogeneity, imprecise status information and relatively high failure rates. In addition, the late job assignment allows dynamic adaptation to changes in grid conditions or user priorities. However, as the scale grows, the central assignment queue may become a bottleneck for the whole system. This article presents a distributed scheduling architecture for late-binding overlays, which addresses this issue by letting execution nodes build a distributed hash table and delegating job matching and assignment to them. This reduces the load on the central server and makes the system much more scalable and robust. Scalability makes fine-grained scheduling possible and enables new functionalities, like the implementation of a distributed data cache on the execution nodes, which helps alleviate the commonly congested grid storage services.

More information in the article:

Delgado Peris, A.; Hernandez, J.M.; Huedo, E., “Distributed scheduling and data sharing in late-binding overlays,” High Performance Computing & Simulation (HPCS), 2014 International Conference on , vol., no., pp.129,136, 21-25 July 2014.
http://dx.doi.org/10.1109/HPCSim.2014.6903678

GridWay 5.10 is now available!

Dear GridWay users,

A new stable release of the GridWay Metascheduler is available for download. It comes with improved drivers for CREAM and BES,the possibility to perform a FHS-compliant installation, changes in the scheduler to enqueue jobs, and other minor features and bugfixes. You can find more information in the release notes and the documentation.

Again, these developments have been funded by the IGE project. Packages for the GridWay core and the different drivers will be included in IGE 2.1.

Try it and send us your feedback!

The GridWay team.

Request for Requirements from EU Globus Users

Dear GridWay users,

The Initiative for Globus in Europe (IGE) is an EU project aimed at meeting the needs of the European Globus Community. As such, we are very interested in hearing about any requirements that you may have regarding Globus technologies, so we can consider them in our future development cycles. We’re here to help you, so if you need something from Globus, this is your chance to let us know!

If you’re aware of others using Globus (perhaps you know they have requirements?), please feel free to forward this message accordingly.

These requirements may originate within your projects or within your wider research communities, and might include (but are not limited to) requests for new features, enhancements of existing features or requests for bug fixes.

If you have any requirements for Globus feel free to register them through our Community Requirements page at http://www.ige-project.eu/hub/rt, or alternatively, email support@ige-project.eu. In either case we’ll get back to you.

We look forward to hearing from you!

Best Regards,

The Initiative for Globus in Europe

2nd Annual EGCF Meeting

The European Globus Community Forum (EGCF) is the organizational body of the Globus community in Europe. Its goal is to boost an integrated approach to collaboration on Globus development and to provide an organizational platform to foster cooperation within Europe and beyond. Its members are users, administrators, and developers, who are applying the Globus Toolkit as their middleware or are interested in doing so.

EGCF is following on the success of its first-ever annual meeting in 2011 by hosting its second forum meeting at Garching (near Munich), Germany on March 26, 2012. The organising committee is seeking Globus developers or users to submit proposals to present at this event.

See you there!

The GridWay team.

GridWay 5.8 released!

A new stable release of the GridWay Metascheduler is available for download.

This release comes with a new installation procedure, improvements in job polling, logging with syslog, forwarding of resource requirements to LRMSs, automatic disposal of jobs, a new execution driver for CREAM (Computing Resource Execution And Management) as used in gLite, a technology preview of an execution driver for OGSA-BES (Open Grid Services Architecture-Basic Execution Service) and, as always, some bug fixing. You can find more information in the release notes.

This stable release is based on a previous development one, 5.7, which has been thoroughly tested. It has been funded by the IGE project (Initiative for Globus in Europe), which aims to support and enhance the provision of Globus Grid software to European e-Infrastructures, like EGI (European Grid Infrastructure). Packages for the GridWay core and its GT5 drivers are included in the IGE Tech Preview.

IGE logo

Enjoy it! Your feedback is more than welcome.

The GridWay team.

There Will Be a Grid of Clouds

Since 2007, when the term cloud computing came into popularity, we have a vision about the future of Grid and Cloud Computing as fully complementary technologies that will coexist and cooperate at different levels of abstraction in e-infrastructures. Pioneering blog posts like Grid and Clouds are Complementary Technologies, A Virtual Infrastructure Layer for Cluster and Grid Computing or Moving Jobs or Moving the Nodes anticipate the full virtualization of Grid sites, the benefits of hybrid Cloud computing for Grid sites or the use of Grid services to federate Cloud sites. Presentations like From Grids to Cloud Services predict the future of compute Grid infrastructures, from infrastructures for sharing basic Grid services to infrastructures for sharing raw resources. I think that this perspective, controversial at that time, was, in part, result of our participation in pioneering initiatives in Cloud computing, such as RESERVOIR, flagship of the EU-funded projects in Cloud computing, or OpenNebula, widely used open-source technology to build clouds.

Unfortunately, at that time, a large part of the Grid computing community saw Cloud as a threat and not as an opportunity, and tried to undervalue the advantages of bringing Cloud to Grid infrastructures, probably because they were not able to understand the potential of the Cloud, or just “scared of the unknown”. However things are now changing. Last week, EGI (European Grid Infrastructure) organized the User Virtualization Workshop to discuss the future of virtualization and cloud computing in the European production infrastructure. The aim of the workshop was to evaluate the challenges to move towards an Infrastructure as a Service model with federation of resource providers. It is only a very first step, but we are very happy to see that EGI is now moving in the right direction and is planning to incorporate virtualization and Cloud computing to enhance Grid computing and to allow other communities to take advantage of the infrastructure. I think this is a first step to adapt to technology and an opportunity to define a leading position in the new infrastructure arena. After all, Grid will be one of the many services that could be executed on the new European Cloud Infrastructure.

GridWay 5.7 released!

A new development release of the GridWay Metascheduler is available for download.

This release comes with a new installation procedure, improvements in job state polling and some bug fixing. It also includes a technology preview of new execution drivers for CREAM (Computing Resource Execution And Management), used in gLite, and OGSA-BES (Open Grid Services Architecture-Basic Execution Service), the OGF specification for remote job management. Find more information in the release notes.

This development campaign has been partially funded by the IGE project (Initiative for Globus in Europe), started in October 2010, which aims to support and enhance the provision of Globus Grid software to European e-Infrastructures, like EGI (European Grid Infrastructure).

IGE logo

Your feedback on the functionality included in this release is highly appreciated, as well as any idea you could have about missing functionality.

The GridWay team.