Category Archives: Grid

GWpilot: Enabling multi-level scheduling in distributed infrastructures with GridWay and pilot jobs

Current systems based on pilot jobs are not exploiting all the scheduling advantages that the technique offers, or they lack compatibility or adaptability. To overcome the limitations or drawbacks in existing approaches, this study presents a different general-purpose pilot system, GWpilot. This system provides individual users or institutions with a more easy-to-use, easy-to-install, scalable, extendable, flexible and adjustable framework to efficiently run legacy applications. The framework is based on the GridWay meta-scheduler and incorporates the powerful features of this system, such as standard interfaces, fair-share policies, ranking, migration, accounting and compatibility with diverse infrastructures. GWpilot goes beyond establishing simple network overlays to overcome the waiting times in remote queues or to improve the reliability in task production. It properly tackles the characterisation problem in current infrastructures, allowing users to arbitrarily incorporate customised monitoring of resources and their running applications into the system. This functionality allows the new framework to implement innovative scheduling algorithms that accomplish the computational needs of a wide range of calculations faster and more efficiently. The system can also be easily stacked under other software layers, such as self-schedulers. The advanced techniques included by default in the framework result in significant performance improvements even when very short tasks are scheduled.

More information in the article:

A.J. Rubio-Montero, E. Huedo, F. Castejón, R. Mayo-García, GWpilot: Enabling multi-level scheduling in distributed infrastructures with GridWay and pilot jobs, Future Generation Computer Systems, Volume 45, April 2015, Pages 25-52, ISSN 0167-739X,

Distributed scheduling and data sharing in late-binding overlays

Pull-based late-binding overlays are used in some of today’s largest computational grids. Job agents are submitted to resources with the duty of retrieving real workload from a central queue at runtime. This helps overcome the problems of these complex environments: heterogeneity, imprecise status information and relatively high failure rates. In addition, the late job assignment allows dynamic adaptation to changes in grid conditions or user priorities. However, as the scale grows, the central assignment queue may become a bottleneck for the whole system. This article presents a distributed scheduling architecture for late-binding overlays, which addresses this issue by letting execution nodes build a distributed hash table and delegating job matching and assignment to them. This reduces the load on the central server and makes the system much more scalable and robust. Scalability makes fine-grained scheduling possible and enables new functionalities, like the implementation of a distributed data cache on the execution nodes, which helps alleviate the commonly congested grid storage services.

More information in the article:

Delgado Peris, A.; Hernandez, J.M.; Huedo, E., “Distributed scheduling and data sharing in late-binding overlays,” High Performance Computing & Simulation (HPCS), 2014 International Conference on , vol., no., pp.129,136, 21-25 July 2014.

GridWay 5.10 is now available!

Dear GridWay users,

A new stable release of the GridWay Metascheduler is available for download. It comes with improved drivers for CREAM and BES,the possibility to perform a FHS-compliant installation, changes in the scheduler to enqueue jobs, and other minor features and bugfixes. You can find more information in the release notes and the documentation.

Again, these developments have been funded by the IGE project. Packages for the GridWay core and the different drivers will be included in IGE 2.1.

Try it and send us your feedback!

The GridWay team.

Request for Requirements from EU Globus Users

Dear GridWay users,

The Initiative for Globus in Europe (IGE) is an EU project aimed at meeting the needs of the European Globus Community. As such, we are very interested in hearing about any requirements that you may have regarding Globus technologies, so we can consider them in our future development cycles. We’re here to help you, so if you need something from Globus, this is your chance to let us know!

If you’re aware of others using Globus (perhaps you know they have requirements?), please feel free to forward this message accordingly.

These requirements may originate within your projects or within your wider research communities, and might include (but are not limited to) requests for new features, enhancements of existing features or requests for bug fixes.

If you have any requirements for Globus feel free to register them through our Community Requirements page at, or alternatively, email In either case we’ll get back to you.

We look forward to hearing from you!

Best Regards,

The Initiative for Globus in Europe

2nd Annual EGCF Meeting

The European Globus Community Forum (EGCF) is the organizational body of the Globus community in Europe. Its goal is to boost an integrated approach to collaboration on Globus development and to provide an organizational platform to foster cooperation within Europe and beyond. Its members are users, administrators, and developers, who are applying the Globus Toolkit as their middleware or are interested in doing so.

EGCF is following on the success of its first-ever annual meeting in 2011 by hosting its second forum meeting at Garching (near Munich), Germany on March 26, 2012. The organising committee is seeking Globus developers or users to submit proposals to present at this event.

See you there!

The GridWay team.

GridWay 5.8 released!

A new stable release of the GridWay Metascheduler is available for download.

This release comes with a new installation procedure, improvements in job polling, logging with syslog, forwarding of resource requirements to LRMSs, automatic disposal of jobs, a new execution driver for CREAM (Computing Resource Execution And Management) as used in gLite, a technology preview of an execution driver for OGSA-BES (Open Grid Services Architecture-Basic Execution Service) and, as always, some bug fixing. You can find more information in the release notes.

This stable release is based on a previous development one, 5.7, which has been thoroughly tested. It has been funded by the IGE project (Initiative for Globus in Europe), which aims to support and enhance the provision of Globus Grid software to European e-Infrastructures, like EGI (European Grid Infrastructure). Packages for the GridWay core and its GT5 drivers are included in the IGE Tech Preview.

IGE logo

Enjoy it! Your feedback is more than welcome.

The GridWay team.

There Will Be a Grid of Clouds

Since 2007, when the term cloud computing came into popularity, we have a vision about the future of Grid and Cloud Computing as fully complementary technologies that will coexist and cooperate at different levels of abstraction in e-infrastructures. Pioneering blog posts like Grid and Clouds are Complementary Technologies, A Virtual Infrastructure Layer for Cluster and Grid Computing or Moving Jobs or Moving the Nodes anticipate the full virtualization of Grid sites, the benefits of hybrid Cloud computing for Grid sites or the use of Grid services to federate Cloud sites. Presentations like From Grids to Cloud Services predict the future of compute Grid infrastructures, from infrastructures for sharing basic Grid services to infrastructures for sharing raw resources. I think that this perspective, controversial at that time, was, in part, result of our participation in pioneering initiatives in Cloud computing, such as RESERVOIR, flagship of the EU-funded projects in Cloud computing, or OpenNebula, widely used open-source technology to build clouds.

Unfortunately, at that time, a large part of the Grid computing community saw Cloud as a threat and not as an opportunity, and tried to undervalue the advantages of bringing Cloud to Grid infrastructures, probably because they were not able to understand the potential of the Cloud, or just “scared of the unknown”. However things are now changing. Last week, EGI (European Grid Infrastructure) organized the User Virtualization Workshop to discuss the future of virtualization and cloud computing in the European production infrastructure. The aim of the workshop was to evaluate the challenges to move towards an Infrastructure as a Service model with federation of resource providers. It is only a very first step, but we are very happy to see that EGI is now moving in the right direction and is planning to incorporate virtualization and Cloud computing to enhance Grid computing and to allow other communities to take advantage of the infrastructure. I think this is a first step to adapt to technology and an opportunity to define a leading position in the new infrastructure arena. After all, Grid will be one of the many services that could be executed on the new European Cloud Infrastructure.

GridWay 5.7 released!

A new development release of the GridWay Metascheduler is available for download.

This release comes with a new installation procedure, improvements in job state polling and some bug fixing. It also includes a technology preview of new execution drivers for CREAM (Computing Resource Execution And Management), used in gLite, and OGSA-BES (Open Grid Services Architecture-Basic Execution Service), the OGF specification for remote job management. Find more information in the release notes.

This development campaign has been partially funded by the IGE project (Initiative for Globus in Europe), started in October 2010, which aims to support and enhance the provision of Globus Grid software to European e-Infrastructures, like EGI (European Grid Infrastructure).

IGE logo

Your feedback on the functionality included in this release is highly appreciated, as well as any idea you could have about missing functionality.

The GridWay team.

StratusLab Releases Open-Source Cloud Solution for Grid Infrastructures

The StratusLab project has just released the first version of its cloud computing distribution, which aims to provide a full cloud solution for grid and cluster computing.

Funded through the European Union Seventh Framework Programme (FP7), the two-year StratusLab project aims to integrate ‘cloud computing’ technologies into ‘grid’ infrastructures. Grids link computers and data that are scattered across the globe to work together for common goals, whilst cloud computing makes software platforms or virtual servers available as a service over the Internet, usually on a commercial basis, and provides a way for organisations to access computing capacity without investing directly in new infrastructure. Linking grid and cloud technologies will result in major benefits for European academic research and is part of the European Commission strategy to develop European computing infrastructures.

The first version of the fully open-source StratusLab distribution was released today (November 9, 2010). The release is a technology preview, and not yet production-ready, but it will give system administrators and users a chance to try out the new features of what will become an integrated solution for cloud management, running grid services within the cloud, and accessing cloud resources and services from the Grid.

While the software, which is based on the OpenNebula open-source toolkit for cloud computing management, can be used as an interface for managing cloud sites, it also provides a range of tools and services specifically designed to facilitate integration of cloud and grid technologies. These include automatic configuration of sites and integration with fabric management tools such as the widely used “Quattor” toolkit, easing deployment of clouds within existing infrastructures. The project also maintains a reference installation where potential users can try out the software without having to install and configure it locally, and an appliance repository containing reference machine virtual images for popular operating systems. The repository will eventually store pre-configured virtual machines for grid services allowing StratusLab users to effortlessly install Grid nodes within their StratusLab clouds.

The StratusLab project brings together six organisations, all key players with recognised leadership, proven expertise, experience and skills in grid and cloud computing. This collaboration presents a balanced combination of academic, research and industrial institutes with complementary capabilities. The participating organisations include the Centre National de la Recherche Scientifique (CNRS), France; DSA-Research at Universidad Complutense de Madrid, Spain; the Greek Research and Technology Network S.A., Greece; SixSq Sárl, Switzerland; Telefonica Investigacion y Desarrollo, Spain, and Trinity College Dublin, Ireland.

Visit for more information or to download the StratusLab distribution.

About the StratusLab Project

The StratusLab project consists of numerous collaborators from six European research institutions. A website can be accessed via the following The project is partially funded by the European Commission through the Grant Agreement RI-261552.

About European Union Framework Programme 7

The Seventh Framework Programme (FP7) bundles all research-related EU initiatives together under a common roof playing a crucial role in reaching the goals of growth, competitiveness and employment. The framework programme runs a number of programmes under the headings Cooperation, Ideas, People and Capacities. All specific programmes work together to promote and encourage the creation of European poles of scientific excellence. More information on FP7 can be obtained from