OpenNebula in Google Summer of Code 2010!

This year OpenNebula has been selected as a Google Summer of Code (GSoC) mentoring organization. GSoC is a program that offers student developers stipends to write code for various open source projects. During the last six years GSoC has brought together nearly 3,400 students and more than 3,000 mentors  from nearly 100 countries worldwide. For more information about the program take a look to the GSoC FAQ.

GsoC2010

We are very excited about this great opportunity to work with very talented and self-motivated students. During the summer the students will be part of our community, and will have the opportunity to learn the basics of virtualization, cloud computing and OpenNebula.

If you are a student, and would be interested in participating in GSoC with OpenNebula as your mentoring organization, please take a look at our GSoC Ideas page.  This page lists projects that OpenNebula has proposed for GSoC, but it is not a closed list.  If you have an idea for a cool project that uses or extends OpenNebula, please contact one of the OpenNebula  GSoC mentors.  Also, if you are teaching distributed/cloud computing or related courses please share this information with your students.

Once you are ready to submit an application, remember that you must do so before April 9th through the GSoC webapp. So come and join us this summer to improve the OpenNebula Cloud Toolkit!

Ruben S. Montero

New website for OpenNebula.org … and much more!

The OpenNebula team is happy to announce  a new website for OpenNebula. This is not only a new look for opennebula.org but a whole new site for the community, a new place to share resources, contribute developments and discuss about components and solutions around OpenNebula.
OpenNebula Ecosystem
Relevant new places in opennebula.org:

  • Community, a whole new space for the OpenNebula community. There you can find information on how to contribute, the OpenNebula users or the ecosystem… Also we have started the ecosystem catalog to collect all the projects and initiatives of the community.
  • Mailing Lists We have created two new mailing lists: announce, a lowtraffic list for announcements; and ecosystem to discuss of cloud components, projects and cloud solutions…
  • New VCS we have migrated the source repository to git. You can view the new git repo or find instructions to check out the source in dev.opennebula.org.

The OpenNebula Team

Journal of Grid Computing: Special Issue on Clouds and Grids

This special issue of the Journal of Grid Computing is dedicated to recent advances in Cloud computing to simplify and optimize the use and operation of existing distributed computing infrastructures in science and engineering. Authors are invited to submit original, unpublished work describing current research and novel ideas in the area of Cloud Computing and its application to Grid and Cluster computing.
Additional information about this special issue can be found at the journal website

Topics of interest include, but are not limited to:

  • Experiences, performance and reliability running scientific applications in Clouds
  • Grid, Cluster and data-intensive computing in Clouds
  • Limitations of Cloud services and technologies for capability and capacity computing
  • Impact of virtualization on the performance of memory, CPU and I/O intensive, and latency sensitive applications, and virtualization support for specialized communication transports
  • Scientific Clouds offering services for the scientific and technical communities
  • Architectures for integration of Cloud technologies and services with Cluster and Grid infrastructures
  • On-demand and utility resource provision models
  • Service and infrastructure scalability and elasticity management for the efficient execution of virtualized Cluster and Grid platforms
  • New paradigms for computing on Cloud
  • Federation, interoperability and portability between Clouds
  • Cloud interfaces, programming models and tools

Paper Submission & Important Dates
The submitted paper must be formatted according to Journal of Grid Computing rules, check the website for more information

Submission: 28 February 2010
Review: 30 April 2010
Revision: 20 June 2010
Final acceptance decision: 10 July 2010
Publication: Sep 2010

Guest Editors
Ignacio M. Llorente
DSA-Research
Universidad Complutense de Madrid
Madrid, 28040, Spain
llorente@dacya.ucm.es

Ruben S. Montero
DSA-Research
Universidad Complutense de Madrid
Madrid, 28040, Spain
rubensm@dacya.ucm.es

Ruben S. Montero

OpenNebula 1.4 Beta 2, Released!

The OpenNebula team is happy to announce the second beta release of OpenNebula 1.4. This Beta 2 is aimed at testers, community members and cloud enthusiasts in order to identify bugs and regressions, so that 1.4 can fully replace OpenNebula 1.2 deployments.

After nearly three months has passed since the feature freeze for OpenNebula 1.4, the OpenNebula team has been working hard on polishing the new features, and solving bugs. While there could be some issues that need to be fixed before the stable release, OpenNebula 1.4 beta 2 shapes up nicely and brings an important number of improvements and innovations in Cloud computing.

Highlights of OpenNebula 1.4 are…

  • EC2 Query API interface for building OpenNebula-based clouds
  • OCCI interface for building OpenNebula-based clouds
  • Support for the VMware Hypervisor family
  • Multiple user support and access-right control for Virtual Machines and Virtual Networks
  • Advance contextualization support to integrate VM packs and implement multi-component services
  • Easy integration with your data-center services and procedures with a new hook system
  • Support block devices as VM images.
  • Many bug fixes, and scalability and performance improvements in several components of the OpenNebula system
  • A whole new set of documentation pages, guides and examples

Quick Links

The OpenNebula Team!

Grids and Clouds at EGEE09

bandeau-accueil

Yesterday we had a couple of quite interesting sessions about how people is using Cloud computing in Grids at the EGEE09 Conference. The speakers addressed two basic points:

  • The use of virtual machines and Clouds as an effective way to provide Grid users with custom execution environments. So the same Grid site can easily support VOs with different (or conflicting) configurations. Obviously, this will improve the quality-of-life of both the users and site admins.
  • Let the users access the Grid resources in a Cloud-way. After more than ten years of large Grid deployments we find out that some user communities just do not get used to the “official” execution work-flow. This users can greatly benefit from accessing “raw” computing capacity. This led to a very interesting discussion on Cloud federation, and the integration of current Grid services (e.g. authorization systems).

The material is available on-line in case you want to find out more.

The closing part of these sessions was about the integration of the RESERVOIR cloud stack  within the EGEE project. This integration would allow EGEE sites to easily meet the users demands (i.e. worker node configurations) or to scale up services to meet peak loads. In other words, to “shape” a EGEE site to meet the highly dynamic demands of the researchers.

Ruben S. Montero

OpenNebula 1.4 Beta 1 codename Hourglass out for testing!

The OpenNebula team is proud to announce the availability of OpenNebula 1.4 Beta1 Hourglass (1.3.80), this is the first preview of next stable release of the OpenNebula Virtual Infrastructure Manager.

OpenNebula focuses on incorporating bleeding edge technologies and innovations in many areas of virtual infrastructure management and Cloud Computing. OpenNebula 1.4 aims to be the swiss-army knife of Cloud Computing, letting you deploy any type of Cloud. The OpenNebula team has been set now to bug fixing mode to provide a robust and stable OpenNebula 1.4.0 to your data centers.

Please note that OpenNebula 1.4 Beta 1 is not suitable for production settings. Its sole purpose is gathering feedback and testing new exciting features…

Highlights of OpenNebula 1.4 are..

  • EC2 Query API interface on top of OpenNebula, so you can turn your OpenNebula installation in a Public Cloud
  • Support for the VMware Hypervisor family
  • Multiple user support and access-right control for Virtual Machines and Virtual Networks
  • Advance contextualization support to integrate VM packs and implement multi-component services
  • A new OpenNebula Cloud API (OCA) to easily develop any cloud interface or cloud application
  • Easy integration with your data-center services and procedures with a new hook system
  • Many bug fixes, and scalability and performance improvements in several components of the OpenNebula system
  • A whole new set of documentation pages, guides and examples

The OpenNebula Team!

REFERENCES:

Libvirt 0.6.5 released… including a OpenNebula driver

Libvirt version 0.6.5 was released last week with a number of bug fixes and new features. The complete list of changes can be viewed at the libvirt web site. This new release includes an OpenNebula driver that provides a libvirt interface to an OpenNebula cluster.

What is it? OpenNebula is a Virtual Infrastructure Manager that controls Virtual Machines (VM) in a pool of distributed resources by orchestrating network, storage and virtualization technologies. The OpenNebula driver lets you manage your private cloud using a standard libvirt interface, including the API as well as the related tools (e.g. virsh) and VM description files.

Why a libvirt interface for your private cloud? Libvirt is evolving into a very rich and widely used interface to manage the virtualization capabilities of a server, including virtual network, storage and domain management. So, libvirt can be a very effective administration interface for a private cloud exposing a complete set of VM and physical node operations. In this way, libvirt + OpenNebula provides a powerful abstraction for your private cloud. More on interfaces for Private Clouds in this post…

Ruben S. Montero

The Infrastructure Quadrant

The Cloud Computing movement is a melting pot of distributed technologies and paradigms that produce new terms at an incredible fast pace. So it is usually difficult for newcomers to figure out how to take advantage of these new technologies or if they fit in their current IT infrastructure at all.

So, let us try to classify the current infrastructure provisioning trends (the IaaS brand of Cloud Computing) using two simple parameters: where do you obtain the resources for your applications (locally or remotely), and how are those resources obtained (physical or virtualized):

  • Own site (Local – Physical): This is the classical provisioning scheme that we’ve been using for years, one service one machine; not much to say here.
  • Grid (Remote – Physical): the resources are obtained from a remote site for an specific service, eg. batch job processing in scientific Grids, or web applications in a typical hosting scenario. In this case you get fixed configurations with a limited control over the remote resources
  • Private Cloud (Local – Virtual): the resources are obtained from your own infrastructure in the form of Virtual Machines. So you can obtained the classical benefits of virtualization (e.g. consolidation, isolation, easy replication of configurations…) but for your infrastructure as a whole and not just for one server.
  • The Cloud (Remote – Virtual): the resources are obtained from an external (cloud) provider. Unlike the Grid and thanks to VMs, you have total control of the resources you are “buying”, you can install what you need. Usually the provider in this case is another company, but it could be a partner, in that case it is called a Community Cloud.

Let’s briefly review three resource provisioning examples in use:

Classical IT Outsourcing (Own site + Grid). This is the well accepted provisioning scheme adopted by many companies. Some of the core services are hosted in the in-house infrastructure and others are moved to an external hosting. Usually research centers use this model to store and analyze big amounts of data, such as those generated by LHC, or to solve grand challenge applications
Cloud Outsourcing (Own site + Cloud). Similar as above but you get VM’s instead of pre-configured environments to support your service workload. In this case, the VMs can be configured to register to the local services(e.g. a clustered web server), so the capacity assigned to the service can grow with its demands.
The Hybrid Cloud (Private Cloud + Cloud). Nowadays the use of Virtual Machines is a common practice, for example to easily get developing and testing environments. This model can be combined with a Cloud if the some of VMs are obtained from a remote provider, typically to satisfy peak demands.

Probably, using this quadrant you can better plan the resource provision strategy for your site or understand what they are trying to sell to you!

Ruben S. Montero

Set Up a Private Cloud in 5 Steps with Ubuntu and OpenNebula

So, do you want to transform your rigid and compartmented infrastructure, into a flexible and agile platform where you can dynamically deploy new services and adjust their capacity?. If the answer is yes, you want to build what is nowadays called a private cloud.

In this mini-howto, you will learn to setup such private cloud in 5 steps with Ubuntu and OpenNebula. We assume that your infrastructure follows a classical cluster-like architecture, with a front-end (cluster01, in the howto) and a set of worker nodes (cluster02 and cluster03).

First you’ll need to add the following PPA if you’re running Ubuntu 8.04 LTS (Hardy) or Ubuntu 8.10 (Intrepid), you should be fine if you are running Ubuntu 9.04 (Jaunty Jackalope):

deb http://ppa.launchpad.net/opennebula-ubuntu/ppa/ubuntu intrepid main

If everything is set up correctly you could see the OpenNebula packages:

$ apt-cache search opennebula
libopennebula-dev - OpenNebula client library - Development
libopennebula1 - OpenNebula client library - Runtime
opennebula - OpenNebula controller
opennebula-common - OpenNebula common files
opennebula-node - OpenNebula node

OK, then. Here we go:

  1. [Front-end (cluster01)] Install the opennebula package:
    $ sudo apt-get install opennebula
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following extra packages will be installed:
      opennebula-common
    The following NEW packages will be installed:
      opennebula opennebula-common
    0 upgraded, 2 newly installed, 0 to remove and 2 not upgraded.
    Need to get 280kB of archives.
    After this operation, 1352kB of additional disk space will be used.
    Do you want to continue [Y/n]?
    
    ...
    Setting up opennebula-common (1.2-0ubuntu1~intrepid1) ...
    ...
    Adding system user `oneadmin' (UID 107) ...
    Adding new user `oneadmin' (UID 107) with group `nogroup' ...
    Generating public/private rsa key pair.
    ...
    Setting up opennebula (1.2-0ubuntu1~intrepid1) ...
    oned and scheduler started

    As you may see from the output of the apt-get command performs several configuration steps: creates a oneadmin account, generates a rsa key pair, and starts the OpenNebula daemon.

  2. [Front-end (cluster01)] Add the cluster nodes to the system. In this case, we’ll be using KVM and no shared storage. This simple configuration should work out-of-the-box with Ubuntu:
    $ onehost add cluster02 im_kvm vmm_kvm tm_ssh
    Success!
    ...
    $ onehost add cluster03 im_kvm vmm_kvm tm_ssh
    Success!
    ...
    

    Now, we have just to follow the wise output of the previous commands

  3. [Front-end (cluster01)] You need to add the cluster nodes to the known hosts list for the oneadmin user:
    sudo -u oneadmin ssh cluster02
    The authenticity of host 'cluster02 (192.168.16.2)' can't be established.
    RSA key fingerprint is 37:41:a5:0c:e0:64:cb:03:3d:ac:86:b3:44:68:5c:f9.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added 'cluster02,192.168.16.2' (RSA) to the list of known hosts.
    oneadmin@cluster02's password:
    

    You don’t need to actually login.

  4. [Worker Node (cluster02,cluster03)] Install the OpenNebula Node package:
    $ sudo apt-get install opennebula-node
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following extra packages will be installed:
      opennebula-common
    The following NEW packages will be installed:
      opennebula-common opennebula-node
    ...
    Setting up opennebula-node (1.2-0ubuntu1~intrepid1) ...
    Adding user `oneadmin' to group `libvirtd' ...
    Adding user oneadmin to group libvirtd
    Done.

    Note that oneadmin is also created at the nodes (no need for NIS here) and added to the libvirtd, so it can manage the VMs.

  5. [Worker Node (cluster02,cluster03)] Trust the oneadmin user at the ssh scope, just copy the command from the onehost output ;) :
    $ sudo  tee /var/lib/one/.ssh/authorized_keys << EOT
    > ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAm9n0E4bS9K8NUL2bWh4F78LZWDo8uj2VZiAeylJzct
    ...7YPgr+Z4+lhOYPYMnZIzVArvbYzlc7HZxczGLzu+pu012a6Mv4McHtrzMqHw== oneadmin@cluster01
    > EOT
    

    You may want to check that you can login the cluster nodes from the front-end, using the oneadmin account without being asked for a password:

    $ sudo -u oneadmin ssh cluster02

You are done! You have your own cloud up and running! Now, you should be able to see your nodes ready to start VMs:

$ onehost list
 HID NAME                      RVM   TCPU   FCPU   ACPU    TMEM    FMEM STAT
   0 cluster02                   0    100    100    100  963992  902768   on
   1 cluster03                   0    100    100    100  963992  907232   on

You may want to check the OpenNebula documentation for further information on configuring the system or to learn how to specify virtual machines.

Ruben S. Montero

When 8 are 26 (or even more…)

Nowadays it is difficult to find a place where virtualization benefits are not praised. One of my favorite ones is consolidation, that has finish with the “one application one server” paradigm. However, when you start placing your services on different hosts, you will find quickly that the new model does not scale per se. Soon (ten VMs is enough), you’ll be trying to answer questions like: Where did I put our web server? Is MAC address already in use? Is this the updated image for the ubuntu server? and the alike.

That was our situation last year; although we organized ourselves with a wiki and a database to store all the information, it became evident that we need a management layer on top of the hypervisors. We started the development of OpenNebula to implement our ideas about the distributed management of Virtual Machines.

Today OpenNebula is a reality, and has reach a production quality status (at least we are using it in production). I’ll briefly show you how we setup our infrastructure with OpenNebula, KVM and Xen.

The Physical Cluster

Our infrastructure is a 8-blade cluster, each one with 8GB and two quad-core CPUs (8 cores/blade). These blades are connected with a private LAN an to the Internet, so they can host public services. The output of the onehost list command is as follows – it’s been colorized for dramatic purposes ;) :

Originally the blades were configured as a classical computing cluster.

The Virtual Infrastructure

Now, thanks to OpenNebula the cluster can be easily configured to host multiple services. For example we are using the 8 blades for:

  • Two grid clusters (4 nodes each) for our grid projects. Needless to say, that we can add/remove nodes to these clusters as we need them
  • A Grid broker based on the Gridway metacheuler
  • A web server, serving this page actually
  • A coupe of teaching clusters for our master students, so they can break anything
  • A cluster devoted to the RESERVOIR project
  • And my favorite, a cluster were we develop OpenNebula. The nodes of this cluster are Xen capable, and they run as KVM virtual machines. So yes, we are running VMs in Xen nodes which are KVM VMs.

This is the output of the onevm list command:

One of the most useful features of the new version of OpenNebula is the networking support. You can define several networks, and OpenNebula will lease you MAC addresses from the network you want. So you do not have to worry about colliding addresses. The MAC addresses are assigned using a simple pattern so you can derive the IP from the MAC (see the OpenNebula Networking guide for more info).

We are using a virtual network for each virtual cluster, output from onevnet list and onevnet show commands:

Thanks to the hypervisors and OpenNebula, we can shape our 8 blades to be any combination of services, and we can resize these services to suite our needs. As can be seen above, you can easily find where is the webserver running, so you can get a console in case you need one, or migrate the VM to other blade…

Although the management is now so easy with OpenNebula, setting up everything (installing the physical blades, the VMs, configure the network…) is a difficult task. Kudos to Javi and Tino for getting the system running!.

Ruben S. Montero