Posts Tagged ‘openstack’

Automating Openstack Testing on Ubuntu

// February 8th, 2012 // 3 Comments » // Uncategorized

During the Ubuntu precise development cycle the Canonical Platform Server Team have been working on automating testing of Openstack on Ubuntu.

The scope of this work was:

  1. Per-commit testing of Openstack trunk to evaluate the current state of the upstream codebase in-conjunction with the current packaging in Ubuntu precise and the current Juju charms to deploy Openstack.
  2. SRU testing for Openstack Diablo on Ubuntu 11.10.

Openstack do a lot of pre-commit testing through the use of gerrit with Jenkins; we wanted to supplement this with Ubuntu focused testing to provide another dimension to the testing already completed upstream.

So grab a coffee and make yourself comfortable; this is not a short read….

Lab Setup

The Ubuntu Openstack QA lab consists of 12 servers; the primary server in the solution is an Ubuntu 11.10 install providing the following functions:

  1. Juju – used to deploy Openstack charms in the Lab
  2. Cobbler to support server provisioning (using the Ubuntu Orchestra packages in Oneiric)
  3. Jenkins CI – provides triggering based on upstream commits to github repositories and general job control and reporting.
  4. Schroots for Oneiric and Precise for building packages locally
  5. A reprepro managed local archive for Oneiric and Precise
  6. Squid based archive caching to reduce installation times in the lab

This server also acts at the gateway into and out of the Lab (it’s setup as a NAT router).

The other 11 servers are registered in Cobbler; All servers are connected to a Sentry CDU (Cabinet Distribution Unit) which allows full power control from Cobbler – thanks goes to Andres Rodriguez for developing the required fence component for Cobbler to support this type of CDU.

Preseeded LVM Snapshot Installs

To initiate a new integration test run requires all machines to be powered down and re-provisioned from scratch.  It is essential that our deployment and test runs can cope the frequency of upstream commits, particularly as the frequency increases as Openstack approaches milestones and releases.   After getting the initial lab setup in place, we were able to tear down all machines, re-provision and deploy Openstack in ~30mins.

It was important that we are able to minimize the time taken to complete the testing cycle.   To do so, we’ve employed the use of LVM snapshotting and restoration of the root partition during the the netboot installation.   The process is as follows:

  1. Test run begins
  2. Juju deploys a service (i.e. nova-compute)
  3. A machine is netbooted and a preseeded LVM-based Ubuntu installation takes place onto /dev/qalab/root
  4. At the end of the installation, the root filesystem is moved to /dev/qalab/pristine-[release]-root and a snapshot created at /dev/qalab/root
  5. The machine reboots, runs Juju and deploys nova-compute as pat of the rest of the Openstack deployment. This deployment is smoke tested.
  6. The next test run begins.  All machines are terminated. Juju redeploys nova-compute, a machine is netbooted and Ubuntu installation kicks off.
  7. The installation checks for the existence of a logical volume at /dev/qalab/pristine-[release]-root.  If it exists, it creates a new snapshot at /dev/qalab/root and reboots. If it does not, continues with installation and goto step 4.
  8. System reboots, Juju installs and redeploys nova-compute to a fresh Ubuntu installation.

This process takes place on all nodes in parallel.  With it in place, we were able to cut down the time it took to tear-down and re-provision a node from ~30 minutes to 10 to 15 minutes depending on the service being deployed.

By taking this approach we are also minimize the chance of any nodes hitting an archive inconsistency during installation. This is a known issue when deploying the development release and halts installation on any node that hits it, failing the entire deployment.

All of this is embedded in debian-installer preseeds via Cobbler snippets.  The snippets and kick starts are available at lp:~openstack-ubuntu-testing/+junk/cobbler-lvm-snapshot.

In the future, we’ll be investigating the use of kexec as an alternative to reboot after snapshot restoration to reduce the time spent waiting on servers to boot.  This should minimize the test cycle even more. Credit to James Blair for the idea (see

Management of Jenkins

All of the projects in Jenkins are managed using Jinja2 XML templates in-conjunction with python-jenkins (python-jenkins); this makes it really easy to setup new jobs in the lab and reconfigure existing ones as required (as well as providing great backup!).

Templates and management scripts can be found in lp:~openstack-ubuntu-testing/+junk/jenkins-qa-lab

Testing Openstack Essex on Ubuntu Precise

This testing was the first to be setup in the lab.  Jenkins (using the git plugin) monitors the upstream repositories for commits on the master branch.  When a change is detected the following process is triggered:


Objective: Validate that upstream trunk still builds OK with current packaging for Ubuntu.

  1. A new snapshot upstream tarball is generated based on the latests commit to the upstream component.
  2. The latest archive packaging for the component is pulled in from lp:~ubuntu-server-dev/<COMPONENT>/essex
  3. Any changes in the testing packaging for the component are merged from lp:~openstack-ubuntu-testing/<COMPONENT>/essex
  4. New changelog entries are automatically created for the new upstream commits.
  5. The source package is generated and built in a clean schroot using sbuild locally.

On the assumption that the package built OK locally:

  1. The source package is uploaded to the Testing PPA (ppa:openstack-ubuntu-testing/testing)
  2. The testing packaging branch is push back to lp:~openstack-ubuntu-testing/<COMPONENT>/essex.
  3. The binary packages from the sbuild are installed into the local reprepro managed archive.

This process is managed by a single script (; Credit to Chuck Short for pulling together this part of the process based on work from Openstack upstream.

For changes to the nova project the deploy phase is then executed.


Objective: Validate that packages install, can be configured and reach a know good state prior to execution of testing.

This phase of testing uses Juju with Cobbler to deploy Openstack into the QA lab infrastructure; It utilizes branches of the Openstack charms to support use of a local archive along with a deployer wrapper around Juju written by Adam Gandelman which executes the actual deployment using Juju and monitors for errors.

The deployer is configured to know where to get the right codebase for the Openstack charms, which services to deploy and which relations to setup between services. As you can see from the above diagram this is non-trivial but the charms and Juju do most of the hard work.

Once Openstack is deployed successfully the test phase is then executed.


Objective: Validate that the Openstack deployment in the lab actually works!

At this point, we can run any integration tests we wish against the newly deployed cloud.  This testing is able to help us achieve multiple goals:

  • Early detection of upstream bugs that break Openstack functionality on Ubuntu
  • Verification that packaging branches in the development version of Ubuntu are compatible with upstream trunk.
  • Using these packages, verification that our Juju charms are deploying a functional Openstack cloud and are up-to-date with any deployment-related configuration changes upstream.

At the moment this phase looks like this:

  1. Configure the Openstack deployment (Adams deployer script provides some utility functions for locating specific services in the environment)
    • Creates network configuration in Nova for the private instance network as well as a pool of public floating IPs.
    • Upload an image into the Glance server for use during testing
    • Creates EC2 credentials in the Keystone server for use during testing.
  2. Run the devstack exercise test scripts which ensure basic functionality of the deployment. Currently, this includes:
    • Basic euca-tools EC2 API for starting and stopping instances
    • EC2 AMI bundle uploads
    • Floating IP allocation, association and connectivity to instance
    • Volume creation and attachment to instance

Note: These are the same sets of tests that are currently run against proposed commits to gerrit upstream.

Longer term we aim to use the Openstack Tempest test suite in the lab; Adam is currently working on getting this up and running.


The Jenkins instance in the QA lab is not publicly accessible; however all jobs run in the lab are published out (using the Jenkins build-publisher plugin) to so that people can see the current state of the testing packaging in Ubuntu precise.

We are also working on setting up email notifications.

Success so far

Juju charms deploy Openstack components in a configuration that is compatible with upstream trunk prior to updates to packaging in Ubuntu.  Previously packages were updated in the archive first while Juju charm updates lagged behind as incompatibilities were uncovered after the fact.

We enabled automated testing 2 days prior to the 3rd Essex milestone release.  We were able to uncover and help fix a handful of bugs upstream before the release, including critical bugs like 921784.  In the past, these bugs were typical uncovered after the release (both upstream and in Ubuntu).

Since E3, there have been even more critical bugs uncovered by this testing and fixed upstream, some of which are only applicable to Ubuntu-specific configurations (not tested upstream) and would have been uncovered by users after code hit the Ubuntu archive (See 922232).

Further Plans for the Lab

Pre-commit  testing of changes to stable branches;  The Ubuntu Server team are  working upstream on maintaining the stable branches of released versions  of OpenStack – this work will validate patches proposed to stable  branches in against the current version of the  packaging in released versions of Ubuntu.  Initially this will target  Diablo on Ubuntu 11.10 but will also support Essex on Ubuntu 12.04 once  released.  Ideally the testing process will provide feedback on to help the stable release team review proposed  patches.


Jenkins job configurations: lp:~openstack-ubuntu-testing/+junk/jenkins-qa-lab

Scripts supporting the lab: lp:~openstack-ubuntu-testing/+junk/jenkins-scripts

LVM snapshot preseeds and Cobbler snippets: lp:~openstack-ubuntu-testing/+junk/cobbler-lvm-snapshot

All other relevant scripts, charm branches, etc:


Overall management of delivery and general whip cracking: Dave Walker

Lab installation and base configuration: Pete Graner, Tim Gardner, Brad Figg, James Page

Fence agent for network power control of servers: Andres Rodriguez

Source package creation and build process: Chuck Short and James Page

Deployment testing using Juju: Adam Gandelman

Testing of Openstack: Adam Gandelman

Jenkins packaging, configuration and management: James Page

Gerrit Plugin for pre-commit testing and generally great ideas: Monty Taylor and James Blair

Writing and reviewing this post: Adam Gandelman, Chuck Short and Dave Walker.

The Hot Topic at SCALE: OpenStack

// February 1st, 2012 // Comments Off on The Hot Topic at SCALE: OpenStack // Uncategorized

The biggest topic at this year’s Southern California Linux Expo (SCALE) conference was the OpenStackTM project. Everyone came away from the show appreciating that OpenStack is only going to get more popular and bigger. OpenStack is building momentum. Jim Ash and Andrei Matei from the HP Cloud Services team stayed busy – talking with and signing up people for our private beta (HP Cloud Compute and HP Cloud Object Storage). To the SCALE attendees, who gave us their opinions, HP’s involvement with OpenStack means that OpenStack will be a serious, viable option for businesses of all sizes and for developers – who want a real choice in the market that competes with the existing proprietary cloud options. People at the conference wanted to know more about the links between HP, OpenStack technology, Linux, and other open source projects. In a nutshell, OpenStack technology is the open source, open API, open development, and open orchestration layer powering HP Cloud Services. And OpenStack technology is built on Linux and open source technology. The OpenStack project and offerings like HP Cloud Services that integrate OpenStack technology bring open source technology and ideals to businesses of all sizes. We were excited about the warm reception HP Cloud Services got from people with a broad range of backgrounds in Linux and cloud – and from developers from all kinds of companies, from the smallest organizations to the largest enterprises. Read more…

OpenStack developers meeting at FOSDEM

// January 27th, 2012 // Comments Off on OpenStack developers meeting at FOSDEM // Uncategorized

On TOSCA and Cloud Standards. MyPOV

// January 23rd, 2012 // Comments Off on On TOSCA and Cloud Standards. MyPOV // Uncategorized

Puccini's Tosca on stage

Recently OASIS standards body started work on the proposed Topology and Orchestration Specification for Cloud Applications (or TOSCA) for short, standards specification. The standard aims to deliver on the long-heralded, but much disputed concept of cloud bursting – the ability to move workloads between public and private infrastructure in a transparent way.

I posted about the initiative on the CloudU LinkedIn group and had a smattering of comments – many of which identified the lack of what insiders would call genuine Cloud vendors. The other common theme from people was the concern that there was more marketing hype in the announcement than any substantive depth.

According to the TOSCA site it works to;

…enhance the portability of cloud applications and services. TOSCA will enable the interoperable description of application and infrastructure cloud services, the relationships between parts of the service, and the operational behavior of these services (e.g., deploy, patch, shutdown)–independent of the supplier creating the service, and any particular cloud provider or hosting technology. TOSCA will also make it possible for higher-level operational behavior to be associated with cloud infrastructure management.

It’s a heady concept, if it actually works, TOSCA could deliver a number of benefits including;

  • Smoother migration of existing applications to the cloud
  • Flexible bursting (consumer choice)
  • Dynamic, multi-cloud provider applications

Many people have noticed the lack of large IaaS vendors on the TOSCA steering group, in a GigaOm post about the initiative, one commenter sagely points out that;

Until customers demand it, why should they [the big vendors be involved]? And they can’t demand it until we show feasibility … Thus, a standards project. This worked the same way with databases and XML export formats. It took a while for the biggest incumbents to admit that standardized export might be OK.


I like standards, and I like the idea that customers are able to shift workload between different cloud providers and between public and private. That said, it looks unlikely that any initiative will truly be able to deliver upon a hypervisor agnostic portability mechanism for a number of reasons. Firstly it’s not in the large vendor’s interests to allow for this portability and second because the technologies being utilized are sufficiently ingrained as to make the creation and adoption of a standard problematic.

In back channel discussions others mentioned the fact that TOSCA has some incumbents around the table who have a vested interest in increasing rather than reducing the complexity to ensure lock-in is enhanced and monopolies are secured. That doesn’t bode well for TOSCA to really deliver upon its goals.

TOSCA also enters the market at a time that OpenStack (see disclosure) is gaining momentum among vendors as the open source infrastructure model of choice. While many question the motives and chance for success of OpenStack, there are enough deployments from different vendors already in the wild for me to feel comfortable with it as a true initiative beyond simple marketing.

I’m prepared to cut TOSCA some slack, but would need to see some significant progress before I was happy to say that it is actually more than a loose marketing venture. I’d also want to see some reference implementations of TOSCA to assess how real this really is. Watch this space.

(Cross-posted @ The Diversity Blog - SaaS, Cloud & Business Strategy)

ActiveState Stackato Now Supports Private PaaS on HP Cloud Services

// January 16th, 2012 // 1 Comment » // Uncategorized

With the multitude of PaaS vendors that now exist, most providing an all-things-to-all-people polyglot solution that is (in my view at least) largely undifferentiated from their competitors, there is an increasing focus on vendors making partnerships that allows them to build both mindshare and market penetration. The latest is ActiveState who has announced that their PaaS, Stackato, is supported on HP Cloud Services. Stackato is a private PaaS that enables deployment, scaling, and management of Java, Python, Ruby, PHP, Perl, Node.js, Scala, and Clojure applications. With Stackato, customers can deploy an application to either a private internal cloud or a third party cloud. Stackato is built atop Cloud Foundry and is already supported on vSphere and EC2 and is soon to roll out OpenStack support. The HP announcement is interesting as, while the HP Cloud Service is still in private beta, cloud pundits watch and wait with interest to see how these offerings from “traditional” vendors play out. The HP solution is a combination of HP hardware and services built on top of OpenStack. So in a roundabout way, with this announcement Stackato gets both the credibility of the HOP angle, but also arguably the greater benefit of the OpenStack support which enables a wealth of new partnerships. This is a good partnership (with the usual limitation that it’s hard to make totally definitive statements about the support for what is still a beta product. PaaS vendors need to broaden what they do, as Bare Copeland from ActiveState says;
There is no one-size-fits-all solution for enterprises that are demanding different options for their cloud infrastructure, adding support for HP Cloud Services to Stackato reinforces our vision of providing the PaaS layer in the cloud that supports any underlying infrastructure, which will greatly benefit enterprise developers and IT/DevOps by giving them the options and flexibility they want.
Now the challenge that ActiveSteate, and other vendors have is to articulate their offering in a differentiated and consistent way. But that’s a challenge for another day.

(Cross-posted @ The Diversity Blog - SaaS, Cloud & Business Strategy)

Virtualization & Cloud devroom at FOSDEM

// January 13th, 2012 // Comments Off on Virtualization & Cloud devroom at FOSDEM // Uncategorized

Ending the year well: OpenStack Essex-2 milestone

// December 20th, 2011 // Comments Off on Ending the year well: OpenStack Essex-2 milestone // Uncategorized

Weekly OpenStack snapshots in 12.04, starting with Essex-2 milestone

// December 17th, 2011 // 1 Comment » // Uncategorized

Though Ubuntu 12.04 is still months away, we’ll mention things in development that will be of interest to cloud users over the next year, such as the work going into OpenStack. Chuck Short tweets:

OpenStack Essex-2 milestone (nova, glance, keystone, horizon, quantum) is now available on the Ubuntu 12.04.

This includes the first bundling of keystone and the first build in upcoming weekly snapshots of OpenStack. From now on Chuck will be including the latest, fresh OpenStack weekly build in 12.04. If you’re interested in looking at OpenStack on the next LTS, then this is an excellent way to play with the latest code in development.

If you’re looking for more information check out the OpenStack blueprint, you can also find more technical details on Chuck’s blog.

Hadoop World: Ubuntu, Hadoop and Juju

// November 15th, 2011 // 2 Comments » // Uncategorized

I’m always interested in what’s happening at Canonical and with Ubuntu.  Last week at Hadoop World I ran into a couple of folks from the company (coincidentally both named Mark but neither Mr. Shuttleworth).  Mark Mims from the server team was willing to chat so I grabbed some time with him to learn about what he was doing at Hadoop World and what in the heck is this “charming” Juju?

Some of the ground Mark covers

  • Making the next version of Ubuntu server better for Hadoop and big data
  • (0:34) What are “charms” and what do they have to do with service orchestration
  • (2:05) Charm school and learning to write Juju charms
  • (2:54)  Where does “Orchestra” fit in and how can it be used to spin up OpenStack
  • (3:40) What’s next for Juju

But wait, there’s more!

Stay tuned for more interviews from last week’s Hadoop world.  On tap are:

  • Todd Papaioannou from Battery Ventures
  • John Gray of Facebook
  • Erik Swan of Splunk
  • Nosh Petigara of 10gen/MongoDB.

Extra-credit reading

Pau for now..

OpenStack Essex-1 milestone

// November 14th, 2011 // Comments Off on OpenStack Essex-1 milestone // Uncategorized