Posts Tagged ‘openstack’

Automating Openstack Testing on Ubuntu

// February 8th, 2012 // 3 Comments » // Uncategorized

During the Ubuntu precise development cycle the Canonical Platform Server Team have been working on automating testing of Openstack on Ubuntu.

The scope of this work was:

  1. Per-commit testing of Openstack trunk to evaluate the current state of the upstream codebase in-conjunction with the current packaging in Ubuntu precise and the current Juju charms to deploy Openstack.
  2. SRU testing for Openstack Diablo on Ubuntu 11.10.

Openstack do a lot of pre-commit testing through the use of gerrit with Jenkins; we wanted to supplement this with Ubuntu focused testing to provide another dimension to the testing already completed upstream.

So grab a coffee and make yourself comfortable; this is not a short read….

Lab Setup

The Ubuntu Openstack QA lab consists of 12 servers; the primary server in the solution is an Ubuntu 11.10 install providing the following functions:

  1. Juju – used to deploy Openstack charms in the Lab
  2. Cobbler to support server provisioning (using the Ubuntu Orchestra packages in Oneiric)
  3. Jenkins CI – provides triggering based on upstream commits to github repositories and general job control and reporting.
  4. Schroots for Oneiric and Precise for building packages locally
  5. A reprepro managed local archive for Oneiric and Precise
  6. Squid based archive caching to reduce installation times in the lab

This server also acts at the gateway into and out of the Lab (it’s setup as a NAT router).

The other 11 servers are registered in Cobbler; All servers are connected to a Sentry CDU (Cabinet Distribution Unit) which allows full power control from Cobbler – thanks goes to Andres Rodriguez for developing the required fence component for Cobbler to support this type of CDU.

Preseeded LVM Snapshot Installs

To initiate a new integration test run requires all machines to be powered down and re-provisioned from scratch.  It is essential that our deployment and test runs can cope the frequency of upstream commits, particularly as the frequency increases as Openstack approaches milestones and releases.   After getting the initial lab setup in place, we were able to tear down all machines, re-provision and deploy Openstack in ~30mins.

It was important that we are able to minimize the time taken to complete the testing cycle.   To do so, we’ve employed the use of LVM snapshotting and restoration of the root partition during the the netboot installation.   The process is as follows:

  1. Test run begins
  2. Juju deploys a service (i.e. nova-compute)
  3. A machine is netbooted and a preseeded LVM-based Ubuntu installation takes place onto /dev/qalab/root
  4. At the end of the installation, the root filesystem is moved to /dev/qalab/pristine-[release]-root and a snapshot created at /dev/qalab/root
  5. The machine reboots, runs Juju and deploys nova-compute as pat of the rest of the Openstack deployment. This deployment is smoke tested.
  6. The next test run begins.  All machines are terminated. Juju redeploys nova-compute, a machine is netbooted and Ubuntu installation kicks off.
  7. The installation checks for the existence of a logical volume at /dev/qalab/pristine-[release]-root.  If it exists, it creates a new snapshot at /dev/qalab/root and reboots. If it does not, continues with installation and goto step 4.
  8. System reboots, Juju installs and redeploys nova-compute to a fresh Ubuntu installation.

This process takes place on all nodes in parallel.  With it in place, we were able to cut down the time it took to tear-down and re-provision a node from ~30 minutes to 10 to 15 minutes depending on the service being deployed.

By taking this approach we are also minimize the chance of any nodes hitting an archive inconsistency during installation. This is a known issue when deploying the development release and halts installation on any node that hits it, failing the entire deployment.

All of this is embedded in debian-installer preseeds via Cobbler snippets.  The snippets and kick starts are available at lp:~openstack-ubuntu-testing/+junk/cobbler-lvm-snapshot.

In the future, we’ll be investigating the use of kexec as an alternative to reboot after snapshot restoration to reduce the time spent waiting on servers to boot.  This should minimize the test cycle even more. Credit to James Blair for the idea (see

Management of Jenkins

All of the projects in Jenkins are managed using Jinja2 XML templates in-conjunction with python-jenkins (python-jenkins); this makes it really easy to setup new jobs in the lab and reconfigure existing ones as required (as well as providing great backup!).

Templates and management scripts can be found in lp:~openstack-ubuntu-testing/+junk/jenkins-qa-lab

Testing Openstack Essex on Ubuntu Precise

This testing was the first to be setup in the lab.  Jenkins (using the git plugin) monitors the upstream repositories for commits on the master branch.  When a change is detected the following process is triggered:


Objective: Validate that upstream trunk still builds OK with current packaging for Ubuntu.

  1. A new snapshot upstream tarball is generated based on the latests commit to the upstream component.
  2. The latest archive packaging for the component is pulled in from lp:~ubuntu-server-dev/<COMPONENT>/essex
  3. Any changes in the testing packaging for the component are merged from lp:~openstack-ubuntu-testing/<COMPONENT>/essex
  4. New changelog entries are automatically created for the new upstream commits.
  5. The source package is generated and built in a clean schroot using sbuild locally.

On the assumption that the package built OK locally:

  1. The source package is uploaded to the Testing PPA (ppa:openstack-ubuntu-testing/testing)
  2. The testing packaging branch is push back to lp:~openstack-ubuntu-testing/<COMPONENT>/essex.
  3. The binary packages from the sbuild are installed into the local reprepro managed archive.

This process is managed by a single script (; Credit to Chuck Short for pulling together this part of the process based on work from Openstack upstream.

For changes to the nova project the deploy phase is then executed.


Objective: Validate that packages install, can be configured and reach a know good state prior to execution of testing.

This phase of testing uses Juju with Cobbler to deploy Openstack into the QA lab infrastructure; It utilizes branches of the Openstack charms to support use of a local archive along with a deployer wrapper around Juju written by Adam Gandelman which executes the actual deployment using Juju and monitors for errors.

The deployer is configured to know where to get the right codebase for the Openstack charms, which services to deploy and which relations to setup between services. As you can see from the above diagram this is non-trivial but the charms and Juju do most of the hard work.

Once Openstack is deployed successfully the test phase is then executed.


Objective: Validate that the Openstack deployment in the lab actually works!

At this point, we can run any integration tests we wish against the newly deployed cloud.  This testing is able to help us achieve multiple goals:

  • Early detection of upstream bugs that break Openstack functionality on Ubuntu
  • Verification that packaging branches in the development version of Ubuntu are compatible with upstream trunk.
  • Using these packages, verification that our Juju charms are deploying a functional Openstack cloud and are up-to-date with any deployment-related configuration changes upstream.

At the moment this phase looks like this:

  1. Configure the Openstack deployment (Adams deployer script provides some utility functions for locating specific services in the environment)
    • Creates network configuration in Nova for the private instance network as well as a pool of public floating IPs.
    • Upload an image into the Glance server for use during testing
    • Creates EC2 credentials in the Keystone server for use during testing.
  2. Run the devstack exercise test scripts which ensure basic functionality of the deployment. Currently, this includes:
    • Basic euca-tools EC2 API for starting and stopping instances
    • EC2 AMI bundle uploads
    • Floating IP allocation, association and connectivity to instance
    • Volume creation and attachment to instance

Note: These are the same sets of tests that are currently run against proposed commits to gerrit upstream.

Longer term we aim to use the Openstack Tempest test suite in the lab; Adam is currently working on getting this up and running.


The Jenkins instance in the QA lab is not publicly accessible; however all jobs run in the lab are published out (using the Jenkins build-publisher plugin) to so that people can see the current state of the testing packaging in Ubuntu precise.

We are also working on setting up email notifications.

Success so far

Juju charms deploy Openstack components in a configuration that is compatible with upstream trunk prior to updates to packaging in Ubuntu.  Previously packages were updated in the archive first while Juju charm updates lagged behind as incompatibilities were uncovered after the fact.

We enabled automated testing 2 days prior to the 3rd Essex milestone release.  We were able to uncover and help fix a handful of bugs upstream before the release, including critical bugs like 921784.  In the past, these bugs were typical uncovered after the release (both upstream and in Ubuntu).

Since E3, there have been even more critical bugs uncovered by this testing and fixed upstream, some of which are only applicable to Ubuntu-specific configurations (not tested upstream) and would have been uncovered by users after code hit the Ubuntu archive (See 922232).

Further Plans for the Lab

Pre-commit  testing of changes to stable branches;  The Ubuntu Server team are  working upstream on maintaining the stable branches of released versions  of OpenStack – this work will validate patches proposed to stable  branches in against the current version of the  packaging in released versions of Ubuntu.  Initially this will target  Diablo on Ubuntu 11.10 but will also support Essex on Ubuntu 12.04 once  released.  Ideally the testing process will provide feedback on to help the stable release team review proposed  patches.


Jenkins job configurations: lp:~openstack-ubuntu-testing/+junk/jenkins-qa-lab

Scripts supporting the lab: lp:~openstack-ubuntu-testing/+junk/jenkins-scripts

LVM snapshot preseeds and Cobbler snippets: lp:~openstack-ubuntu-testing/+junk/cobbler-lvm-snapshot

All other relevant scripts, charm branches, etc:


Overall management of delivery and general whip cracking: Dave Walker

Lab installation and base configuration: Pete Graner, Tim Gardner, Brad Figg, James Page

Fence agent for network power control of servers: Andres Rodriguez

Source package creation and build process: Chuck Short and James Page

Deployment testing using Juju: Adam Gandelman

Testing of Openstack: Adam Gandelman

Jenkins packaging, configuration and management: James Page

Gerrit Plugin for pre-commit testing and generally great ideas: Monty Taylor and James Blair

Writing and reviewing this post: Adam Gandelman, Chuck Short and Dave Walker.

The Hot Topic at SCALE: OpenStack

// February 1st, 2012 // Comments Off // Uncategorized

The biggest topic at this year’s Southern California Linux Expo (SCALE) conference was the OpenStackTM project. Everyone came away from the show appreciating that OpenStack is only going to get more popular and bigger. OpenStack is building momentum. Jim Ash and Andrei Matei from the HP Cloud Services team stayed busy – talking with and signing up people for our private beta (HP Cloud Compute and HP Cloud Object Storage). To the SCALE attendees, who gave us their opinions, HP’s involvement with OpenStack means that OpenStack will be a serious, viable option for businesses of all sizes and for developers – who want a real choice in the market that competes with the existing proprietary cloud options. People at the conference wanted to know more about the links between HP, OpenStack technology, Linux, and other open source projects. In a nutshell, OpenStack technology is the open source, open API, open development, and open orchestration layer powering HP Cloud Services. And OpenStack technology is built on Linux and open source technology. The OpenStack project and offerings like HP Cloud Services that integrate OpenStack technology bring open source technology and ideals to businesses of all sizes. We were excited about the warm reception HP Cloud Services got from people with a broad range of backgrounds in Linux and cloud – and from developers from all kinds of companies, from the smallest organizations to the largest enterprises. Read more…

OpenStack developers meeting at FOSDEM

// January 27th, 2012 // Comments Off // Uncategorized

Next week, the European free and open source software developers will converge to Brussels for FOSDEM. We took this opportunity to apply for an OpenStack developers gathering in the Virtualization and Cloud devroom.

At 6pm on Saturday (last session of the day), in the Chavanne room, we will have a one-hour town hall meeting. If you’re an existing OpenStack contributor, a developer considering to join us, an upstream project developer, a downstream distribution packager, or just curious about OpenStack, you’re welcome to join us ! I’ll be there, Stefano Maffulli (our community manager) will be there, and several OpenStack core developers will be there.

We’ll openly discuss issues and solutions about integration with upstream projects, packaging, governance, development processes, community or release cycles. In particular, we’ll have a distribution panel where every OpenStack distribution will be able to explain how they support OpenStack and discuss what we can improve to make things better for them.

And at the end of the session we can informally continue the discussion around fine Belgian beers or their famous Carbonade !

On TOSCA and Cloud Standards. MyPOV

// January 23rd, 2012 // Comments Off // Uncategorized

Puccini's Tosca on stage

Recently OASIS standards body started work on the proposed Topology and Orchestration Specification for Cloud Applications (or TOSCA) for short, standards specification. The standard aims to deliver on the long-heralded, but much disputed concept of cloud bursting – the ability to move workloads between public and private infrastructure in a transparent way.

I posted about the initiative on the CloudU LinkedIn group and had a smattering of comments – many of which identified the lack of what insiders would call genuine Cloud vendors. The other common theme from people was the concern that there was more marketing hype in the announcement than any substantive depth.

According to the TOSCA site it works to;

…enhance the portability of cloud applications and services. TOSCA will enable the interoperable description of application and infrastructure cloud services, the relationships between parts of the service, and the operational behavior of these services (e.g., deploy, patch, shutdown)–independent of the supplier creating the service, and any particular cloud provider or hosting technology. TOSCA will also make it possible for higher-level operational behavior to be associated with cloud infrastructure management.

It’s a heady concept, if it actually works, TOSCA could deliver a number of benefits including;

  • Smoother migration of existing applications to the cloud
  • Flexible bursting (consumer choice)
  • Dynamic, multi-cloud provider applications

Many people have noticed the lack of large IaaS vendors on the TOSCA steering group, in a GigaOm post about the initiative, one commenter sagely points out that;

Until customers demand it, why should they [the big vendors be involved]? And they can’t demand it until we show feasibility … Thus, a standards project. This worked the same way with databases and XML export formats. It took a while for the biggest incumbents to admit that standardized export might be OK.


I like standards, and I like the idea that customers are able to shift workload between different cloud providers and between public and private. That said, it looks unlikely that any initiative will truly be able to deliver upon a hypervisor agnostic portability mechanism for a number of reasons. Firstly it’s not in the large vendor’s interests to allow for this portability and second because the technologies being utilized are sufficiently ingrained as to make the creation and adoption of a standard problematic.

In back channel discussions others mentioned the fact that TOSCA has some incumbents around the table who have a vested interest in increasing rather than reducing the complexity to ensure lock-in is enhanced and monopolies are secured. That doesn’t bode well for TOSCA to really deliver upon its goals.

TOSCA also enters the market at a time that OpenStack (see disclosure) is gaining momentum among vendors as the open source infrastructure model of choice. While many question the motives and chance for success of OpenStack, there are enough deployments from different vendors already in the wild for me to feel comfortable with it as a true initiative beyond simple marketing.

I’m prepared to cut TOSCA some slack, but would need to see some significant progress before I was happy to say that it is actually more than a loose marketing venture. I’d also want to see some reference implementations of TOSCA to assess how real this really is. Watch this space.

(Cross-posted @ The Diversity Blog - SaaS, Cloud & Business Strategy)

ActiveState Stackato Now Supports Private PaaS on HP Cloud Services

// January 16th, 2012 // 1 Comment » // Uncategorized

With the multitude of PaaS vendors that now exist, most providing an all-things-to-all-people polyglot solution that is (in my view at least) largely undifferentiated from their competitors, there is an increasing focus on vendors making partnerships that allows them to build both mindshare and market penetration. The latest is ActiveState who has announced that their PaaS, Stackato, is supported on HP Cloud Services. Stackato is a private PaaS that enables deployment, scaling, and management of Java, Python, Ruby, PHP, Perl, Node.js, Scala, and Clojure applications. With Stackato, customers can deploy an application to either a private internal cloud or a third party cloud. Stackato is built atop Cloud Foundry and is already supported on vSphere and EC2 and is soon to roll out OpenStack support. The HP announcement is interesting as, while the HP Cloud Service is still in private beta, cloud pundits watch and wait with interest to see how these offerings from “traditional” vendors play out. The HP solution is a combination of HP hardware and services built on top of OpenStack. So in a roundabout way, with this announcement Stackato gets both the credibility of the HOP angle, but also arguably the greater benefit of the OpenStack support which enables a wealth of new partnerships. This is a good partnership (with the usual limitation that it’s hard to make totally definitive statements about the support for what is still a beta product. PaaS vendors need to broaden what they do, as Bare Copeland from ActiveState says;
There is no one-size-fits-all solution for enterprises that are demanding different options for their cloud infrastructure, adding support for HP Cloud Services to Stackato reinforces our vision of providing the PaaS layer in the cloud that supports any underlying infrastructure, which will greatly benefit enterprise developers and IT/DevOps by giving them the options and flexibility they want.
Now the challenge that ActiveSteate, and other vendors have is to articulate their offering in a differentiated and consistent way. But that’s a challenge for another day.

(Cross-posted @ The Diversity Blog - SaaS, Cloud & Business Strategy)

Virtualization & Cloud devroom at FOSDEM

// January 13th, 2012 // Comments Off // Uncategorized

The Free and Open source Software Developers’ European Meeting, or FOSDEM, is an institution that happens every year in Brussels. A busy, free and open event that gets a lot of developers together for two days of presentations and cross-pollination. There are typically the FOSDEM main tracks (a set of presentations chosen by the FOSDEM organization) and a set of devrooms, which are topic-oriented or project-oriented and can organize their own schedule freely.

This year, FOSDEM will host an unusual devroom, the Virtualization and Cloud devroom. It will happen in the Chavanne room, a 550-seat auditorium that was traditionally used for main tracks. And it will last for two whole days, while other devrooms typically last for a day or a half-day.

The Virtualization and Cloud devroom is the result of the merging of three separate devroom requests: Virtualization, Xen and OpenStack devrooms. It gives us a larger space and a lot of potential for cross-pollination across projects ! We had a lot of talks proposed, and here is an overview of what you’ll be able to see there.

Saturday, February 4

Saturday will  be the “cloud” day. We will start with a set of talks about OpenStack, past, present and future. I will do an introduction and retrospective of what happened last year in the project, Soren Hansen will guide new developers to Nova, and Debo Dutta will look into future work on application scheduling and Donabe. Next we’ll have a session on various cloud-related technologies: libguestfs, pacemaker-cloud and OpenNebula. The afternoon will start with a nice session on cloud interoperability, including presentations on the Aeolus, CompatibleOne and Deltacloud efforts. We’ll continue with a session on cloud deployment, with a strong OpenStack focus: Ryan Lane will talk about how Wikimedia maintains infrastructure like an open source project, Mike McClurg will look into Ubuntu+XCP+OpenStack deployments, and Dave Walker will introduce the Orchestra project. The day will end with a town hall meeting for all OpenStack developers, including a panel of distribution packagers: I will blog more about that one in the next weeks.

Sunday, February 5

Sunday is more “virtualization” day ! The day will start early with two presentations by Hans de Goede about Spice and USB redirection over the network. Then we’ll have a session on virtualization management, with Guido Trotter giving more Ganeti news and three talks about oVirt. In the afternoon we’ll have a more technical session around virtualization in development: Antti Kantee will introduce ultralightweight kernel service virtualization with rump kernels, Renzo Davoli will lead a workshop on tracing and virtualization, and Dan Berrange will show how to build application sandboxes on top of LXC and KVM with libvirt. The day will end with another developers meeting, this time the Xen developers will meet around Ian Campbell and his Xen deployment troubleshooting workshop.

All in all, that’s two days packed with very interesting presentations, in a devroom large enough to accomodate a good crowd, so we hope to see you there !

Ending the year well: OpenStack Essex-2 milestone

// December 20th, 2011 // Comments Off // Uncategorized

2011 is almost finished, and what a year it has been. We started it with two core projects and one release behind us. During 2011, we got three releases out of the door, grew from 60 code contributors to about 200, added three new core projects, and met for two design summits.

The Essex-2 milestone was released last week. Here is our now-regular overview of the work that made it to OpenStack core projects since the previous milestone.

Nova was the busiest project. Apart from my work on a new secure root wrapper (detailed on previous articles of this blog), we added a pair of OpenStack API extensions to support the creation of snapshots and backups of volumes, the metadata service can now run separately from the API node, network limits can now be set using a per-network base and a per-flavor multiplier, and a small usability feature lets you retrieve the last error that occurred using nova-manage. But Essex is not about new features, it’s more about consistency and stability. On the consistency front, the HA network mode was extended to support XenServer, KVM compute nodes now report capabilities to zones like Xen ones, and the Quantum network manager now supports NAT. Under the hood, VM state transitions have been strengthened, the network data model has been overhauled, internal interfaces now support UUID instance references, and unused callbacks have been removed from the virt driver.

The other projects were all busy starting larger transitions (Keystone’s RBAC, Horizon new user experience, and Glance 2.0 API), leaving less room for essex-2 features. Glance still saw the addition of  a custom directory for data buffering. Keystone introduced global endpoints templates and swauth-like ACL enforcement. Horizon added UI support for downloading RC files, while migrating under the hood from jquery-ui to bootstrap, and adding a versioning scheme for environment/dependencies.

The next milestone is in a bit more than a month: January 26th, 2012. Happy new year and holidays to all !

Weekly OpenStack snapshots in 12.04, starting with Essex-2 milestone

// December 17th, 2011 // 1 Comment » // Uncategorized

Though Ubuntu 12.04 is still months away, we’ll mention things in development that will be of interest to cloud users over the next year, such as the work going into OpenStack. Chuck Short tweets:

OpenStack Essex-2 milestone (nova, glance, keystone, horizon, quantum) is now available on the Ubuntu 12.04.

This includes the first bundling of keystone and the first build in upcoming weekly snapshots of OpenStack. From now on Chuck will be including the latest, fresh OpenStack weekly build in 12.04. If you’re interested in looking at OpenStack on the next LTS, then this is an excellent way to play with the latest code in development.

If you’re looking for more information check out the OpenStack blueprint, you can also find more technical details on Chuck’s blog.

Hadoop World: Ubuntu, Hadoop and Juju

// November 15th, 2011 // 2 Comments » // Uncategorized

I’m always interested in what’s happening at Canonical and with Ubuntu.  Last week at Hadoop World I ran into a couple of folks from the company (coincidentally both named Mark but neither Mr. Shuttleworth).  Mark Mims from the server team was willing to chat so I grabbed some time with him to learn about what he was doing at Hadoop World and what in the heck is this “charming” Juju?

Some of the ground Mark covers

  • Making the next version of Ubuntu server better for Hadoop and big data
  • (0:34) What are “charms” and what do they have to do with service orchestration
  • (2:05) Charm school and learning to write Juju charms
  • (2:54)  Where does “Orchestra” fit in and how can it be used to spin up OpenStack
  • (3:40) What’s next for Juju

But wait, there’s more!

Stay tuned for more interviews from last week’s Hadoop world.  On tap are:

  • Todd Papaioannou from Battery Ventures
  • John Gray of Facebook
  • Erik Swan of Splunk
  • Nosh Petigara of 10gen/MongoDB.

Extra-credit reading

Pau for now..

OpenStack Essex-1 milestone

// November 14th, 2011 // Comments Off // Uncategorized

Last week saw the delivery of the first milestone of the Essex development cycle for Keystone, Glance, Horizon and Nova. This early milestone collected about two months of post-Diablo work… but it’s not as busy in new features as most would think, since a big part of those last two months was spent releasing OpenStack 2011.3 and brainstorming Essex features.

Keystone delivered their first milestone as a core project, with a few new features like support for additional credentials, service registration and using certificate-based SSL client authentication to authenticate services. It should be easier to upgrade from now on, with support for database migrations.

Glance developers were busy preparing significant changes that will land in the next milestone. Several bugfixes and a few features made it to essex-1 though, including the long-awaited SSL client connections. It also moved to UUID image identifiers.

The Nova essex-1 effort was mostly spent on bugfixing, with 129 bugs fixed. New features include a new XenAPI SM volume driver, DHCP support in the Quantum network manager, and optional deferred deletion of instances. Under the hood, the volume code was significantly cleaned up and XML templates were added to simplify serialization in extensions.

Essex-1 was also the first official OpenStack milestone for Horizon, also known as the Dashboard. New features include a instance details page, support for managing Nova volumes and a new extensible modular architecture. The rest of the effort was spent on catching up with the best of core projects in internationalization, developer documentation, and QA (frontend testing and JS unit tests).

Now, keep your seatbelt fastened, as we are one month away from essex-2, where lots of new development work is expected to land !