If you’re using Juju, Nagios is about to get a lot easier to leverage into your environment. Anyone who has ever tried to automate their Nagios configuration, knows that it can be daunting. Nagios is so flexible and has so many options, its hard to get right when doing it by hand. Automating it requires even more thought. Part of this is because monitoring itself is a bit hard to genercise. There are lots of types of monitors. Nagios really focuses on two of these:

• Service monitoring – Make a script that pretends to be a user and see if your synthetic monitor sees what you expect.
• Resource monitoring – Look at the counters and metrics afforded a user of a normal system.

The trick is, the service monitoring wants to interrogate the real services from outside of the machine, while the resource monitoring wants to see things only visible with privileged access. This is why we have NRPE, or “Nagios Remote Plugin Executor” (and NSCA, and munin, but ignore those for now). NRPE is a little daemon that runs on a server and will run a nagios plugin script, returning the result when asked by Nagios. With this you get those privileged things like how much RAM and disk space is used. Normally when you want to use Nagios, you need to sit down and figure out how to tell it to monitor all of your stuff. This involves creating generic objects, figuring out how to get your list of hosts into nagios’s config files, and how to get the classifications for said hosts into nagios. Does anybody trying to make sure their pager goes off when things are broken actually want to learn Nagios? So, here’s how to get Nagios in your Juju environment. First lets assume you have deployed a stack of applications.

juju deploy mysql wikidb                # single MySQL db server
juju deploy haproxy wikibalancer        # and single haproxy load balancer
juju deploy -n 5 mediawiki wiki-app     # 5 app-server nodes to handle mediawiki
juju deploy memcached wiki-cache        # memcached
juju add-relation wikidb:db wiki-app:db # use wikidb service as r/w db for app
juju add-relation wiki-app wikibalancer # load balance wiki-app behind haproxy
juju add-relation wiki-cache wiki-app   # use wiki-cache service for wiki-app

This gives one a nice stack of services that is pretty common in most applications today, with a DB and cache for persistent and ephemeral storage and then many app nodes to scale the heavy lifting.

Now you have your app running, but what about when it breaks? How will you find out? Well this is where Nagios comes in:

juju deploy nagios                          # custom nagios charm
juju add-relation nagios wikidb             # monitor wikidb via nagios
juju add-relation nagios wiki-app           # ""
juju add-relation nagios wikibalancer       # ""

You now should have nagios monitoring things. You can check it out by exposing it and then browsing to the hostname of the nagios instance at ‘http://x.x.x.x/nagios3′. You can find out the password for the ‘nagiosadmin’ user by catting a file that the charm leaves for this purpose:

juju ssh nagios/0 sudo cat /var/lib/juju/nagios.passwd

Now, the checks are very sparse at the moment. This is because we have used the generic monitoring interface which can just monitor the basic things (SSH, ping, etc). We can add some resource monitoring by deploying NRPE:

juju deploy nrpe                          # create a subordinate NRPE service
juju add-relation nrpe wikibalancer       # Put NRPE on wikibalancer
juju add-relation nrpe wiki-app           # Put NRPE on wiki-app
juju add-relation nrpe:monitors nagios:monitors # Tells Nagios to monitor all NRPEs

Now we will get memory stats, root filesystem, etc.

You may have noticed we left off wikidb, that is because it will show you an ambiguous relation warning when you try this:

juju add-relation nrpe wikidb # Put NRPE on wikidb

ERROR Ambiguous relation 'nrpe mysql'; could refer to:
  'nrpe:general-info mysql:juju-info' (juju-info client / juju-info server)
  'nrpe:local-monitors mysql:local-monitors' (local-monitors client / local-monitors server)

This is because mysql has special support to be able to specify its own local monitors in addition to those in the usual basic group (more on this in part 2). To get around this we use:

juju add-relation nrpe:local-monitors wikidb:local-monitors

This is a perfect example of how Juju’s encapsulation around services pays off for re-usability. By wrapping a service like Nagios in a charm, we can start to really develop a set of best practices for using that service and collaborate around making it better for everyone.

Of course, Chef and Puppet users can get this done with existing Nagios modules. Puppet, in particular, has really great Nagios support. However, I want to take a step back and explain why I think Juju has a place along side those methods and will accelerate systems engineering in new directions.

While there is some level of encapsulation in the methods that Chef and Puppet put forth, they’re not fully encapsulated in the way that they interact with other components in a Chef or Puppet system. In most cases, you still have to edit your own service configs to add specific Nagios integration. This works for the custom case, but it does not make it easy for users to collaborate on the way to deploy well known systems. It will also be hard to swap out components for new, better methods as they emerge. Every time you mention Nagios in your code, you are pushing Nagios deeper into your system engineering.

With the method I’ve outlined above, any charmed service can be monitored for basic stats (including the 80 or so that are in the official charm store). You might ask though, what about custom Nagios plugins, or specifying more elaborate but somewhat generic service checks. That is all coming. I will show some examples in my next post about this. I will also go on later to show how Nagios + NRPE can be replaced with collectd, or some other system, without changing the charms that have implemented rich monitoring support.

So, while this at least starts to bring the official Nagios charm up to par with configuration management’s rich Nagios ability, it also sets the stage for replacing Nagios with other things. The key difference here is that as you’ll see in the next few parts, none of the charms will have to mention “Nagios”. They’ll just describe what things to monitor, and Nagios, Collectd, or whatever other system you have in place will find a way to interpret that and monitor it.

Configuration and Monitoring

All charms come with many options available for configuration.  Each is selected to allow the same tuning you would typically perform on a manually deployed machine.  Configuration options are shown per charm when browsing the Charm Store (jujucharms.com/charms/precise).  The Tomcat charm provides numerous options.  For example, to tweak the JVM options of a running service:

juju set tomcat "java_opts=-Xms768M -Xmx1024M"

This sets the Java heap to a miminum and maximum of 768Mb and 1024Mb respectively.  If you are debugging an application, you may also set:

juju set tomcat "java_opts=-Xms768M -Xmx1024M -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=webapps"

To create a ‘.hprof’ Java heap dump you can inspect with VisualVM or jhat each time an OutOfMemoryError occurs.

To open a remote debugger:

juju set tomcat debug_enabled=True

This will open a JDWP debugger on port 8000 you can use to step-through code from Eclipse, Netbeans etc.  (Note: The debugger is never exposed to the Internet, so you need to access it through a ssh tunnel – ‘ssh -L 8000:localhost:8000 ubuntu@xxx.computer.amazonaws.com’, then connect your IDE to localhost port 8000).

A useful part of the JVM is JMX monitoring.  To enable JMX:

juju set tomcat jmx_enabled=True
juju set tomcat "jmx_control_password=<password>"
juju set tomcat "jmx_monitor_password=<password>"

This will start a remote JMX listener on ports 10001, 10002 and set passwords for the ‘monitorRole’ and ‘controlRole’ users (not setting a password disables that account).  You can now open VisualVM or JConsole to connect to the remote JMX instance (screenshot below).  (Note: JMX is never exposed to the Internet, so you need to access it through a ssh tunnel – ‘ssh -L 10001:localhost:10001 -L 10002:localhost:10002 ubuntu@xxx.computer.amazonaws.com’, then connect your JMX client to port 10001).  You can easily expose your own application specific MBeans for monitoring by adding them to the platform MBeanServer.

Options are applied to services and to all units under a service.  It isn’t possible to apply options to a specific unit.  So if you enable debugging, you enable it for all Tomcat units.  Same with Java options.

Options may also be applied at deployment time.  For example, to use Tomcat 6 (rather than the default Tomcat 7), create a ‘config.yaml’ file containing the following:

tomcat:
  tomcat_version: tomcat6

Then deploy:

juju deploy --config config.yaml cs:~robert-ayres/precise/tomcat

All units added via ‘add-unit’ will also be Tomcat 6.

Session Clustering

Previously, we setup a Juju cluster consisting of two Tomcat units behind HAProxy.  In this configuration, HTTP sessions exist only on individual Tomcat units.  For many production setups, the use of load balancer sticky sessions and a non-replicated session is the most performant where HTTP sessions are either not required or expendable in the event of unit failure.  For setups concerned about availability of sessions, you can enable Tomcat session clustering on your Juju service which will replicate session data between all units in the service.  Should a unit fail, any of the remaining units can pickup the subsequent requests with the previous session state.  To enable session clustering:

juju set tomcat cluster_enabled=True

We have two choices of how the cluster manages membership.  The preferred choice is using multicast traffic, but as EC2 doesn’t allow this, we must use static configuration.  This is the default, but you can switch between either method by changing the value of the ‘multicast’ option.  Like everything else Juju deployed, any new units added or removed via ‘add-unit’ or ‘remove-unit’ are automatically included/excluded from the cluster membership.  This easily allows you to toggle clustering so that you can benchmark precisely what latency/throughput cost you have by using replicated sessions.

In summary, I’ve shown how you can tweak Tomcat configuration including enabling JMX monitoring.  We’ve also seen how to enable session clustering.  In my final post of the series, I shall show how you can add Solr indexing to your application.

Deploy a Memcached service:

juju deploy memcached

Configure Tomcat to map Memcached location under a JNDI name:

juju set tomcat "jndi_memcached_config=param/Memcached:memcached"

This will map the ‘memcached’ service under the JNDI name ‘param/Memcached’.  Whilst Memcached is deploying, you can add the relation ahead of time:

juju add-relation tomcat memcached

We will use the excellent Java Memcached library Spy Memcached (code.google.com/p/spymemcached/) in our application.  Download the ‘spymemcached-x.x.x.jar’ and copy it to ‘juju-example/lib’.
Now edit ‘juju-example/grails-app/conf/spring/resources.groovy’ so it contains the following:

import net.spy.memcached.spring.MemcachedClientFactoryBean
import org.springframework.jndi.JndiObjectFactoryBean

beans = {

    memcachedClient(MemcachedClientFactoryBean) {
        servers = { JndiObjectFactoryBean jndi ->
            jndiName = 'param/Memcached'
            resourceRef = true
        }
    }
}

To make use of our Memcached client, let’s create a simple page counter:

(within 'juju-example' directory)
grails create-controller memcached-count

This will create ‘juju-example/grails-app/controllers/juju/example/MemcachedCountController.groovy’.  Edit it so it contains the following:

package juju.example

class MemcachedCountController {

    def memcachedClient

    def index() {
        def count = memcachedClient.incr('juju-example-count', 1, 1)
        render count
    }
}

When Memcached is deployed and associated with Tomcat, redeploy our application:

(within juju-example directory)
grails clean
grails war

(within parent directory)
cp juju-example/target/juju-example-0.1.war precise/j2ee-deployer/deploy
juju upgrade-charm --repository . juju-example

Once redeployed, you should be able to open http://xxx.compute.amazonaws.com/juju/memcachedCount and refresh the page to an incrementing counter, stored in Memcached.

As with our datasource connection, we utilise a JNDI lookup to instantiate our Memcached client using runtime configuration provided by Juju (a space separated list of Memcached units, provided as a JNDI environment parameter).  With this structure, the developer has total control over integrating external services into their application.  If they want to use a different Memcached library, they can use the Juju configuration to instantiate a different class.

If we want to increase our cache capacity, we can add more units:

juju add-unit -n 2 memcached

This will deploy another 2 Memcached units.  Our Tomcats will update to reflect the new units and restart.
(Note: As you add Memcached units, our example counter may appear to reset as its Memcached key is hashed to another server).

We’ve added Memcached to our Juju cluster and seen how you can integrate external services within your application using JNDI values.
In my next post, I’ll write about how we can enable features of our existing cluster like JMX and utilise Tomcat session clustering.

For this demo I’ll build a sample application using the excellent Grails framework (grails.org).  You can of course use traditional tools of Maven, Ant, etc. to produce your final WAR file.  If you want to try the demo yourself, you’ll need to install Grails and Bazaar.

Firstly let’s demonstrate how to deploy Tomcat using Juju (jujucharms.com/~robert-ayres/precise/tomcat).

Open a terminal on any Ubuntu Precise machine and follow the instructions for bootstrapping a Juju cluster – juju.ubuntu.com/docs/getting-started.html.

With a bootstrapped cluster, let’s deploy a Tomcat service:

juju deploy cs:~robert-ayres/precise/tomcat

This will deploy a Tomcat unit under the service name ‘tomcat’.  Like the bootstrap instance, it will take a short time to launch a new instance, install Tomcat, configure defaults and start.  You can check the progress with ‘juju status’.  When deployed you should see the following output (‘machines’ information purposely removed):

services:
  tomcat:
    charm: cs:~robert-ayres/precise/tomcat-1
    relations:
      cluster:
      - tomcat
    units:
      tomcat/0:
        agent-state: started
        machine: 1
        public-address: xxx.compute.amazonaws.com

Should you wish to investigate the details of any unit you can ssh in – ‘ssh ubuntu@xxx.compute.amazonaws.com’ (Juju will have transferred your public key).

The Tomcat manager applications are installed and secured by default, requiring an admin password to be set.  We can apply configuration to Juju services using ‘juju set <service> “<key>=<value>” …’.  To set the ‘admin’ user password on our Tomcat unit:

juju set tomcat "admin_password=<password>"

Our Tomcat unit isn’t initially exposed to the Internet, we can only access it over a ssh tunnel (see ssh ‘-L’ option).  To expose our Tomcat unit to the Internet:

juju expose tomcat

Now you should be able to open your web browser at http://xxx.computer.amazonaws.com:8080/manager and login to Tomcat’s manager using the credentials we just set.
If we prefer our unit to run on a more traditional web port:

juju set tomcat http_port=80

After a small time of configuration you should now be able to access http://xxx.computer.amazonaws.com/manager with the same credentials.
Over HTTP, our credentials aren’t transmitted securely, so let’s enable HTTPS:

juju set tomcat https_enabled=True https_port=443 [1]

Our Tomcat unit will listen for HTTPS connections on the traditional 443 port using a generated self-signed certificate (to use CA signed certificates, see the Tomcat charm README).  Now we can securely access our manager application at https://xxx.computer.amazonaws.com/manager (you need to ignore any browser warning about a self-signed certificate).  We now have a deployed Tomcat optimised and secured for production use!

Now let’s turn our attention to evolving a simple Grails application to demonstrate further Juju abilities.

With a working Grails installation, create ‘juju-example’ application:

grails create-app juju-example

This will create your application in a directory ‘juju-example’.  Inside is a shell of a Grails application, enough for demonstration purposes.

To suit the directory layout of our deployed Tomcat, we should adjust our application to store stacktrace logs in a designated, writable directory.  Edit ‘juju-example/grails-app/conf/Config.groovy’ and inside the ‘log4j’ block add the following ‘appenders’ block:

log4j = {
    ...
    appenders {
        rollingFile name: "stacktrace", maxFileSize: 1024,
                    file: "logs/juju-example-stacktrace.log"
    }
    ...
}

To build a WAR file run:

(within 'juju-example' directory)
grails dev war

This will build a deployable WAR file ‘juju-example/target/juju-example-0.1.war’.

You have secure access to deploy WAR files directly using the Tomcat manager, but there is a better way – using the J2EE Deployer charm.

The J2EE Deployer charm is a subordinate charm that essentially provides a Juju controlled wrapper around deploying your WAR file into a Juju cluster.  This has the distinct advantage of allowing you to upgrade multiple units using a single command as is shown later.  To use the J2EE Deployer, first download a copy of the wrapper for our example application using bzr:

mkdir precise
bzr export precise/j2ee-deployer lp:~robert-ayres/charms/precise/j2ee-deployer/trunk

This will create a local copy of the wrapper under a directory ‘precise/j2ee-deployer’.  The ‘precise’ parent directory is necessary for Juju when using locally deployed charms.
Copy our war file to the ‘deploy’ directory within:

cp juju-example/target/juju-example-0.1.war precise/j2ee-deployer/deploy

Now deploy our application into Juju:

juju deploy --repository . local:j2ee-deployer juju-example

As with other charms, this will securely upload our application into S3 storage for use by any of our Juju services.  Once the deploy command returns, our application should be available within the cluster under the service name ‘juju-example’.  To deploy to Tomcat, we relate the services:

juju add-relation tomcat juju-example

Our Tomcat unit will download our application locally, stop Tomcat, deploy the application and then start Tomcat.
Issue ‘juju status’ commands to check progress.  Once deployment is complete, you can access http://xxx.computer.amazonaws.com/juju-example-0.1/ and see the default Grails welcome page (screenshot below).

We can use ‘juju set’ to change configuration of our application as we did with the Tomcat service.  For example, to change the deployed path to something simpler:

juju set juju-example war-config=juju-example-0.1:/

Our application will now be redeployed and Tomcat restarted so we can access our application at http://xxx.computer.amazonaws.com/.  Now we have a deployed application!

A web application typically requires access to a RDBMS, so let’s demonstrate how we can connect our application to MySQL.
Firstly, deploy a MySQL service:

juju deploy mysql

Whilst this is deploying, we can set the configuration of the imminent relation between Tomcat and MySQL:

juju set tomcat "jndi_db_config=jdbc/JujuDB:mysql:juju:initialSize=20;maxActive=20;maxIdle=20"

This is a colon separated value that maps the requested database ‘juju’ of the ‘mysql’ service under a JNDI name of ‘jdbc/JujuDB’.  The set of values after the final colon set DBCP connection pooling options.  Here we specify a dedicated pool of 20 connections.
Once our MySQL unit is deployed, we relate our Tomcat service:

juju add-relation tomcat mysql

During this process, our Tomcat unit will request the use of database juju.  Our MySQL unit will create the database and return a set of generated credentials for Tomcat to use.  Once complete, our pooled datasource connection is available to our Tomcat application under JNDI – ‘java:comp/env/jdbc/JujuDB’.  To demonstrate its use within our application, firstly configure Grails to use JNDI for its datasource connection.  Within ‘juju-example/grails-app/conf/DataSource.groovy’, inside the ‘production’/'dataSource’ block, add ‘jndiName = “java:comp/env/jdbc/JujuDB”‘ so it reads as follows:

production {
    dataSource {
        dbCreate = "update"
        jndiName = "java:comp/env/jdbc/JujuDB"
    }
}

Next create a domain class which will serve as an example database object:

(within 'juju-example' directory)
grails create-domain-class Book

Edit ‘juju-example/grails-app/domain/juju/example/Book.groovy’ so it contains the following:

package juju.example

class Book {

    static constraints = {
    }

    String author
    String isbn
    Integer pages
    Date published
    String title
}

Now we can use Grails ‘scaffolding’ to generate pages that allow us to insert Books into our database:

grails generate-all Book

Recompile our application to produce a new WAR file:

grails clean
grails war
(Note: 'grails war' now, no 'dev' option)

Now upgrade our application in Juju:

# copy across new war file
cp juju-example/target/juju-example-0.1.war precise/j2ee-deployer/deploy
# upgrade Juju deployment
juju upgrade-charm --repository . juju-example

This will upload our revised application into S3 again and then deploy to all related services, restarting them in the process.
With our newly deployed application utilising its local JNDI datasource, we can now open our web browser at http://xxx.compute.amazonaws.com/juju/book/list and use the generated page to perform CRUD operations on our Book objects, all persisted to our MySQL database.

A key point to be made is how you should develop your application to be cloud deployable.  If the application is developed to utilise external resources via runtime lookups, the application may be deployed to any number of Juju clusters.  You can observe this yourself by adding a relation between your application and any other Tomcat services.

For this post’s finale, let’s show how we can scale Tomcat.
First, deploy the HAProxy load balancer:

juju deploy haproxy

And associate with Tomcat:

juju add-relation haproxy tomcat

Unexpose Tomcat and expose HAProxy:

juju unexpose tomcat
juju expose haproxy

We can now use the public address of HAProxy to access our application.
Now we’re behind a load balancer, its simple to bolster our web traffic capacity by adding a further Tomcat unit:

juju add-unit tomcat

A second Tomcat unit will be deployed and configured as the first.  Same open ports, same MySQL connection, same web application.  Once deployed, HAProxy will serve traffic to both instances in round robin fashion.  Any future application upgrades will occur on both Tomcat units.  If we want to remove a unit:

juju remove-unit tomcat/<n>

where ‘<n>’ is the unit number (shown in status output).

That’s the end of the demo.  Should you wish to destroy your cluster, run:

juju destroy-environment

This will terminate all EC2 instances including the bootstrap instance.

To summarise, I’ve shown how you can create a Juju cluster containing a load balanced Tomcat with MySQL, serving your web application.  We’ve seen how important it is for the application to be cloud deployable allowing it to utilise managed relations.  I’ve also demonstrated how you can upgrade your application once deployed.

In my next post I shall write about adding Memcached to our cluster.

[1] Due to a current Juju bug (https://bugs.launchpad.net/juju/+bug/979859) with command line boolean variables, you may need to create a separate ‘config.yaml’ file containing the contents:

tomcat:
 https_enabled=True
 https_port=443

and then use:

juju set --config config.yaml tomcat

If you were at OSCON you might have seen Mark Shuttleworth demo the newly landed native OpenStack provider for Juju on HP Cloud. This further expands people’s choices on where they want to deploy their environments. We’ll provide the tools and OS, you choose your providers; Amazon, HP Cloud, Rackspace, Microsoft Azure, your own OpenStack, or just your pile of bare metal.

Not happy with your cloud provider? Export your environment and import it into your new cloud provider, all with the same tools. Providers will compete on things like price, performance, reliability, service, and all the things we love about consumer choice.

jitsu export -e aws | jitsu import -e hpcloud -

Your three tier architecture with database replication, caching, load balancers, all the things that makes up your scalabale architecture. Export out of one environment and into another.

Yeah, really.

It doesn’t make your coffee or do data migration (yet), but this is just the beginning. The fact that this is even possible has us really excited about providing people choice on how they want to run their infrastructure. In the past our OpenStack support was through the EC2 compatible API, but now we support OpenStack natively. This opens the door to deploy on OpenStack-based clouds, starting with HP Cloud.

Here’s the docs if you want to start browsing it now. You’ll find the new provider in the Juju PPA if you want to dive in and get started. Thanks to Martin Packman for working on this highly-requested feature. This feature is still under rapid development, so some of the configuration options will change, but you can certainly play with it now.

Looking for some examples? Our mighty community contributors Brandon Holtsclaw and Marco Ceppi have added some example configs for you to get started with right away:

• How can I configure Juju for deployment on HP Cloud?

Looking for help? Join us in #juju on Freenode if you want to get deploying! And don’t worry, we’re working on Rackspace Cloud still, so look forward to that. We’re supporting Essex and HP’s cloud, we’re just waiting for Folsom to solidify a bit more.

And what are you going to deploy on these clouds? Well that’s really up to you, but we have 90 services ready to go for you.

On June 13th we announced that the us-east-1 region has been switched over to S3 [2]. As of July 5th, 2012, all remaining EC2 regions have been migrated to S3. The immediate effect of this change is that AWS VPC instances will no longer experience bug 615545 [3] and will have access to the default selected Ubuntu mirrors.

Due to a bug in S3′s HTTP implementation, users may experience 403, 404 and 416 HTTP errors during update or installation in some EC2 regions. Notably, Ubuntu 11.04 users will see 403 errors while updating the apt meta-data and Ubuntu 10.04, 11.10 and 12.04 users may see hashsum or size mismatches. If you have not applied software updates, then you will need to manually disable http pipelining usage in apt. This can be done simply with:
$ echo ‘Acquire::http::Pipeline-Depth “0″;’ |
sudo tee /etc/apt/apt.conf.d/99-no-pipelining

Amazon has informed us that the S3 http pipelining bug has been fixed in us-east-1 and is being rolled out across the remaining EC2 regions.

We would like to thank all our beta testers that helped to test these new mirrors and provided valuable feedback.

If you experience any issues with the EC2 mirrors please file a bug against Ubuntu at [4].

Thank you,
Ben Howard

[1] https://lists.ubuntu.com/archives/ubuntu-cloud/2012-April/000752.html
[2] https://lists.ubuntu.com/archives/ubuntu-cloud-announce/2012-June/000026.html
[3] https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/615545
[4] https://bugs.launchpad.net/ubuntu/+filebug

Take this example:

ubuntu@ip-10-4-41-125:~$ ab -k -n10000 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/pool/main/c/cup/cup_0.11a+20060608-2.diff.gzThis is ApacheBench, Version 2.3 
    Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
    Licensed to The Apache Software Foundation, http://www.apache.org/

    Benchmarking us-east-1.ec2.archive.ubuntu.com (be patient)
    Completed 1000 requests
    Completed 2000 requests
    Completed 3000 requests
    Completed 4000 requests
    Completed 5000 requests
    Completed 6000 requests
    Completed 7000 requests
    Completed 8000 requests
    Completed 9000 requests
    Completed 10000 requests
    Finished 10000 requests

    Server Software:        AmazonS3
    Server Hostname:        us-east-1.ec2.archive.ubuntu.com
    Server Port:            80

    Document Path:          /ubuntu/pool/main/c/cup/cup_0.11a+20060608-2.diff.gz
    Document Length:        4416 bytes

    Concurrency Level:      1
    Time taken for tests:   201.732 seconds
    Complete requests:      10000
    Failed requests:        0
    Write errors:           0
    Keep-Alive requests:    10000
    Total transferred:      47940000 bytes
    HTML transferred:       44160000 bytes
    Requests per second:    49.57 [#/sec] (mean)
    Time per request:       20.173 [ms] (mean)
    Time per request:       20.173 [ms] (mean, across all concurrent requests)
    Transfer rate:          232.07 [Kbytes/sec] received

    Connection Times (ms)
                  min  mean[+/-sd] median   max
    Connect:        0    0   0.0      0       1
    Processing:     8   20  49.3     14    3013
    Waiting:        8   19  38.8     14    2559
    Total:          8   20  49.3     14    3013

    Percentage of the requests served within a certain time (ms)
      50%     14
      66%     16
      75%     17
      80%     18
      90%     24
      95%     36
      98%     72
      99%    201
     100%   3013 (longest request)

The following is a test suite for a 21MB file requested 100 times sequentially to illustrate the difference between large and small files:

ubuntu@ip-10-4-41-125:~$ ab -k -n100 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/pool/main/libr/libreoffice/libreoffice-common_3.5.2-2ubuntu1_all.deb
    This is ApacheBench, Version 2.3 
    Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
    Licensed to The Apache Software Foundation, http://www.apache.org/

    Benchmarking us-east-1.ec2.archive.ubuntu.com (be patient).....done

    Server Software:        AmazonS3
    Server Hostname:        us-east-1.ec2.archive.ubuntu.com
    Server Port:            80

    Document Path:          /ubuntu/pool/main/libr/libreoffice/libreoffice-common_3.5.2-2ubuntu1_all.deb
    Document Length:        21500700 bytes

    Concurrency Level:      1
    Time taken for tests:   67.119 seconds
    Complete requests:      100
    Failed requests:        0
    Write errors:           0
    Keep-Alive requests:    100
    Total transferred:      2150109200 bytes
    HTML transferred:       2150070000 bytes
    Requests per second:    1.49 [#/sec] (mean)
    Time per request:       671.192 [ms] (mean)
    Time per request:       671.192 [ms] (mean, across all concurrent requests)
    Transfer rate:          31283.38 [Kbytes/sec] received

    Connection Times (ms)
                  min  mean[+/-sd] median   max
    Connect:        0    0   0.1      0       1
    Processing:   407  671 384.9    574    3172
    Waiting:       16   25  21.9     20     220
    Total:        407  671 384.9    574    3172

    Percentage of the requests served within a certain time (ms)
      50%    574
      66%    653
      75%    711
      80%    726
      90%    926
      95%   1089
      98%   2509
      99%   3172
     100%   3172 (longest request)

The amount of uptake seen with Ubuntu Server over the past year has been extremely rewarding and simply amazing.  Infrastructure as a Service (IaaS), a.k.a. Public Cloud, providers are popping up left and right, all wanting to provide Ubuntu Server…all helping to further cement Ubuntu Server’s position as the OS for the cloud.

With that said, I’ve started to become concerned about the way in which some of these IaaS providers distribute Ubuntu.  Ubuntu developers create, publish, and regularly update images on Amazon Web Services and Microsoft Azure.  Canonical hosts and maintains internal archive mirrors in these clouds to provide a low-latency, low-cost update mechanism to users.  Finally, Canonical engineers purposely designed in a pluggable cloud provider API approach to Ubuntu’s service orchestration application, Juju, to lower the operational barriers that often place limitations on cross-cloud workload and service migrations.  We do all this to help ensure cross-platform consistency for Ubuntu Server users, i.e. workloads and applications ran on Ubuntu Server behave in the same manner on bare metal machines and across IaaS providers.

Some IaaS providers and users have decided to produce and host their own Ubuntu Server images without the involvement of the Ubuntu Project or Canonical.  I won’t go into the legal aspects of this, because I’m no lawyer.  However, I believe there is a real risk to users when these images are modified in some way, but still presented as “official” Ubuntu Server images.  Whether the changes are minor, like redirecting fixes and security updates to internal unofficial mirrors, or major, like making changes to OS and/or applications provided in the images themselves, labeling the images as “official”Ubuntu Server is a misrepresentation of the project and the product.  There is a real and legitimate risk of users losing out on the cross-platform assurance that the Ubuntu project and Canonical work so hard to provide due to the images having untested code or simply being out of sync on fixes and updates.  Furthermore, there’s no guarantee that bug fixes made to these modified images will ever make it into the official distro, thus creating a further fork between expected behavior across both bare metal and cloud platforms.  All of this has the potential to lead to poor user experience that’s very damaging to the reputation of Ubuntu the project and product, not to mention Canonical as it’s sponsor.

We ,within the Ubuntu Server team, work extremely hard to ensure our community can depend on having the same user experience and application execution results across all supported platforms, bare metal or cloud.  So…if you are a IaaS provider, and you elect to produce and distribute modified Ubuntu Server images, please…please ensure your users are aware of this by labeling them as customized derivatives.  Let them know that by using these modified images they potentially run the risk of being delayed in getting bug fixes and security updates…and that differences in OS and application behavior from your changes can lead to higher levels of complexity if/when they have a need to move workloads and services to/from other official Ubuntu Server deployments.

Thanks…we now return you to your regularly scheduled program.

   ben@padfoot:~/Desktop$ host us-east-1.ec2.archive.ubuntu.com
   us-east-1.ec2.archive.ubuntu.com is an alias for us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com.
   us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com is an alias for s3-1-w.amazonaws.com.
   s3-1-w.amazonaws.com has address 72.21.214.200

Yes, that’s right. If you are running an instance in Amazon’s us-east-1 region, your EC2 instance is now being served by a higher-speed, hopefully more robust mirror.

One of the changes that comes with this is that VPC in us-east-1 now has access to the Ubuntu Mirrors; that annoying bug is now cleared.

Some of our users are going to hit some problems, although we have made every effort to address this reasonably. If you see 403′s and 404 errors while running “apt-get update” then you need to read the following:

Hello,

As mentioned earlier [1,2], and in an effort to improve on reliability
of the Ubuntu archive mirrors for EC2 instances, Canonical is replacing
the existing EC2 archive mirrors with mirrors backed by Amazon S3. This
change itself will be done via modification of DNS entries and will be
transparent to users.

However, due to a bug in the http pipelining implementation in S3 [3],
a change to apt configuration needs to be made to avoid download errors.
We have chosen to deliver this change via a package upgrade in cloud-init.

The *action required* is one of the following:
a.) upgrade cloud-init:
$ sudo apt-get update ; sudo apt-get install -y cloud-init
b.) launch official AMI's released after 2012-04-01, which will
have the fix included. Please see
http://cloud-images.ubuntu.com/release for the latest AMI's.
Additionally, the Amazon EC2 Quickstart has been updated with
the latest AMI's. At this time we are still in the process of
updating AMIs for Ubuntu 8.04 LTS and Ubuntu 10.10[4].
Announcements will be sent out when they are available.
c.) manually disable http pipeline use in apt
$ echo 'Acquire::http::Pipeline-Depth "0";' |
sudo tee /etc/apt/apt.conf.d/99-no-pipelining

Should you choose not to take appropriate action, you will likely
experience transient apt downloading errors after the change is
implemented. In order to give appropriate time to apply the change, this
transition will not occur before April 18, 2012.

For users of Ubuntu 8.04 LTS AMIs, you will need to take
options 'b' or 'c'. There is not a package update deliverable fix at this
time. For Ubuntu 10.10, the update will be released on its end-of-life,
which is April 10th, 2012.

If you have any questions, please feel free to follow up to this email
thread.

Thanks,
Ben Howard

--
[1] https://lists.ubuntu.com/archives/ubuntu-cloud/2012-January/000715.html
[2] https://lists.ubuntu.com/archives/ubuntu-cloud/2012-March/000744.html
[3] https://bugs.launchpad.net/ubuntu/+source/apt/+bug/948461

–

• http://packages.debian.org/sid/juju

In case you missed it, Brandon Holtsclaw has also announced the first cut of RPM packages. You might remember Brandon from such hits as the OSX Client.

Do note that this is the juju client, lots of things need to happen in order for juju to spawn other distributions, like they need to support cloud-init, generate cloud-init enabled images and publish those as AMIs, etc. We won’t be pursuing that work for other distributions as it requires knowledge of build and image systems and that sort of stuff. However Clint will be publishing a wiki page with the work needed to accomplish this if another distribution wants to take that work and run with it, we think it should be straightforward.

Thanks to Mark Terranova from Fedora for his help, and to the Debian Developers and ftpmasters who processed it in Debian.

You can find us in #juju on Freenode if you have any feedback on the packages.