Gavin Clark at the Register recently reported that only 10% of people upgrading to 9.10 had a satisfactory experience. Serdar Yegalup at Information Week then reported that 40% of people upgrading to Ubuntu 9.10 had issues that they considered unfixable. Both of these conclusion were based on a poll on the Ubuntu support forums.

Serdar had the wisdom to point out that the poll is self-selecting (but still reported it). However, and I write this so you heard it here first, I don’t think we will have to wait long for a Microsoft shill to report his figure as fact and reference Information Week as evidence.

So let’s look at this poll. While we don’t have exact figures it is reasonable to assume that hundreds of thousands of people upgraded to 9.10 in the last few days. The quoted poll has a sample of 2158.

The poll is also on a support forum.

I upgraded to 9.10 a while ago. Flawlessly. So I saw little need to go tell a forum. This is where people go when they have problems. Gavin and Serdar were shocked to find people with support issues on a support forum. I have no doubt the help line at Microsoft has taken a lot of calls recently, but I would not extrapolate from that a large percentage of Windows users are having upgrade problems.

Tellingly and almost the last word on this are the polls from our previous releases, none of which were considered or reported as upgrade disasters:

Jaunty Poll

Intrepid Poll `

Hardy Poll

Gutsy Poll

A very useful summary of these findings by Nicholas Ipsen is here. I am linking to these polls not because I want to provide evidence that the Karmic upgrade experience is or was good or bad, there are other more qualified to comment on that, but that there is nothing new here.

All this of course is of little consolation if you are affected by an issue. Which is why we have the forums and Launchpad so that we can gather data, isolate the issues and fix them if they are an issue with Ubuntu or alert someone who can if they are not. We do this all the time with every release and we are doing it with Ubuntu 9.10.

So what we seem to have here is a poll that has existed for some 5 releases being ‘discovered’ and the data used to support a pre-disposed position. As we operate in the open and publish feedback good and bad, this is the risk we take I guess. Gavin and Serdar had some broader points that we could engage on regarding the readiness or otherwise of Linux for mainstream computing. But to base or support their arguments on this poll does little to illuminate and a lot to obfuscate.

Gerry Carr

Head of platform marketing, Canonical

Thanks a lot to everyone that came to Skills Matter tonight for my presentation. I have really appreciated the quality of your questions and hope that my answers were satisfying.

read more

Ubuntu Enterprise Cloud presentation at Devopsdays 09

For more pictures see …

An interview I did with Jim Ericson of Information Management…

You’ve built up quite a bit of content blogging about the cloud at your site johnmwillis.com in just a short time.

I’ve been doing IT work for 30 years, including a lot with IBM Tivoli, but I’ve been focused on the cloud stuff for two years. I was lucky to meet some of the real pioneers in this space who gave me a kind of baptism, and since I’d been working with large corporations all my life, I completely got the idea. I started interviewing those same people when just a few of us were out there having this conversation.

Read the rest of the interview here….

Some people may say that I am a statistics junky... Well that's certainly true! But what do you want, being a product manager for a product that does not require ANY form of user registration, you have a tendancy to cling to any piece of data you may find that shows that you are not working in vain. Indeed, and to the opposite of most, if not all, of our competitors, we have absolutely no way to determine what is our install base.  We don't control our mirors, we don't have any ping back home mechanisms, and we are not considering adding any.  So, here I am, collecting as much information I can from outside sources...

Well, october was not too bad in that sense:

read more

"With the help of Amazon Web Services, we’re building an infrastructure capable of generating beautiful interactive maps quickly. We’re using four Amazon services in this workflow: SQS (job queuing), EC2 (tile generation), S3 (storage), and CloudFront (distribution). The figure below illustrates the design."

It was already dead. In some senses, anyway.

Google announced a couple of days ago that they’re advancing into the business of GPS guided navigation, rather than staying with their widely popular offering of mapping and positioning only. This announcement affected the rest of the industry immediately, and some of the industry leaders in the area have quickly taken a hit on their share value.

As usual, Slashdot caught up on the news and asked the question: Will Google and Android kill standalone GPS?

Let me point out that the way the facts were covered by Slashdot was quite misguided. Google may be giving a hand to change the industry dynamics a bit faster, but both Garmin and TomTom, the companies which reportedly had an impact in their share value, have phone-based offerings of their own, so it’s not like Google suddenly had an idea for creating a phone-based navigation software which will replace every other offering. The world is slowly converging towards a multi-purpose device for quite a while, and these multi-purpose devices are putting GPSes in the hands of people that in many cases never considered buying a GPS.

The real reason why these companies are taking a hit in their shares now is because Google announced it will offer for free something that these companies charge quality money for at the moment, being it in a standalone GPS or not.

"Overall we’ve been very pleased with the new queue-based system – we’ve successfully managed to decouple the user interface from a database that’s starting to get a little sluggish."

WARNING!

Though most Ubuntu 9.04 Jaunty systems can upgrade to 9.10 Karmic in place, this is not possible on EC2 and should not be attempted. If you do try this, your system will become unusable on reboot and there will be no recovery and no access to any of the data on the boot disk or ephemeral storage.

Here’s why:

• Ubuntu 9.10 Karmic has a version of udev which requires a newer kernel than you would be running for Ubuntu 9.04 Jaunty (especially on EC2).

• You cannot upgrade the kernel used by a running instance on Amazon EC2 (not even rebooting).

• When an EC2 instance cannot boot (as in the case of the udev/kernel mismatch) your only option is to terminate it, losing the local storage.

How To Upgrade

In order to upgrade to Karmic you will need to start a new EC2 instance running a fresh copy of the appropriate Karmic AMI. I post the latest AMI ids for Karmic in the second table on http://alestic.com/.

Keep your old instance(s) running while you configure and test the new Karmic instances. EC2 makes it easy to have multiple sets of servers running in parallel instead of upgrading in place. When you are confident your new servers are functioning properly, you can discard the old ones.

The Ubuntu 9.10 Karmic AMIs released by Canonical have a number of differences from the community Ubuntu AMIs which have been published on http://alestic.com.

One of the biggest differences is that you will ssh to ubuntu@ instead of to root@ on your instance. You can then sudo to perform commands as the root user. Back in April I wrote a guide about Using sudo, ssh, rsync on the Official Ubuntu Images for EC2.

The Ubuntu server team has put a lot of work into making Ubuntu 9.10 Karmic function beautifully on Amazon EC2 and it’s been a pleasure to have a small part in the process. I’m already using the Karmic AMIs on EC2 for one of my production processes. Please give these AMIs a spin and give feedback.

Amazon Web Services just announced the release of two new instance types for EC2. These new types have 34.2 GB and 68.4 GB of RAM with a decent amount of CPU capacity on modern CPUs to go along with it.

Others have already done a great job of describing the instance types:

Jeff Barr’s AWS blog

RightScale’s blog

but when it comes to flexing the raw power at my fingertips with AWS, sometimes I can’t help myself. So…

sitting on my couch with my laptop watching an episode of “Lie to me” on TiVo I just typed:

ec2-run-instances            \
  --instance-type m2.4xlarge \
  --key KEYPAIR              \
  --instance-count 19        \
  ami-e6f6158f

and in under a minute and about $45 later, I had ssh access to well over 1 TB (1,000 GB) of free memory. To be sure, it was spread over 19 Ubuntu servers, but still, there’s gotta be something I can do with that, no?

Here are the results on a single one of these servers running Ubuntu 8.04 Hardy:

root@domU-12-31-39-08-7F-51:~# free
             total       used       free     shared    buffers     cached
Mem:      71687580    1521464   70166116          0       2632      17704
-/+ buffers/cache:    1501128   70186452
Swap:            0          0          0

root@domU-12-31-39-08-7F-51:~# free -g
             total       used       free     shared    buffers     cached
Mem:            68          1         66          0          0          0
-/+ buffers/cache:          1         66
Swap:            0          0          0

Wait, I’d better do whatever I’m gonna do quick or I’m going to be charged another $45.60 for the next hour’s worth of fun!

Ok, time to cut my losses:

ec2-describe-instances | 
  egrep m2.4xlarge | 
  cut -f2 | 
  xargs ec2-terminate-instances

In case you didn’t feel like spending $2.40 to find out the CPUs on one of these beasts, here’s one of the ones I ran:

vendor_id   : GenuineIntel
cpu family  : 6
model       : 26
model name  : Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping    : 5
cpu MHz     : 2666.760
cache size  : 8192 KB
bogomips    : 5203.00

And remember that there’s 8 of these on the m2.4xlarge instance size. (Exact CPUs not guaranteed, your results my vary, etc.)

Amazon Web Services seems to keep releasing new features in advance of when our growing startup needs them. As we start to think about whether we are going to need to trim some tables or split up the database, here comes an instance type that will let us grow a lot longer just focusing on our core business challenges instead of on the infrastructure.

We are pleased to announce the availability of Landscape 1.4.0!

This version includes new features such as Eucalyptus cloud management and package activities sceduling as well as some polishing. Read on for details.

New features

In Landscape we always work at the same time in new features as well as in improving existing ones. Here are the new features for this release:

Ubuntu Enterprise Cloud (UEC)

Not only can Landscape interact with Amazon’s EC2 service, you can now use Landscape to handle instances in your own Ubuntu Enterprise Cloud, based on Eucalyptus. All the features that work with EC2 also work with UEC, such as instance start/stop, Elastic Block Storage and others. All you need to do is provide the URL to the UEC endpoint and the cloud credentials:

Scheduling

In this development cycle we started implementing scheduling in Landscape. For now, it’s only available for package activities and reboot and shutdown of computers, but expect this to be improved in the next few minor releases:

Access Groups

Access Groups allow us to restrict the computers to which selected administrators will have access to. Administrators that belong to an access group can only manage the machines in the same access group.

For example, if we had an access group called “devel”, administrators belonging to this group would only be able to manage the machines that also belonged to the “devel” group.

We want to make some last minute improvements to this feature, so it will only be available next week on November 2nd, 2009, but here is a sneak preview:

Ajax

It started small, with just a few icons showing the status of the EC2 instances in the computer page, but now we have Ajax all over the place in the user interface. This improves the user experience and makes Landscape faster and more robust.

Improvements

Landscape is always evolving, and we like to take care of existing features as well as introduce new ones. This time we paid considerable attention to packaging.

Package search

The package search page, which is central to all package activities, now uses a better search algorithm. Exact and more prominent matches are displayed first, and we got rid of the limitation of three characters at a minimum for a filter:

Packaging User Interface

This has been in place already, but we would like to highlight it again. The new package interface can now handle upgrades, downgrades, new installations and removals all at the same time, in one place. Not only that, it also handles different versions of each package per computer and groups them all under the package name.

For example, before, when searching for a package called “postfix” among five different Ubuntu distributions, we would get one result for each version of “postfix”. Now it’s all grouped together under the “postfix” name, making the page much simpler to work with:

Now a task like “install postfix on these four different Ubuntu machines” suddenly became much simpler: instead of 4 clicks (one for each version), it’s just one click away.

Cloud management dominates the theme for Landscape 1.4 with new features that allow you to manage your private Ubuntu Enterprise Cloud (UEC) as easily as you manage your instances on Amazon EC2. Also released are new features that help system administrators be more efficient including time based package updates and role based access. These new features will be available this week on both the Hosted and Dedicated Server Editions of Landscape.

Cloud Computing: UEC Support

Building upon our support for Amazon’s EC2 Cloud, users can now start, stop and manage their private Ubuntu instances on their UEC from within Landscape. Users simply enter their credentials directly through Landscape to start, stop and manage an instance. We’ve designed Landscape to work directly with both Amazon and Eucalyptus console pages so you can manage your instances without having to use any other tools. Landscape securely stores your security credentials making is easy to spin up new instances on UEC or EC2 at anytime. Once the instances are initiated, they all display on the same page allowing you manage all your physical, virtuali and Cloud instances in one place.

New Timed Package Updates

Users can now schedule package updates, system reboots and shutdowns in the future with minute increments. This gives System Administrators the flexibility to schedule activities for systems to occur during planned maintenance windows or low use periods to minimise the impact on the network. Now that we have developed the time / scheduling function, look for us to add this to other areas of Landscape where it it makes sense, like scheduling scripts in the future.

Administrator Roles

As Landscape is deployed in larger enterprises the need to delegate and limit access to systems has become a key need. Now, in addition to having a master admin who can manage all systems within a Landscape account, you can delegate access to systems to other administrators. This builds on our tags function (which allows you to tag computers performing the same function and update them with a single command) to allocate full access to systems by admin. Typically this means sys admins in different areas can be given full access to their local systems, but not to those in another area or country. Like tags, assigning systems to other administrators is completely flexible so you can use what ever criteria you need.

The Landscape 1.4 client is available today and is included with Ubuntu 9.10 server edition. Details are at www.canonical.com/landscape

Ken Drachnik, Landscape Manager

In the article Running MySQL on Amazon EC2 with Elastic Block Store I describe the principles involved in using EBS on EC2. Though originally published in 2008, it is still relevant today and is worth reviewing to get context for this article.

In the above tutorial, I included a sample script which followed the basic instructions in the article to initiate EBS snapshots of an XFS file system containing a MySQL database. For the most part this script worked for basic installations with low volume.

Over the last year as I and my co-workers have been using this code in production systems, we identified a number of ways it could be improved. Or, put another way, some serious issues came up when the idealistic world view of the original simplistic script met the complexities which can and do arise in the brutal real world.

We gradually improved the code over the course of the year, until the point where it has been running smoothly on production systems with no serious issues. This doesn’t mean that there aren’t any areas left for improvement, but does seem like it’s ready for the general public to give it a try.

The name of the new program is ec2-consistent-snapshot.

Features

Here are some of the ways in which the ec2-consistent-snapshot program has improved over the original:

• Command line options for passing in AWS keys, MySQL access information, and more.

• Can be run with or without a MySQL database on the file system. This lets you use the command to initiate snapshots for any EBS volume.

• Can be used with or without XFS file systems, though if you don’t use XFS, you run the risk of not having a consistent file system on EBS volume restore.

• Instead of using the painfully slow ec2-create-snapshot command written in Java, this Perl program accesses the EC2 API directly with orders of magnitude speed improvement.

• A preliminary FLUSH is performed on the MySQL database before the FLUSH WITH READ LOCK. This preparation reduces the total time the tables are locked.

• A preliminary sync is performed on the XFS file system before the xfs_freeze. This preparation reduces the total time the file system is locked.

• The MySQL LOCK now has timeouts and retries around it. This prevents horrible blocking interactions between the database lock, long running queries, and normal transactions. The length of the timeout and the number of retries are configurable with command line options.

• The MySQL FLUSH is done in such a way that the statement does not propagate through to slave databases, negatively impacting their performance and/or causing negative blocking interactions with long running queries.

• Cleans up MySQL and XFS locks if it is interrupted, if a timeout happens, or if other errors occur. This prevents a number of serious locking issues when things go wrong with the environment or EC2 API.

• Can snapshot EBS volumes in a region other than the default (e.g., eu-west-1).

• Can initiate snapshots of multiple EBS volumes at the same time while everything is consistently locked. This has been used to create consistent snapshots of RAIDed EBS volumes.

Installation

On Ubuntu, you can install the ec2-consistent-snapshot package using the new Alestic PPA (personal package archive) hosted on Launchpad.net. Here are the steps to set up access to packages in the Alestic PPA:

codename=$(lsb_release -cs)
echo "deb http://ppa.launchpad.net/alestic/ppa/ubuntu $codename main"|
  sudo tee /etc/apt/sources.list.d/alestic-ppa.list    
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BE09C571
sudo apt-get update

Once this is set up, you can install and upgrade packages in the Alestic PPA just as you would with any other Ubuntu package. Here’s the command to install the ec2-consistent-snapshot package:

sudo apt-get install -y ec2-consistent-snapshot

This will also install all of the dependency packages except for the Net::Amazon::EC2 Perl module which does not yet have an Ubuntu package. You can install this with the command:

sudo PERL_MM_USE_DEFAULT=1 cpan Net::Amazon::EC2

Now you can read the documentation using:

man ec2-consistent-snapshot

and run the ec2-consistent-snapshot command itself.

Feedback

If you find any problems with ec2-consistent-snapshot, please create bug reports in launchpad. The same mechanism can be used to submit ideas for improvement, which are especially welcomed if you include a patch.

Other questions and feedback are accepted in the comments section for this article. If you’re reading this on a planet, please click through on the title to read the comments.

We are hosting a couple of events aimed at large businesses in New York City and London in over the coming weeks. NYC is next Monday and there are still (a few) slots left. Registration is free and it should be a great evening.

The aim is to give an overview of our cloud strategy and introduce businesses to the private cloud and to Ubuntu Enterprise Cloud in particular. Al Gillen of IDC has been kind enough to do the introductory overview. Simon Wardley and John Willis, Canonical’s cloud gurus will give their opinions and, for me, the highlight of the night will be the live demo conducted by Nick Barcet the Ubuntu Server product manager

We have a fantastic audience registered already but if you are free next Monday 26th and working at a business interested in exploring cloud please do join us.

The London event takes place on November 10th in 195 Picadilly. Similar agenda but we are delighted to have James Governor of Redmonk do the introduction. We hope we can accommodate you at that event if you are in the London area.

Gerry Carr, Canonical

The ShipIt program has been at the core of the Ubuntu project since its inception. The goal was to make sure that there are no restrictions, as far as was possible, to people having access to Ubuntu. In the last five years we have shipped millions of CDs and seen Ubuntu’s popularity and reach grow in ways that would be impossible without ShipIt.

And that aim continues. We need to make Ubuntu available to as many people who need it, particularly those for whom the download options are limited. The goal has not been to supply a CD to every Ubuntu user of every version of Ubuntu.  Remember, one of the coolest things about Ubuntu is the way you can upgrade from one version of Ubuntu to another – without the need for a CD!

While these CDs are often referred to as “free CDs”, they are of course not free of cost to Canonical.  We want to continue this programme, but Ubuntu’s growth means that some changes are necessary. Therefore we are adjusting how we handle CD requests to try to find the right balance between availability of CDs and the continued viability of the ShipIt programme.

We will continue to supply CDs to LoCo teams and Ubuntu members.  And we hope to make CDs available to everyone who is just discovering Ubuntu.   And we continue to search for additional ways to make Ubuntu and Ubuntu materials available to everyone. But we are limiting shipments to people that we think have alternative paths of getting Ubuntu.  For instance,

* you can upgrade to the new release without a CD
* you can download your own CD for free
* you will be able to download the CD wallet artwork
* becoming an Ubuntu member by contributing to Ubuntu, and thereby becoming eligible for more CDs
* And finally, you can purchase CDs.

We will change the language on the ShipIt site to make it clearer what we are doing. We hope that you support this effort and realise that the intent is to continue to make Ubuntu available on CD to everyone who needs it.

Jane Silber, Canonical

We're still a week away from releasing Ubuntu 9.10, which I'm sure will be a phenomenal server release, with huge strides in virtualization and cloud hosting. The Ubuntu Enterprise Cloud should be the most complete open source cloud hosting solution in the industry.

But we're also beginning to prepare for the Ubuntu Developer Summit in Dallas, Texas next month. At this summit, we will discuss our plans for Ubuntu Lucid Lynx, which will release in April 2010 as Ubuntu 10.04 LTS. This being an LTS release, UDS is incredibly important, as these decisions will affect the Ubuntu landscape for at least 5 years.

As your maintainer of Ubuntu's virtualization stack supported by Canonical, I'm pleased to invite you to provide feedback on virtualization in Ubuntu in this simple, brief, 6-question survey:


We are eager to hear your feedback on a few particular questions about KVM, QEMU, Virsh, Virt-Manager, Xen, VirtualBox, OpenVZ, VMWare, Parallels, Amazon EC2, Eucalyptus, and other virtualization technologies.

Note: Nick Barcet will be conducting a much more comprehensive Ubuntu Server Survey in the near future. Stay tuned!

Thanks!
:-Dustin

These are my links for October 16th from 07:47 to 07:50:

• Twitter Data Analysis: An Investor’s Perspective - This is a guest post by Robert J. Moore, the CEO and co-founder of RJMetrics, a on-demand database analytics and business intelligence startup that helps online businesses measure, manage, and monetize better. He was previously a venture capital analyst and currently serves as an advisor to several New York startups. Robert blogs at The Metric System and can be followed on Twitter at @RJMetrics.This is a guest post by Robert J. Moore, the CEO and co-founder of RJMetrics, a on-demand database analytics and business intelligence startup that helps online businesses measure, manage, and monetize better. He was previously a venture capital analyst and currently serves as an advisor to several New York startups. Robert blogs at The Metric System and can be followed on Twitter at @RJMetrics.
• WebOps: Good prep for becoming a new parent? - I think I’ve said before somewhere that working in the field of web operations prepared me somewhat for being a parent. I thought the other day that I should write down some of this reasoning, because it’s pretty often that I’m reminded of similarities:
• Meanwhile: More Meta-Metrics - Like all sane web organizations, we gather metrics about our infrastructure and applications. As many metrics as we can, as often as we can. These metrics, given the right context, helps us figure out all sorts of things about our application, infrastructure, processes, and business.

Dear Blog,

I am really sorry, I've been unfaithful. I have been seing another blog lately. Please don't be mad...

I have accepted to write a monthly blog on WorksWithU.  A couple entries already made:

read more

In little over a week we have two great conferences scheduled for Atlanta.   We have planned on running the second Cloud Camp Atlanta on Wednesday 10/28/2009 and then over the next two days the first NoSqlEast conference on 10/29-10/30.  The idea is to get two exciting crowds to mingle.  Brad Anderson @boorad has done an outstanding job getting some the most exciting thought leaders in cloud computing to come to Atlanta to talk about “Big Data” or the new type of “storage” in his NoSqlEast conference.  Over at our good old Cloud Camp ,we will be talking about all things cloud and we will also be inviting some of the NoSql speakers to talk.  This could turn out to be a great set of events for Atlanta’s technical community.

Please help me spread the word on both conferences.  Also, if you are interested in volenteering for the Cloud Camp please contact me at botchagalupe at gmail dot com….

Thanks

John Willis

New updates have been released for the Ubuntu and Debian AMIs (EC2 images) published on:

http://alestic.com

The following notes apply to this release:

• The images have been upgraded to use the newest 2.6.21 kernel, ramdisk, and kernel modules from Amazon. This fixes a serious security hole in the previous 2.6.21 kernel.

• The Alestic PPA (personal package archive) has been added to the Ubuntu AMIs. This makes it easy to install software packages listed in this PPA, including ec2-consistent-snapshot.

• The runurl package from the Alestic PPA has been pre-installed on the Ubuntu AMIs. This can be a handy tool for setting up new instances with user-data scripts.

• The EC2 AMI tools have been upgraded to version 1.3-34544.

• The ec2-ami-tools package version has been pinned so it does not get downgraded if the official Ubuntu archives still have older versions.

• All packages have been upgraded to their respective latest versions.

• The Ubuntu Karmic images were not updated and have been removed from the listings at the top of http://alestic.com. If you would like to use Ubuntu Karmic Beta, please test with the AMIs published by Canonical listed a bit lower down on the page.

Please give these new images a spin and let us know if you run into any problems.

Enjoy

Many Ubuntu and Debian images for Amazon EC2 include a hook where scripts passed as user-data will be run as root on the first boot.

At Campus Explorer, we’ve been experimenting with an approach where the actual user-data is a very short script which downloads and runs other scripts. This idea is not new, but I have simplified the process by creating a small tool named runurl which adds a lot of flexibility and convenience when configuring new servers.

Usage

The basic synopsis looks like:

runurl URL [ARGS]...

The first argument to the runurl command is the URL of a script or program which should be run. All following options and arguments are passed verbatim to the program as its options and arguments. The exit code of runurl is the exit code of the program.

The runurl command is a very short and simple script, but it makes the user-data startup scripts even shorter and simpler themselves.

Example 1

If the following content is stored at http://run.alestic.com/demo/echo

#!/bin/bash
echo "$@"

then this command:

runurl run.alestic.com/demo/echo "hello, world"

will itself output:

hello, world

You can specify the “http://” in the URLs, but since it’s using wget to download them, the specifier is not necessary and the code might be easier to read without it.

Example 2

Here’s a more substantial sample user-data script which invokes a number of other remote scripts to upgrade the Ubuntu packages, install the munin monitoring software, install and run the Folding@Home application using origami with credit going to Team Ubuntu. It finally sends an email back home that it’s active.

This sample assumes that runurl is installed on the AMI (e.g., Ubuntu AMIs published on http://alestic.com>). For other AMIs, see below for additional commands to add to the start of the script.

#!/bin/bash -ex
runurl run.alestic.com/apt/upgrade
runurl run.alestic.com/install/munin
cd /root
runurl run.alestic.com/install/folding@home -u ec2 -t 45104 -b small
runurl run.alestic.com/email/start youremail@example.com

Note that the last command passes a parameter to the script, identifying where the email should be sent. Please change this if you test the script.

With the above content stored in a file named folding.user-data, you could start 5 new c1.medium instances running the Folding@Home software using the command:

ec2-run-instances                    \
  --user-data-file folding.user-data \
  --key [KEYPAIR]                    \
  --instance-type c1.medium          \
  --instance-count 5                 \
  ami-ed46a784

You can log on to an instance and monitor the installation with

tail -f /var/log/syslog

Once the Folding@Home application is running, you can monitor its progress with:

/root/origami/origami status

and after 15 minutes, check out the Munin system stats at

http://ec2-HOSTNAME/munin/

Expiring URLs

One of the problems with normal user-data scripts is that the contents exist as long as the instance is running and any user on the instance can read the contents of the user-data. This puts any private or confidential information in the user-data at risk.

If you put your actual startup code in private S3 buckets, you can pass runurl a URL to the contents, where the URL expires shortly after it is run. Or, the script could even delete the contents itself if you set it up correctly. This reduces the exposure to the time it takes for the instance to start up and does not let anybody else access the URL during that time.

Updating

Another benefit of keeping the actual startup code separate from the user-data content itself is that you can modify the startup code stored at the URL without modifying the user-data content.

This can be useful with services like EC2 Auto Scaling, where the specified user-data cannot be dynamically changed in a launch configuration without creating a whole new launch configuration.

If you modify the runurl scripts, the next server to be launched will automatically pick up the new instructions.

Bootstrapping

The runurl tool is pre-installed in the latest Ubuntu AMIs published on http://alestic.com. If you are using an Ubuntu image which does not include this software, you can install it from the Alestic PPA using the following commands at the top of your user-data script:

codename=$(lsb_release -cs)
echo "deb http://ppa.launchpad.net/alestic/ppa/ubuntu $codename main"|
  sudo tee /etc/apt/sources.list.d/alestic-ppa.list    
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BE09C571
sudo apt-get update
sudo apt-get install -y runurl

If you are using a Linux distro other than Ubuntu, you can install runurl using the following commands at the top of your user-data script:

sudo wget -qO/usr/bin/runurl run.alestic.com/runurl
sudo chmod 755 /usr/bin/runurl

The subsequent commands in the user-data script can then use the runurl command as demonstrated in the above example.

SSL

To improve your certainty that you are talking to the right server and getting the right data, you could use SSL (https) in your URLs. If you are talking to S3 buckets, however, you’ll need to use the old style S3 bucket access style like:

runurl https://s3.amazonaws.com/run.alestic.com/demo/echo "hello, mars"

This is probably not as critical when accessing it from an EC2 instance as you’re operating over Amazon’s trusted network.

Caveats

There are a number of things which can go wrong when using a tool like runurl. Here are some to think about:

• Only run content which you control or completely trust.

• Just because you like the content of a URL when you look at it in your browser does not mean that it will still look like that when your instance goes to run it. It could change at any point to something that is broken or even malicious unless it is under your control.

• If you depend on this approach for serious applications, you need to make sure that the content you are downloading is coming from a reliable server. S3 is reasonable (with retries) but you also need to consider the DNS server if you are depending on a non-AWS hostname to access the S3 bucket.

The name run.alestic.com points to an S3 bucket, but the DNS for this name is not redundant or worthy of use by applications with serious uptime requirements. This particular service should be considered my playground for ideas and there is no commitment on my part to make sure that it is up or that the content remains stable.

If you like what you see, please feel free to copy any of the open source content on run.alestic.com and store it on your own reliable and trusted servers. It is all published under the Apache2 license.

Project

I’m using this simple script as an opportunity to come up to speed with hosting projects on Launchpad. You can access the source code and submit bugs at

https://launchpad.net/runurl

You can also use launchpad and bazaar to branch the source into parallel projects and/or submit requests to merge patches into the main development branch.

[Update 2009-10-11: Document use of Alestic PPA]

Sorry for the mix-up.  The post you're looking for is this one.

On Monday, October 3, the San Francisco Cloud Computing Club and Silicon Valley Cloud Computing Club hosted a joint session that was notable for any number of reasons. Someone described it as being involved in a Twitter / Clouderati twitterstorm, but face-to-face. Whatever it felt like, it was a great source of good thought and numerous, mutually respected points of view.

James Watters, of Silcon Angle acted as the MC and moderator for the session, and took it upon himself to capture the spirit of the session. He kindly invited me to add in my take on the meetup and we found ourselves with a jointly authored recollection of the conversation.

Here's a snippet. For the full version, take a look at Silicon Angle's site:

...

Q: What is the impact of internal private clouds on both enterprises and external cloud service providers? (Question submitted by Randy Bias)

James Watters: I got the ’scrunch face’ from Randy Bias, and James Urquhart when I suggested that private clouds need to adhere to public cloud standards to be really useful. I believe this is important because it keeps both the economics and usability innovations of the public cloud proximal to how users evaluate their internal private clouds–or as /Hoff said once, allows public cloud to be the forcing function for change.

If Private or internal clouds get really exotic, with proprietary in-house created management, deployment and consumption functions they won’t play as easily with the coming wealth of interesting solutions created on top of public cloud standards.

The other point is simple: this is what really smart companies already have today. If you sit down with the top investment banking firms in the country many of them have highly sophisticated JeOS optimized application deployment, scaling, patching, and management functions for autonomic computing–but its expensive to create this kind of in-house IP.

Amazon sources tell me that over 40% of their revenues are driven by third party applications built directly atop their API. If you build an internal cloud not compliant to public standards you may be left without access to this increasingly important ecosystem of innovation.

Rich Miller: For better or worse, the adoption of cloud-oriented computing by the Enterprise and Small-Medium Business (SMB) will start as a transition from ‘the way things are done now’ to in-house, on-premise clouds. IT organizations will get religion … in part through the widespread adoption of server virtualization … and start operating their in-house IT organizations like utilities: lots of self-service, pay-as-you-go, multi-tenancy. (Remember: cloud is an operating model, not just a technology model.)

But, in order to get there in an orderly fashion, the path will be evolutionary. And, in order to get there, some of the internal clouds will be mixed-bags of infrastructure-cloud offerings (especially in-house data clouds), platform-cloud offerings and application-cloud offerings.

To your point, James, one way in which coordination and compatibility with public cloud offerings may come about is if the management systems that the enterprise uses for their in-house operations are built to recognized ’standards’… those offered by the most powerful service providers (e.g. Amazon AWS) or technology providers (e.g. VMware). Over a reasonable period of time, the management of an in-house, on-premise cloud will morph easily into managing hybrids (both on- and off-prem). ...

Cloud Computing

Over the years, Amazon has repeatedly recommended that customers who care about the security of their data should consider encrypting information stored on disks, whether ephemeral storage (/mnt) or EBS volumes. This, even though they take pains to ensure that disk blocks are wiped between uses by different customers, and they implement policies which restrict access to disks even by their own employees.

There are a few levels where encryption can take place:

1. File level. This includes tools like GnuPG, freely available on Ubuntu in the gnupg package. If you use this approach, make sure that you don’t store the unencrypted information on the disk before encrypting it.

2. File system level. This includes useful packages like encfs which transparently encrypt files before saving to disk, presenting the unencrypted contents in a virtual file system. This can even be used on top of an s3fs file system letting you store encrypted data on S3 with ease.

3. Block device level. You can place any file system you’d like on top of the encrypted block interface and neither your application nor your file system realize that the hardware disk never sees unencrypted data.

The rest of this article presents a simple way to set up a level of encryption at the block device level using cryptsetup/LUKS. It has been tested on the 32-bit Ubuntu 9.10 Jaunty server AMI listed on http://alestic.com and should work on other Ubuntu AMIs and even other distros with minor changes.

This walkthrough uses the /mnt ephemeral storage, but you can replace /mnt and /dev/sda2 with appropriate mount point and device for 64-bit instance types or EBS volumes.

Setup

Install tools and kernel modules:

sudo apt-get update
sudo apt-get install -y cryptsetup xfsprogs
for i in sha256 dm_crypt xfs; do 
  sudo modprobe $i
  echo $i | sudo tee -a /etc/modules
done

Before you continue, make sure there is nothing valuable on /mnt because we’re going to replace it!

sudo umount /mnt
sudo chmod 000 /mnt

Encrypt the disk and create your favorite file system on it:

sudo luksformat -t xfs /dev/sda2
sudo cryptsetup luksOpen /dev/sda2 crypt-sda2

Remember your passphrase! It is not recoverable!

Update /etc/fstab and replace the /mnt line (or create a new line for an EBS volume):

fstabentry='/dev/mapper/crypt-sda2 /mnt xfs noauto 0 0'
sudo perl -pi -e "s%^.* /mnt .*%$fstabentry%" /etc/fstab

Mount the file system on the encrypted block device:

sudo mount /mnt

You’re now to free to place files on /mnt knowing that the content will be encrypted before it is written to the hardware disk.

After reboot, /mnt will appear empty until you re-mount the encrypted partition, entering your passphrase:

sudo cryptsetup luksOpen /dev/sda2 crypt-sda2
sudo mount /mnt

Notes

See “man cryptsetup” for info on adding keys and getting information from the LUKS disk header.

It is possible to auto-mount the encrypted disk on reboot if you are willing to put your passphrase in the root partition (almost ruins the point of encryption). See the documentation on crypttab and consider adding a line like:

crypt-sda2 /dev/sda2 /PASSPHRASEFILE luks

Study the cryptsetup documentation carefully so that you understand what is going on. Keeping your data private is important, but it’s also important that you know how to get it back in the case of problems.

This article does not attempt to cover all of the possible security considerations you might need to take into account for data leakage on disks. For example, sensitive information might be stored in /tmp, /etc, or log files on the root disk. If you have swap enabled, anything in memory could be saved in the clear to disk whenever the operating system feels like it.

How do you solve your data security challenges on EC2?

This article was based on a post made on the EC2 Ubuntu group.

For more pictures in the “Old Country” set link here…

The Ubuntu One beta is going very well. We have appreciated every bug, IRC message, Launchpad Answers question, and Ubuntu Forums post about the service since the beta launch in early May. This community feedback has been extremely important to the decisions we make and have made in developing the service.

With the release of the Ubuntu 9.10 Beta last week which features Ubuntu One as a default option, we thought we’d share a few more recent updates.

More storage
Ubuntu One offers two subscription plan options: 2 free GB for everyone’s essential storage needs and a $10 USD plan with more capacity. We’re happy to announce that we have increased the size of the paid plan from 10 GB to an incredible 50 GB. Ubuntu One paid subscribers can now backup, sync, and share more of their music, photos, and movies.

Expanded services
Ubuntu One started with files and folders. Now we’re expanding the service to synchronize more desktop applications that people use each day. In Ubuntu 9.10, Ubuntu One will backup and synchronize Tomboy notes, Firefox bookmarks, and Evolution contacts.

Easy setup
Ubuntu 9.10 is the first Ubuntu release with Ubuntu One pre-installed. It now only takes a few clicks to enable automatic file synchronization for your Ubuntu computer or computers.

Subscribe now to try out all of these features and more.

Matt Griffin, Product Manager for Ubuntu One

These are my links for October 4th from 17:25 to 17:30:

• PHP on the Microsoft Azure Platform | Cloudiquity - PHP is now officially supported as a language for developing applications that can run on Windows Azure. This makes a lot of sense from Microsoft’s viewpoint as the number of PHP developers that use Windows environments for PHP is not insignificant. Microsoft have now made it possible to run PHP applications on Microsoft Azure servers and also take advantage of the Azure storage services (Table and Blob Storage).
• 10 Mechanical Turk Tests You Should Run Right Now - Use Mechanical Turk for your business. USE IT NOW. You can never spend too much money on Mechanical Turk. It’s too cheap to do that. If you dont' know what Mechanical Turk is yet, here's an explanation: 1) You add a job. 2) You set a price for that job. 3) People do your job for very very little amounts of money. The jobs that you put up should be fairly simple, and take very little time (usually 1-2 minutes a job).
• Using Amazon EC2 Metadata as a Simple DNS - I use the amazon metadata for creating /etc/hosts and do this on a cron schedule. This does everything I need. Instead of fancy DynDNS tricks or having to run and manage an internal DNS server I just have a ruby script that looks at the metadata ec2 to build /etc/hosts. It's easy. To set it up yourself and try it all you need are 3 easy steps. Step 1- Start each of your instances with unique named key that matches what you want their internal hostname to be. Such as "onion" or "potato" or whatever you want to call them. Step 2- Make sure you have ruby, rubygems and amazon-ec2 (rubygem) installed. Then create a ruby script in /usr/local/sbin/hosts that has the following:
• Amazon Web Services Blog: Don’t Forget: You Can Use Amazon SimpleDB For Free! - We polled the attendees at a recent Amazon SimpleDB webinar and found that over half of them didn't know that they could start using the service for free. That's a shame because SimpleDB is easy to use, scales easily to handle high request rates, and is available in our US and EU regions.

Amazon EC2 recently released a feature which lets you share an EBS snapshot so that other accounts can access it. The snapshot can be shared with specific individual accounts or with the public at large.

You should obviously be careful what files you put on a shared EBS snapshot because other people are going to be able to read them. What may not be so obvious to is that you also need to be wary of what files are not currently on the snapshot but once were.

For example, if you copied some files onto the EBS volume, then realized a few contained sensitive information, you might think it’s sufficient to delete the private files and continue on to create a public EBS snapshot of the volume.

The problem with this is that EBS is an elastic block store device, not an interface at the file system level. Any block which was once written to on the block device will be available on the shared EBS volume, even if it is not being used by a visible file on the file system.

Since popular Linux file systems do not generally wipe data when a file is deleted, it is often possible to recover the contents of the deleted files. Even attempting to overwrite a file may, depending on the application, leave the original content available on the disk.

This means any content that touched your EBS volume at any point may still be available to users of your shared EBS snapshot.

To be clear: I do not consider this to be a security flaw in EC2 or EBS. It is merely a security risk for people who do not understand and take precautions against the combination of interactions with file systems, block devices, EBS volumes, and snapshots.

$100 Reward

To demonstrate the security risk, I have created a simple challenge with a tangible reward. Here is a public EBS snapshot:

snap-d53484bc

This EBS snapshot contains two files. The first file is README-1.txt which has nothing sensitive in it but will let you know that you’ve got the right device mounted on your EC2 instance.

The second file created on the source EBS volume contained an Amazon.com gift certificate for $100. I deleted this second file, then took an EBS snapshot of the volume and released it to the public.

The first person who successfully recovers the deleted file on this shared EBS snapshot and enters the gift certificate code into their Amazon.com account will win the $100 prize. Subsequent solvers will get a notice from Amazon that the certificate has already been redeemed, but you still get credit for solving it and helping demonstrate the risks.

Feel free to post a comment on this blog entry if you recovered the deleted file on the shared EBS snapshot. Recipes for doing so are welcomed even if you were not the first. I tested this, so know it’s possible and that the deleted file is still accessible (but I did not redeem the gift certificate, of course).

Good luck!