Posts Tagged ‘ubuntu’

Ubuntu Hardware Summit

// December 12th, 2011 // 1 Comment » // Uncategorized

Last week saw the third annual Ubuntu Hardware Summit (UHS) taking place in Taipei. With over two hundred attendees present, the show is fast becoming one of the must-attend events within the software, ODM and OEM environments across Taiwan.

With standing room only at the back of the room during the Keynote speech, Canonical set the scene for the next two years including the growth of Ubuntu, the multitude of device enablement and an insight into Ubuntu Cloud and Ubuntu Server. After the Keynote, UHS then went into break-out Tracks which included topics on Ubuntu Cloud, Thin Client solutions, hardware enablement and Ubuntu System Architecture.

You can find all presentations from the day at odm.ubuntu.com and clicking the link ‘Download material’.

Building a Precise Pangolin: A summary of UDS success

// November 21st, 2011 // 1 Comment » // Uncategorized

The Ubuntu Developer Summit – UDS – is a major event in the Canonical calendar. Taking place every six months, it is the Ubuntu event which defines the focus and plans for our up-coming version of Ubuntu. In the first week of November, over 800 people, from Canonical engineers and employees, Ubuntu community members, partners, ISVs, upstreams and many more gathered to discuss and plan for the upcoming Ubuntu 12.04, code-named Precise Pangolin.

UDS covered 420 sessions, under nine tacks, from desktop to design, community to server and cloud. Attendees worked in the usual collaborative and open environment and spent the week pooling their experience and expertise and sharing best practise resulting, as always, in the very best ideas. Right now, those ideas are are represented in hundreds of blueprint documents and are being put into action by developers, community and Canonical, who are already driving forward for April’s launch. As a practical demonstration of that openness you can track our progress here (note, it’s early days!): http://status.ubuntu.com/ubuntu-precise/.

Focus on desktop and the cloud

Over the coming months, we’ll see much more of the fruits of UDS’ labour as new features are developed and collaborations and partnerships formed. Right now, the focus is on refinement, quality and stabilisation. As Ubuntu 12.04 will be a LTS release, which, for the first time, will be supported for five years, getting performance and stability right will be extremely important. For businesses, cloud is becoming ever more important, so we’ll be looking at building out a robust test infrastructure; there will be continued support for the latest releases of OpenStack and much effort will be put into improving Juju and developing the Charms collection.

For our desktop users, refinement of the interface is a continued focus and we’ll regularly run usability testing to make sure Ubuntu looks and feels great. For ubuntu 12.04, there will be a lot of developments for power users, including multi-monitor support, and improvements to boot speed, text-free boot and power consumption. And of course, the community centres around the developer programme, design, governance and loco teams. Engaging and embracing developers continues to be important (for free software) as we seek to bring new and exciting applications to the Ubuntu platform.

Our wonderful sponsors

We also wanted to take this opportunity to extend a special thank you to all of our sponsors who helped us accomplish this monumental task. Cloud Foundry, Rackspace, Google, System 76, Freescale, Nebula, as well as our media partners, Ubuntu User, Linux Pro Magazine, all attended and contributed to the success of UDS in different ways. Some gave plenary sessions;
Brian Thomason and Juan Negron – Cloudfoundry Server deployments using Juju
James Blair and Monty Taylor – Rackspace – Distributed QA in the OpenStack Project

It’s Linaro’s summit too

Also, for the second time, UDS was co-hosted with the Linaro Connect event, where the best software developers met to plan out and code the future of Linux on ARM. Canonical has been actively participating in the Linaro project since it began in 2010, and having both events run in parallel is a good opportunity to share new ideas and collaborate. ARM continues to gain more traction in traditional PC areas, such as the data center and Ubuntu continues to contribute to the enablement of ARM. You can hear more from David Brash’s Linaro plenary, An ARM Technology Update.

And a vision for what’s next

While the focus for Canonical and the Ubuntu community is firmly on the next launch , we’ve already started to think beyond this release. In Mark’s opening keynote, he talked of extending the Ubuntu mission; “‘Linux for Human Beings’ cannot end at the desktop, but needs to take into account the devices that will be used by human beings in the years to come….”. In the coming two years, we’ll start to see Ubuntu powering tablets, phones, TVs and smart screens from the car to the office kitchen, and it will connect those devices cleanly and seamlessly to the desktop, the server and the cloud. You can read more on Mark’s vision for the future of Ubuntu on his blog: or see the full keynote.

For lots more video and insight you can check out the excellent Ubuntu Developers Channel on YouTube

So, roll on Ubuntu 12.04!

Deploying the Minecraft Charm!

// November 16th, 2011 // 6 Comments » // Uncategorized

Ubuntu Hardware Summit 2011 December 8th, Taipei

// November 15th, 2011 // 6 Comments » // Uncategorized

This year’s Ubuntu Hardware Summit (UHS) will take place on December 8th at the Grand Victoria Hotel in Taipei. You can register your place at www.ubuntu.com.uhs

Building on the success of 2010 (with over 200 attendees), the 2011 Ubuntu Hardware Summit promises to deliver more. With keynote speeches from various members of the Canonical team and a more focused technical delivery, UHS is created especially for product managers and engineers at ODMs and OEMs, with interest or responsibility in deploying Ubuntu on new computers and devices.

Highlights will include presentations on Ubuntu Server, deploying Ubuntu Cloud, QA, power management, hardware enablement….and much more! Details of the event can also be found on the new Ubuntu Hardware Debugging website at http://odm.ubuntu.com/portal/

UHS is sponsored by Canonical and free of charge.

To reserve your space, visit www.ubuntu.com/uhs today as registrations will close on 29th November 2011.

Running EC2 Instances on a Recurring Schedule with Auto Scaling

// November 15th, 2011 // 1 Comment » // Uncategorized

Do you want to run short jobs on Amazon EC2 on a recurring schedule, but don’t want to pay for an instance running all the time? Would you like to do this using standard Amazon AWS services without needing an external server to run and terminate the instance? Amazon EC2 Auto Scaling is normally used to keep a reasonable number of instances running to handle measured or expected load (e.g., web site traffic, queue processing). In this article I walk through the steps to create an Auto Scaling configuration that runs an instance on a recurring schedule (e.g., four times a day) starting up a pre-defined task and letting that instance shut itself down when it is finished. We tweak the Auto Scaling group so that this uses the minumum cost in instance run time, even though we may not be able to predict in advance exactly how long it will take to complete the job. Here’s a high level overview for folks familiar with Auto Scaling:
  • The instance is started using a recurring schedule action that raises the min and max to 1.
  • The launch configuration is set to pass in a user-data script that runs the desired job on first boot. There’s no need to build our own AMI unless the software installation takes too long.
  • The user-data script runs shutdown at the end to move the instance to the “stopped” state, suspending hourly run charges.
  • Another recurring schedule action lowers the min and max to 0. This is done long after the job should have completed, just to clean up the stopped instance or to terminate an instance that perhaps failed to shut itself down. It also resets the min/max count so a new instance will be started the next time they are raised.
The missing key I found through experimentation is that we need to also suspend Auto Scaling’s (usually valid) desire to replace any unhealthy instances. If we don’t turn off this behavior, Auto Scaling will start another instance as soon as the first one shuts itself down, causing the job to be run over and over.

Prerequisites

The examples in this article assume that you have:
  1. Installed and set up the EC2 command line tools
  2. Installed and set up the EC2 Auto Scaling command line tools
  3. Tell EC2 about your ssh keys using the approach described here: Uploading Personal ssh Keys to Amazon EC2

User-data Script

The first step in setting up this process is to create a user-data script that performs the tasks you want your scheduled instance to execute each time it runs. This user-data script can do anything you want as long as it runs this command after all the work has beeen completed:
shutdown -h now 
I have created a demo user-data script for this article which you can download to your local computer now, saving it as demo-user-data-script.sh
wget -q https://raw.github.com/alestic/demo-ec2-schedule-instance/master/user-data/demo-user-data-script.sh 
WARNING: Do not attempt to run this user-data script on your local computer! Edit the downloaded demo-user-data-script.sh file and change this line at the top of the script to reflect your email address:
EMAIL=youraddress@example.com 
This demo user-data script:
  • Upgrades the Ubuntu instance
  • Installs Postfix with a generic configuration so that it can send email
  • Sends you a demo informative message about the instance at the email address you edited in the script.
  • Sleeps for 5 minutes giving the email a chance to be delivered
  • Shuts down the EC2 instance
The first time you run this demo, try using the script as it stands, only changing your email address. Then, try adjusting the script little by little to make it do tasks that you would find more useful. If you’d like, you can test the user-data script by itself running an instance of Ubuntu 11.10. Please update the AMI id to use the latest release:
ami_id=ami-a7f539ce # Ubuntu 11.10 Oneiric server ec2-run-instances \ --key $USER \ --instance-type t1.micro \ --instance-initiated-shutdown-behavior terminate \ --user-data-file demo-user-data-script.sh \ $ami_id 
You should see an email from the instance and then it should terminate itself about 5 minutes later. Make sure you terminate it manually if it stays running after 10 minutes.

Auto Scaling Group

With the user-data script in hand, we are now ready to create the Auto Scaling setup. Set some variables used in the commands below. Make sure you are using the latest release of the appropriate AMI.
ami_id=ami-a7f539ce # Ubuntu 11.10 Oneiric server region=us-east-1 # Region for running the demo zone=${region}a # A zone in that region export EC2_URL=https://$region.ec2.amazonaws.com export AWS_AUTO_SCALING_URL=https://autoscaling.$region.amazonaws.com launch_config=demo-launch-config auto_scale_group=demo-auto-scale-group 
This lauch configuration describes how we want our instance run each time including the AMI id, instance type, ssh key, and most importantly, our user-data script we edited above:
as-create-launch-config \ --key $USER \ --instance-type t1.micro \ --user-data-file demo-user-data-script.sh \ --image-id $ami_id \ --launch-config "$launch_config" 
The Auto Scaling group keeps track of many things including how many instances we want to have running, how they should be run (launch config above), and what instances are currently running.
as-create-auto-scaling-group \ --auto-scaling-group "$auto_scale_group" \ --launch-configuration "$launch_config" \ --availability-zones "$zone" \ --min-size 0 \ --max-size 0 
Here’s a non-obvious but key part of this approach! Tell the Auto Scaling group that we don’t want it to restart our instance right after the instance intentionally shuts down (or fails):
as-suspend-processes \ "$auto_scale_group" \ --processes ReplaceUnhealthy 
Now we’re finally ready to tell EC2 Auto Scaling when we want to run the instance launch configuration in the Auto Scaling group. Here’s an example that starts one instance four times a day to run the above user-data script:
# UTC: 1:00, 7:00, 13:00, 19:00 as-put-scheduled-update-group-action \ --name "demo-schedule-start" \ --auto-scaling-group "$auto_scale_group" \ --min-size 1 \ --max-size 1 \ --recurrence "0 01,07,13,19 * * *" 
And, we need to create a matching action to make sure the instance is terminated at some point after the longest time the job could take. For this demo, we’ll trigger it 55 minutes later, but it could just as easily be 3 hours and 55 minutes later:
# UTC: 1:55, 7:55, 13:55, 19:55 as-put-scheduled-update-group-action \ --name "demo-schedule-stop" \ --auto-scaling-group "$auto_scale_group" \ --min-size 0 \ --max-size 0 \ --recurrence "55 01,07,13,19 * * *" 
The recurrence value is in a cron format using UTC. You are welcome to change the specs in the above commands to any time you want to run the demo, especially if you don’t want to wait up to six hours for it to trigger. Before setting new schedules, make sure you delete the existing schedule (see the next section). Don’t forget to make the stop time(s) match up appropriately with the start time(s).

Clean up

Once you’re done with this demo, you can delete the AWS resources we created by following these steps: Delete the schedule start and stop actions:
as-delete-scheduled-action \ --force \ --name "demo-schedule-start" \ --auto-scaling-group "$auto_scale_group" as-delete-scheduled-action \ --force \ --name "demo-schedule-stop" \ --auto-scaling-group "$auto_scale_group" 
Scale the Auto Scaling group down to zero instances. This will terminate any running instances:
as-update-auto-scaling-group \ "$auto_scale_group" \ --min-size 0 \ --max-size 0 
Delete the Auto Scaling group
as-delete-auto-scaling-group \ --force-delete \ --auto-scaling-group "$auto_scale_group" 
Delete the Auto Scaling launch config:
as-delete-launch-config \ --force \ --launch-config "$launch_config" 
You might now want to check to make sure nothing was left over. This works best in a wide terminal:
as-describe-launch-configs --headers as-describe-auto-scaling-groups --headers as-describe-scheduled-actions --headers as-describe-auto-scaling-instances --headers 

Timing

Everything takes a little time to filter through the system including:
  • scheduled action to raise min/max
  • triggering the start of an instance after a min/max is raised
  • starting an instance
  • booting an instance, installing software
  • running your job
  • shutting down an instance
  • scheduled action to lower min/max
  • triggering the termination of an instance after a min/max is lowered
When you set up the schedules, remember to make room for these things. For example, don’t schedule the termination of your instance too early or it could kill your job before it has a chance to complete.

Notes

Here are some great resources from Amazon for getting started with and learning about Auto Scaling: The user-data script logging uses the approach described here: Watching the output of the user-data script lets you monitor its progress as well as debug where things might be going wrong. I haven’t run the above approach except in testing, and would welcome any pointers or improvements folks might have to offer. Original article: http://alestic.com/2011/11/ec2-schedule-instance

Hadoop World: Ubuntu, Hadoop and Juju

// November 15th, 2011 // 2 Comments » // Uncategorized

I’m always interested in what’s happening at Canonical and with Ubuntu.  Last week at Hadoop World I ran into a couple of folks from the company (coincidentally both named Mark but neither Mr. Shuttleworth).  Mark Mims from the server team was willing to chat so I grabbed some time with him to learn about what he was doing at Hadoop World and what in the heck is this “charming” Juju?

Some of the ground Mark covers

  • Making the next version of Ubuntu server better for Hadoop and big data
  • (0:34) What are “charms” and what do they have to do with service orchestration
  • (2:05) Charm school and learning to write Juju charms
  • (2:54)  Where does “Orchestra” fit in and how can it be used to spin up OpenStack
  • (3:40) What’s next for Juju

But wait, there’s more!

Stay tuned for more interviews from last week’s Hadoop world.  On tap are:

  • Todd Papaioannou from Battery Ventures
  • John Gray of Facebook
  • Erik Swan of Splunk
  • Nosh Petigara of 10gen/MongoDB.

Extra-credit reading

Pau for now..


OpenStack Essex-1 milestone

// November 14th, 2011 // Comments Off // Uncategorized

Experimenting with juju and Puppet

// November 8th, 2011 // Comments Off // Uncategorized

I’ve been playing with juju for a few months now in different contexts and I’ve really enjoyed the ease with which it allows me to think about services rather than resources.

More recently I’ve started thinking about best-practices for deploying services using juju, while still using puppet to setup individual units. As a simple experiment, I wrote a juju charm to deploy an irssi service [1] to dig around. Here’s what I’ve found so far [2]. The first is kind of obvious, but worth mentioning:

Install hooks can be trivial:

#!/bin/bash
sudo apt-get -y install puppet

juju-log "Initialising machine state."
puppet apply $PWD/hooks/initial_state.pp

Normally the corresponding manifest (see initial_state.pp) would be a little more complicated, but in this example it’s hardly worth mentioning.

Juju config changes can utilise Puppet’s Facter infrastructure:

This enables juju config options to be passed through to puppet, so that config-changed hooks can be equally simple:

#!/bin/bash
juju-log "Getting config options"
username=`config-get username`
public_key=`config-get public_key`

juju-log "Configuring irssi for user"
# We specify custom facts so that they're accessible in the manifest.
FACTER_username=$username FACTER_public_key=$public_key puppet apply $PWD/hooks/configured_state.pp

In this example, it is the configured state manifest that is more interesting (see configured_state.pp). It adds the user to the system, sets up byobu with an irssi window ready to go, and adds the given public ssh key enabling the user to login.

The same would go for other juju hooks (db-relation-changed etc.), which is quite neat – getting the best of both worlds: the charm user can still think in terms of deploying services, while the charm author can use puppets declarative syntax to define the machine states.

Next up: I hope to experiment with an optional puppet master for a real project (something simple like the Ubuntu App directory), so that

  1. a project can be deployed without the (probably private) puppet-master to create a close-to-production environment, while
  2. configuring a puppet-master in the juju config would enable production deploys (or deploys of exact replicas of production to a separate environment for testing).

If you’re interested in seeing the simple irssi charm, the following 2min video demos:

# Deploy an irssi service
$ juju deploy --repository=/home/ubuntu/mycharms  local:oneiric/irssi
# Configure it so a user can login
$ juju set irssi username=michael public_key=AAAA...
# Login to find irssi already up and running in a byobu window
$ ssh michael@new.ip.address

and the code is on Launchpad.

[1] Yes, irssi is not particularly useful as a juju service (as I don’t want multiple units, or relating it to other services etc.), but it suited my purposes for a simple experiment that also automates something I can use for working in the cloud.

[2] I’m not a puppet or juju expert, so if you’ve got any comments or improvements, don’t hesitate.


Filed under: juju, puppet, ubuntu

Hadoop cluster with Ubuntu server and Juju

// November 8th, 2011 // Comments Off // Uncategorized

A while back I started experimenting with Juju and was intrigued by the notion of services instead of machines.

A bit of background on Juju from their website:

  • Formerly called Ensemble, juju is DevOps DistilledTM. Through the use of charms(renamed from formulas), juju provides you with shareable, re-usable, and repeatable expressions of DevOps best practices. You can use them unmodified, or easily change and connect them to fit your needs. Deploying a charm is similar to installing a package on Ubuntu: ask for it and it’s there, remove it and it’s completely gone.

I come from a DevOps background and know first hand the troubles and tribulations of deploying production services, webapps, etc.  One that's particularly "thorny" is hadoop.

To deploy a hadoop cluster, we would need to download the dependencies ( java, etc. ), download hadoop, configure it and deploy it.  This process is somewhat different depending on the type of node that you're deploying ( ie: namenode, job-tracker, etc. ).  This is a multi-step process that requires too much human intervention.  It is also a process that is difficult to automate and reproduce.  Imagine 10, 20 or 50 node cluster using this method.  It can get frustrating quickly and it is prone to mistake.

With this experience in mind ( and a lot of reading ), I set out to deploy a hadoop cluster using an Juju charm.

First things first, let's install Juju.  Follow the Getting Started documentation on the Juju site here.

According to the Juju documenation, we just need to follow some file naming conventions for what they call "hooks" ( executable scripts in your language of choice that perform certain actions ).  These "hooks" control the installation, relationships, start, stop, etc of your charm.  We also need to summarize the description of the formula in a file called metadata.yaml.  The metadata.yaml file describes the formula, it's interfaces, what it requires and provides among other things.  More on this file later when I show you the one for hadoop-master and hadoop-slave.

Armed with a bit of knowledge and a desire for simplicity, I decided to split the hadoop cluster in two:

  • hadoop-master (namenode and jobtracker )
  • hadoop-slave ( datanode and tasktracker )
I know this is not an all-encompassing list but, this will take care of a good portion of deployments and, the Juju charms are easy enough to modify that you can work your changes into them.

One of my colleagues, Brian Thomason did a lot of packaging for these charms so, my job is now easier.  The configuration for the packages has been distilled down to three questions:

  1. namenode ( leave blank if you are the namenode )
  2. jobtracker ( leave blank if you are the jobtracker )
  3. hdfs data directory ( leave blank to use the default: /var/lib/hadoop-0.20/dfs/data )
Due to the magic of Ubuntu packaging, we can even "preseed" the answers to those questions to avoid being asked about them ( and stopping the otherwise automatic process ). We'll use the utility debconf-set-selections for this.  Here is a piece of the code that I use to preseed the values in my charm:
  • echo debconf hadoop/namenode string ${NAMENODE}| /usr/bin/debconf-set-selections
  • echo debconf hadoop/jobtracker string ${JOBTRACKER}| /usr/bin/debconf-set-selections
  • echo debconf hadoop/hdfsdatadir string ${HDFSDATADIR}| /usr/bin/debconf-set-selections
The variable names should be self explanatory.  

Thanks to Brian's work, I now just have to install the packages ( hadoop-0.20-namenode and hadoop-0.20-jobtracker).  Let's put all of this together into a Juju charm.

  • Create a directory for the hadoop-master formula ( mkdir hadoop-master )
  • Make a directory for the hooks of this charm ( mkdir hadoop-master/hooks )
  • Let's start with the always needed metadata.yaml file ( hadoop-master/metadata.yaml ):
ensemble: formula
name: hadoop-master
revision: 1
summary: Master Node for Hadoop
description: |
  The Hadoop Distributed Filesystem (HDFS) requires one unique server, the
  namenode, which manages the block locations of files on the
  filesystem.  The jobtracker is a central service which is responsible
  for managing the tasktracker services running on all nodes in a
  Hadoop Cluster.  The jobtracker allocates work to the tasktracker
  nearest to the data with an available work slot.
provides:
  hadoop-master:
    interface: hadoop-master

  • Every Juju charm has an install script ( in our case: hadoop-master/hooks/install ).  This is an executable file in your language of choice that Juju will run when it's time to install your charm.  Anything and everything that needs to happen for your charm to install, needs to be inside of that file.  Let's take a look at the install script of hadoop-master:
#!/bin/bash
# Here do anything needed to install the service
# i.e. apt-get install -y foo  or  bzr branch http://myserver/mycode /srv/webroot


##################################################################################
# Set debugging
##################################################################################
set -ux
juju-log "install script"


##################################################################################
# Add the repositories
##################################################################################
export TERM=linux
# Add the Hadoop PPA
juju-log "Adding ppa"
apt-add-repository ppa:canonical-sig/thirdparty
juju-log "updating cache"
apt-get update


##################################################################################
# Calculate our IP Address
##################################################################################
juju-log "calculating ip"
IP_ADDRESS=`hostname -f`
juju-log "Private IP: ${IP_ADDRESS}"


##################################################################################
# Preseed our Namenode, Jobtracker and HDFS Data directory
##################################################################################
NAMENODE="${IP_ADDRESS}"
JOBTRACKER="${IP_ADDRESS}"
HDFSDATADIR="/var/lib/hadoop-0.20/dfs/data"
juju-log "Namenode: ${NAMENODE}"
juju-log "Jobtracker: ${JOBTRACKER}"
juju-log "HDFS Dir: ${HDFSDATADIR}"

echo debconf hadoop/namenode string ${NAMENODE}| /usr/bin/debconf-set-selections
echo debconf hadoop/jobtracker string ${JOBTRACKER}| /usr/bin/debconf-set-selections
echo debconf hadoop/hdfsdatadir string ${HDFSDATADIR}| /usr/bin/debconf-set-selections


##################################################################################
# Install the packages
##################################################################################
juju-log "installing packages"
apt-get install -y hadoop-0.20-namenode
apt-get install -y hadoop-0.20-jobtracker


##################################################################################
# Open the necessary ports
##################################################################################
if [ -x /usr/bin/open-port ];then
   open-port 50010/TCP
   open-port 50020/TCP
   open-port 50030/TCP
   open-port 50105/TCP
   open-port 54310/TCP
   open-port 54311/TCP
   open-port 50060/TCP
   open-port 50070/TCP
   open-port 50075/TCP
   open-port 50090/TCP
fi


  • There a few other files that we need to create ( start and stop ) to get the hadoop-master charm installed.  Let's see those files:
    • start
#!/bin/bash
# Here put anything that is needed to start the service.
# Note that currently this is run directly after install
# i.e. 'service apache2 start'

set -x
service hadoop-0.20-namenode status && service hadoop-0.20-namenode restart || service hadoop-0.20-namenode start
service hadoop-0.20-jobtracker status && service hadoop-0.20-jobtracker restart || service hadoop-0.20-jobtracker start

    • stop
#!/bin/bash
# This will be run when the service is being torn down, allowing you to disable
# it in various ways..
# For example, if your web app uses a text file to signal to the load balancer
# that it is live... you could remove it and sleep for a bit to allow the load
# balancer to stop sending traffic.
# rm /srv/webroot/server-live.txt && sleep 30

set -x
juju-log "stop script"
service hadoop-0.20-namenode stop
service hadoop-0.20-jobtracker stop

Let's go back to the metadata.yaml file and examin it in more detail:

ensemble: formula
name: hadoop-master
revision: 1
summary: Master Node for Hadoop
description: |
  The Hadoop Distributed Filesystem (HDFS) requires one unique server, the
  namenode, which manages the block locations of files on the
  filesystem.  The jobtracker is a central service which is responsible
  for managing the tasktracker services running on all nodes in a
  Hadoop Cluster.  The jobtracker allocates work to the tasktracker
  nearest to the data with an available work slot.
provides:
  hadoop-master:
    interface: hadoop-master

The emphasized section ( provides ) tells juju that this formula provides an interface named hadoop-master that can be used in relationships with other charms ( in our case we'll be using it to connect the hadoop-master with the hadoop-slave charm that we'll be writing a bit later ).  For this relationship to work, we need to let Juju know what to do ( More detailed information about relationships in charms can be found here ).

Per the Juju documentation, we need to name our relationship hooks hadoop-master-relation-joined  and it should also be an executable script in your language of choice.  Let's see what that file looks like:

#!/bin/sh
# This must be renamed to the name of the relation. The goal here is to
# affect any change needed by relationships being formed
# This script should be idempotent.

set -x

juju-log "joined script started"

# Calculate our IP Address
IP_ADDRESS=`unit-get private-address`

# Preseed our Namenode, Jobtracker and HDFS Data directory
NAMENODE="${IP_ADDRESS}"
JOBTRACKER="${IP_ADDRESS}"
HDFSDATADIR="/var/lib/hadoop-0.20/dfs/data"

relation-set namenode="${NAMENODE}" jobtracker="${JOBTRACKER}" hdfsdatadir="${HDFSDATADIR}"



juju-log "$JUJU_REMOTE_UNIT joined"

Your formula charm directory should now look something like this:
natty/hadoop-masternatty/hadoop-master/metadata.yamlnatty/hadoop-master/hooks/installnatty/hadoop-master/hooks/startnatty/hadoop-master/hooks/stopnatty/hadoop-master/hooks/hadoop-master-relation-joined
 This charm should now be complete...  It's not too exciting yet as it doesn't have the hadoop-slave counterpart to it but, it is a complete charm.

The latest version of the hadoop-master charm can be found here if you want to get it.

The hadoop-slave charm is almost the same as the hadoop-master charm with some exceptions.  Those I'll leave as an exercise for the reader.

The hadoop-slave charm can be found here if you want to get it.

Once you have both charm ( hadoop-master and hadoop-slave ) you can easily deploy your cluster by typing:

  • juju bootstrap   # ( creates/bootstraps the ensemble environment)
  • juju deploy --repository . local:natty/hadoop-master # ( deploys hadoop-master )
  • juju deploy --repository . local:natty/hadoop-slave # ( deploys hadoop-slave )
  • juju add-relation hadoop-slave hadoop-master # ( connects the hadoop-slave to the hadoop-master )
As you can see, once you have the charm written and tested, deploying the cluster is really a matter of a few commands.  The above example gives you one hadoop-master ( namenode, jobtracker ) and one hadoop-slave ( datanode, tasktracker ).

To add another node to this existing hadoop cluster, we add:

  • juju add-unit hadoop-slave # ( this adds one more slave )
Run the above command multiple times to continue to add hadoop-slave nodes to your cluster.

Juju allows you to catalog the steps needed to get your service/application installed, configured and running properly.  Once your knowledge has been captured in an Juju charm, it can be re-used by you or others without much knowledge of what's needed to get the application/service running.

In the DevOps world, this code re-usability can save time, effort and money by providing self contained charms that provide a service or application.

Quick UDS-P Wrapup

// November 5th, 2011 // 2 Comments » // Uncategorized

It’s late, I’m tired, so this is going to be brief.  But if I didn’t put something up now, chances are I’d procrastinate to the point where it didn’t matter anymore, so something is better than nothing.

JuJu

So the buzz all week was about Juju and Charms.  It’s a very cool technology that I think is really going to highlight the potential of cloud computing.  Until now I always had people comparing the cloud to virtual machines, telling me they already automate deploying VMs, but with Juju you don’t think about machines anymore, virtual of otherwise.  It’s all about services, which is really what you want, a service that is doing something for you.  You don’t need to care where, or on what, or in combination with some other thing, Juju handles all that automatically.  It’s really neat, and I’m looking forward to using it more.

Summit

Summit worked this week.  In fact, this is the first time in my memory where there wasn’t a problem with the code during UDS.  And that’s not because we left it alone either.  IS actually moved the entire site to a new server the day before UDS started.  We landed several fixes during the week to fix minor inconveniences experienced by IS or the admins.  And that’s not even taking into consideration all the last-minute features that were added by our Linaro developers the week prior.  But through it all, Summit kept working.  That, more than anything else, is testament to the work the Summit developers put in over the last cycle to improve the code quality and development processes, and I am very, very proud that.  But we’re not taking a break this cycle.  In fact, we had two separate sessions this week about ways to improve the user experience, and will be joined by some professional designers to help us towards that goal.

Ubuntu One eBook syncing

So what started off as an casual question to Stuart Langridge turned into a full blown session about how to sync ebook data using Ubuntu One.  We brainstormed several options of what we can sync, including reading position, bookmarks, highlights and notes, as well as ways to sync them in an application agnostic manner.  I missed the session on the upcoming Ubuntu One Database (U1DB), but we settled on that being the ideal way of handling this project, and that this project was an ideal test case for the U1DB.  For reasons I still can’t explain, I volunteered to develop this functionality, at some point during the next cycle.  It’s certainly going to be a learning experience.

Friends

Friends!  It sure was good to catch up with all of you.  Both friends from far-away lands, and those closer to home.  Even though we chat on IRC almost constantly, there’s still nothing quite like being face to face.  I greatly enjoyed working in the same room with the Canonical ISD team, which has some of the smartest people I’ve ever had the pleasure of working with.  It was also wonderful to catch up with all my friends from the community.  I don’t know of any other product or project that brings people together the way Ubuntu does, and I’m amazed and overjoyed that I get to be a part of it.