Posts Tagged ‘aws’

Introducing CloudTask – a cloud batch execution tool

// August 12th, 2011 // Comments Off // Uncategorized

The cloud. Isn't that just a new name everyone on the latest hype bandwagon is slapping on the same old stuff? Yes. Or rather, at least the way some clueless marketing types are using it that is. With so much smoke you'd forgive the cynics for thinking there's no fire. But... there are a few genuinely interesting things an IT guy can do today that just weren't practical a few years back.

Like renting an armada of servers you could never afford to buy to parallelize a computational task and get results in an hour instead of days, weeks or even months, for exactly the same cost. Now that's kind of exciting if you can pull it off.

We came across a modest version of this problem for various routine TurnKey related maintenance tasks such as rebuilding appliances. It sure was nice to be able to fire up a server on-demand, run a batch job and be able upload new images to sourceforge at 100MB/s. Usually we would leave the batch job running overnight and terminate the server the next day or so. That wasn't so bad for non-frequent tasks, but we realized we could do better. On Amazon EC2 running 10 servers for 1 hour costs the same as running 1 server for 10 hours. That's the theory anyhow. In practice launching and controlling many servers by hand can be painful.

Obviously, a bit of clever automation would be just the ticket. I hate reinventing the wheel so I first tried surveying existing solutions, but I couldn't find any that fit our needs.

So I rolled up my sleeves, and about a month or so later cloudtask was born. It's kind of neat. If you've ever had to do this sort of thing by hand or put together an ugly mess of scripts you'll probably find cloudtask an easier and more reliable primitive to build on.

Here's the documentation.

Here's a tacky video demo I whipped up (best viewed full screen):

Unavailable Availability Zones on Amazon EC2

// August 6th, 2011 // Comments Off // Uncategorized

I’m taking a class about using Chef with EC2 by Florian Drescher today and Florian mentioned that he noticed one of the four availability zones in us-east-1 is not currently available for starting new instances.

I’ve confirmed this in my own AWS accounts and found that one of the three availability zones in us-west-1 is also unavailable in addition to one of the four availability zones in us-east-1.

Here’s the error I get when I try to start an instance in the availability zone using an old AWS account:

Client.Unsupported: The requested Availability Zone is no longer supported. Please retry your request by not specifying an Availability Zone or choosing us-east-1d, us-east-1a, us-east-1b.

When I use an AWS account I created two days ago, I don’t even see the fourth availability zone at all:

$ ec2-describe-availability-zones --region us-east-1
AVAILABILITYZONE    us-east-1b  available   us-east-1   
AVAILABILITYZONE    us-east-1c  available   us-east-1   
AVAILABILITYZONE    us-east-1d  available   us-east-1

The exact name of the unavailable availability zones will vary between EC2 accounts. You can read more about that here:

Matching EC2 Availability Zones Across AWS Accounts

The availability zones that are unavailable in my AWS accounts map to the following identifiers using the method described in the above article:

us-east-1x ceb6a579-757c-474b-b09b-52c84b605767
us-west-1x e5a2ff3b-79b4-4217-8c93-ebf1d633dd6e

If my guess is correct based on old accounts I have, I believe these may be the oldest (original) availability zones in their respective regions.

Has there been any communication from Amazon about unsupported availability zones? Is this temporary or permanent? When I searched Google for the above error, I got back one result in Japanese and it appears to be somebody asking what the error is.

No longer supporting an availability zone in EC2 is something that Amazon is allowed to do under the EC2 SLA, especially with the way that they seem to phase them out. The SLA does not kick in until two availability zones are completely unavailable and “unavailable” includes your existing instances having no external connectivity. This is one reason we try to architect services with the ability to quickly move resources from one availability zone to another.

I’d love to hear if other people are able to start instances in these availability zones. Please also mention if you already have instances running in those zones.

Update 2011-08-06: According to a post in May from Amazon this seems to be a normal part of how AWS grows in an orderly manner, and if you already have instances running in a zone, you should be able to continue running instances in that zone. It isn’t clear to me how quickly you might lose a zone after your last remaining instance is stopped or terminated, but according to one user it sounds like it might be nearly immediate.

Original article: http://alestic.com/2011/08/ec2-unavailabiilty-zones

Unavailable Availability Zones on Amazon EC2

// August 6th, 2011 // Comments Off // Uncategorized

I’m taking a class about using Chef with EC2 by Florian Drescher today and Florian mentioned that he noticed one of the four availability zones in us-east-1 is not currently available for starting new instances.

I’ve confirmed this in my own AWS accounts and found that one of the three availability zones in us-west-1 is also unavailable in addition to one of the four availability zones in us-east-1.

Here’s the error I get when I try to start an instance in the availability zone using an old AWS account:

Client.Unsupported: The requested Availability Zone is no longer supported. Please retry your request by not specifying an Availability Zone or choosing us-east-1d, us-east-1a, us-east-1b.

When I use an AWS account I created two days ago, I don’t even see the fourth availability zone at all:

$ ec2-describe-availability-zones --region us-east-1
AVAILABILITYZONE    us-east-1b  available   us-east-1   
AVAILABILITYZONE    us-east-1c  available   us-east-1   
AVAILABILITYZONE    us-east-1d  available   us-east-1

The exact name of the unavailable availability zones will vary between EC2 accounts. You can read more about that here:

Matching EC2 Availability Zones Across AWS Accounts

The availability zones that are unavailable in my AWS accounts map to the following identifiers using the method described in the above article:

us-east-1x ceb6a579-757c-474b-b09b-52c84b605767
us-west-1x e5a2ff3b-79b4-4217-8c93-ebf1d633dd6e

If my guess is correct based on old accounts I have, I believe these may be the oldest (original) availability zones in their respective regions.

Has there been any communication from Amazon about unsupported availability zones? Is this temporary or permanent? When I searched Google for the above error, I got back one result in Japanese and it appears to be somebody asking what the error is.

No longer supporting an availability zone in EC2 is something that Amazon is allowed to do under the EC2 SLA, especially with the way that they seem to phase them out. The SLA does not kick in until two availability zones are completely unavailable and “unavailable” includes your existing instances having no external connectivity. This is one reason we try to architect services with the ability to quickly move resources from one availability zone to another.

I’d love to hear if other people are able to start instances in these availability zones. Please also mention if you already have instances running in those zones.

Update 2011-08-06: According to a post in May from Amazon this seems to be a normal part of how AWS grows in an orderly manner, and if you already have instances running in a zone, you should be able to continue running instances in that zone. It isn’t clear to me how quickly you might lose a zone after your last remaining instance is stopped or terminated, but according to one user it sounds like it might be nearly immediate.

Original article: http://alestic.com/2011/08/ec2-unavailabiilty-zones

Desktop AMI login security with NX

// August 3rd, 2011 // Comments Off // Uncategorized

Update 2011-08-04: Amazon Security did more research and investigated the desktop AMIs. They have confirmed that their software incorrectly flagged the AMIs (false positive) and they caught it in time to stop the warning emails from going out to users.

These AMIs include the NX software for remote desktop operation and the way that NX implement login authentication with ssh is convoluted, but secure. I can easily understand why it might have looked like there were potential problems with the AMIs, and I’m glad things turned out well.

As always, hats off to the hard working folks at AWS and thank for all the great products and services.

Original message:

If Amazon AWS/EC2 contacts you with a warning that one of my AMIs you are running contains a back door security hole with ssh keys or user passwords, please don’t be alarmed.

They just sent me a notice that all of my old Ubuntu and Debian desktop AMIs from 2008-2010 need to be taken down, but it was a misunderstanding of how the NoMachine NX software implements login security and I’m sure those AMIs just got caught up in their automated sweeps.

I sent an email to help Amazon to help explain why a fixed public ssh key in the AMI with a publicly known “private” ssh key is not a security risk in this instance (see command=”/usr/NX/bin/nxserver —login” in the authorized_keys2 file) but they might be sending out notices to users of these AMIs before they get my response.

You can read up on how NX authentication works securely here:

http://www.nomachine.com/ar/view.php?ar_id=AR02C00150

It’s the middle of the night for me, so I’ll publish more later, but I wanted to get the world out just in case there is alarm.

Note: There may be other reasons you should not run some of those AMIs, e.g., if they are of a release that is past end of life, but they don’t have open back doors that let me access them.

I’ve written about building AMIs securely recently and have been preaching this kind of stuff with EC2 for years.

Original article: http://alestic.com/2011/08/ec2-desktop-ami-security

Desktop AMI login security with NX

// August 3rd, 2011 // Comments Off // Uncategorized

Update 2011-08-04: Amazon Security did more research and investigated the desktop AMIs. They have confirmed that their software incorrectly flagged the AMIs (false positive) and they caught it in time to stop the warning emails from going out to users.

These AMIs include the NX software for remote desktop operation and the way that NX implement login authentication with ssh is convoluted, but secure. I can easily understand why it might have looked like there were potential problems with the AMIs, and I’m glad things turned out well.

As always, hats off to the hard working folks at AWS and thank for all the great products and services.

Original message:

If Amazon AWS/EC2 contacts you with a warning that one of my AMIs you are running contains a back door security hole with ssh keys or user passwords, please don’t be alarmed.

They just sent me a notice that all of my old Ubuntu and Debian desktop AMIs from 2008-2010 need to be taken down, but it was a misunderstanding of how the NoMachine NX software implements login security and I’m sure those AMIs just got caught up in their automated sweeps.

I sent an email to help Amazon to help explain why a fixed public ssh key in the AMI with a publicly known “private” ssh key is not a security risk in this instance (see command=”/usr/NX/bin/nxserver —login” in the authorized_keys2 file) but they might be sending out notices to users of these AMIs before they get my response.

You can read up on how NX authentication works securely here:

http://www.nomachine.com/ar/view.php?ar_id=AR02C00150

It’s the middle of the night for me, so I’ll publish more later, but I wanted to get the world out just in case there is alarm.

Note: There may be other reasons you should not run some of those AMIs, e.g., if they are of a release that is past end of life, but they don’t have open back doors that let me access them.

I’ve written about building AMIs securely recently and have been preaching this kind of stuff with EC2 for years.

Original article: http://alestic.com/2011/08/ec2-desktop-ami-security

TurnKey 11.2, free micro instances, EBS backed cloud servers

// August 2nd, 2011 // Comments Off // Uncategorized

TurnKey 11.2: micro instances, EBS support, built-in TurnKey DNS, security updates

We just updated the web site and the TurnKey Hub with the new TurnKey 11.2 maintenance release, which includes:

  1. TurnKey Hub support for micro instances, Amazon's free tier and cloud servers backed by persistent network-attached storage volumes (AKA EBS backed instances).
  2. Built-in support for TurnKey's new dynamic DNS service.
  3. The latest security updates.

TurnKey Micro instances: 2 cents/hour or 0 cents/hour for a year with the free tier

We've added support for micro instances (613 MB RAM), Amazon EC2's smallest cloud server type which costs just 2 cents an hour to run, which is less than $15/month if you run a server 24x7. If that isn't close enough to free for you, Amazon is giving away a year's worth of micro instance usage to new users as part of their free tier program.

This means many of you will now be able to try out TurnKey in the cloud free for a year. Yay!

We would have added support for micro instances as soon as they came out except Amazon designed them to work differently from other instance types we already supported. In particular, we had to add support for EBS backed instances...

EBS backed instances: cloud servers that can be turned off any time

Up until now the TurnKey Hub only supported S3 backed instances. These are non-persistent cloud servers with temporary storage that is lost once you destroy the server. This means you can't just turn off an S3 backed instance to save usage fees when you are not using it, though you could work around this limitation by using TKLBAM to backup a cloud server before destroying it and later restoring its state into a new cloud server.

With the support we've added for EBS backed instances, this limitation has been removed. EBS is what Amazon calls its on-demand Network Attached Storage service. The catch is that the Hub has to pre-allocate a fixed size EBS volume for your cloud server to boot from. Unless you are in the free usage tier you'll have to pay an additional $0.10/GB per month in EBS storage fees for the convenience (e.g., 50GB EBS volume == extra $5/monthly). The ability to turn off servers when not in use may make up for this extra cost though.

Speaking of costs, the pricing structure on the TurnKey side is a bit different for EBS backed instances as Amazon doesn't allow vendors to add a 10% markup to hourly usage fees like we've been doing with S3 backed instances. So instead, we're probably going to be experimenting with a global fixed monthly fee for this feature. After the trial period ends (in a month or so). Currently there is no extra charge.

Note that this future extra monthly charge will not apply to micro instances.

A word of warning: EBS is not a backup replacement and EBS-backed instances still need to be backed up by TKLBAM. EBS volumes just provides data persistence. It's a network hard drive that lives in a specific Amazon datacenter. It is not a replacement for backups. For example, if the data on your EBS volume gets accidentally deleted or corrupted, without a backup system to restore from you will be out of luck. TKLBAM on the other hand provides true incremental backups, so good data can't be accidentally overwritten by bad. Also, TKLBAM uses S3, which is designed by default to provide 11 nines (99.999999999%) of storage reliability, much higher than EBS.

Security updates

As most of you know security updates are already installed automatically on first boot and nightly  (by default). If you're using an older version of TurnKey this means you don't need to do anything to get the latest security fixes. But for new deployments pulling a large number of security updates over the network can take considerable time, so occasional maintenance releases that already include them are a good idea.

We're in the process of upgrading our development process so this sort of update will be easier to do in the future and can be done as frequently as necessary.

TurnKey Domain management & Dynamic DNS

// May 27th, 2011 // Comments Off // Uncategorized

A while ago I was chatting with Liraz and said "wouldn't it be great if when launching a cloud server the Hub would perform some magic and assign the server a friendly name? I'm tired of remembering IP addresses, and logging into our DNS management console to setup records."

Then we thought, "lets make DNS easy, lets make it TurnKey". So we did...

No matter your use case, we got you covered:

Custom domains

Alice uses the Hub to launch and manage her cloud servers. Every time she sets up a new server, she needs to navigate to her DNS management console, log in, go back to the Hub to get the servers IP address, switch back to the DNS console, and setup the appropriate records to point to her server (e.g., www.example.com -> 89.231.194.85).

Not so bad, thats how everyone does it, right? Not anymore!

We have just released new DNS features in the TurnKey Hub. Not only can you now manage your DNS settings using a crisp user interface right in the Hub, backed by the awesomeness of Amazon's Route53 highly available and scalable Domain Name System, but because the Hub also manages your cloud servers, the two systems are tightly integrated.

Hub Domain Management

For example, when launching a cloud server, you specify the hostname to associate with your new server, and as soon as your server is running, the DNS records will be automatically created/updated accordingly. You can also associate a hostname with a running server right from the server listing.

Associate domain

But wait, there's more! Don't want to use Elastic/Static IP's with your cloud servers? Do you manage a server behind a dynamic IP address? We got you covered - see Dynamic DNS below.

Are you using the Hub API to programmatically launch your servers? Do you use launch another server like this one, or launch this backup in the cloud? We got your covered there as well...

TKLAPP.com

Bob, unlike Alice, doesn't own a domain name, so why should he be a second class citizen and not get all these cool new features? He too is tired of remembering IP addresses and sharing them with his friends. He wants an easy to remember name as well.

Enter TKLAPP.com! TKLAPP.com hostnames and available to all Hub users, and they're free! Because there is a limited name space, they are available on first-come-first-serve basis, so go grab your own vanity name (or names) before someone else does.

Launch associate domain

DNS names aren't just user friendly, they are sometimes required. For example, appliances which use domain preseeding (such as WordpressMagento, StatusNet, ejabberd) will now be fully configured and ready to rock right off the bat.

Dynamic DNS

And we didn't forget about Charlie either, who might be running TurnKey on his own hardware, in a VM or at a hosting provider that supports TurnKey. And given the state of free Dynamic DNS services out there, we created HubDNS.

HubDNS is the TurnKey Dynamic DNS client. It supports both custom domains as well as the free TKLAPP.com domain. It's also super simple to set up:

apt-get update
apt-get install hubdns

hubdns-init HUB_APIKEY foo.tklapp.com
hubdns-update
chmod +x /etc/cron.hourly/hubdns-update  # automatic hourly updates

BTW, HubDNS should work without issues on any Debian/Ubuntu based system. Full installation and usage documentation is available here.

Thoughts, comments, feature requests?

TurnKey Domain management & Dynamic DNS

// May 27th, 2011 // Comments Off // Uncategorized

A while ago I was chatting with Liraz and said "wouldn't it be great if when launching a cloud server the Hub would perform some magic and assign the server a friendly name? I'm tired of remembering IP addresses, and logging into our DNS management console to setup records."

Then we thought, "lets make DNS easy, lets make it TurnKey". So we did...

No matter your use case, we got you covered:

Custom domains

Alice uses the Hub to launch and manage her cloud servers. Every time she sets up a new server, she needs to navigate to her DNS management console, log in, go back to the Hub to get the servers IP address, switch back to the DNS console, and setup the appropriate records to point to her server (e.g., www.example.com -> 89.231.194.85).

Not so bad, thats how everyone does it, right? Not anymore!

We have just released new DNS features in the TurnKey Hub. Not only can you now manage your DNS settings using a crisp user interface right in the Hub, backed by the awesomeness of Amazon's Route53 highly available and scalable Domain Name System, but because the Hub also manages your cloud servers, the two systems are tightly integrated.

Hub Domain Management

For example, when launching a cloud server, you specify the hostname to associate with your new server, and as soon as your server is running, the DNS records will be automatically created/updated accordingly. You can also associate a hostname with a running server right from the server listing.

Associate domain

But wait, there's more! Don't want to use Elastic/Static IP's with your cloud servers? Do you manage a server behind a dynamic IP address? We got you covered - see Dynamic DNS below.

Are you using the Hub API to programmatically launch your servers? Do you use launch another server like this one, or launch this backup in the cloud? We got your covered there as well...

TKLAPP.com

Bob, unlike Alice, doesn't own a domain name, so why should he be a second class citizen and not get all these cool new features? He too is tired of remembering IP addresses and sharing them with his friends. He wants an easy to remember name as well.

Enter TKLAPP.com! TKLAPP.com hostnames and available to all Hub users, and they're free! Because there is a limited name space, they are available on first-come-first-serve basis, so go grab your own vanity name (or names) before someone else does.

Launch associate domain

DNS names aren't just user friendly, they are sometimes required. For example, appliances which use domain preseeding (such as WordpressMagento, StatusNet, ejabberd) will now be fully configured and ready to rock right off the bat.

Dynamic DNS

And we didn't forget about Charlie either, who might be running TurnKey on his own hardware, in a VM or at a hosting provider that supports TurnKey. And given the state of free Dynamic DNS services out there, we created HubDNS.

HubDNS is the TurnKey Dynamic DNS client. It supports both custom domains as well as the free TKLAPP.com domain. It's also super simple to set up:

apt-get update
apt-get install hubdns

hubdns-init HUB_APIKEY foo.tklapp.com
hubdns-update
chmod +x /etc/cron.hourly/hubdns-update  # automatic hourly updates

BTW, HubDNS should work without issues on any Debian/Ubuntu based system. Full installation and usage documentation is available here.

Thoughts, comments, feature requests?

Announcing public API for TurnKey Hub

// April 26th, 2011 // Comments Off // Uncategorized

More power, control, flexibility and automation of cloud servers.

Alan Kay once said: "Simple things should be simple, complex things should be possible". We live by those words, and I think we've done a pretty good job up until now.

The Hub makes launching and managing instances on Amazon EC2 really simple, but the one thing that has been missing is a solution to make complex things possible - i.e., programmatic control.

Which brings me to todays announcement of the TurnKey Hub API, and HubTools - Python API bindings and CLI tools.

Some examples to wet your appetite:

Launch a new TurnKey Core appliance in the cloud:

$ hub-launch core

And of course, preseeding is supported, for example:

$ hub-launch lamp --db-pass=foobar

But wait, there's more. Lets say you are developing a new Wordpress website in a local VM which is backed up using TKLBAM, with a backup ID of 2. Restoring the backup to a new cloud server is as simple as:

$ hub-launch 2

So how do you know what backups you have available? Which appliances are available and their preseeding options? The status and related information of your cloud servers?

It's easy:

hub-list-backups
hub-list-appliances
hub-list-servers

If the included CLI tools aren't enough and you need more power, use the Python bindings to develop your own code. It's really simple.

For example, lets say I wanted to launch 10 TKL Core servers:

for i in range(1, 11):
    hub.servers.launch("core", label="TurnKey Core %s" % i)

There is so much you can do with HubTools, it's only limited to your imagination.

The full documentation is available here. If you don't have a free Hub account yet, get one here.

EC2 Reserved Instance Offering IDs Change Over Time

// April 26th, 2011 // Comments Off // Uncategorized

This article is a followup to Matching EC2 Availability Zones Across AWS Accounts written back in 2009. Please read that article first in order to understand any of what I write here.

Since I wrote that article, Amazon has apparently changed the reserved instance offering ids at least once. I haven’t been tracking this, so I don’t know if this was a one time thing or if the offering ids change on a regular basis.

Interestingly, the offering ids still seem to match across accounts and still map to different availability zones, so perhaps they can still be used to map the true, underlying availability zones as the original article proposes.

To document for posterity and comparison, here are my 2009 availability zone offering ids as calculated by the procedure in the above mentioned article:

“Blue” Account (2009):

eu-west-1a 649fd0c8-75d4-4e16-88c7-1ddb83f66062
eu-west-1b 438012d3-0440-480a-9f5c-eb7e55dd5a37
us-east-1a 4b2293b4-1e6c-4eb3-ab74-4493c0e57987
us-east-1b 60dcfab3-a56c-4092-8c90-3677e9da02b7
us-east-1c c48ab04c-c057-457e-a4d8-a0f172f4db2d
us-east-1d c48ab04c-7e96-4ea8-9579-d62194490546

“Red” Account (2009):

eu-west-1a 438012d3-0440-480a-9f5c-eb7e55dd5a37
eu-west-1b 649fd0c8-75d4-4e16-88c7-1ddb83f66062
us-east-1a c48ab04c-c057-457e-a4d8-a0f172f4db2d
us-east-1b 4b2293b4-1e6c-4eb3-ab74-4493c0e57987
us-east-1c 60dcfab3-a56c-4092-8c90-3677e9da02b7
us-east-1d c48ab04c-7e96-4ea8-9579-d62194490546

And here are the new, 2011-05-25 availability zone offering ids.

“Blue” Account (2011):

eu-west-1a c48ab04c-0bd0-4be9-8db5-a4bad61c6c58
eu-west-1b d586503b-7025-44e8-8487-09907b6b0e7e
us-east-1a 438012d3-80c7-42c6-9396-a209c58607f9
us-east-1b 60dcfab3-06bb-4b68-9503-53bf89823b5e
us-east-1c ceb6a579-757c-474b-b09b-52c84b605767
us-east-1d 649fd0c8-5d76-4881-a522-fe5224c10fcc

“Red” Account (2011):

eu-west-1a d586503b-7025-44e8-8487-09907b6b0e7e
eu-west-1b c48ab04c-0bd0-4be9-8db5-a4bad61c6c58
us-east-1a ceb6a579-757c-474b-b09b-52c84b605767
us-east-1b 438012d3-80c7-42c6-9396-a209c58607f9
us-east-1c 60dcfab3-06bb-4b68-9503-53bf89823b5e
us-east-1d 649fd0c8-5d76-4881-a522-fe5224c10fcc

Note: I have excluded the regions and availability zones that Amazon has added since 2009 (quite a few).

Though all of the offering ids have changed since 2009, it appears that the availability zone mappings between these two accounts has remained the same across the years. This is a small sampling and I still think that Amazon could change this, especially for zones where no resources are being used (instances running, EBS volumes created), so don’t depend on it remaining so.

Please let me know if you do any further experimentation or future tracking of these ids. There’s not a ton of practical application, but I still find this interesting.

Original article: http://alestic.com/2011/04/ec2-offering-ids