Archive for August, 2011

TurnKey Hub: Not just adding random features

// August 19th, 2011 // Comments Off // Uncategorized

I started writing a review for cloudtask as a comment on the announcement, but decided it would be better to address a topic that was raised by Jeremiah when we launched the Hub API:

I'm impressed with the way all the pieces of TurnKey just fit together like puzzle pieces. Even as new functionality is added it never seems like something gets patched onto the existing framework.  Instead, it just integrates smoothly with what's already there.

Liraz's reply:

While the new Hub features are coming out incrementally, they really are part of a bigger unifying vision. I don't like to hype up vaporware, but we have a lot of great ideas we're working on and we like to think we're just getting started. There's a big gap between what the Hub does right now and what we envision it doing in the future.

Also note that we're using the Hub internally to help develop and test TurnKey, so we're probably using it more than the typical user and run into its limits sooner. We're scratching our own itch, and the community benefits from that.

To recap, some of the features Liraz was referring to in his comment, and some that came after his comment are:

More features are coming, but this a great milestone to stop, reflect, and talk a little about a pain point we've had, and how the features we've released to this point are helping us solve it.

Building and maintaining exponentially growing library

TurnKey currently has 45 appliances, available in ISO, VMDK, OVF, EC2 S3 and EBS backed, Xen and Eucalyptus.

To sum up, thats 585 images we need to build and maintain. 

The (long overdue) TKL 11 part 2 will double the appliance library. Additionally we are working on supporting more build targets, as well as the new Tokyo EC2 region. Throw 64bit support into the equasion and the number of images to build and maintain will grow to about 3,300!

Automation and a glimpse into a pain point

We try to automate everything we do, but even executing and managing the stuff thats already automated needs automation when you reach scale.

To give you a little glimps into what I mean. When building the Amazon EC2 S3 backed images (in the past), I couldn't do it locally as my upstream kinda sucks are would take me about 2 weeks to build and upload.

So do it in the cloud, right? Correct. But building and uploading from an EC2 instance still takes a couple of days, mainly due to subsequent builds and uploading images built for different regions is quiet slow.

OK, so lets just spawn more servers in the cloud, in the different regions. Yep, this allows us to perform the relative region builds so upload is much faster, and they run in parallel.

But manually launching each instance from the Hub's web UI (pre- hub-api), updating my /etc/hosts file to keep track of the different instances (pre- hub-domains), logging into each instance and transfering the build infrastructure (pre- restore-on-launch), and finally starting the automated build process while making sure nothing breaks is painful to say the least.

It would also be much faster (and cost the same) if we could split the appliance builds amoungst more instances in each region, but try managing that manually and keep your hair - argh!

To throw some salt on an already bloody wound, remember we also have to build the other target formats (EBS, VMDK, OVF, etc...). Try do that without going bald.

Parallel batch execution with auto-launched cloud servers

This is where integrating the features mentioned above, and cloudtask comes into play.

Instead of pulling my hair out and waiting days, yesterday I threw a couple of tasks at cloudtask, which launched about 50 instances spanning the globe, our build infrastructure was automatically setup on each instance, and cloudtask started dishing out jobs to each of them.

Within about 1 hour, I received reports via email that all the builds completed successfully and were published. Pain point gone.

Now thats automation on steriods! And best of all, I get to keep my hair!
 

As mentioned above, we are just getting started. Lots more to come - it's getting exciting...

Caught in the Spam Trap

// August 19th, 2011 // Comments Off // Uncategorized

Spam is a problem.  We’ve all experienced it, our inboxes clogged with it, training spam filters to try and keep up with the ever more creative ways of misspelling various pharmaceuticals.  We’re always on the lookout for a better way to block it.  But have you ever been on the other end of that fight?

Well now I have.  No, I wasn’t sending spam.  Nor was I even the victim of a hacking attack that sent out spam.  No sir, my crime was much more mundane: debugging emails.  While trying out ReadFeeder’s new update algorithm, I had cron sending me regular reports at the end of each run.  Every 10 minutes.  But that’s 6 emails an hour, that’s not a problem is it?

Well it turns out it is.  Not because of the number of emails though, but rather because of the source of those emails.  See, spammers can’t exactly set up shop at a permanent location, otherwise they’ll get blocked in short order, so they tend to look for more dynamic solutions, and something like Amazon’s EC2 or other automated hosting services fit that bill.  In order to prevent them from using these services to dump ads into our collective tubes, spam filters have been blocking emails coming from dynamically allocated host IP addresses.

But ReadFeeder.net needs to send emails, not just cron output to me, but registration information new beta testers.  It just wouldn’t do to have those emails falling into spam traps and never reaching their intended users.  The solution, as I found out, is to use a “smart host”, another email server that isn’t on a dynamic IP address.  Most of us do this all the time, we send email through our ISP’s email servers rather than directly to our recipient’s email server.  Since I registered ReadFeeder.net with Google Apps, I was able to tell Postfix to send outbound emails through Google’s mail servers first, which required that I authenticated over an encrypted connection.

There are a couple dozen results for a search of “use google apps as smarthost for postfix”, and all of them have slightly different configurations, none of which worked exactly for me.  I’d post mine, but I’m pretty sure it wouldn’t work exactly for you either, so I’d just be adding to the noise.  But if you’re trying to get a similar setup and can’t quite make it work, send me an email or reply to this post.

Oh, and if you signed up for the Beta program and haven’t gotten an email letting you know how to get started using it, please let me know, because it’s likely that email got caught in a spam filter somewhere, and I will have to resend it to you.

HPCC with Ubuntu Server and Ensemble

// August 19th, 2011 // Comments Off // Uncategorized

** This is an updated post reflecting the new name of the project formerly known as Ensemble now known as Juju **

Let's start this post with a bit of background on the technologies that I'll be using:
  • What is Ubuntu?
    • Ubuntu is a fast, secure and easy-to-use operating system used by millions of people around the world.  
    • Secure, fast and powerful, Ubuntu Server is transforming IT environments worldwide. Realise the full potential of your infrastructure with a reliable, easy-to-integrate technology platform.
  • What is Juju?
    • Juju is a next generation service orchestration framework. It has been likened to APT for the cloud. With juju, different authors are able to create service charms independently, and make those services coordinate their communication through a simple protocol. Users can then take the product of different authors and very comfortably deploy those services in an environment. The result is multiple machines and components transparently collaborating towards providing the requested service.
  • What is HPCC?
    • HPCC (High Performance Computing Cluster) is a massive parallel-processing computing platform that solves Big Data problems. The platform is now Open Source!
 Now that we are all caught up, let's delve right into it.  I will be discussing the details of my newly created hpcc juju charm.

The hpcc charm has been one of the trickiest one to date to get working properly so, I want to take some time to explain some of the challenges that I encountered.

hpcc seems to use ssh keys for authentication and a single xml file to hold it's configuration.  All nodes that are part of the cluster should have identical keys and xml configuration file.

  • The ssh keys are pretty easy to do ( there is even a script that will do it all for you located at /opt/HPCCSystems/sbin/keygen.sh ).  You can just run: ssh-keygen -f path_where_to_save_keys/id_rsa -N "" -q
  • The configuration file environment.xml is a lot trickier to configure so, I will use cheetah templates to help make a template out of this enormous file.  
    • According to their website:
      • Cheetah is an open source template engine and code generation tool, written in Python. It can be used standalone or combined with other tools and frameworks. Web development is its principle use, but Cheetah is very flexible and is also being used to generate C++ game code, Java, sql, form emails and even Python code.
With cheetah, I can create self contained templates that can be generated into their intended file by just calling cheetah.  This is because we can embed python code inside the template itself, making the template ( environment.tmpl in our case ) more or less a python program that generates a fully functional environment.xml file ready for hpcc to use.

Another, very important, reason to use a template engine is the ability to create identical configuration files from each node without having to pass them around.  In other words, each node can create it's own configuration file and, since all nodes are using the same methods and data to create the file, they will all be exactly the same.

The hpcc configuration file is huge so, I'll just talk about some of the interesting bits of it here:
#import random
#import subprocess
#set $rel_structure = { $subprocess.check_output(['facter', 'install_time']).strip() : { 'name' : $subprocess.check_output(['hostname', '-f']).strip(), 'netAddress' : $subprocess.check_output(['facter','ipaddress']).strip(), 'uuid' : $subprocess.check_output(['facter','uuid']).strip()  } }
#for $member in $subprocess.check_output(['relation-list']).strip().split():
   #set $rel_structure[$subprocess.check_output(['relation-get','install_time', $member]).strip()] = { 'name' : $subprocess.check_output(['relation-get','name', $member]).strip(), 'netAddress' : $subprocess.check_output(['relation-get','netAddress', $member]).strip(), 'uuid' : $subprocess.check_output(['relation-get', 'uuid', $member]).strip() }
#end for
#set $nodes = []
#for $index in $sorted($rel_structure.keys()):
   $nodes.append(($rel_structure[$index]['netAddress'], $rel_structure[$index]['name']))
#end for
The above piece of code is what I am currently using to populate a list with the FQDN and address of each cluster member sorted by install time.  This puts the "master" of the cluster at the top of the list which will become useful when populating certain parts of the configuration file.


As we can see by the code above, the main piece of information that we use in this template is the node list.  Here is a sample of how we use it in the environment.tmpl template file:
 #for $netAddress, $name in $nodes:
  <Computer computerType="linuxmachine"
            domain="localdomain"
            name="$name"
            netAddress="$netAddress"/>
#end for
I encourage you to download the charm here and examine the environment.tmpl file in the templates directory.

Here is the complete environment.tmpl file... I know it's pretty small and, you can just download the charm and read the file at your leasure but, I wanted to give you an idea of the size and complexity of hpcc's configuration file.

====

#import random
#import subprocess
#set $rel_structure = { $subprocess.check_output(['facter', 'install_time']).strip() : { 'name' : $subprocess.check_output(['hostname', '-f']).strip(), 'netAddress' : $subprocess.check_output(['facter','ipaddress']).strip(), 'uuid' : $subprocess.check_output(['facter','uuid']).strip()  } }
#for $member in $subprocess.check_output(['relation-list']).strip().split():
   #set $rel_structure[$subprocess.check_output(['relation-get','install_time', $member]).strip()] = { 'name' : $subprocess.check_output(['relation-get','name', $member]).strip(), 'netAddress' : $subprocess.check_output(['relation-get','netAddress', $member]).strip(), 'uuid' : $subprocess.check_output(['relation-get', 'uuid', $member]).strip() }
#end for
#set $nodes = []
#for $index in $sorted($rel_structure.keys()):
   $nodes.append(($rel_structure[$index]['netAddress'], $rel_structure[$index]['name']))
#end for
<?xml version="1.0" encoding="UTF-8"?>
<!-- Edited with ConfigMgr on ip 71.204.190.179 on 2011-08-16T00:39:16 -->
<Environment>
 <EnvSettings>
  <blockname>HPCCSystems</blockname>
  <configs>/etc/HPCCSystems</configs>
  <environment>environment.xml</environment>
  <group>hpcc</group>
  <home>/home</home>
  <interface>eth0</interface>
  <lock>/var/lock/HPCCSystems</lock>
  <log>/var/log/HPCCSystems</log>
  <path>/opt/HPCCSystems</path>
  <pid>/var/run/HPCCSystems</pid>
  <runtime>/var/lib/HPCCSystems</runtime>
  <sourcedir>/etc/HPCCSystems/source</sourcedir>
  <user>hpcc</user>
 </EnvSettings>
 <Hardware>
 #for $netAddress, $name in $nodes:
  <Computer computerType="linuxmachine"
            domain="localdomain"
            name="$name"
            netAddress="$netAddress"/>
#end for
  <ComputerType computerType="linuxmachine"
                manufacturer="unknown"
                name="linuxmachine"
                opSys="linux"/>
  <Domain name="localdomain" password="" username=""/>
  <Switch name="Switch"/>
 </Hardware>
 <Programs>
  <Build name="community_3.0.4" url="/opt/HPCCSystems">
   <BuildSet installSet="deploy_map.xml"
             name="dafilesrv"
             path="componentfiles/dafilesrv"
             processName="DafilesrvProcess"
             schema="dafilesrv.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="dali"
             path="componentfiles/dali"
             processName="DaliServerProcess"
             schema="dali.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="dfuplus"
             path="componentfiles/dfuplus"
             processName="DfuplusProcess"
             schema="dfuplus.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="dfuserver"
             path="componentfiles/dfuserver"
             processName="DfuServerProcess"
             schema="dfuserver.xsd"/>
   <BuildSet deployable="no"
             installSet="deploy_map.xml"
             name="DropZone"
             path="componentfiles/DropZone"
             processName="DropZone"
             schema="dropzone.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="eclagent"
             path="componentfiles/eclagent"
             processName="EclAgentProcess"
             schema="eclagent_config.xsd"/>
   <BuildSet installSet="deploy_map.xml" name="eclminus" path="componentfiles/eclminus"/>
   <BuildSet installSet="deploy_map.xml"
             name="eclplus"
             path="componentfiles/eclplus"
             processName="EclPlusProcess"
             schema="eclplus.xsd"/>
   <BuildSet installSet="eclccserver_deploy_map.xml"
             name="eclccserver"
             path="componentfiles/configxml"
             processName="EclCCServerProcess"
             schema="eclccserver.xsd"/>
   <BuildSet installSet="eclscheduler_deploy_map.xml"
             name="eclscheduler"
             path="componentfiles/configxml"
             processName="EclSchedulerProcess"
             schema="eclscheduler.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="esp"
             path="componentfiles/esp"
             processName="EspProcess"
             schema="esp.xsd"/>
   <BuildSet deployable="no"
             installSet="deploy_map.xml"
             name="espsmc"
             path="componentfiles/espsmc"
             processName="EspService"
             schema="espsmcservice.xsd">
    <Properties defaultPort="8010"
                defaultResourcesBasedn="ou=SMC,ou=EspServices,ou=ecl"
                defaultSecurePort="18010"
                type="WsSMC">
     <Authenticate access="Read"
                   description="Root access to SMC service"
                   path="/"
                   required="Read"
                   resource="SmcAccess"/>
     <AuthenticateFeature description="Access to SMC service"
                          path="SmcAccess"
                          resource="SmcAccess"
                          service="ws_smc"/>
     <AuthenticateFeature description="Access to thor queues"
                          path="ThorQueueAccess"
                          resource="ThorQueueAccess"
                          service="ws_smc"/>
     <AuthenticateFeature description="Access to super computer environment"
                          path="ConfigAccess"
                          resource="ConfigAccess"
                          service="ws_config"/>
     <AuthenticateFeature description="Access to DFU"
                          path="DfuAccess"
                          resource="DfuAccess"
                          service="ws_dfu"/>
     <AuthenticateFeature description="Access to DFU XRef"
                          path="DfuXrefAccess"
                          resource="DfuXrefAccess"
                          service="ws_dfuxref"/>
     <AuthenticateFeature description="Access to machine information"
                          path="MachineInfoAccess"
                          resource="MachineInfoAccess"
                          service="ws_machine"/>
     <AuthenticateFeature description="Access to SNMP metrics information"
                          path="MetricsAccess"
                          resource="MetricsAccess"
                          service="ws_machine"/>
     <AuthenticateFeature description="Access to remote execution"
                          path="ExecuteAccess"
                          resource="ExecuteAccess"
                          service="ws_machine"/>
     <AuthenticateFeature description="Access to DFU workunits"
                          path="DfuWorkunitsAccess"
                          resource="DfuWorkunitsAccess"
                          service="ws_fs"/>
     <AuthenticateFeature description="Access to DFU exceptions"
                          path="DfuExceptionsAccess"
                          resource="DfuExceptions"
                          service="ws_fs"/>
     <AuthenticateFeature description="Access to spraying files"
                          path="FileSprayAccess"
                          resource="FileSprayAccess"
                          service="ws_fs"/>
     <AuthenticateFeature description="Access to despraying of files"
                          path="FileDesprayAccess"
                          resource="FileDesprayAccess"
                          service="ws_fs"/>
     <AuthenticateFeature description="Access to dkcing of key files"
                          path="FileDkcAccess"
                          resource="FileDkcAccess"
                          service="ws_fs"/>
     <AuthenticateFeature description="Access to files in dropzone"
                          path="FileIOAccess"
                          resource="FileIOAccess"
                          service="ws_fileio"/>
     <AuthenticateFeature description="Access to WS ECL service"
                          path="WsEclAccess"
                          resource="WsEclAccess"
                          service="ws_ecl"/>
     <AuthenticateFeature description="Access to Roxie queries and files"
                          path="RoxieQueryAccess"
                          resource="RoxieQueryAccess"
                          service="ws_roxiequery"/>
     <AuthenticateFeature description="Access to cluster topology"
                          path="ClusterTopologyAccess"
                          resource="ClusterTopologyAccess"
                          service="ws_topology"/>
     <AuthenticateFeature description="Access to own workunits"
                          path="OwnWorkunitsAccess"
                          resource="OwnWorkunitsAccess"
                          service="ws_workunits"/>
     <AuthenticateFeature description="Access to others&apos; workunits"
                          path="OthersWorkunitsAccess"
                          resource="OthersWorkunitsAccess"
                          service="ws_workunits"/>
     <AuthenticateFeature description="Access to ECL direct service"
                          path="EclDirectAccess"
                          resource="EclDirectAccess"
                          service="ecldirect"/>
     <ProcessFilters>
      <Platform name="Windows">
       <ProcessFilter name="any">
        <Process name="dafilesrv"/>
       </ProcessFilter>
       <ProcessFilter name="AttrServerProcess">
        <Process name="attrserver"/>
       </ProcessFilter>
       <ProcessFilter name="DaliProcess">
        <Process name="daserver"/>
       </ProcessFilter>
       <ProcessFilter multipleInstances="true" name="DfuServerProcess">
        <Process name="dfuserver"/>
       </ProcessFilter>
       <ProcessFilter multipleInstances="true" name="EclCCServerProcess">
        <Process name="eclccserver"/>
       </ProcessFilter>
       <ProcessFilter multipleInstances="true" name="EspProcess">
        <Process name="esp"/>
        <Process name="dafilesrv" remove="true"/>
       </ProcessFilter>
       <ProcessFilter name="FTSlaveProcess">
        <Process name="ftslave"/>
       </ProcessFilter>
       <ProcessFilter name="RoxieServerProcess">
        <Process name="ccd"/>
       </ProcessFilter>
       <ProcessFilter name="RoxieSlaveProcess">
        <Process name="ccd"/>
       </ProcessFilter>
       <ProcessFilter name="SchedulerProcess">
        <Process name="scheduler"/>
       </ProcessFilter>
       <ProcessFilter name="ThorMasterProcess">
        <Process name="thormaster"/>
       </ProcessFilter>
       <ProcessFilter name="ThorSlaveProcess">
        <Process name="thorslave"/>
       </ProcessFilter>
       <ProcessFilter name="SashaServerProcess">
        <Process name="saserver"/>
       </ProcessFilter>
      </Platform>
      <Platform name="Linux">
       <ProcessFilter name="any">
        <Process name="dafilesrv"/>
       </ProcessFilter>
       <ProcessFilter name="AttrServerProcess">
        <Process name="attrserver"/>
       </ProcessFilter>
       <ProcessFilter name="DaliProcess">
        <Process name="daserver"/>
       </ProcessFilter>
       <ProcessFilter multipleInstances="true" name="DfuServerProcess">
        <Process name="."/>
       </ProcessFilter>
       <ProcessFilter multipleInstances="true" name="EclCCServerProcess">
        <Process name="."/>
       </ProcessFilter>
       <ProcessFilter multipleInstances="true" name="EspProcess">
        <Process name="."/>
        <Process name="dafilesrv" remove="true"/>
       </ProcessFilter>
       <ProcessFilter name="FTSlaveProcess">
        <Process name="ftslave"/>
       </ProcessFilter>
       <ProcessFilter name="GenesisServerProcess">
        <Process name="httpd"/>
        <Process name="atftpd"/>
        <Process name="dhcpd"/>
       </ProcessFilter>
       <ProcessFilter name="RoxieServerProcess">
        <Process name="ccd"/>
       </ProcessFilter>
       <ProcessFilter name="RoxieSlaveProcess">
        <Process name="ccd"/>
       </ProcessFilter>
       <ProcessFilter name="SchedulerProcess">
        <Process name="scheduler"/>
       </ProcessFilter>
       <ProcessFilter name="ThorMasterProcess">
        <Process name="thormaster"/>
       </ProcessFilter>
       <ProcessFilter name="ThorSlaveProcess">
        <Process name="thorslave"/>
       </ProcessFilter>
       <ProcessFilter name="SashaServerProcess">
        <Process name="saserver"/>
       </ProcessFilter>
      </Platform>
     </ProcessFilters>
    </Properties>
   </BuildSet>
   <BuildSet installSet="deploy_map.xml"
             name="ftslave"
             path="componentfiles/ftslave"
             processName="FTSlaveProcess"
             schema="ftslave_linux.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="hqltest"
             path="componentfiles/hqltest"
             processName="HqlTestProcess"/>
   <BuildSet deployable="no"
             installSet="deploy_map.xml"
             name="ldapServer"
             path="componentfiles/ldapServer"
             processName="LDAPServerProcess"
             schema="ldapserver.xsd"/>
   <BuildSet deployable="no"
             installSet="auditlib_deploy_map.xml"
             name="plugins_auditlib"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet deployable="no"
             installSet="debugservices_deploy_map.xml"
             name="plugins_debugservices"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet deployable="no"
             installSet="fileservices_deploy_map.xml"
             name="plugins_fileservices"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet deployable="no"
             installSet="logging_deploy_map.xml"
             name="plugins_logging"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet deployable="no"
             installSet="parselib_deploy_map.xml"
             name="plugins_parselib"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet deployable="no"
             installSet="stringlib_deploy_map.xml"
             name="plugins_stringlib"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet deployable="no"
             installSet="unicodelib_deploy_map.xml"
             name="plugins_unicodelib"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet deployable="no"
             installSet="workunitservices_deploy_map.xml"
             name="plugins_workunitservices"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet installSet="roxie_deploy_map.xml"
             name="roxie"
             path="componentfiles/configxml"
             processName="RoxieCluster"
             schema="roxie.xsd"/>
   <BuildSet installSet="deploy_map.xml" name="roxieconfig" path="componentfiles/roxieconfig"/>
   <BuildSet installSet="deploy_map.xml"
             name="sasha"
             path="componentfiles/sasha"
             processName="SashaServerProcess"
             schema="sasha.xsd"/>
   <BuildSet deployable="no"
             installSet="deploy_map.xml"
             name="SiteCertificate"
             path="componentfiles/SiteCertificate"
             processName="SiteCertificate"
             schema="SiteCertificate.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="soapplus"
             path="componentfiles/soapplus"
             processName="SoapPlusProcess"
             schema="soapplus.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="thor"
             path="componentfiles/thor"
             processName="ThorCluster"
             schema="thor.xsd"/>
   <BuildSet deployable="no"
             installSet="deploy_map.xml"
             name="topology"
             path="componentfiles/topology"
             processName="Topology"
             schema="topology.xsd"/>
   <BuildSet deployable="no"
             installSet="deploy_map.xml"
             name="ws_ecl"
             path="componentfiles/ws_ecl"
             processName="EspService"
             schema="esp_service_wsecl2.xsd">
    <Properties bindingType="ws_eclSoapBinding"
                defaultPort="8002"
                defaultResourcesBasedn="ou=WsEcl,ou=EspServices,ou=ecl"
                defaultSecurePort="18002"
                plugin="ws_ecl"
                type="ws_ecl">
     <Authenticate access="Read"
                   description="Root access to WS ECL service"
                   path="/"
                   required="Read"
                   resource="WsEclAccess"/>
     <AuthenticateFeature description="Access to WS ECL service"
                          path="WsEclAccess"
                          resource="WsEclAccess"
                          service="ws_ecl"/>
    </Properties>
   </BuildSet>
   <BuildSet deployable="no"
             installSet="deploy_map.xml"
             name="ecldirect"
             path="componentfiles/ecldirect"
             processName="EspService"
             schema="esp_service_ecldirect.xsd">
    <Properties bindingType="EclDirectSoapBinding"
                defaultPort="8008"
                defaultResourcesBasedn="ou=EclDirectAccess,ou=EspServices,ou=ecl"
                defaultSecurePort="18008"
                plugin="ecldirect"
                type="ecldirect">
     <Authenticate access="Read"
                   description="Root access to ECL Direct service"
                   path="/"
                   required="Read"
                   resource="EclDirectAccess"/>
     <AuthenticateFeature description="Access to ECL Direct service"
                          path="EclDirectAccess"
                          resource="EclDirectAccess"
                          service="ecldirect"/>
    </Properties>
   </BuildSet>
  </Build>
 </Programs>
 <Software>
  <DafilesrvProcess build="community_3.0.4"
                    buildSet="dafilesrv"
                    description="DaFileSrv process"
                    name="mydafilesrv"
                    version="1">
#for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/mydafilesrv"
             name="s_$name"
             netAddress="$netAddress"/>
#end for             
  </DafilesrvProcess>
  <DaliServerProcess build="community_3.0.4"
                     buildSet="dali"
                     environment="/etc/HPCCSystems/environment.xml"
                     name="mydali"
                     recoverFromIncErrors="true">                     
   <Instance computer="$nodes[0][1]"
             directory="/var/lib/HPCCSystems/mydali"
             name="s_$nodes[0][1]"
             netAddress="$nodes[0][0]"
             port="7070"/>
  </DaliServerProcess>
  <DfuServerProcess build="community_3.0.4"
                    buildSet="dfuserver"
                    daliServers="mydali"
                    description="DFU Server"
                    monitorinterval="900"
                    monitorqueue="dfuserver_monitor_queue"
                    name="mydfuserver"
                    queue="dfuserver_queue"
                    transferBufferSize="65536">
#for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/mydfuserver"
             name="s_$name"
             netAddress="$netAddress"/>
#end for    
#raw        
   <SSH SSHidentityfile="$HOME/.ssh/id_rsa"
#end raw
        SSHpassword=""
        SSHretries="3"
        SSHtimeout="0"
        SSHusername="hpcc"/>
  </DfuServerProcess>
  <Directories name="HPCCSystems">
   <Category dir="/var/log/[NAME]/[INST]" name="log"/>
   <Category dir="/var/lib/[NAME]/[INST]" name="run"/>
   <Category dir="/etc/[NAME]/[INST]" name="conf"/>
   <Category dir="/var/lib/[NAME]/[INST]/temp" name="temp"/>
   <Category dir="/var/lib/[NAME]/hpcc-data/[COMPONENT]" name="data"/>
   <Category dir="/var/lib/[NAME]/hpcc-data2/[COMPONENT]" name="data2"/>
   <Category dir="/var/lib/[NAME]/hpcc-data3/[COMPONENT]" name="data3"/>
   <Category dir="/var/lib/[NAME]/hpcc-mirror/[COMPONENT]" name="mirror"/>
   <Category dir="/var/lib/[NAME]/queries/[INST]" name="query"/>
   <Category dir="/var/lock/[NAME]/[INST]" name="lock"/>
  </Directories>
  <DropZone build="community_3.0.4"
            buildSet="DropZone"
            computer="$nodes[0][0]"
            description="DropZone process"
            directory="/var/lib/HPCCSystems/dropzone"
            name="mydropzone"/>
  <EclAgentProcess allowedPipePrograms="*"
                   build="community_3.0.4"
                   buildSet="eclagent"
                   daliServers="mydali"
                   description="EclAgent process"
                   name="myeclagent"
                   pluginDirectory="/opt/HPCCSystems/plugins/"
                   thorConnectTimeout="600"
                   traceLevel="0"
                   wuQueueName="myeclagent_queue">
#for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/myeclagent"
             name="s_$name"
             netAddress="$netAddress"/>
#end for  
  </EclAgentProcess>
  <EclCCServerProcess build="community_3.0.4"
                      buildSet="eclccserver"
                      daliServers="mydali"
                      description="EclCCServer process"
                      enableSysLog="true"
                      maxCompileThreads="4"
                      name="myeclccserver"
                      traceLevel="1">
#for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/myeclccserver"
             name="s_$name"
             netAddress="$netAddress"/>
#end for
  </EclCCServerProcess>
  <EclSchedulerProcess build="community_3.0.4"
                       buildSet="eclscheduler"
                       daliServers="mydali"
                       description="EclScheduler process"
                       name="myeclscheduler">
#for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/myeclscheduler"
             name="s_$name"
             netAddress="$netAddress"/>
#end for
  </EclSchedulerProcess>
  <EspProcess build="community_3.0.4"
              buildSet="esp"
              componentfilesDir="/opt/HPCCSystems/componentfiles"
              daliServers="mydali"
              description="ESP server"
              enableSEHMapping="true"
              formOptionsAccess="false"
              httpConfigAccess="true"
              logLevel="1"
              logRequests="false"
              logResponses="false"
              maxBacklogQueueSize="200"
              maxConcurrentThreads="0"
              maxRequestEntityLength="8000000"
              name="myesp"
              perfReportDelay="60"
              portalurl="http://hpccsystems.com/download">
   <Authentication ldapAuthMethod="kerberos"
                   ldapConnections="10"
                   ldapServer=""
                   method="none"/>
   <EspBinding defaultForPort="true"
               defaultServiceVersion=""
               name="myespsmc"
               port="8010"
               protocol="http"
               resourcesBasedn="ou=SMC,ou=EspServices,ou=ecl"
               service="EclWatch"
               workunitsBasedn="ou=workunits,ou=ecl"
               wsdlServiceAddress="">
    <Authenticate access="Read"
                  description="Root access to SMC service"
                  path="/"
                  required="Read"
                  resource="SmcAccess"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to SMC service"
                         path="SmcAccess"
                         resource="SmcAccess"
                         service="ws_smc"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to thor queues"
                         path="ThorQueueAccess"
                         resource="ThorQueueAccess"
                         service="ws_smc"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to super computer environment"
                         path="ConfigAccess"
                         resource="ConfigAccess"
                         service="ws_config"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to DFU"
                         path="DfuAccess"
                         resource="DfuAccess"
                         service="ws_dfu"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to DFU XRef"
                         path="DfuXrefAccess"
                         resource="DfuXrefAccess"
                         service="ws_dfuxref"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to machine information"
                         path="MachineInfoAccess"
                         resource="MachineInfoAccess"
                         service="ws_machine"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to SNMP metrics information"
                         path="MetricsAccess"
                         resource="MetricsAccess"
                         service="ws_machine"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to remote execution"
                         path="ExecuteAccess"
                         resource="ExecuteAccess"
                         service="ws_machine"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to DFU workunits"
                         path="DfuWorkunitsAccess"
                         resource="DfuWorkunitsAccess"
                         service="ws_fs"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to DFU exceptions"
                         path="DfuExceptionsAccess"
                         resource="DfuExceptions"
                         service="ws_fs"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to spraying files"
                         path="FileSprayAccess"
                         resource="FileSprayAccess"
                         service="ws_fs"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to despraying of files"
                         path="FileDesprayAccess"
                         resource="FileDesprayAccess"
                         service="ws_fs"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to dkcing of key files"
                         path="FileDkcAccess"
                         resource="FileDkcAccess"
                         service="ws_fs"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to files in dropzone"
                         path="FileIOAccess"
                         resource="FileIOAccess"
                         service="ws_fileio"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to WS ECL service"
                         path="WsEclAccess"
                         resource="WsEclAccess"
                         service="ws_ecl"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to Roxie queries and files"
                         path="RoxieQueryAccess"
                         resource="RoxieQueryAccess"
                         service="ws_roxiequery"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to cluster topology"
                         path="ClusterTopologyAccess"
                         resource="ClusterTopologyAccess"
                         service="ws_topology"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to own workunits"
                         path="OwnWorkunitsAccess"
                         resource="OwnWorkunitsAccess"
                         service="ws_workunits"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to others&apos; workunits"
                         path="OthersWorkunitsAccess"
                         resource="OthersWorkunitsAccess"
                         service="ws_workunits"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to ECL direct service"
                         path="EclDirectAccess"
                         resource="EclDirectAccess"
                         service="ecldirect"/>
   </EspBinding>
   <EspBinding defaultForPort="true"
               defaultServiceVersion=""
               name="myws_ecl"
               port="8002"
               protocol="http"
               resourcesBasedn="ou=WsEcl,ou=EspServices,ou=ecl"
               service="myws_ecl"
               workunitsBasedn="ou=workunits,ou=ecl"
               wsdlServiceAddress="">
    <Authenticate access="Read"
                  description="Root access to WS ECL service"
                  path="/"
                  required="Read"
                  resource="WsEclAccess"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to WS ECL service"
                         path="WsEclAccess"
                         resource="WsEclAccess"
                         service="ws_ecl"/>
   </EspBinding>
   <EspBinding defaultForPort="true"
               defaultServiceVersion=""
               name="myecldirect"
               port="8008"
               protocol="http"
               resourcesBasedn="ou=EclDirectAccess,ou=EspServices,ou=ecl"
               service="myecldirect"
               workunitsBasedn="ou=workunits,ou=ecl"
               wsdlServiceAddress="">
    <Authenticate access="Read"
                  description="Root access to ECL Direct service"
                  path="/"
                  required="Read"
                  resource="EclDirectAccess"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to ECL Direct service"
                         path="EclDirectAccess"
                         resource="EclDirectAccess"
                         service="ecldirect"/>
   </EspBinding>
   <HTTPS acceptSelfSigned="true"
          CA_Certificates_Path="ca.pem"
          certificateFileName="certificate.cer"
          city=""
          country="US"
          daysValid="365"
          enableVerification="false"
          organization="Customer of HPCCSystems"
          organizationalUnit=""
          passphrase=""
          privateKeyFileName="privatekey.cer"
          regenerateCredentials="false"
          requireAddressMatch="false"
          state=""
          trustedPeers="anyone"/>
   #for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/myesp"
             FQDN=""
             name="s_$name"
             netAddress="$netAddress"/>
   #end for
   <ProtocolX authTimeout="3"
              defaultTimeout="21"
              idleTimeout="3600"
              maxTimeout="36"
              minTimeout="30"
              threadCount="2"/>
  </EspProcess>
  <EspService allowNewRoxieOnDemandQuery="false"
              AWUsCacheTimeout="15"
              build="community_3.0.4"
              buildSet="espsmc"
              description="ESP services for SMC"
              disableUppercaseTranslation="false"
              eclServer="myeclccserver"
              enableSystemUseRewrite="false"
              excludePartitions="/,/dev*,/sys,/usr,/proc/*"
              monitorDaliFileServer="false"
              name="EclWatch"
              pluginsPath="/opt/HPCCSystems/plugins"
              syntaxCheckQueue=""
              viewTimeout="1000"
              warnIfCpuLoadOver="95"
              warnIfFreeMemoryUnder="5"
              warnIfFreeStorageUnder="5">
   <Properties defaultPort="8010"
               defaultResourcesBasedn="ou=SMC,ou=EspServices,ou=ecl"
               defaultSecurePort="18010"
               type="WsSMC">
    <Authenticate access="Read"
                  description="Root access to SMC service"
                  path="/"
                  required="Read"
                  resource="SmcAccess"/>
    <AuthenticateFeature description="Access to SMC service"
                         path="SmcAccess"
                         resource="SmcAccess"
                         service="ws_smc"/>
    <AuthenticateFeature description="Access to thor queues"
                         path="ThorQueueAccess"
                         resource="ThorQueueAccess"
                         service="ws_smc"/>
    <AuthenticateFeature description="Access to super computer environment"
                         path="ConfigAccess"
                         resource="ConfigAccess"
                         service="ws_config"/>
    <AuthenticateFeature description="Access to DFU"
                         path="DfuAccess"
                         resource="DfuAccess"
                         service="ws_dfu"/>
    <AuthenticateFeature description="Access to DFU XRef"
                         path="DfuXrefAccess"
                         resource="DfuXrefAccess"
                         service="ws_dfuxref"/>
    <AuthenticateFeature description="Access to machine information"
                         path="MachineInfoAccess"
                         resource="MachineInfoAccess"
                         service="ws_machine"/>
    <AuthenticateFeature description="Access to SNMP metrics information"
                         path="MetricsAccess"
                         resource="MetricsAccess"
                         service="ws_machine"/>
    <AuthenticateFeature description="Access to remote execution"
                         path="ExecuteAccess"
                         resource="ExecuteAccess"
                         service="ws_machine"/>
    <AuthenticateFeature description="Access to DFU workunits"
                         path="DfuWorkunitsAccess"
                         resource="DfuWorkunitsAccess"
                         service="ws_fs"/>
    <AuthenticateFeature description="Access to DFU exceptions"
                         path="DfuExceptionsAccess"
                         resource="DfuExceptions"
                         service="ws_fs"/>
    <AuthenticateFeature description="Access to spraying files"
                         path="FileSprayAccess"
                         resource="FileSprayAccess"
                         service="ws_fs"/>
    <AuthenticateFeature description="Access to despraying of files"
                         path="FileDesprayAccess"
                         resource="FileDesprayAccess"
                         service="ws_fs"/>
    <AuthenticateFeature description="Access to dkcing of key files"
                         path="FileDkcAccess"
                         resource="FileDkcAccess"
                         service="ws_fs"/>
    <AuthenticateFeature description="Access to files in dropzone"
                         path="FileIOAccess"
                         resource="FileIOAccess"
                         service="ws_fileio"/>
    <AuthenticateFeature description="Access to WS ECL service"
                         path="WsEclAccess"
                         resource="WsEclAccess"
                         service="ws_ecl"/>
    <AuthenticateFeature description="Access to Roxie queries and files"
                         path="RoxieQueryAccess"
                         resource="RoxieQueryAccess"
                         service="ws_roxiequery"/>
    <AuthenticateFeature description="Access to cluster topology"
                         path="ClusterTopologyAccess"
                         resource="ClusterTopologyAccess"
                         service="ws_topology"/>
    <AuthenticateFeature description="Access to own workunits"
                         path="OwnWorkunitsAccess"
                         resource="OwnWorkunitsAccess"
                         service="ws_workunits"/>
    <AuthenticateFeature description="Access to others&apos; workunits"
                         path="OthersWorkunitsAccess"
                         resource="OthersWorkunitsAccess"
                         service="ws_workunits"/>
    <AuthenticateFeature description="Access to ECL direct service"
                         path="EclDirectAccess"
                         resource="EclDirectAccess"
                         service="ecldirect"/>
    <ProcessFilters>
     <Platform name="Windows">
      <ProcessFilter name="any">
       <Process name="dafilesrv"/>
      </ProcessFilter>
      <ProcessFilter name="AttrServerProcess">
       <Process name="attrserver"/>
      </ProcessFilter>
      <ProcessFilter name="DaliProcess">
       <Process name="daserver"/>
      </ProcessFilter>
      <ProcessFilter multipleInstances="true" name="DfuServerProcess">
       <Process name="dfuserver"/>
      </ProcessFilter>
      <ProcessFilter multipleInstances="true" name="EclCCServerProcess">
       <Process name="eclccserver"/>
      </ProcessFilter>
      <ProcessFilter multipleInstances="true" name="EspProcess">
       <Process name="esp"/>
       <Process name="dafilesrv" remove="true"/>
      </ProcessFilter>
      <ProcessFilter name="FTSlaveProcess">
       <Process name="ftslave"/>
      </ProcessFilter>
      <ProcessFilter name="RoxieServerProcess">
       <Process name="ccd"/>
      </ProcessFilter>
      <ProcessFilter name="RoxieSlaveProcess">
       <Process name="ccd"/>
      </ProcessFilter>
      <ProcessFilter name="SchedulerProcess">
       <Process name="scheduler"/>
      </ProcessFilter>
      <ProcessFilter name="ThorMasterProcess">
       <Process name="thormaster"/>
      </ProcessFilter>
      <ProcessFilter name="ThorSlaveProcess">
       <Process name="thorslave"/>
      </ProcessFilter>
      <ProcessFilter name="SashaServerProcess">
       <Process name="saserver"/>
      </ProcessFilter>
     </Platform>
     <Platform name="Linux">
      <ProcessFilter name="any">
       <Process name="dafilesrv"/>
      </ProcessFilter>
      <ProcessFilter name="AttrServerProcess">
       <Process name="attrserver"/>
      </ProcessFilter>
      <ProcessFilter name="DaliProcess">
       <Process name="daserver"/>
      </ProcessFilter>
      <ProcessFilter multipleInstances="true" name="DfuServerProcess">
       <Process name="."/>
      </ProcessFilter>
      <ProcessFilter multipleInstances="true" name="EclCCServerProcess">
       <Process name="."/>
      </ProcessFilter>
      <ProcessFilter multipleInstances="true" name="EspProcess">
       <Process name="."/>
       <Process name="dafilesrv" remove="true"/>
      </ProcessFilter>
      <ProcessFilter name="FTSlaveProcess">
       <Process name="ftslave"/>
      </ProcessFilter>
      <ProcessFilter name="GenesisServerProcess">
       <Process name="httpd"/>
       <Process name="atftpd"/>
       <Process name="dhcpd"/>
      </ProcessFilter>
      <ProcessFilter name="RoxieServerProcess">
       <Process name="ccd"/>
      </ProcessFilter>
      <ProcessFilter name="RoxieSlaveProcess">
       <Process name="ccd"/>
      </ProcessFilter>
      <ProcessFilter name="SchedulerProcess">
       <Process name="scheduler"/>
      </ProcessFilter>
      <ProcessFilter name="ThorMasterProcess">
       <Process name="thormaster"/>
      </ProcessFilter>
      <ProcessFilter name="ThorSlaveProcess">
       <Process name="thorslave"/>
      </ProcessFilter>
      <ProcessFilter name="SashaServerProcess">
       <Process name="saserver"/>
      </ProcessFilter>
     </Platform>
    </ProcessFilters>
   </Properties>
  </EspService>
  <EspService build="community_3.0.4"
              buildSet="ws_ecl"
              description="WS ECL Service"
              name="myws_ecl">
   <Properties bindingType="ws_eclSoapBinding"
               defaultPort="8002"
               defaultResourcesBasedn="ou=WsEcl,ou=EspServices,ou=ecl"
               defaultSecurePort="18002"
               plugin="ws_ecl"
               type="ws_ecl">
    <Authenticate access="Read"
                  description="Root access to WS ECL service"
                  path="/"
                  required="Read"
                  resource="WsEclAccess"/>
    <AuthenticateFeature description="Access to WS ECL service"
                         path="WsEclAccess"
                         resource="WsEclAccess"
                         service="ws_ecl"/>
   </Properties>
  </EspService>
  <EspService build="community_3.0.4"
              buildSet="ecldirect"
              clusterName="hthor"
              description="ESP service for running raw ECL queries"
              name="myecldirect">
   <Properties bindingType="EclDirectSoapBinding"
               defaultPort="8008"
               defaultResourcesBasedn="ou=EclDirectAccess,ou=EspServices,ou=ecl"
               defaultSecurePort="18008"
               plugin="ecldirect"
               type="ecldirect">
    <Authenticate access="Read"
                  description="Root access to ECL Direct service"
                  path="/"
                  required="Read"
                  resource="EclDirectAccess"/>
    <AuthenticateFeature description="Access to ECL Direct service"
                         path="EclDirectAccess"
                         resource="EclDirectAccess"
                         service="ecldirect"/>
   </Properties>
  </EspService>
  <FTSlaveProcess build="community_3.0.4"
                  buildSet="ftslave"
                  description="FTSlave process"
                  name="myftslave"
                  version="1">
#for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/myftslave"
             name="s_$name"
             netAddress="$netAddress"
             program="/opt/HPCCSystems/bin/ftslave"/>
#end for
  </FTSlaveProcess>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_auditlib"
                 description="plugin process"
                 name="myplugins_auditlib"/>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_debugservices"
                 description="plugin process"
                 name="myplugins_debugservices"/>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_fileservices"
                 description="plugin process"
                 name="myplugins_fileservices"/>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_logging"
                 description="plugin process"
                 name="myplugins_logging"/>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_parselib"
                 description="plugin process"
                 name="myplugins_parselib"/>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_stringlib"
                 description="plugin process"
                 name="myplugins_stringlib"/>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_unicodelib"
                 description="plugin process"
                 name="myplugins_unicodelib"/>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_workunitservices"
                 description="plugin process"
                 name="myplugins_workunitservices"/>
  <RoxieCluster allowRoxieOnDemand="false"
                baseDataDir="/var/lib/HPCCSystems/hpcc-data/roxie"
                blindLogging="false"
                blobCacheMem="0"
                build="community_3.0.4"
                buildSet="roxie"
                callbackRetries="3"
                callbackTimeout="500"
                channel0onPrimary="true"
                checkCompleted="true"
                checkFileDate="true"
                checkingHeap="0"
                checkPrimaries="true"
                checkState="false"
                checkVersion="true"
                clusterWidth="$len($nodes)"
                copyResources="true"
                crcResources="false"
                cyclicOffset="1"
                dafilesrvLookupTimeout="10000"
                daliServers="mydali"
                debugPermitted="true"
                defaultConcatPreload="0"
                defaultFetchPreload="0"
                defaultFullKeyedJoinPreload="0"
                defaultHighPriorityTimeLimit="0"
                defaultHighPriorityTimeWarning="5000"
                defaultKeyedJoinPreload="0"
                defaultLowPriorityTimeLimit="0"
                defaultLowPriorityTimeWarning="0"
                defaultMemoryLimit="0"
                defaultParallelJoinPreload="0"
                defaultPrefetchProjectPreload="10"
                defaultSLAPriorityTimeLimit="0"
                defaultSLAPriorityTimeWarning="5000"
                defaultStripLeadingWhitespace="1"
                deleteUnneededFiles="false"
                description="Roxie cluster"
                directory="/var/lib/HPCCSystems/myroxie"
                diskReadBufferSize="65536"
                diskReadStable="true"
                doIbytiDelay="true"
                enableForceKeyDiffCopy="false"
                enableHeartBeat="true"
                enableKeyDiff="true"
                enableSNMP="true"
                enableSysLog="true"
                fastLaneQueue="true"
                fieldTranslationEnabled="false"
                flushJHtreeCacheOnOOM="true"
                forceStdLog="false"
                highTimeout="2000"
                ignoreMissingFiles="false"
                indexReadChunkSize="60000"
                indexReadStable="true"
                initIbytiDelay="100"
                jumboFrames="false"
                keyedJoinFlowLimit="1000"
                keyedJoinStable="true"
                lazyOpen="false"
                leafCacheMem="50"
                linuxYield="true"
                localFilesExpire="-1"
                localSlave="false"
                logFullQueries="false"
                logQueueDrop="32"
                logQueueLen="512"
                lowTimeout="10000"
                maxBlockSize="10000000"
                maxLocalFilesOpen="4000"
                maxLockAttempts="5"
                maxRemoteFilesOpen="1000"
                memoryStatsInterval="60"
                memTraceLevel="1"
                memTraceSizeLimit="0"
                minFreeDiskSpace="1073741824"
                minIbytiDelay="0"
                minLocalFilesOpen="2000"
                minRemoteFilesOpen="500"
                miscDebugTraceLevel="0"
                monitorDaliFileServer="false"
                multicastBase="239.1.1.1"
                multicastLast="239.1.254.254"
                name="myroxie"
                nodeCacheMem="100"
                nodeCachePreload="false"
                numChannels="$len($nodes)"
                numDataCopies="2"
                parallelAggregate="0"
                perChannelFlowLimit="10"
                pingInterval="60"
                pluginsPath="/opt/HPCCSystems/plugins"
                preabortIndexReadsThreshold="100"
                preabortKeyedJoinsThreshold="100"
                preferredSubnet=""
                preferredSubnetMask=""
                remoteFilesExpire="3600000"
                resolveFilesInPackage="false"
                roxieMulticastEnabled="true"
                serverSideCacheSize="0"
                serverThreads="30"
                simpleLocalKeyedJoins="true"
                siteCertificate=""
                slaTimeout="2000"
                slaveConfig="cyclic redundancy"
                slaveThreads="30"
                smartSteppingChunkRows="100"
                soapTraceLevel="1"
                socketCheckInterval="5000"
#raw                
                SSHidentityfile="$HOME/.ssh/id_rsa"
#end raw                
                SSHpassword=""
                SSHretries="3"
                SSHtimeout="0"
                SSHusername="hpcc"
                statsExpiryTime="3600"
                syncCluster="false"
                systemMonitorInterval="60000"
                totalMemoryLimit="1073741824"
                traceLevel="1"
                trapTooManyActiveQueries="true"
                udpFlowSocketsSize="131071"
                udpInlineCollation="false"
                udpInlineCollationPacketLimit="50"
                udpLocalWriteSocketSize="131071"
                udpMaxRetryTimedoutReqs="0"
                udpMaxSlotsPerClient="2147483647"
                udpMulticastBufferSize="131071"
                udpOutQsPriority="0"
                udpQueueSize="100"
                udpRequestToSendTimeout="5"
                udpResendEnabled="true"
                udpRetryBusySenders="0"
                udpSendCompletedInData="false"
                udpSendQueueSize="50"
                udpSnifferEnabled="true"
                udpTraceLevel="1"
                useHardLink="false"
                useLogQueue="true"
                useMemoryMappedIndexes="false"
                useRemoteResources="true"
                useTreeCopy="false">
   <RoxieFarmProcess dataDirectory="/var/lib/HPCCSystems/hpcc-data/roxie"
                     listenQueue="200"
                     name="farm1"
                     numThreads="30"
                     port="9876"
                     requestArrayThreads="5">
#for $netAddress, $name in $nodes:                    
    <RoxieServerProcess computer="$name" name="farm1_$name"/>
#end for
   </RoxieFarmProcess>
#for $netAddress, $name in $nodes:                    
   <RoxieServerProcess computer="$name"
                       dataDirectory="/var/lib/HPCCSystems/hpcc-data/roxie"
                       listenQueue="200"
                       name="farm1_s_$name"
                       netAddress="$netAddress"
                       numThreads="30"
                       port="9876"
                       requestArrayThreads="5"/>
#end for
#for $netAddress, $name in $nodes:                    
   <RoxieSlave computer="$name" name="s_$name">
    <RoxieChannel dataDirectory="/var/lib/HPCCSystems/hpcc-data/roxie" number="$random.randint(1,$len($nodes))"/>
    <RoxieChannel dataDirectory="/var/lib/HPCCSystems/hpcc-data2/roxie" number="$random.randint(1,$len($nodes))"/>
   </RoxieSlave>
#end for
#set $channel = 1
#for $netAddress, $name in $nodes:                    
   <RoxieSlaveProcess channel="$channel"
                      computer="$name"
                      dataDirectory="/var/lib/HPCCSystems/hpcc-data/roxie"
                      name="s_$name"
                      netAddress="$netAddress"/>
#set $channel = $channel + 1                      
#end for
  </RoxieCluster>
  <SashaServerProcess autoRestartInterval="0"
                      build="community_3.0.4"
                      buildSet="sasha"
                      cachedWUat="* * * * *"
                      cachedWUinterval="24"
                      cachedWUlimit="100"
                      coalesceAt="* * * * *"
                      coalesceInterval="1"
                      dafsmonAt="* * * * *"
                      dafsmonInterval="0"
                      dafsmonList="*"
                      daliServers="mydali"
                      description="Sasha Server process"
                      DFUrecoveryAt="* * * * *"
                      DFUrecoveryCutoff="4"
                      DFUrecoveryInterval="12"
                      DFUrecoveryLimit="20"
                      DFUWUat="* * * * *"
                      DFUWUcutoff="14"
                      DFUWUduration="0"
                      DFUWUinterval="24"
                      DFUWUlimit="1000"
                      DFUWUthrottle="0"
                      ExpiryAt="* 3 * * *"
                      ExpiryInterval="24"
                      keepResultFiles="false"
                      LDSroot="LDS"
                      logDir="."
                      minDeltaSize="50000"
                      name="mysasha"
                      recoverDeltaErrors="false"
                      thorQMonInterval="1"
                      thorQMonQueues="*"
                      thorQMonSwitchMinTime="0"
                      WUat="* * * * *"
                      WUbackup="0"
                      WUcutoff="8"
                      WUduration="0"
                      WUinterval="6"
                      WUlimit="1000"
                      WUretryinterval="7"
                      WUthrottle="0"
                      xrefAt="* 2 * * *"
                      xrefCutoff="1"
                      xrefEclWatchProvider="true"
                      xrefInterval="0"
                      xrefList="*">
#for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/mysasha"
             name="s_$name"
             netAddress="$netAddress"
             port="8877"/>
#end for             
  </SashaServerProcess>
  <ThorCluster autoCopyBackup="false"
               build="community_3.0.4"
               buildSet="thor"
               computer="$nodes[0][1]"
               daliServers="mydali"
               description="Thor process"
               localThor="false"
               monitorDaliFileServer="true"
               multiSlaves="false"
               name="mythor"
               pluginsPath="/opt/HPCCSystems/plugins/"
               replicateAsync="true"
               replicateOutputs="true"
               slaves="8"
               watchdogEnabled="true"
               watchdogProgressEnabled="true">
   <Debug/>
#raw   
   <SSH SSHidentityfile="$HOME/.ssh/id_rsa"
#end raw   
        SSHpassword=""
        SSHretries="3"
        SSHtimeout="0"
        SSHusername="hpcc"/>
   <Storage/>
   <SwapNode/>
   <ThorMasterProcess computer="$nodes[0][1]" name="m_$nodes[0][1]"/>
#if $len($nodes) > 1:
#for $netAddress, $name in $nodes[1:]:                    
   <ThorSlaveProcess computer="$name" name="s_$name"/>
#end for   
#end if
   <Topology>
    <Node process="m_$nodes[0][1]">
#if $len($nodes) > 1:
#for $netAddress, $name in $nodes[1:]:                    
     <Node process="s_$name"/>
#end for   
#end if
    </Node>
   </Topology>
  </ThorCluster>
  <Topology build="community_3.0.4" buildSet="topology" name="topology">
   <Cluster name="hthor" prefix="hthor">
    <EclAgentProcess process="myeclagent"/>
    <EclCCServerProcess process="myeclccserver"/>
    <EclSchedulerProcess process="myeclscheduler"/>
   </Cluster>
   <Cluster name="thor" prefix="thor">
    <EclAgentProcess process="myeclagent"/>
    <EclCCServerProcess process="myeclccserver"/>
    <EclSchedulerProcess process="myeclscheduler"/>
    <ThorCluster process="mythor"/>
   </Cluster>
   <Cluster name="roxie" prefix="roxie">
    <EclAgentProcess process="myeclagent"/>
    <EclCCServerProcess process="myeclccserver"/>
    <EclSchedulerProcess process="myeclscheduler"/>
    <RoxieCluster process="myroxie"/>
   </Cluster>
  </Topology>
 </Software>
</Environment>
====


Even just scrolling past the file takes a while!!  This behemoth of a file was tamed thanks to cheetah, I highly encourage you to read up on it.

This charm may require some changes to your environment.yaml file in ~/.juju as hpcc will only run on 64-bit instances.  Make sure that your juju environment has been properly shutdown before you edit this file ( juju destroy-environment ).  Here is my environment.yaml file where I show you the important part to check:
juju: environments

environments:
  sample:
    type: ec2
    access-key: ( removed ... get your own :) )
    secret-key: ( removed ... get your own :) )
    control-bucket: juju-fbb790f292e14a0394353bb4b63a3403
    admin-secret: 604d18a77fd24e3f91e1df398fcbe9f2
The emphasized parts are the important ones.  You can just copy them from here and paste them into your ~/.juju/environment.yaml file.
Now, let's take a look at the charm starting with the metadata.yaml file:
name: hpcc
revision: 1
summary: HPCC (High Performance Computing Cluster)
description: |
  HPCC (High Performance Computing Cluster) is a massive 
  parallel-processing computing platform that solves Big Data problems.
provides:
  hpcc:
    interface: hpcc
requires:
  hpcc-thor:
    interface: hpcc-thor
  hpcc-roxie:
    interface: hpcc-roxie
peers:
  hpcc-cluster:
    interface: hpcc-cluster

 There are various provides and requires interfaces in this metadata.yaml file but, for now, only the peers interface is being used.  I'll work on the other ones as the charm matures.


Let's look at the hpcc-cluster interface.  More specifically the hpcc-cluster-relation-changed hook where the new configuration is created:
#!/bin/bash
CWD=$(dirname $0)
cheetah fill --oext=xml --odir=/etc/HPCCSystems/ ${CWD}/../templates/environment.tmpl
service hpcc-init restart
It's pretty simple isn't it?   Since the "heavy lifting" is being done with the self contained cheetah template , we don't have much to do here but, to generate the configuration file and restart hpcc.

The other files in this charm are pretty self explanatory and simple so, I am leaving the details of them as an exercise to the reader.

All of the complexities in hpcc has been distilled to the following commands:

  • juju bootstrap
  • bzr branch lp:~negronjl/+junk/hpcc
  • juju deploy --repository . hpcc
    • wait a minute of two
  • juju status
    • you should see something similar to this:

negronjl@negronjl-laptop:~/src/juju/charms$ juju status
2011-08-18 16:00:54,413 INFO Connecting to environment.


machines: 
  0: {dns-name: ec2-184-73-109-244.compute-1.amazonaws.com, instance-id: i-6d61460c} 
  1: {dns-name: ec2-50-16-60-94.compute-1.amazonaws.com, instance-id: i-d5694eb4}


services:
  hpcc:
    charm: local:hpcc-1
    relations: {hpcc-cluster: hpcc}
    units:
      hpcc/0:
        machine: 1
        relations: {}
        state: null


2011-08-18 16:00:58,374 INFO 'status' command finished successfully
negronjl@negronjl-laptop:~/src/juju/charms$
The above commands, will give you a single node.

You can access the web interface of your node by pointing your browser to http://<FQDN>:8010 Where FQDN is the Fully Qualified Domain Name or Public IP Address of your hpcc instance.  On the left side, there should be a menu, explore the items on the Topology section.  The Target Clusters section should look something similar to this:


To experience the true power of hpcc, you should probably throw in some more nodes at it.  Let's do just that with:


  • juju add-unit hpcc 
    • do this as many times as you feel comfortable   
    • Each command will give you a new node in the cluster
    • wait a minute or two and you should see something similar to this:

negronjl@negronjl-laptop:~$ juju status
2011-08-18 16:25:55,739 INFO Connecting to environment.
machines:
  0: {dns-name: ec2-184-73-109-244.compute-1.amazonaws.com, instance-id: i-6d61460c}
  1: {dns-name: ec2-50-16-60-94.compute-1.amazonaws.com, instance-id: i-d5694eb4}
  2: {dns-name: ec2-50-19-181-98.compute-1.amazonaws.com, instance-id: i-a5795ec4}
  3: {dns-name: ec2-184-72-147-67.compute-1.amazonaws.com, instance-id: i-25446344}
services:
  hpcc:
    charm: local:hpcc-1
    relations: {hpcc-cluster: hpcc}
    units:
      hpcc/0:
        machine: 1
        relations:
          hpcc-cluster: {state: up}
        state: started
      hpcc/1:
        machine: 2
        relations:
          hpcc-cluster: {state: up}
        state: started
      hpcc/2:
        machine: 3
        relations:
          hpcc-cluster: {state: up}
        state: started
2011-08-18 16:26:01,837 INFO 'status' command finished successfully
Notice how we now have more hpcc nodes :)  Here is what the web interface could look like:


Again....we have more nodes :)

Now that we have a working cluster, let's try it.  We'll first do the mandatory Hello World in ECL.  It looks something like this (hello.ecl):
Output('Hello world');
 We have to compile our hello.ecl so we can use it.  We do that by logging into one of the nodes ( I used juju ssh 1 to log on to the first/master node ) and typing the following:
eclcc hello.ecl -o
We run the file just like we would any other binary:
./hello
... and the output is:
ubuntu@ip-10-111-19-210:~$ ./hello
Hello world
ubuntu@ip-10-111-19-210:~$ 
There are far more interesting examples in the Learning ECL Documentation here.  I highly encourage you to go and read about it.

That's it for now.  Feedback is always welcome of course so, let me know how I'm doing.

-Juan

HPCC with Ubuntu Server and Ensemble

// August 19th, 2011 // 1 Comment » // Uncategorized

Let's start this post with a bit of background on the technologies that I'll be using:
  • What is Ubuntu?
    • Ubuntu is a fast, secure and easy-to-use operating system used by millions of people around the world.  
    • Secure, fast and powerful, Ubuntu Server is transforming IT environments worldwide. Realise the full potential of your infrastructure with a reliable, easy-to-integrate technology platform.
  • What is Ensemble?
    • Ensemble is a next generation service orchestration framework. It has been likened to APT for the cloud. With Ensemble, different authors are able to create service formulas independently, and make those services coordinate their communication through a simple protocol. Users can then take the product of different authors and very comfortably deploy those services in an environment. The result is multiple machines and components transparently collaborating towards providing the requested service.
  • What is HPCC?
    • HPCC (High Performance Computing Cluster) is a massive parallel-processing computing platform that solves Big Data problems. The platform is now Open Source!
 Now that we are all caught up, let's delve right into it.  I will be discussing the details of my newly created hpcc ensemble formula.

The hpcc formula has been one of the trickiest one to date to get working properly so, I want to take some time to explain some of the challenges that I encountered.

hpcc seems to use ssh keys for authentication and a single xml file to hold it's configuration.  All nodes that are part of the cluster should have identical keys and xml configuration file.

  • The ssh keys are pretty easy to do ( there is even a script that will do it all for you located at /opt/HPCCSystems/sbin/keygen.sh ).  You can just run: ssh-keygen -f path_where_to_save_keys/id_rsa -N "" -q
  • The configuration file environment.xml is a lot trickier to configure so, I will use cheetah templates to help make a template out of this enormous file.  
    • According to their website:
      • Cheetah is an open source template engine and code generation tool, written in Python. It can be used standalone or combined with other tools and frameworks. Web development is its principle use, but Cheetah is very flexible and is also being used to generate C++ game code, Java, sql, form emails and even Python code.
With cheetah, I can create self contained templates that can be generated into their intended file by just calling cheetah.  This is because we can embed python code inside the template itself, making the template ( environment.tmpl in our case ) more or less a python program that generates a fully functional environment.xml file ready for hpcc to use.

Another, very important, reason to use a template engine is the ability to create identical configuration files from each node without having to pass them around.  In other words, each node can create it's own configuration file and, since all nodes are using the same methods and data to create the file, they will all be exactly the same.

The hpcc configuration file is huge so, I'll just talk about some of the interesting bits of it here:
#import random
#import subprocess
#set $rel_structure = { $subprocess.check_output(['facter', 'install_time']).strip() : { 'name' : $subprocess.check_output(['hostname', '-f']).strip(), 'netAddress' : $subprocess.check_output(['facter','ipaddress']).strip(), 'uuid' : $subprocess.check_output(['facter','uuid']).strip()  } }
#for $member in $subprocess.check_output(['relation-list']).strip().split():
   #set $rel_structure[$subprocess.check_output(['relation-get','install_time', $member]).strip()] = { 'name' : $subprocess.check_output(['relation-get','name', $member]).strip(), 'netAddress' : $subprocess.check_output(['relation-get','netAddress', $member]).strip(), 'uuid' : $subprocess.check_output(['relation-get', 'uuid', $member]).strip() }
#end for
#set $nodes = []
#for $index in $sorted($rel_structure.keys()):
   $nodes.append(($rel_structure[$index]['netAddress'], $rel_structure[$index]['name']))
#end for
The above piece of code is what I am currently using to populate a list with the FQDN and address of each cluster member sorted by install time.  This puts the "master" of the cluster at the top of the list which will become useful when populating certain parts of the configuration file.


As we can see by the code above, the main piece of information that we use in this template is the node list.  Here is a sample of how we use it in the environment.tmpl template file:
 #for $netAddress, $name in $nodes:
  <Computer computerType="linuxmachine"
            domain="localdomain"
            name="$name"
            netAddress="$netAddress"/>
#end for
I encourage you to download the formula here and examine the environment.tmpl file in the templates directory.

Here is the complete environment.tmpl file... I know it's pretty small and, you can just download the formula and read the file at your leasure but, I wanted to give you an idea of the size and complexity of hpcc's configuration file.

====

#import random
#import subprocess
#set $rel_structure = { $subprocess.check_output(['facter', 'install_time']).strip() : { 'name' : $subprocess.check_output(['hostname', '-f']).strip(), 'netAddress' : $subprocess.check_output(['facter','ipaddress']).strip(), 'uuid' : $subprocess.check_output(['facter','uuid']).strip()  } }
#for $member in $subprocess.check_output(['relation-list']).strip().split():
   #set $rel_structure[$subprocess.check_output(['relation-get','install_time', $member]).strip()] = { 'name' : $subprocess.check_output(['relation-get','name', $member]).strip(), 'netAddress' : $subprocess.check_output(['relation-get','netAddress', $member]).strip(), 'uuid' : $subprocess.check_output(['relation-get', 'uuid', $member]).strip() }
#end for
#set $nodes = []
#for $index in $sorted($rel_structure.keys()):
   $nodes.append(($rel_structure[$index]['netAddress'], $rel_structure[$index]['name']))
#end for
<?xml version="1.0" encoding="UTF-8"?>
<!-- Edited with ConfigMgr on ip 71.204.190.179 on 2011-08-16T00:39:16 -->
<Environment>
 <EnvSettings>
  <blockname>HPCCSystems</blockname>
  <configs>/etc/HPCCSystems</configs>
  <environment>environment.xml</environment>
  <group>hpcc</group>
  <home>/home</home>
  <interface>eth0</interface>
  <lock>/var/lock/HPCCSystems</lock>
  <log>/var/log/HPCCSystems</log>
  <path>/opt/HPCCSystems</path>
  <pid>/var/run/HPCCSystems</pid>
  <runtime>/var/lib/HPCCSystems</runtime>
  <sourcedir>/etc/HPCCSystems/source</sourcedir>
  <user>hpcc</user>
 </EnvSettings>
 <Hardware>
 #for $netAddress, $name in $nodes:
  <Computer computerType="linuxmachine"
            domain="localdomain"
            name="$name"
            netAddress="$netAddress"/>
#end for
  <ComputerType computerType="linuxmachine"
                manufacturer="unknown"
                name="linuxmachine"
                opSys="linux"/>
  <Domain name="localdomain" password="" username=""/>
  <Switch name="Switch"/>
 </Hardware>
 <Programs>
  <Build name="community_3.0.4" url="/opt/HPCCSystems">
   <BuildSet installSet="deploy_map.xml"
             name="dafilesrv"
             path="componentfiles/dafilesrv"
             processName="DafilesrvProcess"
             schema="dafilesrv.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="dali"
             path="componentfiles/dali"
             processName="DaliServerProcess"
             schema="dali.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="dfuplus"
             path="componentfiles/dfuplus"
             processName="DfuplusProcess"
             schema="dfuplus.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="dfuserver"
             path="componentfiles/dfuserver"
             processName="DfuServerProcess"
             schema="dfuserver.xsd"/>
   <BuildSet deployable="no"
             installSet="deploy_map.xml"
             name="DropZone"
             path="componentfiles/DropZone"
             processName="DropZone"
             schema="dropzone.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="eclagent"
             path="componentfiles/eclagent"
             processName="EclAgentProcess"
             schema="eclagent_config.xsd"/>
   <BuildSet installSet="deploy_map.xml" name="eclminus" path="componentfiles/eclminus"/>
   <BuildSet installSet="deploy_map.xml"
             name="eclplus"
             path="componentfiles/eclplus"
             processName="EclPlusProcess"
             schema="eclplus.xsd"/>
   <BuildSet installSet="eclccserver_deploy_map.xml"
             name="eclccserver"
             path="componentfiles/configxml"
             processName="EclCCServerProcess"
             schema="eclccserver.xsd"/>
   <BuildSet installSet="eclscheduler_deploy_map.xml"
             name="eclscheduler"
             path="componentfiles/configxml"
             processName="EclSchedulerProcess"
             schema="eclscheduler.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="esp"
             path="componentfiles/esp"
             processName="EspProcess"
             schema="esp.xsd"/>
   <BuildSet deployable="no"
             installSet="deploy_map.xml"
             name="espsmc"
             path="componentfiles/espsmc"
             processName="EspService"
             schema="espsmcservice.xsd">
    <Properties defaultPort="8010"
                defaultResourcesBasedn="ou=SMC,ou=EspServices,ou=ecl"
                defaultSecurePort="18010"
                type="WsSMC">
     <Authenticate access="Read"
                   description="Root access to SMC service"
                   path="/"
                   required="Read"
                   resource="SmcAccess"/>
     <AuthenticateFeature description="Access to SMC service"
                          path="SmcAccess"
                          resource="SmcAccess"
                          service="ws_smc"/>
     <AuthenticateFeature description="Access to thor queues"
                          path="ThorQueueAccess"
                          resource="ThorQueueAccess"
                          service="ws_smc"/>
     <AuthenticateFeature description="Access to super computer environment"
                          path="ConfigAccess"
                          resource="ConfigAccess"
                          service="ws_config"/>
     <AuthenticateFeature description="Access to DFU"
                          path="DfuAccess"
                          resource="DfuAccess"
                          service="ws_dfu"/>
     <AuthenticateFeature description="Access to DFU XRef"
                          path="DfuXrefAccess"
                          resource="DfuXrefAccess"
                          service="ws_dfuxref"/>
     <AuthenticateFeature description="Access to machine information"
                          path="MachineInfoAccess"
                          resource="MachineInfoAccess"
                          service="ws_machine"/>
     <AuthenticateFeature description="Access to SNMP metrics information"
                          path="MetricsAccess"
                          resource="MetricsAccess"
                          service="ws_machine"/>
     <AuthenticateFeature description="Access to remote execution"
                          path="ExecuteAccess"
                          resource="ExecuteAccess"
                          service="ws_machine"/>
     <AuthenticateFeature description="Access to DFU workunits"
                          path="DfuWorkunitsAccess"
                          resource="DfuWorkunitsAccess"
                          service="ws_fs"/>
     <AuthenticateFeature description="Access to DFU exceptions"
                          path="DfuExceptionsAccess"
                          resource="DfuExceptions"
                          service="ws_fs"/>
     <AuthenticateFeature description="Access to spraying files"
                          path="FileSprayAccess"
                          resource="FileSprayAccess"
                          service="ws_fs"/>
     <AuthenticateFeature description="Access to despraying of files"
                          path="FileDesprayAccess"
                          resource="FileDesprayAccess"
                          service="ws_fs"/>
     <AuthenticateFeature description="Access to dkcing of key files"
                          path="FileDkcAccess"
                          resource="FileDkcAccess"
                          service="ws_fs"/>
     <AuthenticateFeature description="Access to files in dropzone"
                          path="FileIOAccess"
                          resource="FileIOAccess"
                          service="ws_fileio"/>
     <AuthenticateFeature description="Access to WS ECL service"
                          path="WsEclAccess"
                          resource="WsEclAccess"
                          service="ws_ecl"/>
     <AuthenticateFeature description="Access to Roxie queries and files"
                          path="RoxieQueryAccess"
                          resource="RoxieQueryAccess"
                          service="ws_roxiequery"/>
     <AuthenticateFeature description="Access to cluster topology"
                          path="ClusterTopologyAccess"
                          resource="ClusterTopologyAccess"
                          service="ws_topology"/>
     <AuthenticateFeature description="Access to own workunits"
                          path="OwnWorkunitsAccess"
                          resource="OwnWorkunitsAccess"
                          service="ws_workunits"/>
     <AuthenticateFeature description="Access to others&apos; workunits"
                          path="OthersWorkunitsAccess"
                          resource="OthersWorkunitsAccess"
                          service="ws_workunits"/>
     <AuthenticateFeature description="Access to ECL direct service"
                          path="EclDirectAccess"
                          resource="EclDirectAccess"
                          service="ecldirect"/>
     <ProcessFilters>
      <Platform name="Windows">
       <ProcessFilter name="any">
        <Process name="dafilesrv"/>
       </ProcessFilter>
       <ProcessFilter name="AttrServerProcess">
        <Process name="attrserver"/>
       </ProcessFilter>
       <ProcessFilter name="DaliProcess">
        <Process name="daserver"/>
       </ProcessFilter>
       <ProcessFilter multipleInstances="true" name="DfuServerProcess">
        <Process name="dfuserver"/>
       </ProcessFilter>
       <ProcessFilter multipleInstances="true" name="EclCCServerProcess">
        <Process name="eclccserver"/>
       </ProcessFilter>
       <ProcessFilter multipleInstances="true" name="EspProcess">
        <Process name="esp"/>
        <Process name="dafilesrv" remove="true"/>
       </ProcessFilter>
       <ProcessFilter name="FTSlaveProcess">
        <Process name="ftslave"/>
       </ProcessFilter>
       <ProcessFilter name="RoxieServerProcess">
        <Process name="ccd"/>
       </ProcessFilter>
       <ProcessFilter name="RoxieSlaveProcess">
        <Process name="ccd"/>
       </ProcessFilter>
       <ProcessFilter name="SchedulerProcess">
        <Process name="scheduler"/>
       </ProcessFilter>
       <ProcessFilter name="ThorMasterProcess">
        <Process name="thormaster"/>
       </ProcessFilter>
       <ProcessFilter name="ThorSlaveProcess">
        <Process name="thorslave"/>
       </ProcessFilter>
       <ProcessFilter name="SashaServerProcess">
        <Process name="saserver"/>
       </ProcessFilter>
      </Platform>
      <Platform name="Linux">
       <ProcessFilter name="any">
        <Process name="dafilesrv"/>
       </ProcessFilter>
       <ProcessFilter name="AttrServerProcess">
        <Process name="attrserver"/>
       </ProcessFilter>
       <ProcessFilter name="DaliProcess">
        <Process name="daserver"/>
       </ProcessFilter>
       <ProcessFilter multipleInstances="true" name="DfuServerProcess">
        <Process name="."/>
       </ProcessFilter>
       <ProcessFilter multipleInstances="true" name="EclCCServerProcess">
        <Process name="."/>
       </ProcessFilter>
       <ProcessFilter multipleInstances="true" name="EspProcess">
        <Process name="."/>
        <Process name="dafilesrv" remove="true"/>
       </ProcessFilter>
       <ProcessFilter name="FTSlaveProcess">
        <Process name="ftslave"/>
       </ProcessFilter>
       <ProcessFilter name="GenesisServerProcess">
        <Process name="httpd"/>
        <Process name="atftpd"/>
        <Process name="dhcpd"/>
       </ProcessFilter>
       <ProcessFilter name="RoxieServerProcess">
        <Process name="ccd"/>
       </ProcessFilter>
       <ProcessFilter name="RoxieSlaveProcess">
        <Process name="ccd"/>
       </ProcessFilter>
       <ProcessFilter name="SchedulerProcess">
        <Process name="scheduler"/>
       </ProcessFilter>
       <ProcessFilter name="ThorMasterProcess">
        <Process name="thormaster"/>
       </ProcessFilter>
       <ProcessFilter name="ThorSlaveProcess">
        <Process name="thorslave"/>
       </ProcessFilter>
       <ProcessFilter name="SashaServerProcess">
        <Process name="saserver"/>
       </ProcessFilter>
      </Platform>
     </ProcessFilters>
    </Properties>
   </BuildSet>
   <BuildSet installSet="deploy_map.xml"
             name="ftslave"
             path="componentfiles/ftslave"
             processName="FTSlaveProcess"
             schema="ftslave_linux.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="hqltest"
             path="componentfiles/hqltest"
             processName="HqlTestProcess"/>
   <BuildSet deployable="no"
             installSet="deploy_map.xml"
             name="ldapServer"
             path="componentfiles/ldapServer"
             processName="LDAPServerProcess"
             schema="ldapserver.xsd"/>
   <BuildSet deployable="no"
             installSet="auditlib_deploy_map.xml"
             name="plugins_auditlib"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet deployable="no"
             installSet="debugservices_deploy_map.xml"
             name="plugins_debugservices"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet deployable="no"
             installSet="fileservices_deploy_map.xml"
             name="plugins_fileservices"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet deployable="no"
             installSet="logging_deploy_map.xml"
             name="plugins_logging"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet deployable="no"
             installSet="parselib_deploy_map.xml"
             name="plugins_parselib"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet deployable="no"
             installSet="stringlib_deploy_map.xml"
             name="plugins_stringlib"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet deployable="no"
             installSet="unicodelib_deploy_map.xml"
             name="plugins_unicodelib"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet deployable="no"
             installSet="workunitservices_deploy_map.xml"
             name="plugins_workunitservices"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet installSet="roxie_deploy_map.xml"
             name="roxie"
             path="componentfiles/configxml"
             processName="RoxieCluster"
             schema="roxie.xsd"/>
   <BuildSet installSet="deploy_map.xml" name="roxieconfig" path="componentfiles/roxieconfig"/>
   <BuildSet installSet="deploy_map.xml"
             name="sasha"
             path="componentfiles/sasha"
             processName="SashaServerProcess"
             schema="sasha.xsd"/>
   <BuildSet deployable="no"
             installSet="deploy_map.xml"
             name="SiteCertificate"
             path="componentfiles/SiteCertificate"
             processName="SiteCertificate"
             schema="SiteCertificate.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="soapplus"
             path="componentfiles/soapplus"
             processName="SoapPlusProcess"
             schema="soapplus.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="thor"
             path="componentfiles/thor"
             processName="ThorCluster"
             schema="thor.xsd"/>
   <BuildSet deployable="no"
             installSet="deploy_map.xml"
             name="topology"
             path="componentfiles/topology"
             processName="Topology"
             schema="topology.xsd"/>
   <BuildSet deployable="no"
             installSet="deploy_map.xml"
             name="ws_ecl"
             path="componentfiles/ws_ecl"
             processName="EspService"
             schema="esp_service_wsecl2.xsd">
    <Properties bindingType="ws_eclSoapBinding"
                defaultPort="8002"
                defaultResourcesBasedn="ou=WsEcl,ou=EspServices,ou=ecl"
                defaultSecurePort="18002"
                plugin="ws_ecl"
                type="ws_ecl">
     <Authenticate access="Read"
                   description="Root access to WS ECL service"
                   path="/"
                   required="Read"
                   resource="WsEclAccess"/>
     <AuthenticateFeature description="Access to WS ECL service"
                          path="WsEclAccess"
                          resource="WsEclAccess"
                          service="ws_ecl"/>
    </Properties>
   </BuildSet>
   <BuildSet deployable="no"
             installSet="deploy_map.xml"
             name="ecldirect"
             path="componentfiles/ecldirect"
             processName="EspService"
             schema="esp_service_ecldirect.xsd">
    <Properties bindingType="EclDirectSoapBinding"
                defaultPort="8008"
                defaultResourcesBasedn="ou=EclDirectAccess,ou=EspServices,ou=ecl"
                defaultSecurePort="18008"
                plugin="ecldirect"
                type="ecldirect">
     <Authenticate access="Read"
                   description="Root access to ECL Direct service"
                   path="/"
                   required="Read"
                   resource="EclDirectAccess"/>
     <AuthenticateFeature description="Access to ECL Direct service"
                          path="EclDirectAccess"
                          resource="EclDirectAccess"
                          service="ecldirect"/>
    </Properties>
   </BuildSet>
  </Build>
 </Programs>
 <Software>
  <DafilesrvProcess build="community_3.0.4"
                    buildSet="dafilesrv"
                    description="DaFileSrv process"
                    name="mydafilesrv"
                    version="1">
#for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/mydafilesrv"
             name="s_$name"
             netAddress="$netAddress"/>
#end for             
  </DafilesrvProcess>
  <DaliServerProcess build="community_3.0.4"
                     buildSet="dali"
                     environment="/etc/HPCCSystems/environment.xml"
                     name="mydali"
                     recoverFromIncErrors="true">                     
   <Instance computer="$nodes[0][1]"
             directory="/var/lib/HPCCSystems/mydali"
             name="s_$nodes[0][1]"
             netAddress="$nodes[0][0]"
             port="7070"/>
  </DaliServerProcess>
  <DfuServerProcess build="community_3.0.4"
                    buildSet="dfuserver"
                    daliServers="mydali"
                    description="DFU Server"
                    monitorinterval="900"
                    monitorqueue="dfuserver_monitor_queue"
                    name="mydfuserver"
                    queue="dfuserver_queue"
                    transferBufferSize="65536">
#for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/mydfuserver"
             name="s_$name"
             netAddress="$netAddress"/>
#end for    
#raw        
   <SSH SSHidentityfile="$HOME/.ssh/id_rsa"
#end raw
        SSHpassword=""
        SSHretries="3"
        SSHtimeout="0"
        SSHusername="hpcc"/>
  </DfuServerProcess>
  <Directories name="HPCCSystems">
   <Category dir="/var/log/[NAME]/[INST]" name="log"/>
   <Category dir="/var/lib/[NAME]/[INST]" name="run"/>
   <Category dir="/etc/[NAME]/[INST]" name="conf"/>
   <Category dir="/var/lib/[NAME]/[INST]/temp" name="temp"/>
   <Category dir="/var/lib/[NAME]/hpcc-data/[COMPONENT]" name="data"/>
   <Category dir="/var/lib/[NAME]/hpcc-data2/[COMPONENT]" name="data2"/>
   <Category dir="/var/lib/[NAME]/hpcc-data3/[COMPONENT]" name="data3"/>
   <Category dir="/var/lib/[NAME]/hpcc-mirror/[COMPONENT]" name="mirror"/>
   <Category dir="/var/lib/[NAME]/queries/[INST]" name="query"/>
   <Category dir="/var/lock/[NAME]/[INST]" name="lock"/>
  </Directories>
  <DropZone build="community_3.0.4"
            buildSet="DropZone"
            computer="$nodes[0][0]"
            description="DropZone process"
            directory="/var/lib/HPCCSystems/dropzone"
            name="mydropzone"/>
  <EclAgentProcess allowedPipePrograms="*"
                   build="community_3.0.4"
                   buildSet="eclagent"
                   daliServers="mydali"
                   description="EclAgent process"
                   name="myeclagent"
                   pluginDirectory="/opt/HPCCSystems/plugins/"
                   thorConnectTimeout="600"
                   traceLevel="0"
                   wuQueueName="myeclagent_queue">
#for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/myeclagent"
             name="s_$name"
             netAddress="$netAddress"/>
#end for  
  </EclAgentProcess>
  <EclCCServerProcess build="community_3.0.4"
                      buildSet="eclccserver"
                      daliServers="mydali"
                      description="EclCCServer process"
                      enableSysLog="true"
                      maxCompileThreads="4"
                      name="myeclccserver"
                      traceLevel="1">
#for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/myeclccserver"
             name="s_$name"
             netAddress="$netAddress"/>
#end for
  </EclCCServerProcess>
  <EclSchedulerProcess build="community_3.0.4"
                       buildSet="eclscheduler"
                       daliServers="mydali"
                       description="EclScheduler process"
                       name="myeclscheduler">
#for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/myeclscheduler"
             name="s_$name"
             netAddress="$netAddress"/>
#end for
  </EclSchedulerProcess>
  <EspProcess build="community_3.0.4"
              buildSet="esp"
              componentfilesDir="/opt/HPCCSystems/componentfiles"
              daliServers="mydali"
              description="ESP server"
              enableSEHMapping="true"
              formOptionsAccess="false"
              httpConfigAccess="true"
              logLevel="1"
              logRequests="false"
              logResponses="false"
              maxBacklogQueueSize="200"
              maxConcurrentThreads="0"
              maxRequestEntityLength="8000000"
              name="myesp"
              perfReportDelay="60"
              portalurl="http://hpccsystems.com/download">
   <Authentication ldapAuthMethod="kerberos"
                   ldapConnections="10"
                   ldapServer=""
                   method="none"/>
   <EspBinding defaultForPort="true"
               defaultServiceVersion=""
               name="myespsmc"
               port="8010"
               protocol="http"
               resourcesBasedn="ou=SMC,ou=EspServices,ou=ecl"
               service="EclWatch"
               workunitsBasedn="ou=workunits,ou=ecl"
               wsdlServiceAddress="">
    <Authenticate access="Read"
                  description="Root access to SMC service"
                  path="/"
                  required="Read"
                  resource="SmcAccess"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to SMC service"
                         path="SmcAccess"
                         resource="SmcAccess"
                         service="ws_smc"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to thor queues"
                         path="ThorQueueAccess"
                         resource="ThorQueueAccess"
                         service="ws_smc"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to super computer environment"
                         path="ConfigAccess"
                         resource="ConfigAccess"
                         service="ws_config"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to DFU"
                         path="DfuAccess"
                         resource="DfuAccess"
                         service="ws_dfu"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to DFU XRef"
                         path="DfuXrefAccess"
                         resource="DfuXrefAccess"
                         service="ws_dfuxref"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to machine information"
                         path="MachineInfoAccess"
                         resource="MachineInfoAccess"
                         service="ws_machine"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to SNMP metrics information"
                         path="MetricsAccess"
                         resource="MetricsAccess"
                         service="ws_machine"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to remote execution"
                         path="ExecuteAccess"
                         resource="ExecuteAccess"
                         service="ws_machine"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to DFU workunits"
                         path="DfuWorkunitsAccess"
                         resource="DfuWorkunitsAccess"
                         service="ws_fs"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to DFU exceptions"
                         path="DfuExceptionsAccess"
                         resource="DfuExceptions"
                         service="ws_fs"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to spraying files"
                         path="FileSprayAccess"
                         resource="FileSprayAccess"
                         service="ws_fs"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to despraying of files"
                         path="FileDesprayAccess"
                         resource="FileDesprayAccess"
                         service="ws_fs"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to dkcing of key files"
                         path="FileDkcAccess"
                         resource="FileDkcAccess"
                         service="ws_fs"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to files in dropzone"
                         path="FileIOAccess"
                         resource="FileIOAccess"
                         service="ws_fileio"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to WS ECL service"
                         path="WsEclAccess"
                         resource="WsEclAccess"
                         service="ws_ecl"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to Roxie queries and files"
                         path="RoxieQueryAccess"
                         resource="RoxieQueryAccess"
                         service="ws_roxiequery"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to cluster topology"
                         path="ClusterTopologyAccess"
                         resource="ClusterTopologyAccess"
                         service="ws_topology"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to own workunits"
                         path="OwnWorkunitsAccess"
                         resource="OwnWorkunitsAccess"
                         service="ws_workunits"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to others&apos; workunits"
                         path="OthersWorkunitsAccess"
                         resource="OthersWorkunitsAccess"
                         service="ws_workunits"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to ECL direct service"
                         path="EclDirectAccess"
                         resource="EclDirectAccess"
                         service="ecldirect"/>
   </EspBinding>
   <EspBinding defaultForPort="true"
               defaultServiceVersion=""
               name="myws_ecl"
               port="8002"
               protocol="http"
               resourcesBasedn="ou=WsEcl,ou=EspServices,ou=ecl"
               service="myws_ecl"
               workunitsBasedn="ou=workunits,ou=ecl"
               wsdlServiceAddress="">
    <Authenticate access="Read"
                  description="Root access to WS ECL service"
                  path="/"
                  required="Read"
                  resource="WsEclAccess"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to WS ECL service"
                         path="WsEclAccess"
                         resource="WsEclAccess"
                         service="ws_ecl"/>
   </EspBinding>
   <EspBinding defaultForPort="true"
               defaultServiceVersion=""
               name="myecldirect"
               port="8008"
               protocol="http"
               resourcesBasedn="ou=EclDirectAccess,ou=EspServices,ou=ecl"
               service="myecldirect"
               workunitsBasedn="ou=workunits,ou=ecl"
               wsdlServiceAddress="">
    <Authenticate access="Read"
                  description="Root access to ECL Direct service"
                  path="/"
                  required="Read"
                  resource="EclDirectAccess"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to ECL Direct service"
                         path="EclDirectAccess"
                         resource="EclDirectAccess"
                         service="ecldirect"/>
   </EspBinding>
   <HTTPS acceptSelfSigned="true"
          CA_Certificates_Path="ca.pem"
          certificateFileName="certificate.cer"
          city=""
          country="US"
          daysValid="365"
          enableVerification="false"
          organization="Customer of HPCCSystems"
          organizationalUnit=""
          passphrase=""
          privateKeyFileName="privatekey.cer"
          regenerateCredentials="false"
          requireAddressMatch="false"
          state=""
          trustedPeers="anyone"/>
   #for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/myesp"
             FQDN=""
             name="s_$name"
             netAddress="$netAddress"/>
   #end for
   <ProtocolX authTimeout="3"
              defaultTimeout="21"
              idleTimeout="3600"
              maxTimeout="36"
              minTimeout="30"
              threadCount="2"/>
  </EspProcess>
  <EspService allowNewRoxieOnDemandQuery="false"
              AWUsCacheTimeout="15"
              build="community_3.0.4"
              buildSet="espsmc"
              description="ESP services for SMC"
              disableUppercaseTranslation="false"
              eclServer="myeclccserver"
              enableSystemUseRewrite="false"
              excludePartitions="/,/dev*,/sys,/usr,/proc/*"
              monitorDaliFileServer="false"
              name="EclWatch"
              pluginsPath="/opt/HPCCSystems/plugins"
              syntaxCheckQueue=""
              viewTimeout="1000"
              warnIfCpuLoadOver="95"
              warnIfFreeMemoryUnder="5"
              warnIfFreeStorageUnder="5">
   <Properties defaultPort="8010"
               defaultResourcesBasedn="ou=SMC,ou=EspServices,ou=ecl"
               defaultSecurePort="18010"
               type="WsSMC">
    <Authenticate access="Read"
                  description="Root access to SMC service"
                  path="/"
                  required="Read"
                  resource="SmcAccess"/>
    <AuthenticateFeature description="Access to SMC service"
                         path="SmcAccess"
                         resource="SmcAccess"
                         service="ws_smc"/>
    <AuthenticateFeature description="Access to thor queues"
                         path="ThorQueueAccess"
                         resource="ThorQueueAccess"
                         service="ws_smc"/>
    <AuthenticateFeature description="Access to super computer environment"
                         path="ConfigAccess"
                         resource="ConfigAccess"
                         service="ws_config"/>
    <AuthenticateFeature description="Access to DFU"
                         path="DfuAccess"
                         resource="DfuAccess"
                         service="ws_dfu"/>
    <AuthenticateFeature description="Access to DFU XRef"
                         path="DfuXrefAccess"
                         resource="DfuXrefAccess"
                         service="ws_dfuxref"/>
    <AuthenticateFeature description="Access to machine information"
                         path="MachineInfoAccess"
                         resource="MachineInfoAccess"
                         service="ws_machine"/>
    <AuthenticateFeature description="Access to SNMP metrics information"
                         path="MetricsAccess"
                         resource="MetricsAccess"
                         service="ws_machine"/>
    <AuthenticateFeature description="Access to remote execution"
                         path="ExecuteAccess"
                         resource="ExecuteAccess"
                         service="ws_machine"/>
    <AuthenticateFeature description="Access to DFU workunits"
                         path="DfuWorkunitsAccess"
                         resource="DfuWorkunitsAccess"
                         service="ws_fs"/>
    <AuthenticateFeature description="Access to DFU exceptions"
                         path="DfuExceptionsAccess"
                         resource="DfuExceptions"
                         service="ws_fs"/>
    <AuthenticateFeature description="Access to spraying files"
                         path="FileSprayAccess"
                         resource="FileSprayAccess"
                         service="ws_fs"/>
    <AuthenticateFeature description="Access to despraying of files"
                         path="FileDesprayAccess"
                         resource="FileDesprayAccess"
                         service="ws_fs"/>
    <AuthenticateFeature description="Access to dkcing of key files"
                         path="FileDkcAccess"
                         resource="FileDkcAccess"
                         service="ws_fs"/>
    <AuthenticateFeature description="Access to files in dropzone"
                         path="FileIOAccess"
                         resource="FileIOAccess"
                         service="ws_fileio"/>
    <AuthenticateFeature description="Access to WS ECL service"
                         path="WsEclAccess"
                         resource="WsEclAccess"
                         service="ws_ecl"/>
    <AuthenticateFeature description="Access to Roxie queries and files"
                         path="RoxieQueryAccess"
                         resource="RoxieQueryAccess"
                         service="ws_roxiequery"/>
    <AuthenticateFeature description="Access to cluster topology"
                         path="ClusterTopologyAccess"
                         resource="ClusterTopologyAccess"
                         service="ws_topology"/>
    <AuthenticateFeature description="Access to own workunits"
                         path="OwnWorkunitsAccess"
                         resource="OwnWorkunitsAccess"
                         service="ws_workunits"/>
    <AuthenticateFeature description="Access to others&apos; workunits"
                         path="OthersWorkunitsAccess"
                         resource="OthersWorkunitsAccess"
                         service="ws_workunits"/>
    <AuthenticateFeature description="Access to ECL direct service"
                         path="EclDirectAccess"
                         resource="EclDirectAccess"
                         service="ecldirect"/>
    <ProcessFilters>
     <Platform name="Windows">
      <ProcessFilter name="any">
       <Process name="dafilesrv"/>
      </ProcessFilter>
      <ProcessFilter name="AttrServerProcess">
       <Process name="attrserver"/>
      </ProcessFilter>
      <ProcessFilter name="DaliProcess">
       <Process name="daserver"/>
      </ProcessFilter>
      <ProcessFilter multipleInstances="true" name="DfuServerProcess">
       <Process name="dfuserver"/>
      </ProcessFilter>
      <ProcessFilter multipleInstances="true" name="EclCCServerProcess">
       <Process name="eclccserver"/>
      </ProcessFilter>
      <ProcessFilter multipleInstances="true" name="EspProcess">
       <Process name="esp"/>
       <Process name="dafilesrv" remove="true"/>
      </ProcessFilter>
      <ProcessFilter name="FTSlaveProcess">
       <Process name="ftslave"/>
      </ProcessFilter>
      <ProcessFilter name="RoxieServerProcess">
       <Process name="ccd"/>
      </ProcessFilter>
      <ProcessFilter name="RoxieSlaveProcess">
       <Process name="ccd"/>
      </ProcessFilter>
      <ProcessFilter name="SchedulerProcess">
       <Process name="scheduler"/>
      </ProcessFilter>
      <ProcessFilter name="ThorMasterProcess">
       <Process name="thormaster"/>
      </ProcessFilter>
      <ProcessFilter name="ThorSlaveProcess">
       <Process name="thorslave"/>
      </ProcessFilter>
      <ProcessFilter name="SashaServerProcess">
       <Process name="saserver"/>
      </ProcessFilter>
     </Platform>
     <Platform name="Linux">
      <ProcessFilter name="any">
       <Process name="dafilesrv"/>
      </ProcessFilter>
      <ProcessFilter name="AttrServerProcess">
       <Process name="attrserver"/>
      </ProcessFilter>
      <ProcessFilter name="DaliProcess">
       <Process name="daserver"/>
      </ProcessFilter>
      <ProcessFilter multipleInstances="true" name="DfuServerProcess">
       <Process name="."/>
      </ProcessFilter>
      <ProcessFilter multipleInstances="true" name="EclCCServerProcess">
       <Process name="."/>
      </ProcessFilter>
      <ProcessFilter multipleInstances="true" name="EspProcess">
       <Process name="."/>
       <Process name="dafilesrv" remove="true"/>
      </ProcessFilter>
      <ProcessFilter name="FTSlaveProcess">
       <Process name="ftslave"/>
      </ProcessFilter>
      <ProcessFilter name="GenesisServerProcess">
       <Process name="httpd"/>
       <Process name="atftpd"/>
       <Process name="dhcpd"/>
      </ProcessFilter>
      <ProcessFilter name="RoxieServerProcess">
       <Process name="ccd"/>
      </ProcessFilter>
      <ProcessFilter name="RoxieSlaveProcess">
       <Process name="ccd"/>
      </ProcessFilter>
      <ProcessFilter name="SchedulerProcess">
       <Process name="scheduler"/>
      </ProcessFilter>
      <ProcessFilter name="ThorMasterProcess">
       <Process name="thormaster"/>
      </ProcessFilter>
      <ProcessFilter name="ThorSlaveProcess">
       <Process name="thorslave"/>
      </ProcessFilter>
      <ProcessFilter name="SashaServerProcess">
       <Process name="saserver"/>
      </ProcessFilter>
     </Platform>
    </ProcessFilters>
   </Properties>
  </EspService>
  <EspService build="community_3.0.4"
              buildSet="ws_ecl"
              description="WS ECL Service"
              name="myws_ecl">
   <Properties bindingType="ws_eclSoapBinding"
               defaultPort="8002"
               defaultResourcesBasedn="ou=WsEcl,ou=EspServices,ou=ecl"
               defaultSecurePort="18002"
               plugin="ws_ecl"
               type="ws_ecl">
    <Authenticate access="Read"
                  description="Root access to WS ECL service"
                  path="/"
                  required="Read"
                  resource="WsEclAccess"/>
    <AuthenticateFeature description="Access to WS ECL service"
                         path="WsEclAccess"
                         resource="WsEclAccess"
                         service="ws_ecl"/>
   </Properties>
  </EspService>
  <EspService build="community_3.0.4"
              buildSet="ecldirect"
              clusterName="hthor"
              description="ESP service for running raw ECL queries"
              name="myecldirect">
   <Properties bindingType="EclDirectSoapBinding"
               defaultPort="8008"
               defaultResourcesBasedn="ou=EclDirectAccess,ou=EspServices,ou=ecl"
               defaultSecurePort="18008"
               plugin="ecldirect"
               type="ecldirect">
    <Authenticate access="Read"
                  description="Root access to ECL Direct service"
                  path="/"
                  required="Read"
                  resource="EclDirectAccess"/>
    <AuthenticateFeature description="Access to ECL Direct service"
                         path="EclDirectAccess"
                         resource="EclDirectAccess"
                         service="ecldirect"/>
   </Properties>
  </EspService>
  <FTSlaveProcess build="community_3.0.4"
                  buildSet="ftslave"
                  description="FTSlave process"
                  name="myftslave"
                  version="1">
#for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/myftslave"
             name="s_$name"
             netAddress="$netAddress"
             program="/opt/HPCCSystems/bin/ftslave"/>
#end for
  </FTSlaveProcess>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_auditlib"
                 description="plugin process"
                 name="myplugins_auditlib"/>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_debugservices"
                 description="plugin process"
                 name="myplugins_debugservices"/>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_fileservices"
                 description="plugin process"
                 name="myplugins_fileservices"/>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_logging"
                 description="plugin process"
                 name="myplugins_logging"/>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_parselib"
                 description="plugin process"
                 name="myplugins_parselib"/>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_stringlib"
                 description="plugin process"
                 name="myplugins_stringlib"/>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_unicodelib"
                 description="plugin process"
                 name="myplugins_unicodelib"/>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_workunitservices"
                 description="plugin process"
                 name="myplugins_workunitservices"/>
  <RoxieCluster allowRoxieOnDemand="false"
                baseDataDir="/var/lib/HPCCSystems/hpcc-data/roxie"
                blindLogging="false"
                blobCacheMem="0"
                build="community_3.0.4"
                buildSet="roxie"
                callbackRetries="3"
                callbackTimeout="500"
                channel0onPrimary="true"
                checkCompleted="true"
                checkFileDate="true"
                checkingHeap="0"
                checkPrimaries="true"
                checkState="false"
                checkVersion="true"
                clusterWidth="$len($nodes)"
                copyResources="true"
                crcResources="false"
                cyclicOffset="1"
                dafilesrvLookupTimeout="10000"
                daliServers="mydali"
                debugPermitted="true"
                defaultConcatPreload="0"
                defaultFetchPreload="0"
                defaultFullKeyedJoinPreload="0"
                defaultHighPriorityTimeLimit="0"
                defaultHighPriorityTimeWarning="5000"
                defaultKeyedJoinPreload="0"
                defaultLowPriorityTimeLimit="0"
                defaultLowPriorityTimeWarning="0"
                defaultMemoryLimit="0"
                defaultParallelJoinPreload="0"
                defaultPrefetchProjectPreload="10"
                defaultSLAPriorityTimeLimit="0"
                defaultSLAPriorityTimeWarning="5000"
                defaultStripLeadingWhitespace="1"
                deleteUnneededFiles="false"
                description="Roxie cluster"
                directory="/var/lib/HPCCSystems/myroxie"
                diskReadBufferSize="65536"
                diskReadStable="true"
                doIbytiDelay="true"
                enableForceKeyDiffCopy="false"
                enableHeartBeat="true"
                enableKeyDiff="true"
                enableSNMP="true"
                enableSysLog="true"
                fastLaneQueue="true"
                fieldTranslationEnabled="false"
                flushJHtreeCacheOnOOM="true"
                forceStdLog="false"
                highTimeout="2000"
                ignoreMissingFiles="false"
                indexReadChunkSize="60000"
                indexReadStable="true"
                initIbytiDelay="100"
                jumboFrames="false"
                keyedJoinFlowLimit="1000"
                keyedJoinStable="true"
                lazyOpen="false"
                leafCacheMem="50"
                linuxYield="true"
                localFilesExpire="-1"
                localSlave="false"
                logFullQueries="false"
                logQueueDrop="32"
                logQueueLen="512"
                lowTimeout="10000"
                maxBlockSize="10000000"
                maxLocalFilesOpen="4000"
                maxLockAttempts="5"
                maxRemoteFilesOpen="1000"
                memoryStatsInterval="60"
                memTraceLevel="1"
                memTraceSizeLimit="0"
                minFreeDiskSpace="1073741824"
                minIbytiDelay="0"
                minLocalFilesOpen="2000"
                minRemoteFilesOpen="500"
                miscDebugTraceLevel="0"
                monitorDaliFileServer="false"
                multicastBase="239.1.1.1"
                multicastLast="239.1.254.254"
                name="myroxie"
                nodeCacheMem="100"
                nodeCachePreload="false"
                numChannels="$len($nodes)"
                numDataCopies="2"
                parallelAggregate="0"
                perChannelFlowLimit="10"
                pingInterval="60"
                pluginsPath="/opt/HPCCSystems/plugins"
                preabortIndexReadsThreshold="100"
                preabortKeyedJoinsThreshold="100"
                preferredSubnet=""
                preferredSubnetMask=""
                remoteFilesExpire="3600000"
                resolveFilesInPackage="false"
                roxieMulticastEnabled="true"
                serverSideCacheSize="0"
                serverThreads="30"
                simpleLocalKeyedJoins="true"
                siteCertificate=""
                slaTimeout="2000"
                slaveConfig="cyclic redundancy"
                slaveThreads="30"
                smartSteppingChunkRows="100"
                soapTraceLevel="1"
                socketCheckInterval="5000"
#raw                
                SSHidentityfile="$HOME/.ssh/id_rsa"
#end raw                
                SSHpassword=""
                SSHretries="3"
                SSHtimeout="0"
                SSHusername="hpcc"
                statsExpiryTime="3600"
                syncCluster="false"
                systemMonitorInterval="60000"
                totalMemoryLimit="1073741824"
                traceLevel="1"
                trapTooManyActiveQueries="true"
                udpFlowSocketsSize="131071"
                udpInlineCollation="false"
                udpInlineCollationPacketLimit="50"
                udpLocalWriteSocketSize="131071"
                udpMaxRetryTimedoutReqs="0"
                udpMaxSlotsPerClient="2147483647"
                udpMulticastBufferSize="131071"
                udpOutQsPriority="0"
                udpQueueSize="100"
                udpRequestToSendTimeout="5"
                udpResendEnabled="true"
                udpRetryBusySenders="0"
                udpSendCompletedInData="false"
                udpSendQueueSize="50"
                udpSnifferEnabled="true"
                udpTraceLevel="1"
                useHardLink="false"
                useLogQueue="true"
                useMemoryMappedIndexes="false"
                useRemoteResources="true"
                useTreeCopy="false">
   <RoxieFarmProcess dataDirectory="/var/lib/HPCCSystems/hpcc-data/roxie"
                     listenQueue="200"
                     name="farm1"
                     numThreads="30"
                     port="9876"
                     requestArrayThreads="5">
#for $netAddress, $name in $nodes:                    
    <RoxieServerProcess computer="$name" name="farm1_$name"/>
#end for
   </RoxieFarmProcess>
#for $netAddress, $name in $nodes:                    
   <RoxieServerProcess computer="$name"
                       dataDirectory="/var/lib/HPCCSystems/hpcc-data/roxie"
                       listenQueue="200"
                       name="farm1_s_$name"
                       netAddress="$netAddress"
                       numThreads="30"
                       port="9876"
                       requestArrayThreads="5"/>
#end for
#for $netAddress, $name in $nodes:                    
   <RoxieSlave computer="$name" name="s_$name">
    <RoxieChannel dataDirectory="/var/lib/HPCCSystems/hpcc-data/roxie" number="$random.randint(1,$len($nodes))"/>
    <RoxieChannel dataDirectory="/var/lib/HPCCSystems/hpcc-data2/roxie" number="$random.randint(1,$len($nodes))"/>
   </RoxieSlave>
#end for
#set $channel = 1
#for $netAddress, $name in $nodes:                    
   <RoxieSlaveProcess channel="$channel"
                      computer="$name"
                      dataDirectory="/var/lib/HPCCSystems/hpcc-data/roxie"
                      name="s_$name"
                      netAddress="$netAddress"/>
#set $channel = $channel + 1                      
#end for
  </RoxieCluster>
  <SashaServerProcess autoRestartInterval="0"
                      build="community_3.0.4"
                      buildSet="sasha"
                      cachedWUat="* * * * *"
                      cachedWUinterval="24"
                      cachedWUlimit="100"
                      coalesceAt="* * * * *"
                      coalesceInterval="1"
                      dafsmonAt="* * * * *"
                      dafsmonInterval="0"
                      dafsmonList="*"
                      daliServers="mydali"
                      description="Sasha Server process"
                      DFUrecoveryAt="* * * * *"
                      DFUrecoveryCutoff="4"
                      DFUrecoveryInterval="12"
                      DFUrecoveryLimit="20"
                      DFUWUat="* * * * *"
                      DFUWUcutoff="14"
                      DFUWUduration="0"
                      DFUWUinterval="24"
                      DFUWUlimit="1000"
                      DFUWUthrottle="0"
                      ExpiryAt="* 3 * * *"
                      ExpiryInterval="24"
                      keepResultFiles="false"
                      LDSroot="LDS"
                      logDir="."
                      minDeltaSize="50000"
                      name="mysasha"
                      recoverDeltaErrors="false"
                      thorQMonInterval="1"
                      thorQMonQueues="*"
                      thorQMonSwitchMinTime="0"
                      WUat="* * * * *"
                      WUbackup="0"
                      WUcutoff="8"
                      WUduration="0"
                      WUinterval="6"
                      WUlimit="1000"
                      WUretryinterval="7"
                      WUthrottle="0"
                      xrefAt="* 2 * * *"
                      xrefCutoff="1"
                      xrefEclWatchProvider="true"
                      xrefInterval="0"
                      xrefList="*">
#for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/mysasha"
             name="s_$name"
             netAddress="$netAddress"
             port="8877"/>
#end for             
  </SashaServerProcess>
  <ThorCluster autoCopyBackup="false"
               build="community_3.0.4"
               buildSet="thor"
               computer="$nodes[0][1]"
               daliServers="mydali"
               description="Thor process"
               localThor="false"
               monitorDaliFileServer="true"
               multiSlaves="false"
               name="mythor"
               pluginsPath="/opt/HPCCSystems/plugins/"
               replicateAsync="true"
               replicateOutputs="true"
               slaves="8"
               watchdogEnabled="true"
               watchdogProgressEnabled="true">
   <Debug/>
#raw   
   <SSH SSHidentityfile="$HOME/.ssh/id_rsa"
#end raw   
        SSHpassword=""
        SSHretries="3"
        SSHtimeout="0"
        SSHusername="hpcc"/>
   <Storage/>
   <SwapNode/>
   <ThorMasterProcess computer="$nodes[0][1]" name="m_$nodes[0][1]"/>
#if $len($nodes) > 1:
#for $netAddress, $name in $nodes[1:]:                    
   <ThorSlaveProcess computer="$name" name="s_$name"/>
#end for   
#end if
   <Topology>
    <Node process="m_$nodes[0][1]">
#if $len($nodes) > 1:
#for $netAddress, $name in $nodes[1:]:                    
     <Node process="s_$name"/>
#end for   
#end if
    </Node>
   </Topology>
  </ThorCluster>
  <Topology build="community_3.0.4" buildSet="topology" name="topology">
   <Cluster name="hthor" prefix="hthor">
    <EclAgentProcess process="myeclagent"/>
    <EclCCServerProcess process="myeclccserver"/>
    <EclSchedulerProcess process="myeclscheduler"/>
   </Cluster>
   <Cluster name="thor" prefix="thor">
    <EclAgentProcess process="myeclagent"/>
    <EclCCServerProcess process="myeclccserver"/>
    <EclSchedulerProcess process="myeclscheduler"/>
    <ThorCluster process="mythor"/>
   </Cluster>
   <Cluster name="roxie" prefix="roxie">
    <EclAgentProcess process="myeclagent"/>
    <EclCCServerProcess process="myeclccserver"/>
    <EclSchedulerProcess process="myeclscheduler"/>
    <RoxieCluster process="myroxie"/>
   </Cluster>
  </Topology>
 </Software>
</Environment>
====


Even just scrolling past the file takes a while!!  This behemoth of a file was tamed thanks to cheetah, I highly encourage you to read up on it.

This formula may require some changes to your environment.yaml file in ~/.ensemble as hpcc will only run on 64-bit instances.  Make sure that your ensemble environment has been properly shutdown before you edit this file ( ensemble shutdown ).  Here is my environment.yaml file where I show you the important part to check:
ensemble: environments

environments:
  sample:
    type: ec2
    access-key: ( removed ... get your own :) )
    secret-key: ( removed ... get your own :) )
    control-bucket: ensemble-fbb790f292e14a0394353bb4b63a3403
    admin-secret: 604d18a77fd24e3f91e1df398fcbe9f2
    default-image-id: ami-8303c5ea
    default-instance-type: m1.large
The emphasized parts are the important ones.  You can just copy them from here and paste them into your ~/.ensemble/environment.yaml file.
Now, let's take a look at the formula starting with the metadata.yaml file:
ensemble: formula
name: hpcc
revision: 1
summary: HPCC (High Performance Computing Cluster)
description: |
  HPCC (High Performance Computing Cluster) is a massive 
  parallel-processing computing platform that solves Big Data problems.
provides:
  hpcc:
    interface: hpcc
requires:
  hpcc-thor:
    interface: hpcc-thor
  hpcc-roxie:
    interface: hpcc-roxie
peers:
  hpcc-cluster:
    interface: hpcc-cluster

 There are various provides and requires interfaces in this metadata.yaml file but, for now, only the peers interface is being used.  I'll work on the other ones as the formula matures.


Let's look at the hpcc-cluster interface.  More specifically the hpcc-cluster-relation-changed hook where the new configuration is created:
#!/bin/bash
CWD=$(dirname $0)
cheetah fill --oext=xml --odir=/etc/HPCCSystems/ ${CWD}/../templates/environment.tmpl
service hpcc-init restart
It's pretty simple isn't it?   Since the "heavy lifting" is being done with the self contained cheetah template , we don't have much to do here but, to generate the configuration file and restart hpcc.

The other files in this formula are pretty self explanatory and simple so, I am leaving the details of them as an exercise to the reader.

All of the complexities in hpcc has been distilled to the following commands:

  • ensemble bootstrap
  • bzr branch lp:~negronjl/+junk/hpcc
  • ensemble deploy --repository . hpcc
    • wait a minute of two
  • ensemble status
    • you should see something similar to this:

negronjl@negronjl-laptop:~/src/ensemble/formulas$ ensemble status
2011-08-18 16:00:54,413 INFO Connecting to environment.


machines: 
  0: {dns-name: ec2-184-73-109-244.compute-1.amazonaws.com, instance-id: i-6d61460c} 
  1: {dns-name: ec2-50-16-60-94.compute-1.amazonaws.com, instance-id: i-d5694eb4}


services:
  hpcc:
    formula: local:hpcc-1
    relations: {hpcc-cluster: hpcc}
    units:
      hpcc/0:
        machine: 1
        relations: {}
        state: null


2011-08-18 16:00:58,374 INFO 'status' command finished successfully
negronjl@negronjl-laptop:~/src/ensemble/formulas$
The above commands, will give you a single node.

You can access the web interface of your node by pointing your browser to http://<FQDN>:8010 Where FQDN is the Fully Qualified Domain Name or Public IP Address of your hpcc instance.  On the left side, there should be a menu, explore the items on the Topology section.  The Target Clusters section should look something similar to this:


To experience the true power of hpcc, you should probably throw in some more nodes at it.  Let's do just that with:


  • ensemble add-unit hpcc 
    • do this as many times as you feel comfortable   
    • Each command will give you a new node in the cluster
    • wait a minute or two and you should see something similar to this:

negronjl@negronjl-laptop:~$ ensemble status
2011-08-18 16:25:55,739 INFO Connecting to environment.
machines:
  0: {dns-name: ec2-184-73-109-244.compute-1.amazonaws.com, instance-id: i-6d61460c}
  1: {dns-name: ec2-50-16-60-94.compute-1.amazonaws.com, instance-id: i-d5694eb4}
  2: {dns-name: ec2-50-19-181-98.compute-1.amazonaws.com, instance-id: i-a5795ec4}
  3: {dns-name: ec2-184-72-147-67.compute-1.amazonaws.com, instance-id: i-25446344}
services:
  hpcc:
    formula: local:hpcc-1
    relations: {hpcc-cluster: hpcc}
    units:
      hpcc/0:
        machine: 1
        relations:
          hpcc-cluster: {state: up}
        state: started
      hpcc/1:
        machine: 2
        relations:
          hpcc-cluster: {state: up}
        state: started
      hpcc/2:
        machine: 3
        relations:
          hpcc-cluster: {state: up}
        state: started
2011-08-18 16:26:01,837 INFO 'status' command finished successfully
Notice how we now have more hpcc nodes :)  Here is what the web interface could look like:


Again....we have more nodes :)

Now that we have a working cluster, let's try it.  We'll first do the mandatory Hello World in ECL.  It looks something like this (hello.ecl):
Output('Hello world');
 We have to compile our hello.ecl so we can use it.  We do that by logging into one of the nodes ( I used ensemble ssh 1 to log on to the first/master node ) and typing the following:
eclcc hello.ecl -o
We run the file just like we would any other binary:
./hello
... and the output is:
ubuntu@ip-10-111-19-210:~$ ./hello
Hello world
ubuntu@ip-10-111-19-210:~$ 
There are far more interesting examples in the Learning ECL Documentation here.  I highly encourage you to go and read about it.

That's it for now.  Feedback is always welcome of course so, let me know how I'm doing.

-Juan


PowerNap Your Data Center! (LinuxCon 2011 Vancouver)

// August 18th, 2011 // 3 Comments » // Uncategorized


I was honored to speak at LinuxCon North America in beautiful Vancouver yesterday, about one of my favorite topics -- energy efficiency opportunities using Ubuntu Servers in the data center (something I've blogged before).

I'm pleased to share those slides with you today!  The talk is entitled PowerNap Your Data Center, and focused on Ubuntu's innovative PowerNap suite, from the system administrator's or data center manager's perspective.

We discussed the original, Cloud motivations for PowerNap, its evolution from the basic process monitoring and suspend/hibernate methods of PowerNap1, to our complete rewrite of PowerNap2 (thanks, Andres!) which added nearly a dozen monitors and the ubiquitously useful PowerSave mode.  PowerNap is now more useful and configurable than ever!

Flip through the presentation below, or download the PDF here.



Get Adobe Flash player


Stay tuned for another PowerNap presentation I'm giving at Linux Plumbers next month in California.  That one should be a bit deeper dive into the technical implementation, and hopefully generate some plumbing layer discussion and improvement suggestions.

:-Dustin

Testing of OpenStack

// August 18th, 2011 // Comments Off // Uncategorized

ReadFeeder Beta begins

// August 18th, 2011 // Comments Off // Uncategorized

I started sending out ReadFeeder.net beta invitations this week!  I won’t lie and say it’s been smooth sailing, but the bugs that have been caught were easily fixed, and I’ve personally fully switch over to it.  Since I documented the entire journey of creating ReadFeeder, it seems fitting to document this next stage in it’s life.  So on that note, this is the first of a continuation of a series involving the life and times of ReadFeeder.net

Bugs

So it turns out there were still some lingering bugs from the Big Refactor, which caused things like the OPML import to not work properly, and displaying the feed’s original name instead of the name the user gave it.  Fixing those caused a couple of other bugs to appear, specifically regarding the Queue Bookmarks.  But those have all been fixed now, and I’m pleased to say that things are currently quiet on that front.

Feed Updating

During the Big Refactor I put in place some extra fields to be used during the regularly scheduled feed updates that would help make the update process smarter.  Amazon’s t1.micro instances don’t give you dedicated CPU time, but they do let you “burst” up to as many as two processing “units” for short periods of time.  In order to stop any one person’s apps from hogging the CPU all the time, the t1.micro instance will penalize you if you use it too much or too often.  Previously this meant that I had to slow down and space out my updates, which meant that it would take up to several hours to finish updating all the feeds, otherwise Amazon would stick me in the penalty box.

I initially started ReadFeeder.net on the same schedule, updating all the feeds approximately every two hours, which works up until it reaches about 400 feeds.  Even after adding only a few people to the beta program, it quickly surpassed 500 feeds.  Some of the Big Refactor helped here, since two or more people subscribed to the same feed wouldn’t cause it to update the feed multiple times, but because I selected beta users of different backgrounds and reading habits, the overlap wasn’t that high.

But, I was now collecting both the last time a feed was checked, and the last time that feed had a new article in it.  Using these two fields, the update process will now calculate a “staleness” factor for a feed,  which is just the time between it’s last article and the last time ReadFeeder checked it (how long since it was “fresh”).  It also knows the “age” of a feed, basically the time since it’s last check.  The current, and very simple, algorithm has ReadFeeder only checking for updates to a feed if it’s “age” is more than half of it’s “staleness”.  That means that if a feed hasn’t been updated in an hour, it’ll be checked again in 30 minutes.  However, if it wasn’t updated in the last day, it won’t be checked again for another 12 hours.

The effect was nothing short of astonishing.  Rather than checking 500 feeds every 2 hours, ReadFeeder now checks between 10 and 20 feeds every 10 minutes, and the CPU load has been barely high enough to register on Amazon’s monitoring charts, and is far from hitting the penalty limits.  Yet, at the same time, feeds that produce a large quantity of content are being updated more often than before.  And because ReadFeeder always places recently imported articles at the top of your list, even if a stale site isn’t checked as often, those articles will still show up at the top even if they were published before others from a feed that gets checked more often.  It’s a small tradeoff in latency on articles from slow feeds, but a huge win for latency on articles from heavy feeds, and makes an even bigger impact on the amount of users that can be supported on the same resources.

I plan on continuing to monitor and tweak the updating algorithm to squeeze as much performance out of ReadFeeder as I can, while improving the user experience, but this has already been such a massive improvement that I will be able to start adding beta testers at a much faster pace than I had expected.  So whether you are a heavy news reader or just looking for a different way of managing a smaller number of feeds, please take a minute and signup for the Beta program.

Cloud Foundry Platform as a Service (PaaS) in Ubuntu 11.10

// August 17th, 2011 // Comments Off // Uncategorized

Great news that Ubuntu 11.10 will include Cloud Foundry, an open source Platform as a Service (PaaS) that enables developers to build, deploy and run Cloud applications. Other PaaS offerings have typically locked you into using specific frameworks and services defined by the PaaS vendor as well as proprietary host platforms. This has made migrating applications between different providers as well as moving applications back into your own data center difficult if not impossible.

Cloud Foundry is the world’s first Open PaaS with your choice of frameworks, application services and Cloud deployment platforms. Cloud Foundry is made to be extensible so that while Spring, Rails, Sinatra and Node.js apps are supported today more frameworks can be added in the future as they gain in popularity. Even more exciting is that you will be able to run your PaaS in any cloud or behind your own firewall if you choose. No lock-in to a single development framework or Cloud vendor!

In Ubuntu 11.10 we’ve added client and server deployment tools using Ensemble that allow you to easily deploy a single node server in minutes as well as a distributed, multi node environment quickly and easily to create a production quality PaaS. You can deploy applications in AWS, Openstack or on your own internal servers.

Stay tuned as we’ll be doing more exciting work with Cloud Foundry. We encourage you to check out Cloud Foundry in Ubuntu 11.10 and learn more about the project at www.cloudfoundry.org

Cloud Foundry Community Logo

 

Ensemble Progress: Sprintin’ in Austin, y’all

// August 16th, 2011 // 2 Comments » // Uncategorized

This past week the Ensemble team sprinted in Austin, Texas. The sprint was colocated with members from the Ubuntu server team working on Orchestra and LXC containers. Here’s the summary of the work so far.

Ensemble Core

  • Jim Baker landed the “expose” and “unexpose” functionality, allowing you to control which ports are exposed on your instances easily. Here’s an overview.
  • William Reade worked on the orchestra provider, it can now communicate with a Cobbler server and launch machines. This is an important step to managing physical hardware using Ensemble
  • Benjamin Saller merged his branch to allow for `–placement
    ` on deploy and add-unit where policy determines how units are allocated relative to machines. Currently this defaults to the existing policy of using unassigned machines for new units, but provides the options of `local` style placement which places new units on a single initial node. This will help facilitate local formula develop ment through later branches.
  • Kapil Thangavelu continues his security work, he’s merged in his security-otp-principal branch that stores the credentials for a named persistent principal in a node protected by an ACL referencing a separate OTP credentials as well as implementing security groups as a stored/persistant principal with membership denoting ACL to utilize the group node.
  • Kapil Thangavelu implemented an ACL abstraction for manipulating a zookeeper’s node ACL.
  • Kapil and Ben Saller also worked on local development and LXC support.
  • Dustin Kirkland and Andres Rodriguez spent the week developing the Orchestra/Ensemble integration, from installing the initial Orchestra server, to deploying Ubuntu onto physical nodes, and eventually bootstrapping Ensemble and deploying formulas to the baremetal. They used a series of Dell Vostro laptops (our portable cloud) to do this development:

  • Mark Mims started figuring out how to spell out formula interfaces… making it easier for users to figure out how to plug various formulas together within an ensemble
  • Clint Byrum and Mark Mims started expanding openstack testing.

You can keep an eye on the current set of active reviews to see what else is coming down the pipe.

New Formulas

Cool new formulas in progress and tutorials:

The name “principia” will be dropped and instead the collection of formulas will just be called “ensemble formula collection“, though this rename is not complete.

Are you deploying services on Ubuntu? We could always se help from people who want to write formulas for software they care about. We;ve got instructions on how to get started writing formulas, as well as a list of easy bitesize bugs for formulas that should be relatively straight forward.

Are you or your team writing something that runs on EC2? We’re here to help make it easy for people to try your software, feel free to contact the list and we’ll get started!

Ubuntu Server for ARM processors

// August 16th, 2011 // 1 Comment » // Uncategorized

Fitting more computing capacity into a limited power envelope is one of the key challenges facing data centre designers today. It impacts companies whether they are building a £500 million data centre or simply working how much compute that can fit into a couple of racks at a shared facility.

One of the emerging approaches to solving this problem is to look at the technologies in low power-consumption appliances like phones and applying them to dense clusters in server-like configurations. Whether it is in smartphones, tablets or other embedded systems, the processor at the heart of these low power devices is generally ARM-based.

With Ubuntu Server becoming the de-facto standard for cloud infrastructure and big data solutions, we recognise that power consumption is key to efficient scaling. Building on four years of working with ARM, we are now taking the step of supporting Ubuntu Server on ARM.  We expect these processors to be used in a variety of use cases including microservers.

This is a first step and there will be many revisions of processors, hardware designs and of software as the performance and supported server workloads optimised for ARM grow over the next four years. It is, however, a first crucial step towards a new technology and one where yet again open-source innovation leads.

The new addition to the Ubuntu family | First release in October 2011

In October, the Ubuntu Server 11.10 release will be simultaneously available for x86, x86-64 and ARM-based architectures. The base image of the releases will be the same across architectures with a common kernel baseline. The ARM architecture will also be part of the long-term support (LTS) version of Ubuntu Server in 12.04 and other future releases.

Initial development focus and optimisation will be around the most popular Ubuntu workloads of web/network infrastructure and distributed data processing via NoSQL or big data applications where workloads typically use hundreds or thousands of systems.

Get involved

The Ubuntu Server on ARM initiative is a multi-year initiative gathering broad participation. More project information can be found on the Ubuntu wiki

Hardware partners, ISVs and open-source application developers keen to join existing partners around Ubuntu Server on ARM should contact us via our partner enquiry form.