Ensemble: the Service Orchestration framework for hard core DevOps

// August 19th, 2011 // Comments Off // Uncategorized

sudo apt-get install your-web-app

...

sudo apt-get errrrrrr....yeah, not gonna work :-(

ensemble bootstrap

ensemble deploy your-web-app

... 

HPCC with Ubuntu Server and Ensemble

// August 19th, 2011 // Comments Off // Uncategorized

 Now that we are all caught up, let's delve right into it.  I will be discussing the details of my newly created hpcc juju charm.

The hpcc charm has been one of the trickiest one to date to get working properly so, I want to take some time to explain some of the challenges that I encountered.

hpcc seems to use ssh keys for authentication and a single xml file to hold it's configuration.  All nodes that are part of the cluster should have identical keys and xml configuration file.

• The ssh keys are pretty easy to do ( there is even a script that will do it all for you located at /opt/HPCCSystems/sbin/keygen.sh ).  You can just run: ssh-keygen -f path_where_to_save_keys/id_rsa -N "" -q
• The configuration file environment.xml is a lot trickier to configure so, I will use cheetah templates to help make a template out of this enormous file.  
• According to their website:
• Cheetah is an open source template engine and code generation tool, written in Python. It can be used standalone or combined with other tools and frameworks. Web development is its principle use, but Cheetah is very flexible and is also being used to generate C++ game code, Java, sql, form emails and even Python code.

With cheetah, I can create self contained templates that can be generated into their intended file by just calling cheetah.  This is because we can embed python code inside the template itself, making the template ( environment.tmpl in our case ) more or less a python program that generates a fully functional environment.xml file ready for hpcc to use.

Another, very important, reason to use a template engine is the ability to create identical configuration files from each node without having to pass them around.  In other words, each node can create it's own configuration file and, since all nodes are using the same methods and data to create the file, they will all be exactly the same.

The hpcc configuration file is huge so, I'll just talk about some of the interesting bits of it here:

#import random

#import subprocess

#set $rel_structure = { $subprocess.check_output(['facter', 'install_time']).strip() : { 'name' : $subprocess.check_output(['hostname', '-f']).strip(), 'netAddress' : $subprocess.check_output(['facter','ipaddress']).strip(), 'uuid' : $subprocess.check_output(['facter','uuid']).strip()  } }

#for $member in $subprocess.check_output(['relation-list']).strip().split():

   #set $rel_structure[$subprocess.check_output(['relation-get','install_time', $member]).strip()] = { 'name' : $subprocess.check_output(['relation-get','name', $member]).strip(), 'netAddress' : $subprocess.check_output(['relation-get','netAddress', $member]).strip(), 'uuid' : $subprocess.check_output(['relation-get', 'uuid', $member]).strip() }

#end for

#set $nodes = []

#for $index in $sorted($rel_structure.keys()):

   $nodes.append(($rel_structure[$index]['netAddress'], $rel_structure[$index]['name']))

#end for

The above piece of code is what I am currently using to populate a list with the FQDN and address of each cluster member sorted by install time.  This puts the "master" of the cluster at the top of the list which will become useful when populating certain parts of the configuration file.


As we can see by the code above, the main piece of information that we use in this template is the node list.  Here is a sample of how we use it in the environment.tmpl template file:

 #for $netAddress, $name in $nodes:

  <Computer computerType="linuxmachine"

            domain="localdomain"

            name="$name"

            netAddress="$netAddress"/>

#end for

I encourage you to download the charm here and examine the environment.tmpl file in the templates directory.

Here is the complete environment.tmpl file... I know it's pretty small and, you can just download the charm and read the file at your leasure but, I wanted to give you an idea of the size and complexity of hpcc's configuration file.

====

#import random
#import subprocess
#set $rel_structure = { $subprocess.check_output(['facter', 'install_time']).strip() : { 'name' : $subprocess.check_output(['hostname', '-f']).strip(), 'netAddress' : $subprocess.check_output(['facter','ipaddress']).strip(), 'uuid' : $subprocess.check_output(['facter','uuid']).strip()  } }
#for $member in $subprocess.check_output(['relation-list']).strip().split():
   #set $rel_structure[$subprocess.check_output(['relation-get','install_time', $member]).strip()] = { 'name' : $subprocess.check_output(['relation-get','name', $member]).strip(), 'netAddress' : $subprocess.check_output(['relation-get','netAddress', $member]).strip(), 'uuid' : $subprocess.check_output(['relation-get', 'uuid', $member]).strip() }
#end for
#set $nodes = []
#for $index in $sorted($rel_structure.keys()):
   $nodes.append(($rel_structure[$index]['netAddress'], $rel_structure[$index]['name']))
#end for
<?xml version="1.0" encoding="UTF-8"?>
<!-- Edited with ConfigMgr on ip 71.204.190.179 on 2011-08-16T00:39:16 -->
<Environment>
 <EnvSettings>
  <blockname>HPCCSystems</blockname>
  <configs>/etc/HPCCSystems</configs>
  <environment>environment.xml</environment>
  <group>hpcc</group>
  <home>/home</home>
  <interface>eth0</interface>
  <lock>/var/lock/HPCCSystems</lock>
  <log>/var/log/HPCCSystems</log>
  <path>/opt/HPCCSystems</path>
  <pid>/var/run/HPCCSystems</pid>
  <runtime>/var/lib/HPCCSystems</runtime>
  <sourcedir>/etc/HPCCSystems/source</sourcedir>
  <user>hpcc</user>
 </EnvSettings>
 <Hardware>
 #for $netAddress, $name in $nodes:
  <Computer computerType="linuxmachine"
            domain="localdomain"
            name="$name"
            netAddress="$netAddress"/>
#end for
  <ComputerType computerType="linuxmachine"
                manufacturer="unknown"
                name="linuxmachine"
                opSys="linux"/>
  <Domain name="localdomain" password="" username=""/>
  <Switch name="Switch"/>
 </Hardware>
 <Programs>
  <Build name="community_3.0.4" url="/opt/HPCCSystems">
   <BuildSet installSet="deploy_map.xml"
             name="dafilesrv"
             path="componentfiles/dafilesrv"
             processName="DafilesrvProcess"
             schema="dafilesrv.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="dali"
             path="componentfiles/dali"
             processName="DaliServerProcess"
             schema="dali.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="dfuplus"
             path="componentfiles/dfuplus"
             processName="DfuplusProcess"
             schema="dfuplus.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="dfuserver"
             path="componentfiles/dfuserver"
             processName="DfuServerProcess"
             schema="dfuserver.xsd"/>
   <BuildSet deployable="no"
             installSet="deploy_map.xml"
             name="DropZone"
             path="componentfiles/DropZone"
             processName="DropZone"
             schema="dropzone.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="eclagent"
             path="componentfiles/eclagent"
             processName="EclAgentProcess"
             schema="eclagent_config.xsd"/>
   <BuildSet installSet="deploy_map.xml" name="eclminus" path="componentfiles/eclminus"/>
   <BuildSet installSet="deploy_map.xml"
             name="eclplus"
             path="componentfiles/eclplus"
             processName="EclPlusProcess"
             schema="eclplus.xsd"/>
   <BuildSet installSet="eclccserver_deploy_map.xml"
             name="eclccserver"
             path="componentfiles/configxml"
             processName="EclCCServerProcess"
             schema="eclccserver.xsd"/>
   <BuildSet installSet="eclscheduler_deploy_map.xml"
             name="eclscheduler"
             path="componentfiles/configxml"
             processName="EclSchedulerProcess"
             schema="eclscheduler.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="esp"
             path="componentfiles/esp"
             processName="EspProcess"
             schema="esp.xsd"/>
   <BuildSet deployable="no"
             installSet="deploy_map.xml"
             name="espsmc"
             path="componentfiles/espsmc"
             processName="EspService"
             schema="espsmcservice.xsd">
    <Properties defaultPort="8010"
                defaultResourcesBasedn="ou=SMC,ou=EspServices,ou=ecl"
                defaultSecurePort="18010"
                type="WsSMC">
     <Authenticate access="Read"
                   description="Root access to SMC service"
                   path="/"
                   required="Read"
                   resource="SmcAccess"/>
     <AuthenticateFeature description="Access to SMC service"
                          path="SmcAccess"
                          resource="SmcAccess"
                          service="ws_smc"/>
     <AuthenticateFeature description="Access to thor queues"
                          path="ThorQueueAccess"
                          resource="ThorQueueAccess"
                          service="ws_smc"/>
     <AuthenticateFeature description="Access to super computer environment"
                          path="ConfigAccess"
                          resource="ConfigAccess"
                          service="ws_config"/>
     <AuthenticateFeature description="Access to DFU"
                          path="DfuAccess"
                          resource="DfuAccess"
                          service="ws_dfu"/>
     <AuthenticateFeature description="Access to DFU XRef"
                          path="DfuXrefAccess"
                          resource="DfuXrefAccess"
                          service="ws_dfuxref"/>
     <AuthenticateFeature description="Access to machine information"
                          path="MachineInfoAccess"
                          resource="MachineInfoAccess"
                          service="ws_machine"/>
     <AuthenticateFeature description="Access to SNMP metrics information"
                          path="MetricsAccess"
                          resource="MetricsAccess"
                          service="ws_machine"/>
     <AuthenticateFeature description="Access to remote execution"
                          path="ExecuteAccess"
                          resource="ExecuteAccess"
                          service="ws_machine"/>
     <AuthenticateFeature description="Access to DFU workunits"
                          path="DfuWorkunitsAccess"
                          resource="DfuWorkunitsAccess"
                          service="ws_fs"/>
     <AuthenticateFeature description="Access to DFU exceptions"
                          path="DfuExceptionsAccess"
                          resource="DfuExceptions"
                          service="ws_fs"/>
     <AuthenticateFeature description="Access to spraying files"
                          path="FileSprayAccess"
                          resource="FileSprayAccess"
                          service="ws_fs"/>
     <AuthenticateFeature description="Access to despraying of files"
                          path="FileDesprayAccess"
                          resource="FileDesprayAccess"
                          service="ws_fs"/>
     <AuthenticateFeature description="Access to dkcing of key files"
                          path="FileDkcAccess"
                          resource="FileDkcAccess"
                          service="ws_fs"/>
     <AuthenticateFeature description="Access to files in dropzone"
                          path="FileIOAccess"
                          resource="FileIOAccess"
                          service="ws_fileio"/>
     <AuthenticateFeature description="Access to WS ECL service"
                          path="WsEclAccess"
                          resource="WsEclAccess"
                          service="ws_ecl"/>
     <AuthenticateFeature description="Access to Roxie queries and files"
                          path="RoxieQueryAccess"
                          resource="RoxieQueryAccess"
                          service="ws_roxiequery"/>
     <AuthenticateFeature description="Access to cluster topology"
                          path="ClusterTopologyAccess"
                          resource="ClusterTopologyAccess"
                          service="ws_topology"/>
     <AuthenticateFeature description="Access to own workunits"
                          path="OwnWorkunitsAccess"
                          resource="OwnWorkunitsAccess"
                          service="ws_workunits"/>
     <AuthenticateFeature description="Access to others&apos; workunits"
                          path="OthersWorkunitsAccess"
                          resource="OthersWorkunitsAccess"
                          service="ws_workunits"/>
     <AuthenticateFeature description="Access to ECL direct service"
                          path="EclDirectAccess"
                          resource="EclDirectAccess"
                          service="ecldirect"/>
     <ProcessFilters>
      <Platform name="Windows">
       <ProcessFilter name="any">
        <Process name="dafilesrv"/>
       </ProcessFilter>
       <ProcessFilter name="AttrServerProcess">
        <Process name="attrserver"/>
       </ProcessFilter>
       <ProcessFilter name="DaliProcess">
        <Process name="daserver"/>
       </ProcessFilter>
       <ProcessFilter multipleInstances="true" name="DfuServerProcess">
        <Process name="dfuserver"/>
       </ProcessFilter>
       <ProcessFilter multipleInstances="true" name="EclCCServerProcess">
        <Process name="eclccserver"/>
       </ProcessFilter>
       <ProcessFilter multipleInstances="true" name="EspProcess">
        <Process name="esp"/>
        <Process name="dafilesrv" remove="true"/>
       </ProcessFilter>
       <ProcessFilter name="FTSlaveProcess">
        <Process name="ftslave"/>
       </ProcessFilter>
       <ProcessFilter name="RoxieServerProcess">
        <Process name="ccd"/>
       </ProcessFilter>
       <ProcessFilter name="RoxieSlaveProcess">
        <Process name="ccd"/>
       </ProcessFilter>
       <ProcessFilter name="SchedulerProcess">
        <Process name="scheduler"/>
       </ProcessFilter>
       <ProcessFilter name="ThorMasterProcess">
        <Process name="thormaster"/>
       </ProcessFilter>
       <ProcessFilter name="ThorSlaveProcess">
        <Process name="thorslave"/>
       </ProcessFilter>
       <ProcessFilter name="SashaServerProcess">
        <Process name="saserver"/>
       </ProcessFilter>
      </Platform>
      <Platform name="Linux">
       <ProcessFilter name="any">
        <Process name="dafilesrv"/>
       </ProcessFilter>
       <ProcessFilter name="AttrServerProcess">
        <Process name="attrserver"/>
       </ProcessFilter>
       <ProcessFilter name="DaliProcess">
        <Process name="daserver"/>
       </ProcessFilter>
       <ProcessFilter multipleInstances="true" name="DfuServerProcess">
        <Process name="."/>
       </ProcessFilter>
       <ProcessFilter multipleInstances="true" name="EclCCServerProcess">
        <Process name="."/>
       </ProcessFilter>
       <ProcessFilter multipleInstances="true" name="EspProcess">
        <Process name="."/>
        <Process name="dafilesrv" remove="true"/>
       </ProcessFilter>
       <ProcessFilter name="FTSlaveProcess">
        <Process name="ftslave"/>
       </ProcessFilter>
       <ProcessFilter name="GenesisServerProcess">
        <Process name="httpd"/>
        <Process name="atftpd"/>
        <Process name="dhcpd"/>
       </ProcessFilter>
       <ProcessFilter name="RoxieServerProcess">
        <Process name="ccd"/>
       </ProcessFilter>
       <ProcessFilter name="RoxieSlaveProcess">
        <Process name="ccd"/>
       </ProcessFilter>
       <ProcessFilter name="SchedulerProcess">
        <Process name="scheduler"/>
       </ProcessFilter>
       <ProcessFilter name="ThorMasterProcess">
        <Process name="thormaster"/>
       </ProcessFilter>
       <ProcessFilter name="ThorSlaveProcess">
        <Process name="thorslave"/>
       </ProcessFilter>
       <ProcessFilter name="SashaServerProcess">
        <Process name="saserver"/>
       </ProcessFilter>
      </Platform>
     </ProcessFilters>
    </Properties>
   </BuildSet>
   <BuildSet installSet="deploy_map.xml"
             name="ftslave"
             path="componentfiles/ftslave"
             processName="FTSlaveProcess"
             schema="ftslave_linux.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="hqltest"
             path="componentfiles/hqltest"
             processName="HqlTestProcess"/>
   <BuildSet deployable="no"
             installSet="deploy_map.xml"
             name="ldapServer"
             path="componentfiles/ldapServer"
             processName="LDAPServerProcess"
             schema="ldapserver.xsd"/>
   <BuildSet deployable="no"
             installSet="auditlib_deploy_map.xml"
             name="plugins_auditlib"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet deployable="no"
             installSet="debugservices_deploy_map.xml"
             name="plugins_debugservices"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet deployable="no"
             installSet="fileservices_deploy_map.xml"
             name="plugins_fileservices"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet deployable="no"
             installSet="logging_deploy_map.xml"
             name="plugins_logging"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet deployable="no"
             installSet="parselib_deploy_map.xml"
             name="plugins_parselib"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet deployable="no"
             installSet="stringlib_deploy_map.xml"
             name="plugins_stringlib"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet deployable="no"
             installSet="unicodelib_deploy_map.xml"
             name="plugins_unicodelib"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet deployable="no"
             installSet="workunitservices_deploy_map.xml"
             name="plugins_workunitservices"
             path="componentfiles/configxml"
             processName="PluginProcess"
             schema="plugin.xsd"/>
   <BuildSet installSet="roxie_deploy_map.xml"
             name="roxie"
             path="componentfiles/configxml"
             processName="RoxieCluster"
             schema="roxie.xsd"/>
   <BuildSet installSet="deploy_map.xml" name="roxieconfig" path="componentfiles/roxieconfig"/>
   <BuildSet installSet="deploy_map.xml"
             name="sasha"
             path="componentfiles/sasha"
             processName="SashaServerProcess"
             schema="sasha.xsd"/>
   <BuildSet deployable="no"
             installSet="deploy_map.xml"
             name="SiteCertificate"
             path="componentfiles/SiteCertificate"
             processName="SiteCertificate"
             schema="SiteCertificate.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="soapplus"
             path="componentfiles/soapplus"
             processName="SoapPlusProcess"
             schema="soapplus.xsd"/>
   <BuildSet installSet="deploy_map.xml"
             name="thor"
             path="componentfiles/thor"
             processName="ThorCluster"
             schema="thor.xsd"/>
   <BuildSet deployable="no"
             installSet="deploy_map.xml"
             name="topology"
             path="componentfiles/topology"
             processName="Topology"
             schema="topology.xsd"/>
   <BuildSet deployable="no"
             installSet="deploy_map.xml"
             name="ws_ecl"
             path="componentfiles/ws_ecl"
             processName="EspService"
             schema="esp_service_wsecl2.xsd">
    <Properties bindingType="ws_eclSoapBinding"
                defaultPort="8002"
                defaultResourcesBasedn="ou=WsEcl,ou=EspServices,ou=ecl"
                defaultSecurePort="18002"
                plugin="ws_ecl"
                type="ws_ecl">
     <Authenticate access="Read"
                   description="Root access to WS ECL service"
                   path="/"
                   required="Read"
                   resource="WsEclAccess"/>
     <AuthenticateFeature description="Access to WS ECL service"
                          path="WsEclAccess"
                          resource="WsEclAccess"
                          service="ws_ecl"/>
    </Properties>
   </BuildSet>
   <BuildSet deployable="no"
             installSet="deploy_map.xml"
             name="ecldirect"
             path="componentfiles/ecldirect"
             processName="EspService"
             schema="esp_service_ecldirect.xsd">
    <Properties bindingType="EclDirectSoapBinding"
                defaultPort="8008"
                defaultResourcesBasedn="ou=EclDirectAccess,ou=EspServices,ou=ecl"
                defaultSecurePort="18008"
                plugin="ecldirect"
                type="ecldirect">
     <Authenticate access="Read"
                   description="Root access to ECL Direct service"
                   path="/"
                   required="Read"
                   resource="EclDirectAccess"/>
     <AuthenticateFeature description="Access to ECL Direct service"
                          path="EclDirectAccess"
                          resource="EclDirectAccess"
                          service="ecldirect"/>
    </Properties>
   </BuildSet>
  </Build>
 </Programs>
 <Software>
  <DafilesrvProcess build="community_3.0.4"
                    buildSet="dafilesrv"
                    description="DaFileSrv process"
                    name="mydafilesrv"
                    version="1">
#for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/mydafilesrv"
             name="s_$name"
             netAddress="$netAddress"/>
#end for             
  </DafilesrvProcess>
  <DaliServerProcess build="community_3.0.4"
                     buildSet="dali"
                     environment="/etc/HPCCSystems/environment.xml"
                     name="mydali"
                     recoverFromIncErrors="true">                     
   <Instance computer="$nodes[0][1]"
             directory="/var/lib/HPCCSystems/mydali"
             name="s_$nodes[0][1]"
             netAddress="$nodes[0][0]"
             port="7070"/>
  </DaliServerProcess>
  <DfuServerProcess build="community_3.0.4"
                    buildSet="dfuserver"
                    daliServers="mydali"
                    description="DFU Server"
                    monitorinterval="900"
                    monitorqueue="dfuserver_monitor_queue"
                    name="mydfuserver"
                    queue="dfuserver_queue"
                    transferBufferSize="65536">
#for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/mydfuserver"
             name="s_$name"
             netAddress="$netAddress"/>
#end for    
#raw        
   <SSH SSHidentityfile="$HOME/.ssh/id_rsa"
#end raw
        SSHpassword=""
        SSHretries="3"
        SSHtimeout="0"
        SSHusername="hpcc"/>
  </DfuServerProcess>
  <Directories name="HPCCSystems">
   <Category dir="/var/log/[NAME]/[INST]" name="log"/>
   <Category dir="/var/lib/[NAME]/[INST]" name="run"/>
   <Category dir="/etc/[NAME]/[INST]" name="conf"/>
   <Category dir="/var/lib/[NAME]/[INST]/temp" name="temp"/>
   <Category dir="/var/lib/[NAME]/hpcc-data/[COMPONENT]" name="data"/>
   <Category dir="/var/lib/[NAME]/hpcc-data2/[COMPONENT]" name="data2"/>
   <Category dir="/var/lib/[NAME]/hpcc-data3/[COMPONENT]" name="data3"/>
   <Category dir="/var/lib/[NAME]/hpcc-mirror/[COMPONENT]" name="mirror"/>
   <Category dir="/var/lib/[NAME]/queries/[INST]" name="query"/>
   <Category dir="/var/lock/[NAME]/[INST]" name="lock"/>
  </Directories>
  <DropZone build="community_3.0.4"
            buildSet="DropZone"
            computer="$nodes[0][0]"
            description="DropZone process"
            directory="/var/lib/HPCCSystems/dropzone"
            name="mydropzone"/>
  <EclAgentProcess allowedPipePrograms="*"
                   build="community_3.0.4"
                   buildSet="eclagent"
                   daliServers="mydali"
                   description="EclAgent process"
                   name="myeclagent"
                   pluginDirectory="/opt/HPCCSystems/plugins/"
                   thorConnectTimeout="600"
                   traceLevel="0"
                   wuQueueName="myeclagent_queue">
#for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/myeclagent"
             name="s_$name"
             netAddress="$netAddress"/>
#end for  
  </EclAgentProcess>
  <EclCCServerProcess build="community_3.0.4"
                      buildSet="eclccserver"
                      daliServers="mydali"
                      description="EclCCServer process"
                      enableSysLog="true"
                      maxCompileThreads="4"
                      name="myeclccserver"
                      traceLevel="1">
#for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/myeclccserver"
             name="s_$name"
             netAddress="$netAddress"/>
#end for
  </EclCCServerProcess>
  <EclSchedulerProcess build="community_3.0.4"
                       buildSet="eclscheduler"
                       daliServers="mydali"
                       description="EclScheduler process"
                       name="myeclscheduler">
#for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/myeclscheduler"
             name="s_$name"
             netAddress="$netAddress"/>
#end for
  </EclSchedulerProcess>
  <EspProcess build="community_3.0.4"
              buildSet="esp"
              componentfilesDir="/opt/HPCCSystems/componentfiles"
              daliServers="mydali"
              description="ESP server"
              enableSEHMapping="true"
              formOptionsAccess="false"
              httpConfigAccess="true"
              logLevel="1"
              logRequests="false"
              logResponses="false"
              maxBacklogQueueSize="200"
              maxConcurrentThreads="0"
              maxRequestEntityLength="8000000"
              name="myesp"
              perfReportDelay="60"
              portalurl="http://hpccsystems.com/download">
   <Authentication ldapAuthMethod="kerberos"
                   ldapConnections="10"
                   ldapServer=""
                   method="none"/>
   <EspBinding defaultForPort="true"
               defaultServiceVersion=""
               name="myespsmc"
               port="8010"
               protocol="http"
               resourcesBasedn="ou=SMC,ou=EspServices,ou=ecl"
               service="EclWatch"
               workunitsBasedn="ou=workunits,ou=ecl"
               wsdlServiceAddress="">
    <Authenticate access="Read"
                  description="Root access to SMC service"
                  path="/"
                  required="Read"
                  resource="SmcAccess"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to SMC service"
                         path="SmcAccess"
                         resource="SmcAccess"
                         service="ws_smc"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to thor queues"
                         path="ThorQueueAccess"
                         resource="ThorQueueAccess"
                         service="ws_smc"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to super computer environment"
                         path="ConfigAccess"
                         resource="ConfigAccess"
                         service="ws_config"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to DFU"
                         path="DfuAccess"
                         resource="DfuAccess"
                         service="ws_dfu"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to DFU XRef"
                         path="DfuXrefAccess"
                         resource="DfuXrefAccess"
                         service="ws_dfuxref"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to machine information"
                         path="MachineInfoAccess"
                         resource="MachineInfoAccess"
                         service="ws_machine"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to SNMP metrics information"
                         path="MetricsAccess"
                         resource="MetricsAccess"
                         service="ws_machine"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to remote execution"
                         path="ExecuteAccess"
                         resource="ExecuteAccess"
                         service="ws_machine"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to DFU workunits"
                         path="DfuWorkunitsAccess"
                         resource="DfuWorkunitsAccess"
                         service="ws_fs"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to DFU exceptions"
                         path="DfuExceptionsAccess"
                         resource="DfuExceptions"
                         service="ws_fs"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to spraying files"
                         path="FileSprayAccess"
                         resource="FileSprayAccess"
                         service="ws_fs"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to despraying of files"
                         path="FileDesprayAccess"
                         resource="FileDesprayAccess"
                         service="ws_fs"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to dkcing of key files"
                         path="FileDkcAccess"
                         resource="FileDkcAccess"
                         service="ws_fs"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to files in dropzone"
                         path="FileIOAccess"
                         resource="FileIOAccess"
                         service="ws_fileio"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to WS ECL service"
                         path="WsEclAccess"
                         resource="WsEclAccess"
                         service="ws_ecl"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to Roxie queries and files"
                         path="RoxieQueryAccess"
                         resource="RoxieQueryAccess"
                         service="ws_roxiequery"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to cluster topology"
                         path="ClusterTopologyAccess"
                         resource="ClusterTopologyAccess"
                         service="ws_topology"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to own workunits"
                         path="OwnWorkunitsAccess"
                         resource="OwnWorkunitsAccess"
                         service="ws_workunits"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to others&apos; workunits"
                         path="OthersWorkunitsAccess"
                         resource="OthersWorkunitsAccess"
                         service="ws_workunits"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to ECL direct service"
                         path="EclDirectAccess"
                         resource="EclDirectAccess"
                         service="ecldirect"/>
   </EspBinding>
   <EspBinding defaultForPort="true"
               defaultServiceVersion=""
               name="myws_ecl"
               port="8002"
               protocol="http"
               resourcesBasedn="ou=WsEcl,ou=EspServices,ou=ecl"
               service="myws_ecl"
               workunitsBasedn="ou=workunits,ou=ecl"
               wsdlServiceAddress="">
    <Authenticate access="Read"
                  description="Root access to WS ECL service"
                  path="/"
                  required="Read"
                  resource="WsEclAccess"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to WS ECL service"
                         path="WsEclAccess"
                         resource="WsEclAccess"
                         service="ws_ecl"/>
   </EspBinding>
   <EspBinding defaultForPort="true"
               defaultServiceVersion=""
               name="myecldirect"
               port="8008"
               protocol="http"
               resourcesBasedn="ou=EclDirectAccess,ou=EspServices,ou=ecl"
               service="myecldirect"
               workunitsBasedn="ou=workunits,ou=ecl"
               wsdlServiceAddress="">
    <Authenticate access="Read"
                  description="Root access to ECL Direct service"
                  path="/"
                  required="Read"
                  resource="EclDirectAccess"/>
    <AuthenticateFeature authenticate="Yes"
                         description="Access to ECL Direct service"
                         path="EclDirectAccess"
                         resource="EclDirectAccess"
                         service="ecldirect"/>
   </EspBinding>
   <HTTPS acceptSelfSigned="true"
          CA_Certificates_Path="ca.pem"
          certificateFileName="certificate.cer"
          city=""
          country="US"
          daysValid="365"
          enableVerification="false"
          organization="Customer of HPCCSystems"
          organizationalUnit=""
          passphrase=""
          privateKeyFileName="privatekey.cer"
          regenerateCredentials="false"
          requireAddressMatch="false"
          state=""
          trustedPeers="anyone"/>
   #for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/myesp"
             FQDN=""
             name="s_$name"
             netAddress="$netAddress"/>
   #end for
   <ProtocolX authTimeout="3"
              defaultTimeout="21"
              idleTimeout="3600"
              maxTimeout="36"
              minTimeout="30"
              threadCount="2"/>
  </EspProcess>
  <EspService allowNewRoxieOnDemandQuery="false"
              AWUsCacheTimeout="15"
              build="community_3.0.4"
              buildSet="espsmc"
              description="ESP services for SMC"
              disableUppercaseTranslation="false"
              eclServer="myeclccserver"
              enableSystemUseRewrite="false"
              excludePartitions="/,/dev*,/sys,/usr,/proc/*"
              monitorDaliFileServer="false"
              name="EclWatch"
              pluginsPath="/opt/HPCCSystems/plugins"
              syntaxCheckQueue=""
              viewTimeout="1000"
              warnIfCpuLoadOver="95"
              warnIfFreeMemoryUnder="5"
              warnIfFreeStorageUnder="5">
   <Properties defaultPort="8010"
               defaultResourcesBasedn="ou=SMC,ou=EspServices,ou=ecl"
               defaultSecurePort="18010"
               type="WsSMC">
    <Authenticate access="Read"
                  description="Root access to SMC service"
                  path="/"
                  required="Read"
                  resource="SmcAccess"/>
    <AuthenticateFeature description="Access to SMC service"
                         path="SmcAccess"
                         resource="SmcAccess"
                         service="ws_smc"/>
    <AuthenticateFeature description="Access to thor queues"
                         path="ThorQueueAccess"
                         resource="ThorQueueAccess"
                         service="ws_smc"/>
    <AuthenticateFeature description="Access to super computer environment"
                         path="ConfigAccess"
                         resource="ConfigAccess"
                         service="ws_config"/>
    <AuthenticateFeature description="Access to DFU"
                         path="DfuAccess"
                         resource="DfuAccess"
                         service="ws_dfu"/>
    <AuthenticateFeature description="Access to DFU XRef"
                         path="DfuXrefAccess"
                         resource="DfuXrefAccess"
                         service="ws_dfuxref"/>
    <AuthenticateFeature description="Access to machine information"
                         path="MachineInfoAccess"
                         resource="MachineInfoAccess"
                         service="ws_machine"/>
    <AuthenticateFeature description="Access to SNMP metrics information"
                         path="MetricsAccess"
                         resource="MetricsAccess"
                         service="ws_machine"/>
    <AuthenticateFeature description="Access to remote execution"
                         path="ExecuteAccess"
                         resource="ExecuteAccess"
                         service="ws_machine"/>
    <AuthenticateFeature description="Access to DFU workunits"
                         path="DfuWorkunitsAccess"
                         resource="DfuWorkunitsAccess"
                         service="ws_fs"/>
    <AuthenticateFeature description="Access to DFU exceptions"
                         path="DfuExceptionsAccess"
                         resource="DfuExceptions"
                         service="ws_fs"/>
    <AuthenticateFeature description="Access to spraying files"
                         path="FileSprayAccess"
                         resource="FileSprayAccess"
                         service="ws_fs"/>
    <AuthenticateFeature description="Access to despraying of files"
                         path="FileDesprayAccess"
                         resource="FileDesprayAccess"
                         service="ws_fs"/>
    <AuthenticateFeature description="Access to dkcing of key files"
                         path="FileDkcAccess"
                         resource="FileDkcAccess"
                         service="ws_fs"/>
    <AuthenticateFeature description="Access to files in dropzone"
                         path="FileIOAccess"
                         resource="FileIOAccess"
                         service="ws_fileio"/>
    <AuthenticateFeature description="Access to WS ECL service"
                         path="WsEclAccess"
                         resource="WsEclAccess"
                         service="ws_ecl"/>
    <AuthenticateFeature description="Access to Roxie queries and files"
                         path="RoxieQueryAccess"
                         resource="RoxieQueryAccess"
                         service="ws_roxiequery"/>
    <AuthenticateFeature description="Access to cluster topology"
                         path="ClusterTopologyAccess"
                         resource="ClusterTopologyAccess"
                         service="ws_topology"/>
    <AuthenticateFeature description="Access to own workunits"
                         path="OwnWorkunitsAccess"
                         resource="OwnWorkunitsAccess"
                         service="ws_workunits"/>
    <AuthenticateFeature description="Access to others&apos; workunits"
                         path="OthersWorkunitsAccess"
                         resource="OthersWorkunitsAccess"
                         service="ws_workunits"/>
    <AuthenticateFeature description="Access to ECL direct service"
                         path="EclDirectAccess"
                         resource="EclDirectAccess"
                         service="ecldirect"/>
    <ProcessFilters>
     <Platform name="Windows">
      <ProcessFilter name="any">
       <Process name="dafilesrv"/>
      </ProcessFilter>
      <ProcessFilter name="AttrServerProcess">
       <Process name="attrserver"/>
      </ProcessFilter>
      <ProcessFilter name="DaliProcess">
       <Process name="daserver"/>
      </ProcessFilter>
      <ProcessFilter multipleInstances="true" name="DfuServerProcess">
       <Process name="dfuserver"/>
      </ProcessFilter>
      <ProcessFilter multipleInstances="true" name="EclCCServerProcess">
       <Process name="eclccserver"/>
      </ProcessFilter>
      <ProcessFilter multipleInstances="true" name="EspProcess">
       <Process name="esp"/>
       <Process name="dafilesrv" remove="true"/>
      </ProcessFilter>
      <ProcessFilter name="FTSlaveProcess">
       <Process name="ftslave"/>
      </ProcessFilter>
      <ProcessFilter name="RoxieServerProcess">
       <Process name="ccd"/>
      </ProcessFilter>
      <ProcessFilter name="RoxieSlaveProcess">
       <Process name="ccd"/>
      </ProcessFilter>
      <ProcessFilter name="SchedulerProcess">
       <Process name="scheduler"/>
      </ProcessFilter>
      <ProcessFilter name="ThorMasterProcess">
       <Process name="thormaster"/>
      </ProcessFilter>
      <ProcessFilter name="ThorSlaveProcess">
       <Process name="thorslave"/>
      </ProcessFilter>
      <ProcessFilter name="SashaServerProcess">
       <Process name="saserver"/>
      </ProcessFilter>
     </Platform>
     <Platform name="Linux">
      <ProcessFilter name="any">
       <Process name="dafilesrv"/>
      </ProcessFilter>
      <ProcessFilter name="AttrServerProcess">
       <Process name="attrserver"/>
      </ProcessFilter>
      <ProcessFilter name="DaliProcess">
       <Process name="daserver"/>
      </ProcessFilter>
      <ProcessFilter multipleInstances="true" name="DfuServerProcess">
       <Process name="."/>
      </ProcessFilter>
      <ProcessFilter multipleInstances="true" name="EclCCServerProcess">
       <Process name="."/>
      </ProcessFilter>
      <ProcessFilter multipleInstances="true" name="EspProcess">
       <Process name="."/>
       <Process name="dafilesrv" remove="true"/>
      </ProcessFilter>
      <ProcessFilter name="FTSlaveProcess">
       <Process name="ftslave"/>
      </ProcessFilter>
      <ProcessFilter name="GenesisServerProcess">
       <Process name="httpd"/>
       <Process name="atftpd"/>
       <Process name="dhcpd"/>
      </ProcessFilter>
      <ProcessFilter name="RoxieServerProcess">
       <Process name="ccd"/>
      </ProcessFilter>
      <ProcessFilter name="RoxieSlaveProcess">
       <Process name="ccd"/>
      </ProcessFilter>
      <ProcessFilter name="SchedulerProcess">
       <Process name="scheduler"/>
      </ProcessFilter>
      <ProcessFilter name="ThorMasterProcess">
       <Process name="thormaster"/>
      </ProcessFilter>
      <ProcessFilter name="ThorSlaveProcess">
       <Process name="thorslave"/>
      </ProcessFilter>
      <ProcessFilter name="SashaServerProcess">
       <Process name="saserver"/>
      </ProcessFilter>
     </Platform>
    </ProcessFilters>
   </Properties>
  </EspService>
  <EspService build="community_3.0.4"
              buildSet="ws_ecl"
              description="WS ECL Service"
              name="myws_ecl">
   <Properties bindingType="ws_eclSoapBinding"
               defaultPort="8002"
               defaultResourcesBasedn="ou=WsEcl,ou=EspServices,ou=ecl"
               defaultSecurePort="18002"
               plugin="ws_ecl"
               type="ws_ecl">
    <Authenticate access="Read"
                  description="Root access to WS ECL service"
                  path="/"
                  required="Read"
                  resource="WsEclAccess"/>
    <AuthenticateFeature description="Access to WS ECL service"
                         path="WsEclAccess"
                         resource="WsEclAccess"
                         service="ws_ecl"/>
   </Properties>
  </EspService>
  <EspService build="community_3.0.4"
              buildSet="ecldirect"
              clusterName="hthor"
              description="ESP service for running raw ECL queries"
              name="myecldirect">
   <Properties bindingType="EclDirectSoapBinding"
               defaultPort="8008"
               defaultResourcesBasedn="ou=EclDirectAccess,ou=EspServices,ou=ecl"
               defaultSecurePort="18008"
               plugin="ecldirect"
               type="ecldirect">
    <Authenticate access="Read"
                  description="Root access to ECL Direct service"
                  path="/"
                  required="Read"
                  resource="EclDirectAccess"/>
    <AuthenticateFeature description="Access to ECL Direct service"
                         path="EclDirectAccess"
                         resource="EclDirectAccess"
                         service="ecldirect"/>
   </Properties>
  </EspService>
  <FTSlaveProcess build="community_3.0.4"
                  buildSet="ftslave"
                  description="FTSlave process"
                  name="myftslave"
                  version="1">
#for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/myftslave"
             name="s_$name"
             netAddress="$netAddress"
             program="/opt/HPCCSystems/bin/ftslave"/>
#end for
  </FTSlaveProcess>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_auditlib"
                 description="plugin process"
                 name="myplugins_auditlib"/>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_debugservices"
                 description="plugin process"
                 name="myplugins_debugservices"/>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_fileservices"
                 description="plugin process"
                 name="myplugins_fileservices"/>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_logging"
                 description="plugin process"
                 name="myplugins_logging"/>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_parselib"
                 description="plugin process"
                 name="myplugins_parselib"/>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_stringlib"
                 description="plugin process"
                 name="myplugins_stringlib"/>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_unicodelib"
                 description="plugin process"
                 name="myplugins_unicodelib"/>
  <PluginProcess build="community_3.0.4"
                 buildSet="plugins_workunitservices"
                 description="plugin process"
                 name="myplugins_workunitservices"/>
  <RoxieCluster allowRoxieOnDemand="false"
                baseDataDir="/var/lib/HPCCSystems/hpcc-data/roxie"
                blindLogging="false"
                blobCacheMem="0"
                build="community_3.0.4"
                buildSet="roxie"
                callbackRetries="3"
                callbackTimeout="500"
                channel0onPrimary="true"
                checkCompleted="true"
                checkFileDate="true"
                checkingHeap="0"
                checkPrimaries="true"
                checkState="false"
                checkVersion="true"
                clusterWidth="$len($nodes)"
                copyResources="true"
                crcResources="false"
                cyclicOffset="1"
                dafilesrvLookupTimeout="10000"
                daliServers="mydali"
                debugPermitted="true"
                defaultConcatPreload="0"
                defaultFetchPreload="0"
                defaultFullKeyedJoinPreload="0"
                defaultHighPriorityTimeLimit="0"
                defaultHighPriorityTimeWarning="5000"
                defaultKeyedJoinPreload="0"
                defaultLowPriorityTimeLimit="0"
                defaultLowPriorityTimeWarning="0"
                defaultMemoryLimit="0"
                defaultParallelJoinPreload="0"
                defaultPrefetchProjectPreload="10"
                defaultSLAPriorityTimeLimit="0"
                defaultSLAPriorityTimeWarning="5000"
                defaultStripLeadingWhitespace="1"
                deleteUnneededFiles="false"
                description="Roxie cluster"
                directory="/var/lib/HPCCSystems/myroxie"
                diskReadBufferSize="65536"
                diskReadStable="true"
                doIbytiDelay="true"
                enableForceKeyDiffCopy="false"
                enableHeartBeat="true"
                enableKeyDiff="true"
                enableSNMP="true"
                enableSysLog="true"
                fastLaneQueue="true"
                fieldTranslationEnabled="false"
                flushJHtreeCacheOnOOM="true"
                forceStdLog="false"
                highTimeout="2000"
                ignoreMissingFiles="false"
                indexReadChunkSize="60000"
                indexReadStable="true"
                initIbytiDelay="100"
                jumboFrames="false"
                keyedJoinFlowLimit="1000"
                keyedJoinStable="true"
                lazyOpen="false"
                leafCacheMem="50"
                linuxYield="true"
                localFilesExpire="-1"
                localSlave="false"
                logFullQueries="false"
                logQueueDrop="32"
                logQueueLen="512"
                lowTimeout="10000"
                maxBlockSize="10000000"
                maxLocalFilesOpen="4000"
                maxLockAttempts="5"
                maxRemoteFilesOpen="1000"
                memoryStatsInterval="60"
                memTraceLevel="1"
                memTraceSizeLimit="0"
                minFreeDiskSpace="1073741824"
                minIbytiDelay="0"
                minLocalFilesOpen="2000"
                minRemoteFilesOpen="500"
                miscDebugTraceLevel="0"
                monitorDaliFileServer="false"
                multicastBase="239.1.1.1"
                multicastLast="239.1.254.254"
                name="myroxie"
                nodeCacheMem="100"
                nodeCachePreload="false"
                numChannels="$len($nodes)"
                numDataCopies="2"
                parallelAggregate="0"
                perChannelFlowLimit="10"
                pingInterval="60"
                pluginsPath="/opt/HPCCSystems/plugins"
                preabortIndexReadsThreshold="100"
                preabortKeyedJoinsThreshold="100"
                preferredSubnet=""
                preferredSubnetMask=""
                remoteFilesExpire="3600000"
                resolveFilesInPackage="false"
                roxieMulticastEnabled="true"
                serverSideCacheSize="0"
                serverThreads="30"
                simpleLocalKeyedJoins="true"
                siteCertificate=""
                slaTimeout="2000"
                slaveConfig="cyclic redundancy"
                slaveThreads="30"
                smartSteppingChunkRows="100"
                soapTraceLevel="1"
                socketCheckInterval="5000"
#raw                
                SSHidentityfile="$HOME/.ssh/id_rsa"
#end raw                
                SSHpassword=""
                SSHretries="3"
                SSHtimeout="0"
                SSHusername="hpcc"
                statsExpiryTime="3600"
                syncCluster="false"
                systemMonitorInterval="60000"
                totalMemoryLimit="1073741824"
                traceLevel="1"
                trapTooManyActiveQueries="true"
                udpFlowSocketsSize="131071"
                udpInlineCollation="false"
                udpInlineCollationPacketLimit="50"
                udpLocalWriteSocketSize="131071"
                udpMaxRetryTimedoutReqs="0"
                udpMaxSlotsPerClient="2147483647"
                udpMulticastBufferSize="131071"
                udpOutQsPriority="0"
                udpQueueSize="100"
                udpRequestToSendTimeout="5"
                udpResendEnabled="true"
                udpRetryBusySenders="0"
                udpSendCompletedInData="false"
                udpSendQueueSize="50"
                udpSnifferEnabled="true"
                udpTraceLevel="1"
                useHardLink="false"
                useLogQueue="true"
                useMemoryMappedIndexes="false"
                useRemoteResources="true"
                useTreeCopy="false">
   <RoxieFarmProcess dataDirectory="/var/lib/HPCCSystems/hpcc-data/roxie"
                     listenQueue="200"
                     name="farm1"
                     numThreads="30"
                     port="9876"
                     requestArrayThreads="5">
#for $netAddress, $name in $nodes:                    
    <RoxieServerProcess computer="$name" name="farm1_$name"/>
#end for
   </RoxieFarmProcess>
#for $netAddress, $name in $nodes:                    
   <RoxieServerProcess computer="$name"
                       dataDirectory="/var/lib/HPCCSystems/hpcc-data/roxie"
                       listenQueue="200"
                       name="farm1_s_$name"
                       netAddress="$netAddress"
                       numThreads="30"
                       port="9876"
                       requestArrayThreads="5"/>
#end for
#for $netAddress, $name in $nodes:                    
   <RoxieSlave computer="$name" name="s_$name">
    <RoxieChannel dataDirectory="/var/lib/HPCCSystems/hpcc-data/roxie" number="$random.randint(1,$len($nodes))"/>
    <RoxieChannel dataDirectory="/var/lib/HPCCSystems/hpcc-data2/roxie" number="$random.randint(1,$len($nodes))"/>
   </RoxieSlave>
#end for
#set $channel = 1
#for $netAddress, $name in $nodes:                    
   <RoxieSlaveProcess channel="$channel"
                      computer="$name"
                      dataDirectory="/var/lib/HPCCSystems/hpcc-data/roxie"
                      name="s_$name"
                      netAddress="$netAddress"/>
#set $channel = $channel + 1                      
#end for
  </RoxieCluster>
  <SashaServerProcess autoRestartInterval="0"
                      build="community_3.0.4"
                      buildSet="sasha"
                      cachedWUat="* * * * *"
                      cachedWUinterval="24"
                      cachedWUlimit="100"
                      coalesceAt="* * * * *"
                      coalesceInterval="1"
                      dafsmonAt="* * * * *"
                      dafsmonInterval="0"
                      dafsmonList="*"
                      daliServers="mydali"
                      description="Sasha Server process"
                      DFUrecoveryAt="* * * * *"
                      DFUrecoveryCutoff="4"
                      DFUrecoveryInterval="12"
                      DFUrecoveryLimit="20"
                      DFUWUat="* * * * *"
                      DFUWUcutoff="14"
                      DFUWUduration="0"
                      DFUWUinterval="24"
                      DFUWUlimit="1000"
                      DFUWUthrottle="0"
                      ExpiryAt="* 3 * * *"
                      ExpiryInterval="24"
                      keepResultFiles="false"
                      LDSroot="LDS"
                      logDir="."
                      minDeltaSize="50000"
                      name="mysasha"
                      recoverDeltaErrors="false"
                      thorQMonInterval="1"
                      thorQMonQueues="*"
                      thorQMonSwitchMinTime="0"
                      WUat="* * * * *"
                      WUbackup="0"
                      WUcutoff="8"
                      WUduration="0"
                      WUinterval="6"
                      WUlimit="1000"
                      WUretryinterval="7"
                      WUthrottle="0"
                      xrefAt="* 2 * * *"
                      xrefCutoff="1"
                      xrefEclWatchProvider="true"
                      xrefInterval="0"
                      xrefList="*">
#for $netAddress, $name in $nodes:                    
   <Instance computer="$name"
             directory="/var/lib/HPCCSystems/mysasha"
             name="s_$name"
             netAddress="$netAddress"
             port="8877"/>
#end for             
  </SashaServerProcess>
  <ThorCluster autoCopyBackup="false"
               build="community_3.0.4"
               buildSet="thor"
               computer="$nodes[0][1]"
               daliServers="mydali"
               description="Thor process"
               localThor="false"
               monitorDaliFileServer="true"
               multiSlaves="false"
               name="mythor"
               pluginsPath="/opt/HPCCSystems/plugins/"
               replicateAsync="true"
               replicateOutputs="true"
               slaves="8"
               watchdogEnabled="true"
               watchdogProgressEnabled="true">
   <Debug/>
#raw   
   <SSH SSHidentityfile="$HOME/.ssh/id_rsa"
#end raw   
        SSHpassword=""
        SSHretries="3"
        SSHtimeout="0"
        SSHusername="hpcc"/>
   <Storage/>
   <SwapNode/>
   <ThorMasterProcess computer="$nodes[0][1]" name="m_$nodes[0][1]"/>
#if $len($nodes) > 1:
#for $netAddress, $name in $nodes[1:]:                    
   <ThorSlaveProcess computer="$name" name="s_$name"/>
#end for   
#end if
   <Topology>
    <Node process="m_$nodes[0][1]">
#if $len($nodes) > 1:
#for $netAddress, $name in $nodes[1:]:                    
     <Node process="s_$name"/>
#end for   
#end if
    </Node>
   </Topology>
  </ThorCluster>
  <Topology build="community_3.0.4" buildSet="topology" name="topology">
   <Cluster name="hthor" prefix="hthor">
    <EclAgentProcess process="myeclagent"/>
    <EclCCServerProcess process="myeclccserver"/>
    <EclSchedulerProcess process="myeclscheduler"/>
   </Cluster>
   <Cluster name="thor" prefix="thor">
    <EclAgentProcess process="myeclagent"/>
    <EclCCServerProcess process="myeclccserver"/>
    <EclSchedulerProcess process="myeclscheduler"/>
    <ThorCluster process="mythor"/>
   </Cluster>
   <Cluster name="roxie" prefix="roxie">
    <EclAgentProcess process="myeclagent"/>
    <EclCCServerProcess process="myeclccserver"/>
    <EclSchedulerProcess process="myeclscheduler"/>
    <RoxieCluster process="myroxie"/>
   </Cluster>
  </Topology>
 </Software>
</Environment>

====

Even just scrolling past the file takes a while!!  This behemoth of a file was tamed thanks to cheetah, I highly encourage you to read up on it.

This charm may require some changes to your environment.yaml file in ~/.juju as hpcc will only run on 64-bit instances.  Make sure that your juju environment has been properly shutdown before you edit this file ( juju destroy-environment ).  Here is my environment.yaml file where I show you the important part to check:

juju: environments

environments:

  sample:

    type: ec2

    access-key: ( removed ... get your own :) )

    secret-key: ( removed ... get your own :) )

    control-bucket: juju-fbb790f292e14a0394353bb4b63a3403

    admin-secret: 604d18a77fd24e3f91e1df398fcbe9f2

The emphasized parts are the important ones.  You can just copy them from here and paste them into your ~/.juju/environment.yaml file.

Now, let's take a look at the charm starting with the metadata.yaml file:

name: hpcc

revision: 1

summary: HPCC (High Performance Computing Cluster)

description: |

  HPCC (High Performance Computing Cluster) is a massive 

  parallel-processing computing platform that solves Big Data problems.

provides:

  hpcc:

    interface: hpcc

requires:

  hpcc-thor:

    interface: hpcc-thor

  hpcc-roxie:

    interface: hpcc-roxie

peers:

  hpcc-cluster:

    interface: hpcc-cluster

 There are various provides and requires interfaces in this metadata.yaml file but, for now, only the peers interface is being used.  I'll work on the other ones as the charm matures.


Let's look at the hpcc-cluster interface.  More specifically the hpcc-cluster-relation-changed hook where the new configuration is created:

#!/bin/bash

CWD=$(dirname $0)

cheetah fill --oext=xml --odir=/etc/HPCCSystems/ ${CWD}/../templates/environment.tmpl

service hpcc-init restart

It's pretty simple isn't it?   Since the "heavy lifting" is being done with the self contained cheetah template , we don't have much to do here but, to generate the configuration file and restart hpcc.

The other files in this charm are pretty self explanatory and simple so, I am leaving the details of them as an exercise to the reader.

All of the complexities in hpcc has been distilled to the following commands:

• juju bootstrap
• bzr branch lp:~negronjl/+junk/hpcc
• juju deploy --repository . hpcc
• wait a minute of two
• juju status
• you should see something similar to this:

negronjl@negronjl-laptop:~/src/juju/charms$ juju status
2011-08-18 16:00:54,413 INFO Connecting to environment.


machines: 
  0: {dns-name: ec2-184-73-109-244.compute-1.amazonaws.com, instance-id: i-6d61460c} 
  1: {dns-name: ec2-50-16-60-94.compute-1.amazonaws.com, instance-id: i-d5694eb4}


services:
  hpcc:
    charm: local:hpcc-1
    relations: {hpcc-cluster: hpcc}
    units:
      hpcc/0:
        machine: 1
        relations: {}
        state: null


2011-08-18 16:00:58,374 INFO 'status' command finished successfully
negronjl@negronjl-laptop:~/src/juju/charms$

The above commands, will give you a single node.

You can access the web interface of your node by pointing your browser to http://<FQDN>:8010 Where FQDN is the Fully Qualified Domain Name or Public IP Address of your hpcc instance.  On the left side, there should be a menu, explore the items on the Topology section.  The Target Clusters section should look something similar to this:

To experience the true power of hpcc, you should probably throw in some more nodes at it.  Let's do just that with:

• juju add-unit hpcc 
• do this as many times as you feel comfortable   
• Each command will give you a new node in the cluster
• wait a minute or two and you should see something similar to this:

negronjl@negronjl-laptop:~$ juju status

2011-08-18 16:25:55,739 INFO Connecting to environment.

machines:

  0: {dns-name: ec2-184-73-109-244.compute-1.amazonaws.com, instance-id: i-6d61460c}

  1: {dns-name: ec2-50-16-60-94.compute-1.amazonaws.com, instance-id: i-d5694eb4}

  2: {dns-name: ec2-50-19-181-98.compute-1.amazonaws.com, instance-id: i-a5795ec4}

  3: {dns-name: ec2-184-72-147-67.compute-1.amazonaws.com, instance-id: i-25446344}

services:

  hpcc:

    charm: local:hpcc-1

    relations: {hpcc-cluster: hpcc}

    units:

      hpcc/0:

        machine: 1

        relations:

          hpcc-cluster: {state: up}

        state: started

      hpcc/1:

        machine: 2

        relations:

          hpcc-cluster: {state: up}

        state: started

      hpcc/2:

        machine: 3

        relations:

          hpcc-cluster: {state: up}

        state: started

2011-08-18 16:26:01,837 INFO 'status' command finished successfully

Notice how we now have more hpcc nodes :)  Here is what the web interface could look like:

Again....we have more nodes :)

Now that we have a working cluster, let's try it.  We'll first do the mandatory Hello World in ECL.  It looks something like this (hello.ecl):

Output('Hello world');

 We have to compile our hello.ecl so we can use it.  We do that by logging into one of the nodes ( I used juju ssh 1 to log on to the first/master node ) and typing the following:

eclcc hello.ecl -o

We run the file just like we would any other binary:

./hello

... and the output is:

ubuntu@ip-10-111-19-210:~$ ./hello
Hello world
ubuntu@ip-10-111-19-210:~$ 

There are far more interesting examples in the Learning ECL Documentation here.  I highly encourage you to go and read about it.

That's it for now.  Feedback is always welcome of course so, let me know how I'm doing.

-Juan

Ensemble security and firewall enhancements

// August 16th, 2011 // 3 Comments » // Uncategorized

A lot of work is going into making sure Ensemble is more secure and enterprise ready. As part of that, all deployed services are now firewalled by default and for a formula deployed service to be publically accessible, the formula author has to specify which ports are open and when, as well as the operator needs to signal wanting to open that port. All formulas that expose ports should use open-port (and optionally close-port) diligently. Here’s what you need to know

Updating formulas for the new expose functionality

This is the only change necessary for the WordPress/MySQL example, in example/wordpress/hooks/db-relation-changed:

  # Make it publicly visible, once the wordpress service is exposed
  open-port 80/tcp

It is important that formulas open ports only when ready. So in the WordPress example, you wouldn’t want to do this port opening until Apache has been successfully configured and restarted. Otherwise, there’s a chance that users might see “It works!” before the desired page is available.

Firewall changes also are a two-step process. The hooks for a service unit need to open ports (and they can also close ports), but the Ensemble administrator must also expose the service. For the WordPress example, you can expose it any time after the service has been deployed with the following:

  ensemble expose wordpress

Just expose the services you’re interested in exposing, possibly as soon as immediately after deployment. Again, it’s the formula author’s responsibility to ensure that port opening is done at the right time.

The service can be subsequently unexposed with

  ensemble unexpose wordpress

You can see if a service is exposed with ensemble status. This would result in output similar to the following:

$ ensemble status
2011-08-09 17:59:29,704 INFO Connecting to environment.
machines:
  0: {dns-name: ec2-50-18-5-80.us-west-1.compute.amazonaws.com,
instance-id: i-1119e556}
  1: {dns-name: ec2-50-18-73-159.us-west-1.compute.amazonaws.com,
instance-id: i-531ae614}
  2: {dns-name: ec2-50-18-139-254.us-west-1.compute.amazonaws.com,
instance-id: i-671ae620}
services:
  mysql:
    formula: local:mysql-11
    relations: {db: wordpress}
    units:
      mysql/0:
        machine: 1
        relations:
          db: {state: up}
        state: started
  wordpress:
    exposed: true
    formula: local:wordpress-30
    relations: {db: mysql}
    units:
      wordpress/0:
        machine: 2
        open-ports: [80/tcp]
        relations:
          db: {state: up}
        state: started
2011-08-09 17:59:36,031 INFO 'status' command finished successfully

This work is only part of the effort to ensure Ensemble uses secure mechanisms in its operations. Recent work also made sure all state information between cloud nodes are properly access controlled to avoid leaking any confidential data. Ensemble is rapidly progressing, and now is a great time to start playing with the technology, and to start writing your own formulas!

Interested? Join the friendly Ensemble community at #ubuntu-ensemble on IRC freenode, drop in, say hi, and grab me (kim0) for any questions

MongoDB replicaset on Amazon ec2 with Ensemble

// August 15th, 2011 // 3 Comments » // Uncategorized

MongoDB is such a great piece of open-source technology. It supports some very interesting features such as sharding and replica-sets. I have seen demos of MongoDB, where the speaker happily calls creating the replica-set cluster a “one hour thing“! I decided to sprinkle some Ensemble magic on this problem, using Jaun’s formulas, the problem becomes a “10 second thing” basically! Spinning up a Mongo replica-set cluster could not be easier! Check this video out

If you can’t see the embedded video, here’s a direct link http://youtu.be/CVgMA6Hi7rw

Yep that’s how simple it is! If you want to create more read-slaves, you only need to ask Ensemble to do it for you:

$ ensemble add-unit mongodb

If you’re interested to learn more about exactly how this “magic” works, check out this indepth guide dissecting how the Mongo Ensemble formulas exactly works by “Juan Negron” the formula author.

So was this useful? Will you be deploying your next mongodb servers with Ensemble?
Leave me a comment, let me know your thoughts! Also let me know what you’d like to see deployed next with Ensemble. Be sure to drop in to #ubuntu-ensemble on freenode irc and say hi

Easy cassandra deployments with Ubuntu Server and Juju

// August 12th, 2011 // Comments Off // Uncategorized

Easy cassandra deployments with Ubuntu Server and Ensemble

// August 12th, 2011 // 2 Comments » // Uncategorized

MongoDB Replica Sets with Ubuntu Server and Juju

// August 11th, 2011 // Comments Off // Uncategorized

• stand alone deployment. 
• replica sets.  More information about replica sets can be found here.
• master and server relationships
• Don't try to solve all deployment scenarios just concentrate on the above ones for now.

sudo add-apt-repository ppa:juju/pkgs
sudo apt-get update
sudo apt-get install charm-tools

• charm create mongodb

mongodb:
mongodb/metadata.yaml
mongodb/hooks
mongodb/hooks/install
mongodb/hooks/start
mongodb/hooks/stop
mongodb/hooks/relation-name-relation-joined
mongodb/hooks/relation-name-relation-departed
mongodb/hooks/relation-name-relation-changed
mongodb/hooks/relation-name-relation-broken

MongoDB Replica Sets with Ubuntu Server and Ensemble

// August 11th, 2011 // 2 Comments » // Uncategorized

• stand alone deployment. 
• replica sets.  More information about replica sets here.
• master and server relationships
• Don't try to solve all deployment scenarios just concentrate on the above ones for now.

sudo add-apt-repository ppa:ensemble/ppa
sudo apt-get update
sudo apt-get install principia-tools

• principia formulate mongodb

mongodb:
mongodb/metadata.yaml
mongodb/hooks
mongodb/hooks/install
mongodb/hooks/start
mongodb/hooks/stop
mongodb/hooks/relation-name-relation-joined
mongodb/hooks/relation-name-relation-departed
mongodb/hooks/relation-name-relation-changed
mongodb/hooks/relation-name-relation-broken

Hadoop cluster with Ubuntu server and Ensemble

// August 8th, 2011 // 7 Comments » // Uncategorized

• Ensemble is a next generation service orchestration framework. It has been likened to APT for the cloud. With Ensemble, different authors are able to create service formulas independently, and make those services coordinate their communication through a simple protocol. Users can then take the product of different authors and very comfortably deploy those services in an environment. The result is multiple machines and components transparently collaborating towards providing the requested service.

• hadoop-master (namenode and jobtracker )
• hadoop-slave ( datanode and tasktracker )

1. namenode ( leave blank if you are the namenode )
2. jobtracker ( leave blank if you are the jobtracker )
3. hdfs data directory ( leave blank to use the default: /var/lib/hadoop-0.20/dfs/data )

hadoop-master
hadoop-master/metadata.yaml
hadoop-master/hooks/install
hadoop-master/hooks/start
hadoop-master/hooks/stop
hadoop-master/hooks/hadoop-master-relation-joined

• ensemble bootstrap   # ( creates/bootstraps the ensemble environment)
• ensemble deploy --repository . hadoop-master # ( deploys hadoop-master )
• ensemble deploy --repository . hadoop-slave # ( deploys hadoop-slave )
• ensemble add-relation hadoop-slave hadoop-master # ( connects the hadoop-slave to the hadoop-master )

• ensemble add-unit hadoop-slave # ( this adds one more slave )

Ensemble meets Hadoop on the cloud

// August 8th, 2011 // 8 Comments » // Uncategorized

So you wanted to play with hadoop to crunch on some big-data problems, except that, well getting a hadoop cluster up and running in not exactly a one minute thing! Let me show you how to make it “a one minute thing” using Ensemble! Since Ensemble now has formulas for creating hadoop master and slave nodes, thanks to the great work of Juan Negron. Spinning up a hadoop cluster could not be easier! Check this video out

If you can’t see the embedded video, here’s a direct link http://youtu.be/e8IKkWJj7bA

Yep that’s how simple it is! If you want to scale-out the cluster, you only need to ask Ensemble to do it for you:
$ ensemble add-unit hadoop-slave

If you’re interested to learn more about exactly how this “magic” works, check out this indepth guide dissecting how the hadoop Ensemble formulas exactly work by non-other than Juan Negron, the formula author.

So is this easier than configuring a hadoop cluster manually? Leave me a comment, let me know your thoughts! Also let me know what you’d like to see deployed next with Ensemble. Be sure to drop in to #ubuntu-ensemble on freenode irc and say hi