Using Hiera with Puppet (to manage different environments)

Recently I had to install Apache Stratos using Puppet. The requirement was to install 3 Stratos environments (Lets say Dev, Test, Prod). Inorder to manage these three environment I had two options,
1. Use Puppet environments
2. Use Hiera

If I used the first method, I have to keep separate site.pp for each environment. And in Puppet agent, I have to define the environment (Dev, Test or Prod). So I decided to use the second method.

If you want to know what is Hiera and how to install/configure puppetlabs have a very detailed document with examples[1]
So in this blog post is all about how I used Hiera to manage the Dev,Test, Prod Stratos environments.

Hiera Configs

/etc/puppet/hiera.yaml file
---
:backends: 
  - yaml
:yaml:
  :datadir: /etc/puppet/hieradata
:hierarchy:
  - "stratos_%{::stratos_env}" 
I created three files in /etc/puppet/hieradata/for each environment (Dev,Test,Prod).

/etc/puppet/hieradata/stratos_dev.yaml
---
sc_ip: ip-10-0-0-1.ec2.internal
sc_port: 9443
mb_ip: ip-10-0-0-2.ec2.internal
mb_port: 5672
cep_ip: ip-10-0-0-3.ec2.internal
cep_port: 7621
cc_ip: ip-10-0-0-4.ec2.internal
cc_port: 9463
as_ip: ip-10-0-0-5.ec2.internal
as_port: 9473

truststore_password: wso2carbon
internal_repo_user: admin
internal_repo_password: admin123

stratos_datasource:
  hostname: stratosdb.manula.us-east-1.rds.amazonaws.com
  database: dbSManagerDevConfig
  username: DbUserDev
  password: dbuserDev123

registry_datasource:
  hostname: stratosdb.manula.us-east-1.rds.amazonaws.com
  database: dbGovernanceS4Dev
  username: GovDevUser
  password: GovDevUser123
 
/etc/puppet/hieradata/stratos_test.yaml
---
sc_ip: ip-10-1-0-1.ec2.internal
sc_port: 9443
mb_ip: ip-10-1-0-2.ec2.internal
mb_port: 5672
cep_ip: ip-10-1-0-3.ec2.internal
cep_port: 7621
cc_ip: ip-10-1-0-4.ec2.internal
cc_port: 9463
as_ip: ip-10-1-0-5.ec2.internal
as_port: 9473

truststore_password: wso2carbon
internal_repo_user: admin
internal_repo_password: admin123

stratos_datasource:
  hostname: stratosdb.manula.us-east-1.rds.amazonaws.com
  database: dbSManagerTestConfig
  username: DbUserTest
  password: dbuserTest123

registry_datasource:
  hostname: stratosdb.manula.us-east-1.rds.amazonaws.com
  database: dbGovernanceS4Test
  username: GovTestUser
  password: GovTestUser123
 

How it works

If you check my hiera.yaml file you can see that there is only one datasource with 'stratos_env' variable. This stratos_env is an external fact[2] which I have set in the puppet agent node.

* This setup uses pre-built images (backed-images) with setup of scripts. These scripts configure the instances and start the puppet agent at the boot time. One important thing is when launching an instance we set custom facts as the user-data. The script I mentioned downloads the user-data and export them as facts. To start and Stratos Dev instance all I have to do is set the stratos_env=dev when launching an instance.

How Puppet work with Hiera

To get data from hiera to puppet manifests, we have to use 'hiera' function call. I have only used the hiera function calls in the nodes.pp (I've imported nodes.pp file from the sites.pp) file. If I didn't use hiera I have to define all the variables in nodes.pp file and change the variables in the templates or use separate environment as I mentioned above.

This is how my nodes.pp file.
node 'base' {
  $package_repo         = 'https://products.wso2.com/packs'
  $local_package_dir    = '/mnt/packs'

  $domain              = 'cloudpreview.wso2.com'

  $admin_username      = 'admin'
  $admin_password      = 'admin123'
  $carbon_db_password  = 'wso2carbon'

  $registry_datasource = hiera('registry_datasource')

  $puppet_ip           = '10.4.0.1'
  $git_ip              = '10.4.0.2'

  $sc_ip               = hiera('sc_ip')
  $sc_port             = hiera('sc_port')
  $mb_ip               = hiera('mb_ip')
  $mb_port             = hiera('mb_port')
  $cep_ip              = hiera('cep_ip')
  $cep_port            = hiera('cep_port')
  $cc_ip               = hiera('cc_ip')
  $cc_port             = hiera('cc_port')
  $as_ip               = hiera('as_ip')
  $as_port             = hiera('as_port')

  $truststore_password      = hiera('truststore_password')
  $internal_repo_user       = hiera('internal_repo_user')
  $internal_repo_password   = hiera('internal_repo_password')
}


[1] http://docs.puppetlabs.com/hiera/1/index.html
[2] http://docs.puppetlabs.com/guides/custom_facts.html


Port mirroring with OpenvSwitch

I got a chance to play with Openvswitch again. This time the requirement was to configure a network tap. The first time I saw the requirement, I thought of using a bridge with ageing 0 (Then it works like a hub - this setting is to configure MAC address timeout). We can set the aging time using the command below.
brctl setageing bridge1 0
But later I decided to use Openvswitch as setting the ageing time isn't going to be a good solution.
Setting a port mirror is very easy. It could be done by running the command below.
ovs-vsctl -- set Bridge bridge1 mirrors=@m -- --id=@vif1.0 get Port vif1.0 -- --id=@vif2.0 get Port vif2.0 -- --id=@m create Mirror name=mirror1 select-dst-port=@vif1.0 select-src-port=@vif1.0 output-port=@vif2.0
Here what I'm doing is mirroring the port vif1.0 to to vif2.0 (out-put port - so that vif2.0 can see traffic coming in or going out of vif1.0)
But after setting up the mirror I cannot access the vif2.0 So I attached another interface to that VM (it would be vif2.1).

Note: Virtual interface id consists of - "vif{$DOMID}.${DEVID}". DOMID is the domain id of the VM. DEVID is the device id for example the first interface is 0 second is 1.

So with the above port mirroring command I'm monitoring network traffic in VM1(vm with domain id 1) from VM2(vm with domain id 2).
We can simply check this by loggin in toVM2 and running tcpdump command(or any other monitoring tool) on interface 1.
But later I faced another problem. Which is if we reboot the VMs the port mirroring doesn't work. The reason is when a VM reboots the VM ID changes so does the Virtual interface id. To fix that I had to write a shell script.

References
[1] http://git.openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob_plain;f=FAQ;hb=HEAD
[2] http://n40lab.wordpress.com/2013/02/23/openvswitch-port-mirroring/






Monitoring a HTTPS URL with Nagios

By default Nagios supports URL monitoring and there is a command in /etc/nagios-plugins/config/http.cfg (In debian based systems). But if you want to monitor an URL like https://esb.wso2.com:9443/services/echo?wsdl2 you cannot use the default command.
We can use a command like below to monitor a URL like the above one.
define command{
   command_name    check_https_url
   command_line    /usr/lib/nagios/plugins/check_http --ssl -H '$HOSTADDRESS$' -p '$ARG1$' -u '$ARG2$'
}

To add the comnmand to Nagios create a file named https_url.cfg in /etc/nagios-plugins/config/ with the above code (and reload nagios server).

To use the check_https_url add the code below to the host configuration file. For example (in /etc/nagios3/conf.d/server.cfg)

define service {
    host_name           appserver.wso2.com
    service_description WEB SERVICE
    check_command       check_https_url!8243!/services/echo%3Fwsdl2
    use                 generic-service
}
Command arguments are port and the URI

Script to wait for a Service (or a download link)

I wrote a script to download the OpenStack metadata at the boot time. But the downloaded file was empty. But when I executed wget after booting it downloaded the metadata file. So I added the following to the script file to wait for the metadata service.

#!/bin/bash
while ! wget --spider http://169.254.169.254/latest/user-data 2>&1 | grep --quiet "200 OK"
do
  echo "Not available"
  sleep 2
done

But later I found out that the reason for not downloading metadata :). The VM Operating System was SUSE and I executed the script from boot.local (/etc/init.d/boot.local In other operating systems I add the script to rc.local). And this boot.local runs in run level 2 (with no networking :D). You can find out more about run levels from here.

How to add a custom laucher to Gnome Shell Dock (Favourites menu)

I installed Firefox on Debian wheezy. But the problem was I couldn't add the launcher to gnome shell dock (favorites menu).

First thing I did was opened the "dconf editor" and checked the gnome-shell configuration parameters (org -> gnome -> shell).  There I found the favorite-apps variable with several apps. (Check the screenshot below)
BTW the new screenshot tool is great :)

Then I executed locate gnome-terminal.desktop and found the location of the .desktop files. (There were two /usr/share/app-install/desktop/gnome-terminal.desktop and /usr/share/applications/gnome-terminal.desktop)

I copied the iceweasel.desktop file and created the firefox.desktop file (in both locations).

[Desktop Entry]
Encoding=UTF-8
Name=Firefox
Comment=Browse the World Wide Web
GenericName=Web Browser
X-GNOME-FullName=Firefox Web Browser
Exec=/opt/firefox/firefox %u
Terminal=false
X-MultipleArgs=false
Type=Application
Icon=firefox
Categories=Network;WebBrowser;
MimeType=text/html;text/xml;application/xhtml+xml;application/xml;application/vnd.mozilla.xul+xml;application/rss+xml;application/rdf+xml;image/gif;image/jpeg;image/png;x-scheme-handler/http;x-scheme-handler/https;
StartupWMClass=Firefox
StartupNotify=true

I opened the dconf editor and added the firefox.desktop to favorite-apps. The firefox launcher was added to the gnome dock. :)

TIP: When I tried to download firefox from firefox website it downloaded the 32 bit version. So what I did was copied the download link , removed the last part,  browsed the available downloads and downloaded the 64 bit version. (http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/22.0/linux-x86_64/en-US/).

How to run Jenkins under a different user in Linux [Redhat]

When I wanted to change the Jenkins user I first checked /etc/init.d/jenkins script.  There I found two important variables $JENKINS_CONFIG(=/etc/sysconfig/jenkins) and $JENKINS_USER. So I you want you can change the JENKINS_USER variable in the /etc/init.d/jenkins file; but it is not the correct way to do.

To change the jenkins user open the /etc/sysconfig/jenkins (in debian this file is created in /etc/default) and change the JENKINS_USER to whatever you want (make sure that user is in the /etc/passwd file).
$JENKINS_USER="manula"
Then change the ownership of the Jenkins home, Jenkins webroot and logs.
chown -R manula:manula /var/lib/jenkins 
chown -R manula:manula /var/cache/jenkins
chown -R manula:manula /var/log/jenkins
Then restarted the Jenkins jenkins and check the user has changed using a ps command
/etc/init.d/jenkins restart
ps -ef | grep jenkins

Monitoring Hadoop Cluster With Ganglia

Setting up Ganglia Monitoring server


Install gmetad and ganglia-webfrontend on the monitoring server.
apt-get install gmetad ganglia-webfrontend
Copy or create a symlink to /etc/ganglia-webfrontend/apache.conf and reload apache
ln -s /etc/ganglia-webfrontend/apache.conf /etc/apache2/conf.d/ganglia.conf
/etc/init.d/apache2 reload 
To monitor hadoop cluster add the following line to the configuration file of the gmetad. (/etc/ganglia/gmetad.conf)
data_source "Hadoop" 10.0.0.1
(10.0.0.1 is the IP address of the Cluster data collector)

Configuring Hadoop nodes

In this setup hadoop nodes send their statistics to one node. Then the monitoring server fetches the all(cluster) statistics from that node.

Installing gmond
apt-get install ganglia-monitor

Cluster data collector
Change gmond config file to match the following settings.(/etc/ganglia/gmond.conf)
cluster {
  name = "Hadoop"
  owner = "unspecified"
  latlong = "unspecified"
  url = "unspecified"
}

udp_recv_channel {
  port = 8649
  bind = 10.0.0.1
} 
Hadoop node (Other nodes)
Change the gmond config file to match the following settings.(/etc/ganglia/gmond.conf)
 
cluster {
  name = "Hadoop"
  owner = "unspecified"
  latlong = "unspecified"
  url = "unspecified"
}

udp_send_channel {
  host = 10.0.0.1
  port = 8649
}
Hadoop settings (all hadoop nodes)
To enable monitoring hadoop with ganglia uncomment the necessary lines in $HADOOP_HOME/conf/hadoop-metrics2.properties file.
*.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
# for Ganglia 3.1 support
*.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31
*.sink.ganglia.period=10

# default for supportsparse is false
*.sink.ganglia.supportsparse=true
*.sink.ganglia.slope=jvm.metrics.gcCount=zero,jvm.metrics.memHeapUsedM=both
*.sink.ganglia.dmax=jvm.metrics.threadsBlocked=70,jvm


Start the hadoop cluster. Restart the gmetad on monitoring server and gmond in hadoop nodes if you didn't restart it after changes.
Ganglia web interface can be accessed by http:///ganglia
[Web interface path can be found in /etc/apache2/conf.d/ganglia.conf]