Hadoop

Hadoop is an open source software for storing and processing large data sets using a cluster of commodity servers. This article shows how sFlow agents can be installed to monitor the performance of a Hadoop cluster.

Note: This example is based on the commercially Hadoop distribution from Cloudera. The virtual machine downloads from Cloudera are a convenient way to experiment with Hadoop - providing a fully functional Hadoop system on a single virtual machine.

First, install Host sFlow on each node in the Hadoop cluster, see Installing Host sFlow on a Linux server. The Host sFlow agents export standard CPU, memory, disk and network statistics from the servers. In this case, the Host sFlow agent was installed on the Cloudera virtual machine.

Since Hadoop is implemented in Java, the next step involves installing and configuring the sFlow java agent, see Java virtual machine. In this case, the Hadoop java virtual machines are configured in the /etc/hadoop/conf/hadoop-env.sh file. The following fragment from hadoop-env.sh highlights the additional java command line options needed to implement sFlow monitoring of each Hadoop daemon:

# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-javaagent:/etc/hadoop/conf/sflowagent.jar -Dsflow.
hostname=hadoop.namenode -Dsflow.dsindex=50070 -Dcom.sun.management.jmxremote $H
ADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-javaagent:/etc/hadoop/conf/sflowagent.jar
 -Dsflow.hostname=hadoop.secondarynamenode -Dsflow.dsindex=50090 -Dcom.sun.manag
ement.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-javaagent:/etc/hadoop/conf/sflowagent.jar -Dsflow.
hostname=hadoop.datanode -Dsflow.dsindex=50075 -Dcom.sun.management.jmxremote $H
ADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-javaagent:/etc/hadoop/conf/sflowagent.jar -Dsflow.
hostname=hadoop.balancer -Dsflow.dsindex=50060 -Dcom.sun.management.jmxremote $H
ADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-javaagent:/etc/hadoop/conf/sflowagent.jar -Dsflo
w.hostname=hadoop.jobtracker -Dsflow.dsindex=50030 -Dcom.sun.management.jmxremote
 $HADOOP_JOBTRACKER_OPTS"

Note: The sflowagent.jar file was placed in the /etc/hadoop/conf directory. Each daemon was given a descriptive sflow.hostname value corresponding to the daemon name. In addition, a unique sflow.dsindex value must be assigned to each daemon - the index values only need to be unique within a single server - all the servers in the cluster can share the same configuration settings. In this case the port that each daemon listens on is used as the index.

The Host sFlow and Java sFlow agents share configuration, see Host sFlow distributed agent, simplifying the task of configuring sFlow monitoring within each server and across the cluster.

Note: If you are using Ganglia monitor performance of the cluster, then you need to set multiple_jvm_instances = yes since there is more than one Java daemon running on each server, see Using Ganglia to monitor Java virtual machines.

Finally, Hadoop is sensitive to network congestion and generates large traffic volumes. Fortunately, most switch vendors support the sFlow standard. Combining sFlow from the Hadoop servers with sFlow from the switches offers comprehensive visibility into the performance of a Hadoop cluster.

Host sFlow distributed agent

The sFlow standard uses a distributed push model in which each host autonomously sends a continuous stream of performance metrics to a central sFlow collector. The push model is highly scalable and is particularly suited to cloud environments.

The distributed architecture extends to the sFlow agents within a host. The following diagram shows how sFlow agent coordinate on each host to provide a comprehensive and consistent view of performance.

Typically, all hosts in the data center will share identical configuration settings. The Host sFlow daemon, hsflowd, provides a single point of configuration. By default hsflowd uses DNS Service Discover (DNS-SD) to learn configuration settings. Alternatively, settings can be distributed using orchestration software such as Puppet or Chef to update each host's hsflowd configuration file, /etc/hsflowd.conf, and restart the daemon.

The hsflowd daemon writes configuration information that it receives via DNS-SD or through its configuration file to the /etc/hsflowd.auto file. Other sFlow agents running on the host automatically detect changes to the /etc/hsflowd.auto file and apply the configuration settings.

The sFlow protocol allows each agent to operate autonomously and send an independent stream of metrics to the sFlow collector. Distributing monitoring among the agents eliminates dependencies and synchronization challenges that would increase the complexity of the agents.

Each of the sFlow agents is responsible for exporting a different set of metrics:

• hsflowd The Host sFlow daemon exports CPU, memory, disk and network IO performance metrics and can also export per virtual machine statistics when run on a Xen, XenServer or KVM hypervisor. In addition, traffic monitoring using iptables/LOG is supported on Linux platforms.
• Open vSwitch The Open vSwitch is "the default switch in XenServer 6.0, the Xen Cloud Platform and also supports Xen, KVM, Proxmox VE and VirtualBox. It has also been integrated into many virtual management systems including OpenStack, openQRM, and OpenNebula." Enabling the build-in sFlow monitoring on the virtual switch offers the same visibility as sFlow on physical switches and provides a unified end-to-end view of network performance across the physical and virtual infrastructure. The sflowovsd daemon ships with Host sFlow and is used to configure sFlow monitoring on the Open vSwitch using the ovs-vsctl command. Similar integrated sFlow support has also been demonstrated for the Microsoft extensible virtual switch that is part of the upcoming Windows 8 version of Hyper-V.
• java The Java sFlow agent exports performance statistics about java threads, heap/non-heap memory, garbage collection, compilation and class loading.
• httpd An sFlow agent embedded in the web server exports key performance metrics along with detailed transaction data that can be used to monitor top URLs, top Referers, top clients, response times etc. Currently, there are implementation of sFlow for Apache, NGINX, Tomcat and node.js web servers, see http://host-sflow.sourceforge.net/relatedlinks.php.
• Memcached An sFlow agent embedded in the Memcache server exports performance metrics along with detailed data on Memcache operations that can be used to monitor hot keys, missed keys, top clients etc. Currently, there is an implementation of sFlow for Memcached, see http://host-sflow.sourceforge.net/relatedlinks.php.

Together the agents running on each host, along with sFlow agents embedded within the network infrastructure form an integrated monitoring system that provides a unified view view of network, system, storage and application performance that can scale to hundreds of thousands of servers.

Using Ganglia to monitor Java virtual machines

The Ganglia charts show the standard sFlow Java virtual machine metrics. The combination of Ganglia and sFlow provides a highly scalable solution for monitoring the performance of clustered Java application servers. The sFlow Java agent for stand-along Java services, or Tomcat sFlow for web-based servlets, simplify deployments by eliminating the need to poll for metrics using a Java JMX client. Instead, metrics are pushed directly from each Java virtual machine to the central Ganglia collector.

Note: The Tomcat sFlow agent also allows Ganglia to report HTTP performance metrics.

The article, Ganglia 3.2 released, describes the basic steps needed to configure Ganglia as an sFlow collector. Once configured, Ganglia will automatically discover and track new servers as they are added to the network. The articles, Java virtual machine and Tomcat, describes the steps needed to instrument existing Java applications and Apache Tomcat servlet engines respectively. In both cases the sFlow agent is included when starting the Java virtual machine and requires minimal configuration and no change to the application code.

Note: To try out Ganglia's sFlow/Java reporting, you will need to download Ganglia 3.3.

By default, Ganglia will automatically start displaying the Java virtual machine metrics. However, there are two optional configuration settings available in the gmond.conf file that can be used to modify how Ganglia handles the sFlow Java metrics.

sflow{
  accept_jvm_metrics = yes
  multiple_jvm_instances = no
}

Setting the accept_jvm_metrics flag to no will cause Ganglia to ignore Java virtual machine metrics.

The multiple_jvm_instances setting must be set to yes in cases where there are multiple Java virtual machine instances running on each server in the cluster. Charts associated with each java virtual machine instance will be identified by a unique "hostname" included in the title of its charts. For example, the following chart is identified as being associated with the apache-tomcat java virtual machine on host xenvm4.sf.inmon.com:

Ganglia and sFlow offers a comprehensive view of the performance of a cluster of Java servers, providing not just Java related metrics, but also the server CPU, memory, disk and network IO performance metrics needed to fully characterize cluster performance.

Java virtual machine

The jmx-sflow-agent project is an open source implementation of sFlow monitoring for Java Virtual Machines (JVM). Instrumenting Java using sFlow provides scalable, real-time monitoring for applications such as Hadoop, Cassandra and Tomcat that typically involve large numbers of virtual machines.

The sFlow Host Structures extension describes a set of standard metrics for monitoring physical and virtual machines. The jmx-sflow-agent extends the set of physical server metrics exported by a Host sFlow agent which exports the physical server metrics.

The jmx-sflow-agent exports the generic virtual machine structures. In addition, the jmx-sflow-agent uses the Java Management Extensions (JMX) interface to collect performance statistics about java threads, heap/non-heap memory, garbage collection, compilation and class loading. These additional metrics are exported along with the generic virtual machine statistics.

The jmx-sflow-agent software is designed to integrate with the Host sFlow agent to provide a complete picture of server performance. Download, install and configure Host sFlow before proceeding to install the tomcat-sflow-valve - see Installing Host sFlow on a Linux Server. There are a number of options for analyzing cluster performance using Host sFlow, including Ganglia and sFlowTrend.

Next, download the sflowagent.jar file from https://github.com/sflow/jmx-sflow-agent. Copy the sflowagent.jar file into the same directory as your Java application. Enabling the sFlow instrumentation involves including -javaagent:sflowagent.jar argument when starting the Java application.

The following example shows how instrumentation is added to MyApp:

java -javaagent:sflowagent.jar\
     -Dsflow.hostname=vm.test\
     -Dsflow.uuid=564dd23c-453d-acd2-fa64-85da86713620\
     -Dsflow.dsindex=80001\
     MyApp

Arguments can be passed to the sFlow module as system properties:

• sflow.hostname, optionally assign a "hostname" to identify the virtual machine. Choose a naming strategy that is helpful in identifying virtual machine instances, for example: "hadoop.node1" etc. The hostname is exported in the sFlow host_descr structure.
• sflow.uuid, optionally assign a UUID to the virtual machine so that it can be uniquely identified. The UUID is exported in the sFlow host_descr structure.
• sflow.dsindex, uniquely identifies the data source associated with this virtual machine on this server. The dsindex number only needs to be set if more than one virtual machine is running on the server. For virtual machines offering network services, use TCP/UDP port number associated with the service as the dsindex, otherwise, use numbers in the range 65536-99999 to avoid clashes with other sFlow agents on the server.

The sFlow module shares Host sFlow's configuration and will automatically send data to the set of sFlow analyzers specified in the Host sFlow configuration file, picking up any configuration changes without the need to restart the Java application.

Finally, the real potential of of the Java virtual machine sFlow agent is as part of a broader sFlow monitoring system providing scaleable, real-time, visibility into applications, servers, storage and networking across the entire data center.

Tomcat

The tomcat-sflow-valve project is an open source implementation of sFlow monitoring for Apache Tomcat, an open source web server implementing the Java Servlet and JavaServer Pages (JSP) specifications.

The advantage of using sFlow is the scalability it offers for monitoring the performance of large web server clusters or load balancers where request rates are high and conventional logging solutions generate too much data or impose excessive overhead. Real-time monitoring of HTTP provides essential visibility into the performance of large-scale, complex, multi-layer services constructed using Representational State Transfer (REST) architectures. In addition, monitoring HTTP services using sFlow is part of an integrated performance monitoring solution that provides real-time visibility into applications, servers and switches (see sFlow Host Structures).

The tomcat-sflow-valve software is designed to integrate with the Host sFlow agent to provide a complete picture of server performance. Download, install and configure Host sFlow before proceeding to install the tomcat-sflow-valve - see Installing Host sFlow on a Linux Server. There are a number of options for analyzing cluster performance using Host sFlow, including Ganglia and sFlowTrend.

Next, download the tomcat-sflow-valve software from http://tomcat-sflow-valve.googlecode.com/. The following steps install the SFlowValve in a Tomcat 7 server.

tar -xvzf tomcat-sflow-valve-0.5.1.tar.gz
cp sflowvalve.jar $TOMCAT_HOME/lib/

Edit the $TOMCAT_HOME/conf/server.xml file and insert the following Valve statement in the Host section of the Tomcat configuration file:

<Host>
...
  <Valve className="com.sflow.catalina.SFlowValve" />
</Host>

Restart Tomcat:

/sbin/service tomcat restart

Once installed, the sFlow Valve will stream measurements to a central sFlow Analyzer. Currently the only software that can decode HTTP sFlow is sflowtool. Download, compile and install the latest sflowtool sources on the system your are using to receive sFlow from the servers in the Tomcat cluster.

Running sflowtool will display output of the form:

[pp@test]$ /usr/local/bin/sflowtool
startDatagram =================================
datagramSourceIP 10.0.0.111
datagramSize 116
unixSecondsUTC 1294273499
datagramVersion 5
agentSubId 6486
agent 10.0.0.150
packetSequenceNo 6
sysUpTime 44000
samplesInPacket 1
startSample ----------------------
sampleType_tag 0:2
sampleType COUNTERSSAMPLE
sampleSequenceNo 6
sourceId 3:65537
counterBlock_tag 0:2201
http_method_option_count 0
http_method_get_count 247
http_method_head_count 0
http_method_post_count 2
http_method_put_count 0
http_method_delete_count 0
http_method_trace_count 0
http_methd_connect_count 0
http_method_other_count 0
http_status_1XX_count 0
http_status_2XX_count 214
http_status_3XX_count 35
http_status_4XX_count 0
http_status_5XX_count 0
http_status_other_count 0
endSample   ----------------------
startSample ----------------------
sampleType_tag 0:1
sampleType FLOWSAMPLE
sampleSequenceNo 3434
sourceId 3:65537
meanSkipCount 2
samplePool 7082
dropEvents 0
inputPort 0
outputPort 1073741823
flowBlock_tag 0:2100
extendedType socket4
socket4_ip_protocol 6
socket4_local_ip 10.0.0.150
socket4_remote_ip 10.1.1.63
socket4_local_port 80
socket4_remote_port 61401
flowBlock_tag 0:2201
flowSampleType http
http_method 2
http_protocol 1001
http_uri /favicon.ico
http_host 10.0.0.150
http_referrer http://10.0.0.150/membase.php
http_useragent Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_5; en-us) AppleW
http_bytes 284
http_duration_uS 335
http_status 404
endSample   ----------------------
endDatagram   =================================

The -H option causes sflowtool to output the HTTP request samples using the combined log format:

[pp@test]$ /usr/local/bin/sflowtool -H
10.1.1.63 - - [05/Jan/2011:22:39:50 -0800] "GET /membase.php HTTP/1.1" 200 3494 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_5; en-us) AppleW"
10.1.1.63 - - [05/Jan/2011:22:39:50 -0800] "GET /favicon.ico HTTP/1.1" 404 284 "http://10.0.0.150/membase.php" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_5; en-us) AppleW"

Converting sFlow to combined logfile format allows existing log analyzers to be used to analyze the sFlow data. For example, the following commands use sflowtool and webalizer to create reports:

/usr/local/bin/sflowtool -H | rotatelogs log/http_log &
webalizer -o report log/*

The resulting webalizer report shows top URLs:

Finally, the real potential of HTTP sFlow is as part of a broader performance management system providing real-time visibility into applications, servers, storage and networking across the entire data center.