Cluster Monitoring

This sample explains how a cluster of servers can be monitored and can obtain the useful analysis out of the collected data. This sample concentrates on the statistical analysis of a clustered environment. Throughout the sample we'll assume a clustered system as follows;

There are two data centeres and each having two clusters; One is a ESB(Enterprise Service Bus) cluster and other one is the AS (Application Server) cluster.For the this kind of situation WSO2 BAM can be configured in such a way the request count, response count, etc for each cluster can be monitored. The below given diagram explains the architecture how WSO2 BAM can be connected the cluster in the above scenario.

Cluster Monitoring sample setup

As shown in the above diagram, the statistics from the each cluster is pushed in WSO2 BAM and the data is directly stored in the Cassendra data store. Analyzers can be written in order to process the gathered data by WSO2 BAM. The result of analyzers will be summarized and stored back in the cassendra data store. The summarized data can be presented in the gadget dashboard according to his/her requirement by using the Gadget-Ide in the WSO2 BAM.

Each of the above steps will be described in detail in the following steps.

Step 1 - Start BAM Server

  1. Download the WSO2-BAM Server
  2. Unzip the WSO2-BAM.zip in the location where ever you prefer
  3. Go to WSO2-BAM-HOME/bin folder in comman prompt. And type
    • wso2server.sh (For ubuntu, linux, mac OS)
    • wso2server.bat (For Windows)


Step 2 - Sending events to BAM server from clusters

Pre-requisites: Apache-ant

Make sure you machine has apache-ant, if not please install the apache-ant.(http://ant.apache.org/)

  1. Go to location from command prompt WSO2-BAM-HOME/samples/cluster-monitor
  2. In this folder you can see the build.xml. Type 'ant' command from the command propmt. This will simultate the data center siutation as showed in the figure and will pump the events to BAM

The incoming data from the cluster to BAM will be as below;

"dataCenter" : "",
   "cluster" : "",
   "request_count" : "3",
   "response_count" : "2",
   "fault_count" : "1",
   "response_time" : "3.5",
   "clusterStream" : "cluster",
   ....}



Step 3 - Create Index (ClusterStreamSelector)

According to this situation we need to create two indexes.

At this moment we need to create the ClusterStreamSelector. Follow the below given steps to create the index.

  1. Login to the BAM management console. The link for the management console will be there in the server startup log. You can click on it and log in to the management console. Or you can go to https://localhost:9443/ in your web browser, this will take you to the management console.
  2. Go to Configure → Connection Parameters. There you can provide the connection parameter for the cassendra source. Default username, password is admin, admin. Click on Save button after filling out the connection paramters.
  3. Go to Configure → BAM Database


  4. Add index main page

  5. Click on the Add Index.

  6. Create Index named ClusterStreamSelector, which filters out the clusterStream, as shown below. And click Save.


  7. Add Index


Step 4 - Create Analyzer

In the analyzer you need to write the analytics such that it analyzes the gathered data by the BAM according to you requirement. In this sample we are concerned about the request count, response count, fault count and response time of each cluster. Therefore the analyzer sequence for this requirement will be,

<analyzerSequence name="ClusterSummarizer">
    <trigger cron="1 * * * * ? *"/>
    <analyzers>
        <get name="BASE" batchSize="1000">
           <where index="ClusterStreamSelector">                                                                         
             <range column="clusterStream" start="" end=""/>
           </where>
        </get> 
        <groupBy>
            <field name='dataCenter'/>
            <field name='cluster'/>
        </groupBy>
        <drop type="column">
            <fieldSet>
                <field name="correlationKey1"/>
                <field name="metaKey1"/>
                <field name="timestamp"/>
                <field name="ClusterStream"/>
                <field name="timeStamp"/>
            </fieldSet>
        </drop>
        <aggregate>
            <measure name="request_count" aggregationType="SUM"/>
            <measure name="response_count" aggregationType="SUM"/>
            <measure name="fault_count" aggregationType="SUM"/>
            <measure name="response_time" aggregationType="AVG"/>
        </aggregate>
        <put name="clusterSummary" dataSourceType="CASSANDRA">
            <onExist>
                <aggregate>
                    <measure name="request_count" aggregationType="SUM"/>
                    <measure name="response_count" aggregationType="SUM"/>
            	    <measure name="fault_count" aggregationType="SUM"/>
                    <measure name="response_time" aggregationType="AVG"/>
                </aggregate>
            </onExist>
        </put>
        <get name="clusterSummary"/>
        <log/>
    </analyzers>
</analyzerSequence>

The above given analyzer sequence is given in the WSO2BAM/Samples/cluster-monitor/ClusterSummarizer.xml. Follow the below given steps to ad dthe analyzer sequence to the BAM server.

  1. Go to WSO2BAM/samples/cluster-monitor folder
  2. Open the ClusterSummarixer.xml and copy the content
  3. Go to Configure → Analyzer Framework


  4. Add analyzer main page

  5. Click on 'Add sequence' and go to the source view of the analyzer sequence.


  6. Paste the content from the ClusterSummarizer.xml in the step-2, and click on 'Save All' button.


  7. Add analyzer

After adding the summarizer, you can see the below summarized data in the server log.


Summarized server log




Step 5 - Create Index (ClusterIndex)

This index is used to query the summarized data and create the gadget using that. This is similar to the step 3 where we created the ClusterStreamSelector.

  1. Go to Configure → BAM Database
  2. Click on Add Index
  3. Create the index as in the below figure. Click on the 'Add column' to add the second column field.


  4. Create Index - ClusterIndex

Now you can see two index from the index table as below.



All Index table



Step 6 - Create Gadget

In this section we cna see how to design your own gadgets according to your requirement. In this section we created bar chart, pie chart and table to visulaize the data created by BAM.

The below given steps will explain how to create the gadgets and visualize the data collected from the clusters of servers.

  1. Go to Configure → Gadget IDE
  2. In order to obtain the unique data center names, there is a web service in BAM. To communicate with web service, click on the 'Web Service (WSDL)' in the Gadget IDE.

  3. Web sevice gadget

  4. You can see the WSDL of the web service mentioned above, in https://localhost:9443/services/IndexAdminService?wsdl2. Insert this WSDL file url in the gadget, as below. And click on 'load' button.


  5. Web service gadget after load

  6. Select 'getIndexValues' for the field 'Operation' in the WSDL gadget.


  7. Web service gadget after selecting getIndexvalues operation

  8. Click on the Text and connect as below. Note the two Text gadgets have been used. One text field here will mention the indexName, and other one will mention the indexedColum which as two parameters which is required to to the 'getIndexValues' operation.


  9. Web service gadget with text gadget

  10. Now click on 'Combo box'. It'll immediately take to the design view, there you can resize and locate the combo box where ever you like. Then switch back to the 'DataFlow' view. Connect the 'return' node from the WSDL gadget to the 'text' node of the combo box as shown below.


  11. Web service gadget with combo box

  12. This states the return value from the web service method is used to fill the combo box. Now go to the design view, there you can see the unique names of the data centers. (which is returned from the web service)


  13. Combo box in design view

  14. Click on the Cassendra sources. And fill it as the following and click on load button
    • Tablename: ClusterSummary (Where the summarizes data is stored. Refer the analyzer sequence <put> tag)
    • TableIndexName: ClusterIndex (Name of the index which was created to query the ClusterSummary)
    • Index: dataCenter(Name of the field for which the index was created)


    Cassendra source gadget



  15. Click on the lexical string and and connect as below shown.
    • Combo box-text → Cassendra source-from0
    • Combo box-text → lexicalString-text
    • lexicalString-text → cassendra source-to


    Lexical String gadget



  16. Click on the bar chart. It'll take to the design view, locate the gadget as you prefer.


  17. Switch to the Dataflow view. Connect
    • Cassendra Source-Cluster → lable-Bar Chart.
    • Cassendra Source-request count→ value-Bar Chart.


    Bar chart - Data flow



  18. Switch to design view and see the barchart. You can select the data center from the combo box and the bar chart will show the values for that data center
    • Bar chart output of Grid sys data center


    • Bar chart - GridSys

    • Bar chart output of NetVest data center


    • Bar chart - NetVest


  19. Go to the DtaaFlow view and Click on the 'Pie chart'. Then position the pie-chart in the design view as previouse steps and switch back to the DataFlow view. Connect,
    • Cassendra Source-Cluster → lable-Pie Chart.
    • Cassendra Source-fault count→ value-Bar Chart.


    Pie chart - Data Flow view



  20. Now switch to the Design view and see the pie chart. You can select the data center as mentioned in the above step form the combo box.


  21. Pie chart - Design view

  22. Go to DataFlow view and Click on the table. Position the table in the design view. And rhen connect,
    Table-rows → row-Cassendra Source.


  23. Table - Data flow

  24. Switch to the design view and see the table. Here also you can select the data center from combo box and see the respective data center values.
    • Final gadget view for NetVest data center


    • NetVest - All gadgets

    • Final gadget view for GridSys data center


    • GridSys - All gadgets

  25. Now you can see the created gadget from the main dash board.

Step 7 - Publish the gadget to dashboard

  1. Go to the settings in gadget ide side pane and click on the publish


  2. Publish gadget

  3. Type a name of the gadget and click on publish. A URL will be popped out. Copy the URL.
    Eg: /registry/resource/_system/config/repository/dashboards/gadgets/ClusterSummary.xml
  4. Go to BAM server home page, and click on the BAM dashboard.
  5. Click on the Add Gadget, and paste the URL copied which publishing the gadget.


  6. Add gadget to dashboard

  7. Add click on the add gadget.

Now we have come to the end of the sample and now you know the way of generating the custom gadgets from BAM and monitoring your services in clustered environment.