WSO2 Business Activity Monitor

Introduction

WSO2 Business Activity Monitor (WSO2 BAM) is a comprehensive framework designed to solve the problems in the wide area of business activity monitoring. WSO2 BAM comprises of many modules to give the best of performance, scalability and customizability. This allows to achieve requirements of business users, dev ops, CEOs without spending countless months on customizing the solution without sacrificing performance or the ability to scale.

Aggregate, Analyze and Present

Business Activity Monitoring (BAM) is a term coined by Gartner. It stresses on aggregating, analyzing and presenting information about business activities. This definition is paramount when designing a solution to address a business activity monitoring use case. The aggregation refers to the collection of data. The analysis refers to the manipulation of data to extract information. Presentation refers to representing this data visually or in other ways such as alerts. The BAM architecture reflects this natural flow in its design.

BAM Design Flow

BAM Architecture Overview

The BAM architecture is broken down into three main modules. In addition there should be a client entity to send data into BAM. These are known as data agents and are available as installable features and also provide an API to publish any custom events. Documentation on this is available at The three modules are:

  1. The Event Receiver (Aggregate)
  2. The Analyzer Framework (Analyze)
  3. The Dashboard and Reports (Presentation)

BAM Architecture

Data that needs to be monitored goes through these three modules in order. If we take a look at the data flow, an event will be constructed from the data and sent from the data agent to the BAM server, where the Event Receiver will process and store it in the data store. Then, the Analyzer Framework will start to analyze this data according to defined analytic queries. This will usually follow a pattern of retrieving the events from the data store, performing some sort of data operation such as an addition and storing it back in a data store. Finally, the dashboard UI elements will query the data store for the analyzed data and show it in the UI. A WYSIWYG tool names the gadget IDE is provided with BAM to make generating visualizations very easy.

NoSQL Storage

The BAM storage has to accomodate large volumes of data and be ready to scale out as the data load increases. Also, performance on writes and reads are critical. Apache Cassandra is the storage solution used to achieve both these requirements. Cassandra can be clustered in minutes and has impressive performance numbers. WSO2 BAM uses a multi tenented, cloud ready Cassandra to gain a significant performance gain for its storage. Sometimes, you may prefer working with your own choice of DBMS. Even though, BAM stores in Cassandra to gain performance, these can be transferred to a DBMS of choice during the 'analyze' phase.

Scalability

BAM is designed to scale in four main aspects. These are :

  1. Data Storage
  2. Event Receiver
  3. Analyzer Framework
  4. Dashboard
Let's consider each of these aspects separately.

Data Storage

Scaling up the BAM data storage refers to scaling up Cassandra. This is very simple to do following Cassandra documentation, and can be done in a few minutes. The advantage here is that most NoSQL storages makes are easily to scale by design, hence making the scaling of data storage effortless.

Event Receiver

NOTE: This is WIP and will not be available in the 2.0.0-alpha version The Event Receiver is optimized to receive events over the wire and store it in Cassandra as fast as possible. Scaling up the Event Receiver might seem attractive if you have a large number of events per second (10,000,000/s). A single Event Receiver might be unable to handle this and a cluster would be needed to introduce load balancing.

Analyzer Framework

NOTE: This is WIP and will not be available in the 2.0.0-alpha version The analyzer framework will perform analytics on stored events. Scaling up analytics in sensible when the data volume becomes too large to be analyzed by a single node. Apache Hadoop is used for conducting scalable analytics. Hadoop has a comprehensive framework to handle big data analytics. BAM integrates Hadoop to scale out analytics to a set of Hadoop nodes.

Dashboards

If there are a large number of users who will be viewing the dashboards, a single server for dashboards will not suffice. Multiple dashboard servers can then be deployed to load balance the traffic.

The modular architecture of WSO2 BAM allows each module to exist independently in a separate server and scaled up individually. This can be done conveniently using the power of feature installation/uninstallation in the WSO2 Carbon platform. Thereby, allowing these modules can run in separate JVMs. For example, if you have a large event load of around ten million events per second (10,000,000/s), it would make sense to scale up the Event Receiver. To create such a server, take a vanilla BAM server, and uninstall all features except for the Event Receiver feature, and create a cluster of Event Receiver servers to handle this load.

Performance

BAM achieves performance in a number of ways. One reason is the use of a NoSQL storage as mentioned above. The other is the use of extremely fast binary protocols. The one that is used by default is called Apache Thrift, which has very impressive performance numbers and allows us to serve extremely high loads even without load balancing. Below is a list of the reasons why BAM has achieved a huge performance gain:

  1. Apache Thrift based events
  2. Apache Cassandra as NoSQL storage
  3. Smart worker pools that expand and contract according to the event load
  4. Modularized architecture that separates heavy data processing from event transfer

Example Deployment

The following figure shows a sample deployment using WSO2 BAM to monitor WSO2 WSAS, WSO2 ESB as well as generic/custom data sources.

WSO2 BAM Example Deployment