Data Publishers

Data publishers are software components, based on WSO2 Carbon component architecture, that implement WS-Eventing based publisher/subscriber model. Events are fired based on various thresholds that could be configured against various data publishers. The configured thresholds are evaluated in while service messages are invoked, hence the data publication happens real time.

There are three types of data publishers.

  1. Service Statistics Data publisher

    Can be deployed with WSO2 AS and is capable of capturing and publishing service invocation data to BAM.
  2. Mediation Statistics Data publisher

    Can be deployed with WSO2 ESB and is capturing and publishing mediation data for endpoints, proxy services and sequences
  3. Activity Data publishers

    Captures message data across a message flow in WSO2 AS and WSO2 ESB. There are two activity data publishers, Activity Service Data publisher and Activity Mediation Data publisher. Activity Service Data publisher is used with WSO2 AS to capture message data while Activity Mediation Data publisher is used with WSO2 ESB.

Service Statistics Data publisher

Service Statistics Data publisher can be deployed with WSO2 AS and comes as a feature consisting of two WSO2 Carbon Components, the core component and the UI component. Using the configuration user interface, eventing can be enabled or disabled. If eventing is disabled, no events will be fired at all. If eventing is enabled, events will be fired based on the System request count threshold value being satisfied.

System request count threshold

An event will be fired with system wide data, when the difference between the system request count at which the last event was fired and the current system request count becomes greater than this threshold value. The system here refers to the WSO2 AS instance with which the service data publisher is deployed.

The general guideline for configuring the threshold value is that, if the frequency of request rate is high, then the threshold values should be set to have large values, otherwise lower. And the correct values for the thresholds should be set based on the experience gained after monitoring the systems for some time. Also, note that, you will have more requests hitting the server in peak business hours, compared to the off peak hours. Hence the averages between the peak and off peak request counts should be taking into account when deciding on the threshold values. In general, it would be ideal to have an event fired between every minute (60 samples per hour) to five minutes (12 samples per hour) time range.

The event fired will contain data for three level of statistics system, service and operation of the server at that given point.

Event Format

The XML schema for server data event is given below:

            <?xml version="1.0" encoding="utf-8" ?>
            <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
            targetNamespace="http://wso2.org/ns/2009/09/bam/service/statistics/data"
            tns="http://wso2.org/ns/2009/09/bam/service/statistics/data">
            
	            <xsd:element name="Event">
	                <xsd:complexType>
	                    <xsd:sequence>
	                        <xsd:element name="ServiceStatisticsData">
	                            <xsd:complexType>
	                                <xsd:sequence>
	                                    <xsd:element name="ServerName" type="xsd:string"/>
	                                    <xsd:element name="AverageResponseTime" type="xsd:decimal"/>
	                                    <xsd:element name="MinimumResponseTime" type="xsd:long"/>
	                                    <xsd:element name="MaximumResponseTime" type="xsd:long"/>
	                                    <xsd:element name="RequestCount" type="xsd:integer"/>
	                                    <xsd:element name="ResponseCount" type="xsd:integer"/>
	                                    <xsd:element name="FaultCount" type="xsd:integer"/>
	                                    <xsd:element name="ServiceName" type="xsd:string" minOccurs="0"/>
	                                    <xsd:element name="OperationName" type="xsd:string" minOccurs="0"/>
	                                </xsd:sequence>
	                            </xsd:complexType>
	                        </xsd:element>
	                    </xsd:sequence>
	                </xsd:complexType>
	            </xsd:element>
            </xsd:schema>
        

Following is an example event with data that adheres to the above event format:

            <statdata:Event xmlns:statdata="http://wso2.org/ns/2009/09/bam/service/statistics/data">
	            <statdata:ServiceStatisticsData>
		            <statdata:ServerName>http://127.0.0.1:9763</statdata:ServerName>
		            <statdata:AverageResponseTime>16.4</statdata:AveageResponseTime>
		            <statdata:MinimumResponseTime>0</statdata:MinimumResponseTime>
		            <statdata:MaximumResponseTime>109</statdata:MaximumResponseTime>
		            <statdata:RequestCount>22</statdata:RequestCount>
		            <statdata:ResponseCount>20</statdata:ResponseCount>
		            <statdata:FaultCount>2</statdata:FaultCount>
		            <statdata:ServiceName>HelloService</statdata:ServiceName>
		            <statdata:OperationName>greet</statdata:OperationName>
	            </statdata:ServiceStatisticsData>
            </statdata:Event>
        

This event format would be useful if you are to publish events to the service data subscriber endpoint form a custom data publisher on your own. For example, if you are to publish service data from a Microsoft .NET service into WSO2 BAM, you can use the event format information given here, and publish that data into the subscriber endpoint described below.

Subscriber Endpoint

The subscriber endpoint address for service event data that resides within a WSO2 BAM instance has the following format:

https://[host name]:[port]/<[Web context/]>services/BAMServiceStatisticsSubscriberService

for example, the subscriber endpoint can look like the following

https://localhost:9444/bam/services/BAMServiceStatisticsSubscriberService

This subscriber endpoint address is of significance if you are to publish data from a server other than WSO2 AS. You can implement your own custom data publisher to publish events, adhering to the event format given in the above section, to this endpoint. If your custom event publisher implements WS-Eventing, you can manage the subscription from the WSO2 BAM monitored server page. Else, you can add a custom server to WSO2 BAM and publish data to this endpoint on your own.

Installing

The following installation instructions apply to WSO2 AS version 3.2.0

  1. Start WSO2 AS On Linux
    wso2server.sh
    On MS Windows
    wso2server.bat
  2. After starting the server, go to AS management console Configure-->Features-->Settings menu.
    Give P2 repo link as, http://dist.wso2.org/p2/carbon/releases/3.2.0/
    Then click 'Add'. WSO2 p2 repo
  3. Go to 'Available Features' tab and select the above repository. Click 'Find Features'.
    User can see all available features. From those,select 'WSO2 Carbon - BAM Service Statistics Data Publisher Feature' and click 'Install'Service stat feature
  4. Click 'Next' and accept the license agreement.
  5. Restart the WSO2 AS server.

Mediation Statistics Data Publisher

Mediation Statistics Data Publisher can be deployed with WSO2 ESB and comes as installable feature having two WSO2 Carbon components, the core component and the UI component. Using the configuration user interface, eventing can be enabled or disabled at global level, as well as the three threshold values be set. If eventing is disabled, no events will be fired at all. If eventing is enabled, events will be fired based on the threshold values being satisfied. The three threshold values are:

  1. Sequence request count threshold

    An event will be fired with sequence data, when the difference between the sequence request count at which the last event was fired and the current sequence request count becomes greater than this threshold value.
  2. Endpoint request count threshold

    An event will be fired with endpoint data, when the difference between the request count for an endpoint at which the last event was fired for the endpoint and the current request count for that endpoint, becomes greater than this threshold value.
  3. Proxy service request count threshold

    An event will be fired when the difference between the request count for a proxy service at which the last event was fired for that proxy service and the current request count for that proxy service becomes greater than this threshold value.

The threshold value for sequences applies to all sequences present with the WSO2 ESB instance in general. This threshold cannot be configured for individual sequences to have different values. While this may look to be a limitation, it guards against potential performance hits in checking for the sequence names and sequence specific threshold for each sequence invocation. Hence, it is by design that this threshold is common for all sequences hosted with a WSO2 ESB instance.

Like in the case of sequence request count threshold, endpoint request count threshold and proxy service request count threshold are common for all endpoints and proxy services respectively, and cannot be configured to have different values for individual endpoints or proxy services.

Note that the system wide request counts for each monitored WSO2 ESB instance is not provided in the current implementation of WSO2 BAM. You can consider the total request count for a WSO2 ESB instance to be the sum of the request counts for sequences, endpoints and proxy services. However, note that, for it to be possible for the mediation data publisher to populate data, the WSO2 ESB administrator must have enabled statistics on the sequence, endpoint or the proxy service to be monitored. Hence, if statistics are not enabled for any of the features in the ESB, then the assumption that sum of the individual counts to be equal the system count would not hold.

As in the case of service statistics data publisher, the general guideline for configuring the threshold values is that, if the frequency of request rate is high, then the threshold values should be set to have large values, otherwise lower. And the correct values for the thresholds should be set based on the experience gained after monitoring the systems for some time. Also, note that, you will have more requests hitting the server in peak business hours, compared to the off peak hours. Hence the averages between the peak and off peak request counts should be taking into account when deciding on the threshold values. In general, it would be ideal to have an event fired between every minute (60 samples per hour) to five minutes (12 samples per hour) time range.

Note that you can disable firing events at each level, by providing zero or a negative value as the threshold. For example, if you are interested in only the sequence data, you can set a positive value only for the sequence request count threshold and set endpoint and proxy service thresholds to zero. Alternatively, if you are interested in only the proxy service information, that request count threshold can be set to a positive value and sequence and endpoint threshold values set to zero.

However, it should be also noted that, for enhanced monitoring of the system, it would be ideal to have data for all three categories.

Each event for each category will have the following information:

In addition, the event will also have

Server name

TenantID

Sequence, endpoint or the proxy service name

Event Format

Mediation data is published into WSO2 BAM using the user defined event format. The event format would be of interest, if you want to publish mediation data from your own custom publisher. See the section on user defined data for more information. However, if you are using the mediation data publisher, you do not have to worry about the event format, as the data publisher would handle the events for you.

Installing

The following installation instructions apply to WSO2 ESB version 3.0.0

  1. Start WSO2 ESB On Linux
    wso2server.sh
    On MS Windows
    wso2server.bat
  2. After starting the server, go to ESB management console Configure-->Features-->Settings menu.
    Give P2 repo link as, http://dist.wso2.org/p2/carbon/releases/3.2.0/
    Then click 'Add'.
  3. Go to 'Available Features' tab and select the above repository. Click 'Find Features'.
    User can see all available features. From those,select 'WSO2 Carbon - BAM Mediation Statistics Data Publisher Feature' and click 'Install'
  4. Click 'Next' and accept the license agreement.
  5. Restart the WSO2 ESB server.

User Defined Data

User defined data model, supported by WSO2 BAM, provides an extension point for users to allow them publish any custom data into WSO2 BAM database. This allows users to publish custom data as key/value pairs and then later process them as they wish, for monitoring custom situations, outside what is supported by WSO2 BAM by default. This is the event format used for mediation statistics data publishing as well.

Event Format

Following is the XML schema for user defined data event format.

            <?xml version="1.0" encoding="utf-8" ?>
            <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
            targetNamespace="http://wso2.org/ns/2009/09/bam/server/user-defined/data"
            tns="http://wso2.org/ns/2009/09/bam/service/statistics/data">
            
	            <xsd:element name="Event">
		            <xsd:complexType>
			            <xsd:sequence>
				            <xsd:element name="ServerUserDefinedData">
					            <xsd:complexType>
						            <xsd:sequence>
							            <xsd:element name="ServerName" type="xsd:string"/>
								    <xsd:element name="TenantID" type="xsd:string"/>
							            <xsd:element name="Data" minOccurs="0" maxOccurs="unbounded">
								            <xsd:complexType>
									            <xsd:sequence>
										            <xsd:element name="Key" type="xsd:string"/>
										            <xsd:element name="Value" type="xsd:string"/>
									            </xsd:sequence>
								            </xsd:complexType>
							            </xsd:element>
						            </xsd:sequence>
					            </xsd:complexType>
				            </xsd:element>
			            </xsd:sequence>
		            </xsd:complexType>
	            </xsd:element>
            </xsd:schema>
        

As you can see, the event has provision to provide an array of key/value pairs. The mediation data publisher is implemented using this model. Following is an example message, where the event contains information on an endpoint:

            <svrusrdata:Event xmlns:svrusrdata="http://wso2.org/ns/2009/09/bam/server/user-defined/data">
	            <svrusrdata:ServerUserDefinedData>
		            <svrusrdata:ServerName>http://127.0.0.1:8280</svrusrdata:ServerName>
			    <svrusrdata:TenantID>0</svrusrdata:TenantID>
		            <svrusrdata:Data>
			            <svrusrdata:Key>EndpointInMaxProcessingTime-simple</svrusrdata:Key>
			            <svrusrdata:Value>15</svrusrdata:Value>
		            </svrusrdata:Data>
		            <svrusrdata:Data>
			            <svrusrdata:Key>EndpointInAvgProcessingTime-simple</svrusrdata:Key>
			            <svrusrdata:Value>15.0</svrusrdata:Value>
		            </svrusrdata:Data>
		            <svrusrdata:Data>
			            <svrusrdata:Key>EndpointInMinProcessingTime-simple</svrusrdata:Key>
			            <svrusrdata:Value>15</svrusrdata:Value>
		            </svrusrdata:Data>
		            <svrusrdata:Data>
			            <svrusrdata:Key>EndpointInCount-simple</svrusrdata:Key>
			            <svrusrdata:Value>1</svrusrdata:Value>
		            </svrusrdata:Data>
		            <svrusrdata:Data>
			            <svrusrdata:Key>EndpointInFaultCount-simple</svrusrdata:Key>
			            <svrusrdata:Value>0</svrusrdata:Value>
		            </svrusrdata:Data>
		            <svrusrdata:Data>
			            <svrusrdata:Key>EndpointInID</svrusrdata:Key>
			            <svrusrdata:Value>simple</svrusrdata:Value>
		            </svrusrdata:Data>
		            <svrusrdata:Data>
			            <svrusrdata:Key>EndpointInCumulativeCount-simple</svrusrdata:Key>
			            <svrusrdata:Value>3</svrusrdata:Value>
		            </svrusrdata:Data>
		            <svrusrdata:Data>
			            <svrusrdata:Key>EndpointOutCumulativeCount-simple</svrusrdata:Key>
			            <svrusrdata:Value>0</svrusrdata:Value>
		            </svrusrdata:Data>
	            </svrusrdata:ServerUserDefinedData>
            </svrusrdata:Event>
        

You can use this event format to publish data to the user defined data subscriber endpoint, described below.

Subscriber Endpoint

The subscriber endpoint address for user defined event data that resides within a WSO2 BAM instance has the following format:

https://[host name]:[port]/<[Web context/]>services/BAMServerUserDefinedDataSubscriberService

for example, the subscriber endpoint can look like the following

https://localhost:9444/bam/services/BAMServerUserDefinedDataSubscriberService

You can implement your own custom data publisher to publish events, adhering to the event format given in the above section, to this endpoint. If your custom event publisher implements WS-Eventing, you can manage the subscription from the WSO2 BAM monitored server page. Else, you can add a custom server to WSO2 BAM and publish data to this endpoint on your own.

Activity Data Publishers

Activity Data Publishers capture a message flow through monitored servers and lets the user examine the message flow details at BAM side. In order to cope with high volume message flows BAM provides message filtering capabilities so that user can narrow down to the messages of interest with certain filterings. Activity data publishers are two fold.

  1. Activity Service Data Publisher
  2. Activity Mediation Data Publisher

Activity Service Data Publisher

This publisher can be used to capture details of messages flowing through WSO2 Application Server, WSO2 Enterprise Service Bus and WSO2 Data Service Server. This publisher is based on an Axis2 handler implmentation which captures messages as they pass through handler chain and publish required details to BAM.

Installing

The following installation instructions apply to WSO2 ESB 4.0.0, WSO2 AS 4.1.0 and WSO2 DSS 2.6.0.

  1. Start the respective carbon server On Linux
    wso2server.sh
    On MS Windows
    wso2server.bat
  2. After starting the server, go to management console Configure-->Features-->Settings menu.
    Give P2 repo link as, http://dist.wso2.org/p2/carbon/releases/3.2.0/
    Then click 'Add'.
  3. Go to 'Available Features' tab and select the above repository. Click 'Find Features'.
    User can see all available features. From those,select 'WSO2 Carbon - BAM Activity Service Data Publisher Feature' and click 'Install'
  4. Click 'Next' and accept the license agreement.
  5. Restart the server.

Activity Mediation Data Publisher

This publisher can be used to capture details of messages flowing through WSO2 Enterprise Service Bus. This publisher is based on a Synapse mediator implmentation which captures messages as they pass through the mediator sequence and publish required details to BAM. Eventhough the event details published would be similar to that of Activity Service Data Publisher engaged to ESB, the difference of this publisher is the flexibility with which this publisher can be engaged inside ESB. Unlike Activity Service Publisher this publisher can be engaged at multiple locations inside ESB. For example if it is required to monitor a message flow within a sequence of the ESB the class mediator of the publisher can be engaged at the points which message information should be published to BAM.

Installing

The following installation instructions apply to WSO2 ESB 4.0.0.

  1. Start WSO2 ESB On Linux
    wso2server.sh
    On MS Windows
    wso2server.bat
  2. After starting the server, go to ESB management console Configure-->Features-->Settings menu.
    Give P2 repo link as, http://dist.wso2.org/p2/carbon/releases/3.2.0/
    Then click 'Add'.
  3. Go to 'Available Features' tab and select the above repository. Click 'Find Features'.
    User can see all available features. From those,select 'WSO2 Carbon - BAM Activity Mediation Data Publisher Feature' and click 'Install'
  4. Click 'Next' and accept the license agreement.
  5. Restart the ESB server.

Activity Data Event

A similar event format is used in both Activity Service and Activity Mediation Data Publishers.

Event Format

Following is the XML schema for user defined data event format.

            <?xml version="1.0" encoding="utf-8" ?>
            <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
            targetNamespace="http://wso2.org/ns/2009/09/bam/service/activity/data"
            tns="http://wso2.org/ns/2009/09/bam/service/activity/data">
            
	            <xsd:element name="Event">
		            <xsd:complexType>
			            <xsd:sequence>
				            <xsd:element name="ActivityData">
					            <xsd:complexType>
						            <xsd:sequence>
							            <xsd:element name="ServerName" type="xsd:string"/>
								    <xsd:element name="ActivityID" type="xsd:string"/>
							            <xsd:element name="MessageID" type="xsd:string"/>
								    <xsd:element name="ServiceName" type="xsd:string"/>
								    <xsd:element name="OperationName" type="xsd:string"/>
								    <xsd:element name="ActivityName" type="xsd:string"/>
								    <xsd:element name="ActivityDescription" type="xsd:string"/>
								    <xsd:element name="RemoteIPAddress" type="xsd:string"/>
								    <xsd:element name="UserAgent" type="xsd:string"/>
								    <xsd:element name="MessageDirection" type="xsd:string"/>
								    <xsd:element name="MessageBody" type="xsd:string" minOccurs="0"/>
								    <xsd:element name="TimeStamp" type="xsd:dateTime"/>
							            <xsd:element name="XPathExpressions" minOccurs="0">
								            <xsd:complexType>
									            <xsd:sequence>
							            			<xsd:element name="XPathExpression" minOccurs="0"  												maxOccurs="unbounded">
												<xsd:complexType>
									            			<xsd:sequence> 															<xsd:element name="ExpressionKey" 														        type="xsd:string"/> 	
														<xsd:element name="Expression" 														        type="xsd:string"/>
														<xsd:element name="Alias" 														        type="xsd:string"/>
														<xsd:element name="XPathValue" 														        type="xsd:string"/>
														<xsd:element name="Namespaces" 															minOccurs="0">
															<xsd:complexType>
									            						<xsd:sequence>
																	<xsd:element 																	name="Namespace" 																		type="xsd:string"
																	minOccurs="0" 																		maxOccurs="unbounded"/>
																</xsd:sequence>
									            					</xsd:complexType>
														</xsd:element>
													</xsd:sequence>
									            		</xsd:complexType>
											</xsd:element>
									            </xsd:sequence>
								            </xsd:complexType>
							            </xsd:element>
							            <xsd:element name="ActivityProperties" minOccurs="0">
								            <xsd:complexType>
									            <xsd:sequence>
							            			<xsd:element name="ActivityProperty" minOccurs="0"  												maxOccurs="unbounded">
												<xsd:complexType>
									            			<xsd:sequence> 															<xsd:element name="ActivityPropertyChild" 														        type="xsd:string"/> 	
														<xsd:element name="PropertyValue" 														        type="xsd:string"/>
													</xsd:sequence>
									            		</xsd:complexType>
											</xsd:element>
									            </xsd:sequence>
								            </xsd:complexType>
							            </xsd:element>
								    <xsd:element name="OutMessageID" type="xsd:string minOccurs="0""/>
								    <xsd:element name="RequestMessageStatus" type="xsd:string" minOccurs="0"/>
								    <xsd:element name="ResponseMessageStatus" type="xsd:string" minOccurs="0"/>
						            </xsd:sequence>
					            </xsd:complexType>
				            </xsd:element>
			            </xsd:sequence>
		            </xsd:complexType>
	            </xsd:element>
            </xsd:schema>
        

Following is a sample event for this schema.

		<soapenv:Body xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelo pe/">
			<activitydata:Event xmlns:activitydata="http://wso2.org/ns/2009/09/bam/service/activity/data">
					<activitydata:ActivityData>
					<activitydata:ServerName>https://10.10 0.1.143:9443</activitydata:ServerName>
					<activitydata:ActivityID>prash491-a03a-426 0-b177-c493ab7a2dbb</activitydata:ActivityID>
					<activitydata:MessageID />
					<activity data:ServiceName>SampleHelloService</activitydata:ServiceName>
					<activitydata:OperationName>sayHello</activitydata:OperationName>
					<activitydata:ActivityName>hello2 </activitydata:ActivityName>
					<activitydata:ActivityDescription>say hello2</activitydata:ActivityDescription>
					<activitydata:RemoteIPAddress>127.0.0.1</activitydata:RemoteIPAddress>
					<activitydata:UserAgent>%%%%%</activitydata:UserAgent>  // Optional
					<activitydata:MessageDirection>request</activitydata:MessageDirection>
					<activitydata:MessageBody>
						<soap:Body xmlns:soap="http://www.w3.org/2003/05/soap-envelope">
					 		<typ:greet xmlns:typ="http://www.wso2.org/types">
					 			<name>John</name>
					 		</typ:greet>
						</soap:Body> 
					</activitydata:MessageBody>  // Optional
					<activitydata:TimeStamp>2011-04-28 19:00:54</activitydata:TimeStamp>
					<activitydata:XPathExpressions> // Optional
						<activitydata:XPathExpression> // Optional
							<activitydata:ExpressionKey>Name</activitydata:ExpressionKey>
							<activitydata:Expression>/typ:greet/name</activitydata:Expression>
							<activitydata:Alias>GreetingName</activitydata:Alias>
							<activitydata:Namespaces> // Optional
								<activitydata:Namespace>typ@http://www.wso2.org/types</activitydata:Namespace>
							</activitydata:Namespaces>
							<activitydata:XPathValue>John</activitydata:XPathValue>
						</activitydata:XPathExpression>
					</activitydata:XPathExpressions>
					<activitydata:ActivityProperties> // Optional
						<activitydata:ActivityProperty> // Optional
							<activitydata:ActivityPropertyChild>Name</activitydata:ActivityPropertyChild>
							<activitydata:PropertyValue>//name</activitydata:PropertyValue>
						</activitydata:ActivityProperty>
					</activitydata:ActivityProperties>
					<activitydata:OutMessageID/>   // Optional
					<activitydata:RequestMessageStatus>0</activitydata:MessageDirection>  // Optional
					<activitydata:ResponseMessageStatus/>   // Optional
				</activitydata:ActivityData>
 			</activitydata:Event>
		</soapenv:Body>
        

You can use this event format to publish data to the activity data subscriber endpoint, described below.

Subscriber Endpoint

The subscriber endpoint address for activity event data that resides within a WSO2 BAM instance has the following format:

https://[host name]:[port]/<[Web context/]>services/BAMActivityDataStatisticsSubscriberService

for example, the subscriber endpoint can look like the following

https://localhost:9444/bam/services/BAMActivityDataStatisticsSubscriberService