Deploying the WSO2 ESB correctly on your infrastructure is a vital task for the success of the usage of the ESB. You should first analyze the requirement and then select the best matching deployment mode. There are different set of possible deployment modes with different configurations.
Standalone Deployment Guide : Describes the default standalone installation and deployment of the ESB
Webapp Deployment Guide : Contains the information on deploying the ESB on an Application Server, this particular guide explains the deployment of an existing Tomcat Servlet Container.
This document describes the steps you need to follow to setup a working WSO2 ESB cluster in your environment.
Clustering the ESB has two different aspects.
The configuration sharing is done using the WSO2 Governance Registry. All the ESB nodes in the cluster are pointed to one instance of WSO2 G-Reg and through this registry configuration is shared.
WSO2 G-Reg consists of three registry spaces.
So, when it comes to ESB clustering, we will be using local registry per each instance, mounting the configuration registry of each ESB instance to the same config registry and sharing the governance space among GReg and 2-ESB instances.
First lets see how to set up ESB instances pointing to a single G-Reg.
WSO2 ESB Cluster Deployment Diagram
Setting UP WSO2 ESB cluster with GReg
(This documentation http://wso2.org/project/registry/3.5.0/docs/user_guide/registry_config.html contains database configurations with different databases and we recommend you to go through it if you are using a different database server other than mysql.)
ESB cluster database configuration
| Instance Name | Registry Space | Instance Name Registry Space Mysql database name/mounted path (x.x.x.x) |
| ESB Master (x.x.x.x) | local | amamaster |
| config | amagregdb /_system/esbConfig | |
| governance | amagregdb /_system/governance | |
| user-manager | amausermgtdb | |
| ESB Slave1 (x.x.x.x) | local | amaslave1 |
| config | amagregdb /_system/esbConfig | |
| governance | amagregdb /_system/governance | |
| user-manager | amausermgtdb | |
| GREG (x.x.x.x) | All | amagregdb |
In the above table there are 5 separate databases with following names
If you are having your database remotely, please configure mysql instance to allow remote connections to accept.
Running Multiple instance of the WSO2 Carbon Servers in the same machine
If you are running multiple instance same or different WSO2 Product Instances, then you need to configure ports for each instance. You can configure $CARBON_HOME/repository/conf/carbon.xml by using portOffset
<Offset>0</Offset>
Please provide + Offset value
E.g: If Offset=2 and HTTPS port=9443, will be set the effective HTTPS port to 9445
If you see the table above you can see that we are using a single database for all the queries in GREG instance. So we need to change the configuration in registry.xml and user-mgt.xml.
Go to GREG_HOME/repository/conf/registry.xml file remove the original dbConfig element and add the database configuration below. (IP addresses and database urls have to be changed acccording to your setup).
<currentDBConfig>wso2registry</currentDBConfig>
<readOnly>false</readOnly>
<registryRoot>/</registryRoot>
<dbConfig name="wso2registry">
<url>jdbc:mysql://x.x.x.x:3306/amagregdb?autoReconnect=true</url>
<userName>root</userName>
<password>root123</password>
<driverName>com.mysql.jdbc.Driver</driverName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<minIdle>5</minIdle>
</dbConfig>
Go to GREG_HOME/repository/conf/user-mgt.xml file and change the database configuration as below(IP addresses and urls have to be changed according to your setup).
<Property name="url">jdbc:mysql://x.x.x.x:3306/amagregdb?autoReconnect=true</Property>
<Property name="userName">root</Property>
<Property name="password">root123</Property>
<Property name="driverName">com.mysql.jdbc.Driver</Property>
<Property name="maxActive">50</Property>
<Property name="maxWait">60000</Property>
<Property name="minIdle">5</Property>
Copy mysql jdbc driver library to GREG_HOME/repository/components/lib direcotry since by default we do not ship the mysql drivers in GREG.
If we examine the above configuration we can see that we are pointing the same database (amagregdb)for registry and user store .
Start the GREG instance with -Dsetup (This will create tables for user manager database and registry database(pick the database configuration from currentDBConfig element in registry.xml since there might be number of database configurations in your registry.xml which we put to perform mounting from other databases but only one database configuration is configured as currentDBConfig)).
After successfully starting GREG instance create following directory from GREG's management console. Go to management console-->Registry browser and create "esbConfig" collection under "system" path. Configurations will be mounted by ESB instances running in the cluster to this directory.( /_system/esbConfig)
Here we are changing the default database configuration to point master node's local registry to amamaster and will be poinetd the user manager database to amausermgtdb . Here ESB master node's configuration and Governance registriy's configuration also will be mounted from GREG database.
Let's first add local registry configuration and user-manager configuration.
Open the ESB_HOME/repository/conf/registry.xml file remove the original dbConfig element and add the database configuration below. (IP addresses and database urls have to be changed according to your setup).
<currentDBConfig>wso2registry</currentDBConfig>
<readOnly>false</readOnly>
<registryRoot>/</registryRoot>
<dbConfig name="wso2registry">
<url>jdbc:mysql://x.x.x.x:3306/amamaster?autoReconnect=true</url>
<userName>root</userName>
<password>root123</password>
<driverName>com.mysql.jdbc.Driver</driverName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<minIdle>5</minIdle>
</dbConfig>
Open the ESB_HOME/repository/conf/user-mgt.xml file and change the database configuration as below(IP addresses and urls have to be changed according to your setup).
<Property name="url">jdbc:mysql://x.x.x.x:3306/amausermgtdb?autoReconnect=true</Property>
<Property name="userName">root</Property>
<Property name="password">root123</Property>
<Property name="driverName">com.mysql.jdbc.Driver</Property>
<Property name="maxActive">50</Property>
<Property name="maxWait">60000</Property>
<Property name="minIdle">5</Property>
Now let's configure the mounting configuration. For that we need to add more parameters to registry.xml file.
Open the ESB_HOME/repository/conf/registry.xml file and add the following mount database configuration below. (IP addresses and database urls have to be changed acccording to your setup).
<dbConfig name="esbMountRegistry">
<url>jdbc:mysql://x.x.x.x:3306/amagregdb?autoReconnect=true</url>
<userName>root</userName>
<password>root123</password>
<driverName>com.mysql.jdbc.Driver</driverName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<minIdle>5</minIdle>
</dbConfig>
<remoteInstance url="https://x.x.x.x:9443/registry">
<id>Mount1</id>
<dbConfig>esbMountRegistry</dbConfig>
<readOnly>false</readOnly>
<registryRoot>/</registryRoot>
</remoteInstance>
<mount path="/_system/config" overwrite="true">
<instanceId>Mount1</instanceId>
<targetPath>/_system/esbConfig</targetPath>
</mount>
<mount path="/_system/governance" overwrite="true">
<instanceId>Mount1</instanceId>
<targetPath>/_system/governance</targetPath>
</mount>
Please change the "remoteInstance URL" according to the configuration of the GREG instance. Please note that "InstanceIds", "id" and "dbConfig" elements should be mapped properly if you are using different names for them.
Copy mysql jdbc driver library to ESB_HOME/repository/components/lib direcotry since by default we do not ship the mysql drivers in ESB.
Here we are changing the default database configuration, to point slvae node's local registry to amaslave1 and will be pointed the user manager database to amausermgtdb . Here ESB slave node's configuration and Governance registriy's configuration also will be mounted from GREG database.
Let's first add local registry configuration and user-manager configuration.
Open the ESB_SLAVE_HOME/repository/conf/registry.xml file , remove the original dbConfig element and add the database configuration below. (IP addresses and database urls have to be changed acccording to your setup).
<currentDBConfig>wso2registry</currentDBConfig>
<readOnly>false</readOnly>
<registryRoot>/</registryRoot>
<dbConfig name="wso2registry">
<url>jdbc:mysql://x.x.x.x:3306/amaslave1?autoReconnect=true</url>
<userName>root</userName>
<password>root123</password>
<driverName>com.mysql.jdbc.Driver</driverName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<minIdle>5</minIdle>
</dbConfig>
Open the ESB_SLAVE_HOME/repository/conf/user-mgt.xml file and change the database configuration as below(IP addresses and urls have to be changed according to your setup).
Note : Here we are changing the mode to 'readOnly' in database configuration.
<ReadOnly>true</ReadOnly>
<Property name="url">jdbc:mysql://x.x.x.x:3306/amausermgtdb?autoReconnect=true</Property>
<Property name="userName">root</Property>
<Property name="password">root123</Property>
<Property name="driverName">com.mysql.jdbc.Driver</Property>
<Property name="maxActive">50</Property>
<Property name="maxWait">60000</Property>
<Property name="minIdle">5</Property>
Now let's configure the mounting configuration. For that we need to add more parameters to registry.xml file.
Open the ESB_SLAVE_HOME//repository/conf/registry.xml file and add the following mount database configuration below. (IP addresses and database urls have to be changed acccording to your setup).
<dbConfig name="esbMountRegistry">
<url>jdbc:mysql://x.x.x.x:3306/amagregdb?autoReconnect=true</url>
<userName>root</userName>
<password>root123</password>
<driverName>com.mysql.jdbc.Driver</driverName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<minIdle>5</minIdle>
</dbConfig>
<remoteInstance url="https://x.x.x.x:9443/registry">
<id>Mount1</id>
<dbConfig>esbMountRegistry</dbConfig>
<readOnly>true</readOnly>
<registryRoot>/</registryRoot>
</remoteInstance>
<mount path="/_system/config" overwrite="true">
<instanceId>Mount1</instanceId>
<targetPath>/_system/esbConfig</targetPath>
</mount>
<mount path="/_system/governance" overwrite="true">
<instanceId>Mount1</instanceId>
<targetPath>/_system/governance</targetPath>
</mount>
Please change the remoteInstance URL according to the configuration of the GREG running machine. Please note that "InstanceIds", "id" and "dbConfig" elements should be mapped properly if you are using different names for them.
Note: If you compare the configuration with master node, we are only changing the configuration of local registry and all the other configurations are same with the master node, but the registry access mode in mounting is readonly but the local registry configuration should be read-write since it is specific for each node.
Copy mysql jdbc driver library to ESB_SLAVE_HOME/repository/components/lib direcotry since by default we do not ship the mysql drivers in ESB.
Before starting the slave ESB node you need to load registry database schema to amaslave1 database. You can find the required SQL script at ESB_SLAVE_HOME/dbscripts directory.
After successful configurations, run the wso2server.sh or wso2server.bat of master node based on your platform with '-Dsetup' option for the first time start up to create the database tables, and after creating database tables, do not use '-Dsetup' option when run the server.
sh CARBON_HOME/bin/wso2server.sh -Dsetup
You don't need to specify the -Dsetup option for ESB slave node because we have created DB schema in the previous step.
Note: -Dsetup doesn't work with ESB slave node because we share a same database for user manager and -Dsetup option will try to create user manager tables at the startup if you use -Dsetup option.
After setting up the ESB's pointing to a single Registry we have two options for sharing the configuration.
In this mode ESB's can be configured to load the configuration from the Registry instead of the file system. The configuration loading happens only when the ESB starts up. So if a change is made to the master ESB node, other ESB's has to be restarted to get the new configuration. Lot of users prefer this mode because it guarantees a consistent configuration at the start up and configuration is not changed while the ESB is running. By default ESB loads the configuration from the File System. If the configuration has to be loaded from the Registry following configuration has to be uncommented and changed in the carbon.xml
<MediationConfig>
<LoadFromRegistry>false</LoadFromRegistry>
<SaveToFile>false</SaveToFile>
<Persistence>enabled</Persistence>
<RegistryPersistence>enabled</RegistryPersistence>
</MediationConfig>
LoadFromRegistry parameter has to be changed to true in the slave nodes.
The next approach is to use the Deployment Synchronizer shipped with the ESB. With the deployment synchronizer, configurations can be updated in the slave nodes at the runtime.
WSO2 ESB flows are stateless for the most part. But in very rare deployments users have requested to share some of the runtime information among the cluster nodes. For example Caching can be used with runtime state sharing.
Runtime state replication is based on the Tribes group management system. It provides group membership handling and group communication for clustered Carbon Server instances. Although ESB ships with this built in Tribes based implementation, other clustering implementations based on different group management systems can be plugged in easily. The only thing you have to do is to implement a set of clustering interfaces provided by Carbon Server.
ESB clustering currently does not support distributed locking for session data replication. Therefore, we have to deploy primary backup clusters for stateful services. This restriction does not apply to stateless services and you can direct client requests to any node in the cluster for such services.
ESB clustering is configured using the axis2.xml file. So you have to enable clustering in the axis2.xml of each node.
For more details about Carbon Server clustering, please see WSO Carbon Clustering Configuration Language .
NWe can enable clustering by changing the configuration in the axis2.xml. By default, clustering is turned off to avoid additional overhead for individual deployments. Open the axis2.xml and set the value 'true' for the "enable" attribute of the "clustering" element as follows.
<clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="true">
You may also change clustering properties to suit your deployment as explianed in the above article. However, the default configuration is sufficient for the demonstration.
The WSO2 Oxygen Tank Library : The Library contains articles, tutorials, presentations, and other knowledge base items published on The Oxygen Tank on different styles of deployments.