The deployment model of this scenario will be like bellow
In this model we have two rings, Cassandra ring and ZooKeeper ring. There is a common Message broker node for these rings. The reason for having a common broker node is , there should be at least on cassandra server and one zoo keeper server to start WSO2 Message broker in clustered mode Apart from that we have a message broker server which points to both of these rings.Since we are shipping a cassandra server and zoo keeper server with Message broker product we can use them and create a cassandra ring and a zoo keeper ring in the cluster.
As in the image, We have four Message Broker servers in four different hosts. In that we use two servers to create a cassandra ring, another two servers to create a zoo keeper ring and another server uses above created rings as in the diagram.
Servers used to create cassandra ring
Servers used to create zoo keeper ring
Standalone server
Note that 192.168.0.101 server is common to both rings !
First we need to configure the HostName in carbon.xml file which is located in "wso2mb-2.0.1/repository/conf/" directory. Specify the ip of the host as follows
<!-- Host name or IP address of the machine hosting this server e.g. www.wso2.org, 192.168.1.10 This is will become part of the End Point Reference of the services deployed on this server instance. --> <HostName>192.168.0.101</HostName>
We need to configure cassandra servers in these MB servers to form a cassandra ring. For that we need to change the following entries in the "Cassandra.yaml" file which is located in the "wso2mb-2.0.1/repository/conf/etc" directory.
# seeds is actually a comma-delimited list of addresses. # Ex: "<ip1>,<ip2>,<ip3>" - seeds: "192.168.0.100,192.168.0.101"
listen_address: 192.168.0.100
rpc_address: 192.168.0.100
You need to change this entry in the 192.168.0.101 message broker server also.
Configuring Zookeeper Cluster
Then we need to configure the Zoo Keeper Servers to create a ring within among all broker nodes. We can do it by adding client port address to the "zoo.cfg" which is located at "wso2mb-2.0.1/repository/conf/etc " .
After modifying the file , it will look like bellow.
tickTime=2000 dataDir=repository/data/zookeeper clientPort=2181 start_zk_server=true clientPortAddress=192.168.0.101 server.1=192.168.0.101:2888:3888 server.2=192.168.0.102:2888:3888
Above zoo.cfg file is for the server 192.168.0.101. You can copy the same zoo.cfg file to other two servers and change the entry "clientPortAddress" to the ip of that particular server.
Then you need to create file "myid" in the data directory (repository/data/zookeeper) of each server. In that file you need to specify the id of the server.
eg: Id of the the server 192.168.0.101 will be "1". You need only to have the number 1 in that myid file.
You can find more information on this in zookeeper clustering setup guide
You need to do above configuration in each of the message broker servers appropriately.
You need to change this entry in the 192.168.0.102 message broker server also.
Then you need to do the following configuration changes in 192.168.0.100,192.168.0.101 (Servers with cassandra) Message broker nodes and 192.168.0.103 node.
Modify qpid-config.xml file which is located at : /repository/conf/advanced directory as bellow
<clustering> <enabled>true</enabled> <OnceInOrderSupportEnabled>false</OnceInOrderSupportEnabled> <externalCassandraServerRequired>false</externalCassandraServerRequired> <coordination> <!-- Apache Zookeeper Address --> <ZooKeeperConnection>192.168.0.101:2181,192.168.0.102:2181</ZooKeeperConnection> <!-- Format yyyy-MM-dd HH:mm:ss --> <ReferenceTime>2012-02-29 08:08:08</ReferenceTime> </coordination>
Then you need to do the following configuration changes in 192.168.0.101,192.168.0.102 (Servers with zookeeper) Message broker nodes and 192.168.0.103 node.
Modify qpid-virtualhosts.xml file which is located at :/repository/conf/advanced directory as bellow
<store> <class>org.wso2.andes.server.store.CassandraMessageStore</class> <username>admin</username> <password>admin</password> <cluster>ClusterOne</cluster> <idGenerator>org.wso2.andes.server.cluster.coordination.TimeStampBasedMessageIdGenerator</idGenerator> <connectionString>192.168.0.101:9160</connectionString> </store>
Now you can start node common to both of the rings (192.168.0.101) which have cassandra servers and zookeeper server and then you can start all the other nodes.
The WSO2 Oxygen Tank Library : The Library contains articles, tutorials, presentations, and other knowledge base items published on The Oxygen Tank on different styles of deployments.