[ Documentation Index ]

Use inbuilt cassandra server and zoo keeper server for all the broker nodes

The deployment model of this scenario will be like bellow

Cluster Setup Scenario 04

In this model we have two rings, Cassandra ring and ZooKeeper ring. Since we are shipping a cassandra server and zoo keeper server with Message broker product we can use them and create a cassandra ring and a zoo keeper ring in the cluster.

We have three Message Broker servers in three different hosts as and all of them are having a cassandra server in it

  • MB Server 01 - 192.168.0.100
  • MB Server 02 - 192.168.0.101
  • MB Server 03 - 192.168.0.102

First we need to configure the HostName in carbon.xml file which is located in "wso2mb-2.0.1/repository/conf/" directory. Specify the ip of the host as follows. You need to configure this for all the three servers.

          <!--
                 Host name or IP address of the machine hosting this server
                 e.g. www.wso2.org, 192.168.1.10
                 This is will become part of the End Point Reference of the
                 services deployed on this server instance.
              -->
              <HostName>192.168.0.100</HostName>

      

Then We need to configure cassandra servers in these MB servers to form a cassandra ring. For that we need to change the following entries in the "Cassandra.yaml" file which is located in the "wso2mb-2.0.1/repository/conf/etc" directory.

  • Change the seeds in the ring. You can get a better idea on setting a cassandra from following web page. Here we are configuring two seed nodes. (MB Server 01 and 02)
  •               # seeds is actually a comma-delimited list of addresses.
                  # Ex: "<ip1>,<ip2>,<ip3>"
                  - seeds: "192.168.0.101,192.168.0.102"
    
              
  • Change listen address of the server. By default it is "localhost", change it to the ip address of the host where you have the cassandra server. You need to change the listen address of the each server to it's ip address. Here for the server 192.168.0.102, it will like bellow.
  •              listen_address: 192.168.0.102
              
  • Change RPC address of the server.By default it is "localhost", change it to the ip address of the host where you have the cassandra server. You need to change the RPC address of the each server to it's ip address. Here for the server 192.168.0.103, it will like bellow.
  •              rpc_address: 192.168.0.103
              

Configuring Zookeeper Cluster

Then we need to configure the Zoo Keeper Servers to create a ring within among all broker nodes. We can do it by adding client port address to the "zoo.cfg" which is located at "wso2mb-2.0.1/repository/conf/etc " .

After modifying the file , it will look like bellow.

                tickTime=2000
                dataDir=repository/data/zookeeper
                clientPort=2181
               start_zk_server=true
               clientPortAddress=192.168.0.101
               server.1=192.168.0.100:2888:3888
               server.2=192.168.0.101:2888:3888
               server.3=192.168.0.102:2888:3888
           

Above zoo.cfg file is for the server 192.168.0.101. You can copy the same zoo.cfg file to other two servers and change the entry "clientPortAddress" to the ip of that particular server.

Then you need to create file "myid" in the data directory (repository/data/zookeeper) of each server. In that file you need to specify the id of the server.

eg: Id of the the server 192.168.0.100 will be "1". You need only to have the number 1 in that myid file.

You can find more information on this in zookeeper clustering setup guide

You need to do above configuration in each of the message broker servers appropriately.

Then you need to do the following configuration changes in each Message broker nodes

Modify qpid-config.xml file which is located at : /repository/conf/advanced directory as bellow

  • Enable clustering
  • Change the server of the Zookeeper
          <clustering>

          <enabled>true</enabled>
          <OnceInOrderSupportEnabled>false</OnceInOrderSupportEnabled>
          <externalCassandraServerRequired>false</externalCassandraServerRequired>

          <coordination>
              <!-- Apache Zookeeper Address -->
              <ZooKeeperConnection>192.168.0.100:2181,192.168.0.101:2181,192.168.0.102:2181</ZooKeeperConnection>
              <!-- Format yyyy-MM-dd HH:mm:ss -->
              <ReferenceTime>2012-02-29 08:08:08</ReferenceTime>
          </coordination>
      

Now you can start all three Message Broker nodes.

Other Resources

The WSO2 Oxygen Tank Library : The Library contains articles, tutorials, presentations, and other knowledge base items published on The Oxygen Tank on different styles of deployments.