[ Documentation Index ]

Use inbuilt cassandra server and zoo keeper server in a customized manner

The deployment model of this scenario will be like bellow

Cluster Setup Scenario 05

In this model we have two rings, Cassandra ring and ZooKeeper ring. There is a common Message broker node for these rings. The reason for having a common broker node is , there should be at least on cassandra server and one zoo keeper server to start WSO2 Message broker in clustered mode Apart from that we have a message broker server which points to both of these rings.Since we are shipping a cassandra server and zoo keeper server with Message broker product we can use them and create a cassandra ring and a zoo keeper ring in the cluster.

As in the image, We have four Message Broker servers in four different hosts. In that we use two servers to create a cassandra ring, another two servers to create a zoo keeper ring and another server uses above created rings as in the diagram.

Servers used to create cassandra ring

  • MB Server 01 - 192.168.0.100
  • MB Server 02 - 192.168.0.101

Servers used to create zoo keeper ring

  • MB Server 01 - 192.168.0.101
  • MB Server 02 - 192.168.0.102

Standalone server

  • MB Server 01 - 192.168.0.103

Note that 192.168.0.101 server is common to both rings !

First we need to configure the HostName in carbon.xml file which is located in "wso2mb-2.0.1/repository/conf/" directory. Specify the ip of the host as follows

          <!--
                 Host name or IP address of the machine hosting this server
                 e.g. www.wso2.org, 192.168.1.10
                 This is will become part of the End Point Reference of the
                 services deployed on this server instance.
              -->
              <HostName>192.168.0.101</HostName>

      

We need to configure cassandra servers in these MB servers to form a cassandra ring. For that we need to change the following entries in the "Cassandra.yaml" file which is located in the "wso2mb-2.0.1/repository/conf/etc" directory.

  • Change the seeds in the ring. You can get a better idea on setting a cassandra from following web page. Here we are configuring two seed nodes. (MB Server 01 and 02)
  •               # seeds is actually a comma-delimited list of addresses.
                  # Ex: "<ip1>,<ip2>,<ip3>"
                  - seeds: "192.168.0.100,192.168.0.101"
    
              
  • Change listen address of the server. By default it is "localhost", change it to the ip address of the host where you have the cassandra server. You need to change the listen address of the each server to it's ip address. Here for the server 192.168.0.100, it will like bellow.
  •              listen_address: 192.168.0.100
              
  • Change RPC address of the server.By default it is "localhost", change it to the ip address of the host where you have the cassandra server. You need to change the RPC address of the each server to it's ip address. Here for the server 192.168.0.100, it will like bellow.
  •              rpc_address: 192.168.0.100
              

You need to change this entry in the 192.168.0.101 message broker server also.

Configuring Zookeeper Cluster

Then we need to configure the Zoo Keeper Servers to create a ring within among all broker nodes. We can do it by adding client port address to the "zoo.cfg" which is located at "wso2mb-2.0.1/repository/conf/etc " .

After modifying the file , it will look like bellow.

                tickTime=2000
                dataDir=repository/data/zookeeper
                clientPort=2181
               start_zk_server=true
               clientPortAddress=192.168.0.101
               server.1=192.168.0.101:2888:3888
               server.2=192.168.0.102:2888:3888
           

Above zoo.cfg file is for the server 192.168.0.101. You can copy the same zoo.cfg file to other two servers and change the entry "clientPortAddress" to the ip of that particular server.

Then you need to create file "myid" in the data directory (repository/data/zookeeper) of each server. In that file you need to specify the id of the server.

eg: Id of the the server 192.168.0.101 will be "1". You need only to have the number 1 in that myid file.

You can find more information on this in zookeeper clustering setup guide

You need to do above configuration in each of the message broker servers appropriately.

You need to change this entry in the 192.168.0.102 message broker server also.

Then you need to do the following configuration changes in 192.168.0.100,192.168.0.101 (Servers with cassandra) Message broker nodes and 192.168.0.103 node.

Modify qpid-config.xml file which is located at : /repository/conf/advanced directory as bellow

  • Enable clustering
  • Change the server of the Zookeeper
          <clustering>

          <enabled>true</enabled>
          <OnceInOrderSupportEnabled>false</OnceInOrderSupportEnabled>
          <externalCassandraServerRequired>false</externalCassandraServerRequired>

          <coordination>
              <!-- Apache Zookeeper Address -->
              <ZooKeeperConnection>192.168.0.101:2181,192.168.0.102:2181</ZooKeeperConnection>
              <!-- Format yyyy-MM-dd HH:mm:ss -->
              <ReferenceTime>2012-02-29 08:08:08</ReferenceTime>
          </coordination>
      

Then you need to do the following configuration changes in 192.168.0.101,192.168.0.102 (Servers with zookeeper) Message broker nodes and 192.168.0.103 node.

Modify qpid-virtualhosts.xml file which is located at :/repository/conf/advanced directory as bellow

  • Change the connection string by pointing to the cassandra server
          <store>
              <class>org.wso2.andes.server.store.CassandraMessageStore</class>
              <username>admin</username>
              <password>admin</password>
              <cluster>ClusterOne</cluster>
              <idGenerator>org.wso2.andes.server.cluster.coordination.TimeStampBasedMessageIdGenerator</idGenerator>
              <connectionString>192.168.0.101:9160</connectionString>
          </store>
      

Now you can start node common to both of the rings (192.168.0.101) which have cassandra servers and zookeeper server and then you can start all the other nodes.

Other Resources

The WSO2 Oxygen Tank Library : The Library contains articles, tutorials, presentations, and other knowledge base items published on The Oxygen Tank on different styles of deployments.