[ Documentation Index ]

Starting external cassandra server and use inbuilt zoo keeper server in ring model with broker nodes

In this model you need to get downloaded following apache products

Deployment model will be like bellow

Cluster Setup Scenario 03

In this model we have a common cassandra server. Since we are shipping a zoo keeper server with Message broker product we can use that and create a zoo keeper ring in the cluster.

  • Cassandra Server - 192.168.0.100

We have three Message Broker servers in three different hosts as

  • MB Server 01 - 192.168.0.101
  • MB Server 02 - 192.168.0.102
  • MB Server 03 - 192.168.0.103

First we need to configure the HostName in carbon.xml file which is located in "wso2mb-2.0.1/repository/conf/" directory. Specify the ip of the host as follows. You need to configure this for all the three servers.

               <!--
                      Host name or IP address of the machine hosting this server
                      e.g. www.wso2.org, 192.168.1.10
                      This is will become part of the End Point Reference of the
                      services deployed on this server instance.
                   -->
                   <HostName>192.168.0.101</HostName>

           

Then we need to configure cassandra server to listen to all the broker nodes.For that we need to change the following entries in the "Cassandra.yaml" file which is located in the "apache-cassandra-1.0.1/conf/" directory.

  • Change the data storage directories of cassandra. Here by default it will point to the location "/var/lib/". But it will be better, if you can keep the data locally inside the folder where you have the server setup. So change it to "./repository/database" off all three entries as
  •               # directories where Cassandra should store data on disk.
                  data_file_directories:
                      - ./repository/database/cassandra/data
    
                  # commit log
                  commitlog_directory: ./repository/database/cassandra/commitlog
    
                  # saved caches
                  saved_caches_directory: ./repository/database/cassandra/saved_caches
    
              
  • Change listen address of the server. By default it is "localhost", change it to the ip address of the host where you have the cassandra server.
  •              listen_address: 192.168.0.100
              
  • Change RPC address of the server.By default it is "localhost", change it to the ip address of the host where you have the cassandra server.
  •              rpc_address: 192.168.0.100
              

Now save the cassandra.yaml file and you can start the cassandra server by executing the script "cassandra" with the option -f. Eg: "./cassandra -f"

Configuring Zookeeper Cluster

Then we need to configure the Zoo Keeper Servers to create a ring within among all broker nodes. We can do it by adding client port address to the "zoo.cfg" which is located at "wso2mb-2.0.1/repository/conf/etc " .

After modifying the file , it will look like bellow.

           tickTime=2000
           dataDir=repository/data/zookeeper
           clientPort=2181
          start_zk_server=true
          clientPortAddress=192.168.0.101
          server.1=192.168.0.100:2888:3888
          server.2=192.168.0.101:2888:3888
          server.3=192.168.0.102:2888:3888
      

Above zoo.cfg file is for the server 192.168.0.101. You can copy the same zoo.cfg file to other two servers and change the entry "clientPortAddress" to the ip of that particular server.

Then you need to create file "myid" in the data directory (repository/data/zookeeper) of each server. In that file you need to specify the id of the server.

eg: Id of the the server 192.168.0.100 will be "1". You need only to have the number 1 in that myid file.

You can find more information on this in zookeeper clustering setup guide

You need to do above configuration in each of the message broker servers appropriately.

Then you need to do the following configuration changes in each Message broker nodes

Modify qpid-config.xml file which is located at : /repository/conf/advanced directory as bellow

  • Enable clustering
  • Change the server ip of the Zookeeper by adding all zoo keeper server details
          <clustering>

          <enabled>true</enabled>
          <OnceInOrderSupportEnabled>false</OnceInOrderSupportEnabled>
          <externalCassandraServerRequired>true</externalCassandraServerRequired>

          <coordination>
              <!-- Apache Zookeeper Address -->
              <ZooKeeperConnection>192.168.0.101:2181,192.168.0.102:2181,192.168.0.103:2181</ZooKeeperConnection>
              <!-- Format yyyy-MM-dd HH:mm:ss -->
              <ReferenceTime>2012-02-29 08:08:08</ReferenceTime>
          </coordination>
      

Modify qpid-virtualhosts.xml file which is located at :/repository/conf/advanced directory as bellow

  • Change the connection string by pointing to the cassandra server
          <store>
              <class>org.wso2.andes.server.store.CassandraMessageStore</class>
              <username>admin</username>
              <password>admin</password>
              <cluster>ClusterOne</cluster>
              <idGenerator>org.wso2.andes.server.cluster.coordination.TimeStampBasedMessageIdGenerator</idGenerator>
              <connectionString>192.168.0.100:9160</connectionString>
          </store>
      

Now you can start all three Message Broker nodes.

Other Resources

The WSO2 Oxygen Tank Library : The Library contains articles, tutorials, presentations, and other knowledge base items published on The Oxygen Tank on different styles of deployments.