Using nodetool setstreamthroughput to adjust decomission and other streaming peformance

Posted on August 3rd, 2012

Cassandra uses streaming in several places. Streaming is used when a node joins or leaves the ring. For example, when a node is decomissioned using nodetool decommission it streams its data to other nodes before leaving the ring. This value can be adjusted in both the confirguration file and during runtime using nodetool or JMX.

How to do it…

In the conf/cassandra.yaml file the default value can be set:

# When unset, the default is 400 Mbps or 50 MB/s.
# stream_throughput_outbound_megabits_per_sec: 400

This command can be adjusted at runtime using nodetool. The value here is in MB/s

bin/nodetool -h cdbeq01 -p 8585 setstreamthroughput 51

Note: These changes need to be done per machine.

How it works…

setstreamthroughput is a rate limit. The default is 50 Mb/second. If you have a high speed network such as gigabit ethernew this value can be set higher to speed-up certain operations. For example, when doing nodetool decommission the decomission will finsih sooner if the configuration value is raised, assuming no other limiting factor is involved. However, you may wish to set the value lower to researve bandwidth and other resources for low latency operations.

There is more…

Remember that a network with Gigabit ethernet ports may not be capable of supporting 100% utilization on each port at the same time. Consult your network engineer and consider how many of these streaming or other operations bandwidth intensive are happening at once when adjusting this setting cluster wide.

Filed under Chapter 4 - Performance Tuning, Chapter 7 Administration | No Comments »

Raising the number of processes for Cassandra

Posted on July 9th, 2012

Redhat linux 6 and derivatives like CENTOS 6 lower the number of processes a user process can start to avoid accidental fork-bomls. Under high request rate this can cause the operating system to kill the cassandra process.

How to do it

Change the system limits from 1024 to 10240. Start a new shell for these changes to take effect.

vi /etc/security/limits.d/90-nproc.conf
*          soft    nproc     10240

How it works

An active cassandra server can create many threads and processes. Normally Cassandra daemonizes and runs as a standard user not root. This setting allows non root users to create more processes.

Filed under Chapter 4 - Performance Tuning, Chapter 7 Administration | No Comments »

Using nodetool to temporarily disable a node

Posted on February 6th, 2012

There are times when you would like a Cassandra node to stop receiving requests or appear like it is down to outside clients. You may want to use this to disable a node and adjust your log4j properties for more debugging, or your may be running some administrative action like repair or rebuildsstables and you want this action to complete as quickly as possible. Being able to disable Cassandra without shutting it down completely can be very handy.

How to do it:

Use nodetool to disable Cassandra only on a specific node.

/usr/local/apache-cassandra-1.0.7/bin/nodetool -h cassandraserver -p 8585 disablegossip
/usr/local/apache-cassandra-1.0.7/bin/nodetool -h cassandraserver -p 8585 disablethrift

How it works:

Gossip is the protocol that Cassandra nodes use to determine which nodes are UP or DOWN. Cassandra nodes gossip to each other and maintain a state of the cluster. By disabling gossip other Cassandra nodes will stop forwarding read and write requests to this node because they will believe the node is down. Client using thrift will still be able to connect to a node in this state. Disablethrift disables the client port 9160 which prevents clients from connecting to this node. Typically you will enable both disablethrift and disablegossip together.

There is more:

To re-enable gossip and thrift use the following commands:

/usr/local/apache-cassandra-1.0.7/bin/nodetool -h cassandraserver -p 8585 enablegossip
/usr/local/apache-cassandra-1.0.7/bin/nodetool -h cassandraserver -p 8585 enablethrift

Tags: cassandra, enablegossip, enablethrift, gossip, nodetool, thrift
Filed under Chapter 7 Administration | 65 Comments »

Deploying Cassandra with Puppet

Posted on August 6th, 2011

(this is update remake of the puppet recipe on my blog)
Cassandra is a peer-to-peer architecture which is typically deployed on a large number of servers. Deploying, managing, and upgrading these systems is more administrative time especially as your cluster grows. Puppet provides a simple way to install Cassandra.

Getting ready

For this recipe you will need a server running puppet.

How to do it…

Puppet has a file server built in. By default the root of the file server is /var/lib/puppet/files. Create a folder for cassandra. Download a release into that folder, rename the files that are different per cluster.

mkdir /var/lib/puppet/file/cassandra
cd /var/lib/puppet/files/cassandra
wget apache-cassandra-0.8.1.tar.gz
tar -xf apache-cassandra-0.8.1.tar.gz
mv apache-cassandra-0.8.1/conf/cassandra.yaml apache-cassandra-0.8.1/conf/cassandra.yaml.bak
mv apache-cassandra-0.8.1/conf/cassandra-env.sh apache-cassandra-0.8.1/conf/cassandra-env.sh.bak

create a puppet manifest for Cassandra /etc/puppet/manifests/cassandra.pp. Create the base requirements that and Cassandra instance would share.

class cassandra_base {
package { "jdk" :
ensure => installed
}
group { "cassandra":  }
user { "cassandra":
home => "/home/cassandra",
shell => "/bin/bash",
require => Group["cassandra"]
}
file { ["/var/lib/cassandra","/var/log/cassandra","/var/lib/cassandra/saved_caches"]:
type => directory,
owner => cassandra,
group => cassandra,
ensure => directory,
mode =>755,
require => User["cassandra"]
}
file { "/etc/init.d/cassandra":
path => "/etc/init.d/cassandra",
mode => 744,
source => "puppet:///cassandra/cassandra.init",
require => Package["jdk"]
}
}

Now extend the base definition for a specific version of Cassandra.

class cassandra_0_8_2 inherits cassandra_base {
file {
"/usr/local/apache-cassandra-0.8.2":
owner => root,
group => root,
path => "/usr/local/apache-cassandra-0.8.2",
source => "puppet:///mainfiles/cassandra/apache-cassandra-0.8.2",
recurse => true
}
file {
"/usr/local/apache-cassandra-0.8.2/bin":
mode => 755,
path => "/usr/local/apache-cassandra-0.8.2/bin",
source => "puppet:///mainfiles/cassandra/apache-cassandra-0.8.2/bin",
recurse => true,
require => File [ "/usr/local/apache-cassandra-0.8.2" ]
}
file { "/usr/local/cassandra":
ensure => link,
target => "/usr/local/apache-cassandra-0.8.2",
require => File[ "/usr/local/apache-cassandra-0.8.2" ]
}
}

Extend this one more time per cluster. Use the fqdn function in puppet so each node can get its own copy of the configuration file.

class cassandra_lab_0_8_2 inherits cassandra_0_8_2 {
file { "/usr/local/cassandra/conf/cassandra-env.sh":
path => "/usr/local/cassandra/conf/cassandra-env.sh",
owner => "root",
group => "root",
source => "puppet:///cassandra/lab-0.8.2/cassandra-env.sh"
}
file { "/usr/local/cassandra/conf/cassandra.yaml":
path => "/usr/local/cassandra/conf/cassandra.yaml",
owner => "root",
group => "root",
source => "puppet:///cassandra/lab-0.8.2/cassandra.yaml."+fqdn()
}
}

Now setup the configuration files for this cluster.

mkdir /var/lib/puppet/files/cassandra/lab-0.8.2
cp /var/lib/puppet/files/cassandra/cassandra-0.8.2/conf/cassandra.yaml.bak /var/lib/puppet/files/cassandra/lab-0.8.2/cassandra.yaml.server1.domain.com

For each server in the cluster include the class we created.

node 'server1.domain.com','server2.domain.com'{
include cassandra_lab_0_8_2
}

How it works…

The puppet configuration management system copies files from the server to clients. This makes it easy to replicate installations across large clusters of servers. Puppet file directives copy since files or recursive directories from the puppet server to the client. Puppet also installs system users and groups as well as packages in a platform independent way.

Tags: cassandra, nosql, puppet
Filed under Chapter 7 Administration | 59 Comments »