Using nodetool setstreamthroughput to adjust decomission and other streaming peformance

Posted on August 3rd, 2012

Cassandra uses streaming in several places. Streaming is used when a node joins or leaves the ring. For example, when a node is decomissioned using nodetool decommission it streams its data to other nodes before leaving the ring. This value can be adjusted in both the confirguration file and during runtime using nodetool or JMX.

How to do it…

In the conf/cassandra.yaml file the default value can be set:

# When unset, the default is 400 Mbps or 50 MB/s.
# stream_throughput_outbound_megabits_per_sec: 400

This command can be adjusted at runtime using nodetool. The value here is in MB/s

bin/nodetool -h cdbeq01 -p 8585 setstreamthroughput 51

Note: These changes need to be done per machine.

How it works…

setstreamthroughput is a rate limit. The default is 50 Mb/second. If you have a high speed network such as gigabit ethernew this value can be set higher to speed-up certain operations. For example, when doing nodetool decommission the decomission will finsih sooner if the configuration value is raised, assuming no other limiting factor is involved. However, you may wish to set the value lower to researve bandwidth and other resources for low latency operations.

There is more…

Remember that a network with Gigabit ethernet ports may not be capable of supporting 100% utilization on each port at the same time. Consult your network engineer and consider how many of these streaming or other operations bandwidth intensive are happening at once when adjusting this setting cluster wide.

Filed under Chapter 4 - Performance Tuning, Chapter 7 Administration | No Comments »

Raising the number of processes for Cassandra

Posted on July 9th, 2012

Redhat linux 6 and derivatives like CENTOS 6 lower the number of processes a user process can start to avoid accidental fork-bomls. Under high request rate this can cause the operating system to kill the cassandra process.

How to do it

Change the system limits from 1024 to 10240. Start a new shell for these changes to take effect.

vi /etc/security/limits.d/90-nproc.conf
*          soft    nproc     10240

How it works

An active cassandra server can create many threads and processes. Normally Cassandra daemonizes and runs as a standard user not root. This setting allows non root users to create more processes.

Filed under Chapter 4 - Performance Tuning, Chapter 7 Administration | No Comments »