The Bar Eats Foo

Well, my posterous is timing out so I decided to start a blog here. I guess it’s about time to switch. This, as usually, is just a brain dump to stop me from forgetting.

= The problem

Some actions are failing on my cassandra cluster. truncate and drop column family. cassandra is complaining about unreachable nodes. describe cluster reveals the offending IPs. The odd thing is, nodetool ring doesn’t. These are old nodes that were removed from the cluster a while ago, they are stuck in state REMOVED. odd..

= The solution

After visiting IRC, the illustrious driftx shed some light on the subject.

 https://issues.apache.org/jira/browse/CASSANDRA-3337 

If you aren’t a java pro, it’s not exactly obvious what needs to be done. But I have been trying to tune cassandra and I do have mx4j-tools.jar installed. You can read about it on the invaluable Operations page of the cassandra wiki. This opens a web server on port 8081 which is a noob friendly interface to JMX.

Navigating to http://cassnode00:8081/mbean?objectname=org.apache.cassandra.net%3Atype%3DGossiper

Plug the IP addr of the bad nodes into the Value field and invoke the unsafeAssassinateEndpoint.

feel like a ninja.