After more than 6 years doing MySQL Support for MySQL AB, Sun Microsystems, and Oracle, it’s time for a change. Time to get back to development! As of November 2011 I’ll be working full-time on MySQL Connector/Python and other goodies within the MySQL development team at Oracle. Before, this was more or less a pet project done after working hours. However, with the birth of our son Tomas more than a year ago, I’ve been slacking and family got priority.

This post explains how to debug MySQL Cluster 7.1, installed using the RPM packages, using gdb on a Linux box (Red Hat, Oracle Enterprise Linux, CentOS, ..). When a data node crashes lots of information goes into the error log, trace files and out log. However, it makes sometimes sense when you can repeat the crash, to run the data node in debug mode, or using gdb. First, using RPMs and a Linux distribution, make sure you have the ‘debuginfo’ package installed.

There is a log file called ndb_<NodeID>_out.log created by the MySQL Cluster data nodes which can become quite big overtime. There is, unlike the cluster logs created by the management nodes, no rotation build in. So you have to revert to the basics and copy the file away, truncating the old one. For example, if you want to ‘rotate’ the log file of data node with NodeID 3: $ mv ndb_3_out.

This is a reply to the blog post Poor man’s MySQL replication monitoring. Haidong Ji had a few problems using MySQLdb (could use the ‘dict’ cursor) and apparently he doesn’t want to much dependencies. I agree that using the [mysql](http://dev.mysql.com/doc/refman/5.5/en/mysql.html) client tool is a nice alternative if you don’t want to use any 3rd party Python modules. And the MySQL client tools are usually and should be installed with the server.

The MySQL Cluster data node log files can become very big. The best solution is to actually fix the underlying problem. But if you know what you are doing, you can work around it and filter out these annoying log entries. An example of ‘annoying’ entries is when you run MySQL Cluster on virtual machines (not good!) and disks and OS can’t follow any more; a few lines from the ndb_X_out.