HOWTO: Making a Corresponding Test Case for your Patch

Recently, we've seen a number of excellent contributions to the MySQL server, and we're talking with Kevin Burton about possibly integrating his excellent lbpool work into the Connector suite. Jeremy Cole has written a number of patches for small, but very useful, feature enhancements to the main server — namely the ability to do a SHOW PROFILE trace of calls during statement execution, host cache improvements for 5.0, and some nice utility functions for trigger/procedure diagnostics. Bill Karwin has submitted a patch which adds functionality to the server by providing the SHA-2 function, which is required by certain government mandates concerning digest encryption.

One of the things that Bill mentioned in his recent blog post about his SHA-2 patch is figuring out how to properly write a corresponding test case which demonstrates the bug-free behaviour of the patch. I wrote an email to Bill describing the steps to do this, and I figured it was good information to put out there for the general public for anyone interested in submitting feature or bug patches. Recently, I've dug in my heels a bit and volunteered to fix (very) minor bugs in the source code, and this process (of writing a test case and testing the patch) was the first thing I had to learn and understand. Below are some helpful hints about getting a proper build environment up and running on your machine and running the test suite properly. In order to best show how things are done, I will walk everyone through the exact process I went through in fixing my first bug, a very small fix to the INET_ATON() server function.

I welcome MySQL employees to comment on this blog entry (or even expand on it in their own articles) with helpful tips for those of us just getting into bug fixing or patching the MySQL sources.

The BUILD directory and Scripts

Before you test, you've obviously got to get a sane build environment set up to compile the server and your patch. For better or worse, building MySQL requires a number of very up-to-date build tools in order for you to properly compile the server from source. I think many folks tend to downplay the work that goes into providing binaries and (somewhat) snapshots for so many platforms. I learned fairly quickly when working on my first bug fix that compiling MySQL is a complex, error-ridden, and oft-frustrating process. Nevertheless, here are the things you should have installed to avoid headaches:

• automake version 1.9.6
• autoconf version 2.60
• bison version 2.3
• m4 version 1.4.4
• libtool version 1.5.22

Additionally, I have the following versions of development tools installed, though I am less certain that these versions are as absolutely critical as the ones above:

• gcc version 4.1.2
• make version 3.81

Note that it is important what order you install these tools! libtool must be installed last to avoid linking with older versions of the other tools. If you don't, you get weird errors. Thanks to Chad Miller from the MySQL Maintenance team for figuring that one out. Bison is important; if you don't have it installed or a current version, you get weird errors corresponding to AM_ macros referencing Yacc. Thanks to Elliot Murphy for helping me get through that obstacle.

Once you have those tools updated and installed, the first thing you will want to do is set up a directory you will use for development. I use ~/dev:

#> cd ~; mkdir dev

Next, you will want to bring in the sources via bitkeeper. While I am using the non-free Bitkeeper version which allows for direct checkin and commit, the free Bitkeeper client allows you to clone from a tree just fine and start your development. First thing first, you need to download and install the free BK client. Grab the client from this URL. Or just execute the following:

#> wget http://www.bitmover.com/bk-client.shar
#> sh bk-client.shar
#> cd bk_client-1.1
#> make all
#> PATH=$PWD:$PATH

OK, so you've got the free BK free client installed. Now, you've got to clone an appropriate tree. So, you would do the following:

#> sfioball -r+ bk://mysql.bkbits.net/mysql-5.1 mysql-5.1

At this point, wait for BK to download the source tree (this can take a while so go get a cup of coffee). When completed, you know want to build the source. If you change directory into the newly created source tree, you will notice a subdirectory named BUILD. This directory contains various utility scripts which make your life much easier. Personally, I do development on my laptop, which is a Pentium Centrino. So, in order to build the server, I execute the corresponding compile-pentium-debug script:

#> cd mysql-5.1
#> ./BUILD/compile-pentium-debug

And... wait. The script essentially automates building the server with debug symbols, setting up the configuration properly (which, by default, does not build InnoDB). If you get errors in the build process, and they reference things like "undefined ... AM_XXX" it probably means you don't have up-to-date versions of the build tools listed above. Hopefully, if all goes well, the build succeeds. If you get that far, great job! You now know that the server builds properly and that any changes you make from this point on are on a fresh, buildable source — meaning, if you get errors from now on, it's your fault!

The mysql-test Directory and Scripts

When fixing bugs, the very first thing you want to do is create a test case which demonstrates the bug's behaviour (if possible) so that you have a starting point from which to test any changes you make to the source. In my case, the bug was concerning the INET_ATON() function returning a signed integer, instead of the expected unsigned integer. So, I needed to make a test which showed the behaviour of the bug. My first instinct was to create a new test case for my bug, but, as Chad pointed out, a better option was to use an existing test case called "func_misc.test" which contained bug fix tests for a variety of miscellaneous functions.

The MySQL test suite is contained in the /mysql-test subdirectory of the source tree. The mysql-test directory has two subdirectories of utmost concern to you, the bug fixer: the /t directory and the /r directory (for "tests" and "results" respectively). All the tests are found in the /t directory. So, I opened up the relevant test file with gVim:

#> cd mysql-test
#> gvim t/func_misc.test

After skipping to the end of the file (Shift-G for those vi newbies), I added the following lines to the file:

#
# Bug #21466: INET_ATON() returns signed, not unsigned
#

create table t1 select INET_ATON('255.255.0.1') as `a`;
show create table t1;
drop table t1;

directly before the command:

--echo End of 5.0 tests

which simply marks the end of the test run for this test.

The three line test simply creates a new test table "t1" from the result of the SELECT INET_ATON('255.255.0.1') as `a` statement. The next line show us the result of that CREATE TABLE statement. Lastly, we cleanup the test by dropping the test table.

OK, so great, you say. What's the big deal? This is where the mysql-test suite comes into play. We need to make a test result file based on our newly modified test case. We do so with the --record option passed to the mysql-test-run script in the /mysql-test directory. The following output gives you an idea of what happens when we execute this:

#> ./mysql-test-run --record func_misc
Logging: ./mysql-test-run --record func_misc
Using binlog format 'stmt'
Skipping ndbcluster, mysqld not compiled with ndbcluster
Skipping SSL, mysqld not compiled with SSL
Binaries are debug compiled
Using MTR_BUILD_THREAD      = 0
Using MASTER_MYPORT         = 9306
Using MASTER_MYPORT1        = 9307
Using SLAVE_MYPORT          = 9308
Using SLAVE_MYPORT1         = 9309
Using SLAVE_MYPORT2         = 9310
Using NDBCLUSTER_PORT       = 9310
Using NDBCLUSTER_PORT_SLAVE = 9311
Using IM_PORT               = 9312
Using IM_MYSQLD1_PORT       = 9313
Using IM_MYSQLD2_PORT       = 9314
Killing Possible Leftover Processes
Removing Stale Files
Installing Master Database
Installing Master Database
Stopping All Servers
Shutting-down Instance Manager
=======================================================
Starting Tests in the 'main' suite

TEST                           RESULT         TIME (ms)
-------------------------------------------------------

func_misc                      [ pass ]          12217
-------------------------------------------------------
Stopping All Servers
Shutting-down Instance Manager
All 1 tests were successful.
The servers were restarted 0 times
Spent 12.217 seconds actually executing testcases

Anyone who has ever compiled from source and actually run the test suite will recognize this output. The mysql-test suite handily starts all the local servers (typically within the /mysql-test/var/ directory) and runs the specified test — the func_misc test case. Note that you do not pass the file name, but rather just the name of the test. The suite will find the corresponding .test file as long as it is named correctly.

Modifying Our Test Result File

Whenever you run the mysql-test-run script with the --record option, chances are, barring a complete catastrophe, your test case will pass, just like above. Why? Because you aren't actually testing anything. Instead, you are saying "just write the test result file as if everything passed."

The output of the --record test run is placed in the /mysql-test/r/ subdirectory. A file named "xxx.result" will have been created, where "xxx" is the name of the test. In our case, a file called func_misc.result was created. Let's check it out:

#> gvim r/func_misc.result

At the end of the result file we see:

create table t1 select INET_ATON('255.255.0.1') as `a`;
show create table t1;
Table	Create Table
t1	CREATE TABLE `t1` (
  `a` bigint(21) DEFAULT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1
drop table t1;

This is almost the correct output. By default the test suite will output the SQL statements executed in the test as well as the results of executing an SELECT or SHOW commands in the client. Above, you see the three exact statements we put into our test, along with the output of the SHOW CREATE TABLE t1 statement. Everything is correct, except that the `a` field has a signed bigint data type, not an unsigned.

So, what we want to do is change our test case result file so that the output of SHOW CREATE TABLE t1 matches what we need it to. So, we change it to:

create table t1 select INET_ATON('255.255.0.1') as `a`;
show create table t1;
Table	Create Table
t1	CREATE TABLE `t1` (
  `a` bigint(21) unsigned DEFAULT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1
drop table t1;

Re-running our test without the --record option, we expect to see the test now fail, as the current behaviour is not the correct behaviour. And, indeed, we do see a failure when we run the test normally:

#> ./mysql-test-run func_misc
Logging: ./mysql-test-run func_misc
-- snip --
=======================================================
Starting Tests in the 'main' suite

TEST                           RESULT         TIME (ms)
-------------------------------------------------------

func_misc                      [ fail ]

Errors are (from -- snip --) :
mysqltest: Result length mismatch
(the last lines may be the most important ones)
Below are the diffs between actual and expected results:
-------------------------------------------------------
*** r/func_misc.result  2006-10-05 01:49:39.000000000 +0300
--- r/func_misc.reject  2006-10-05 01:50:00.000000000 +0300
***************
*** 138,144 ****
  show create table t1;
  Table Create Table
  t1    CREATE TABLE `t1` (
!   `a` bigint(21) unsigned DEFAULT NULL
  ) ENGINE=MyISAM DEFAULT CHARSET=latin1
  drop table t1;
  End of 5.0 tests
--- 138,144 ----
  show create table t1;
  Table Create Table
  t1    CREATE TABLE `t1` (
!   `a` bigint(21) DEFAULT NULL
  ) ENGINE=MyISAM DEFAULT CHARSET=latin1
  drop table t1;
  End of 5.0 tests
-------------------------------------------------------
Please follow the instructions outlined at
http://www.mysql.com/doc/en/Reporting_mysqltest_bugs.html
to find the reason to this problem and how to report this.

Result from queries before failure can be found in r/func_misc.log

Aborting: func_misc failed in default mode. To continue, re-run with '--force'.
Stopping All Servers
Shutting-down Instance Manager

Nicely, our failure above was an expected one. Now, let's fix the code.

The Source of the INET_ATON() Bug

Originally, I had spent quite some time trying to figure out how to make the INET_ATON() function return an unsigned integer. In fact, I traced through the MySQL 5.1 Doxygen output, along with a few hours fumbling through the source code, thinking that the Item_func_inet_aton class, defined in item_func.h and implemented in item_func.cc would simply need to inherit from Item_func_unsigned instead of Item_func_signed. I was, unfortunately, quite mistaken. Jim Winstead, always an excellent source of detailed information on the source code, pointed out that the only thing I needed to do was set the unsigned_flag member field of the Item_func_inet_aton class in the fix_length_and_dec() function on line 1316 of item_func.h. Hmmm, so much for a complicated solution. I changed the following line (1316) from:

void fix_length_and_dec() { decimals = 0; max_length = 21; maybe_null=1;}

to this:

void fix_length_and_dec() { decimals = 0; max_length = 21; maybe_null=1; unsigned_flag= 1;}

Testing the Fix

So, after the above file change, I compiled the server again with the compile-pentium-debug script, and ran the test case again:

#> ./mysql-test-run func_misc
Logging: ./mysql-test-run func_misc
-- snip --
=======================================================
Starting Tests in the 'main' suite

TEST                           RESULT         TIME (ms)
-------------------------------------------------------

func_misc                      [ pass ]          12181
-------------------------------------------------------
Stopping All Servers
Shutting-down Instance Manager
All 1 tests were successful.
The servers were restarted 0 times
Spent 12.181 seconds actually executing testcases

All Done! The only thing left was to submit the changes. I hope this HOWTO helps others in building and properly creating test cases for server bugs as well as new features.