Cwgordon7's blog

Today we began the code sprint! This is an area of , as one of our weaknesses has proven to be the number of testers we have, particularly how small that number is. Automated testing will provide an easy way to maintain the quality of core— each time a patch is created for core, it will be sent off to a where the patch will be applied to a copy of the Drupal core and all the SimpleTests run on it. If there are failures, the patch would be marked as such and whoever submitted the patch would be forced to hang their head in shame.

What we accomplished today

Today, Dries Buytaert, Rok Žlender, Károly Négyesi, Jimmy Berry, Kevin Bridges, Douglas Hubler, Miglius Alaburda and I got together for 10 hours for the sole purpose of improving automated testing in core. It was great. We bounced ideas off one another, and came up with a great plan for unit testing with mock functions and classes without runkit. We made our test coverage tests work, although they are still running :). We gave a presentation to the people at Drupal Camp Paris on how to write basic functional tests for both core and contributed modules. We talked to Dries about the possibility of .

Unit testing with mock functions and objects

Here's the difference between functional testing and unit testing: functional testing makes sure the code functions the way you expect it to, e.g. if you load a page, is it the page you expected to get? Most of our current tests our functional tests. However, there is another kind of test that is valuable as well: unit testing. Unit tests know about certain internal or API functions that exist in Drupal, and make sure those functions work as expected.

The problem with efficient unit testing is that there are so many conditions that have to be accounted for, and in many cases, these conditions are hard to achieve. For example, if I were calling cache_get(), I have to account for this line:

$cache_flush = variable_get('cache_flush', 0);

which is used in further if() statements. In order to fully test it, I should call it both when this for all possible values of $cache_flush. In order to do this, I need to override the variable_get function which is used to generate this variable's value. Solutions such as would work, but we want to have tests that all Drupal users can use, not just the small percentage with the ability to set up an environment with runkit.

How can we override functions? Once they are defined in their files, they cannot be undefined. However, that is just the key: don't include the files at all. Only include the specific function you are testing, and everything else can be represented by mock functions, used to return a certain value that can be set within the SimpleTest test case. In essence you're plucking a function out of its nest, making it think it's safe at home, and making sure it performs backflips properly. This awesome idea can be attributed to chx and Jimmy Berry. Basically you'd read all of the Drupal code base without parsing it, have PHP tokenize it, find all the functions (and classes and interfaces), cycle through and define them as:

function $function_name() {
  \$args = func_get_args();
  return simpletest_handle_mock_function($function_name, \$args);
}

This is an amazing code snippet. You basically make every single function in the Drupal code base call one centralized function from which you make all your mock-function settings.

We hope to work on this more on the next few days and hopefully get it into a working condition. This will also, hopefully, be made easier with the addition of .

Drupal SimpleTest test coverage

In his blog, Dries promised that we could have three extra months of development time with complete core test coverage. Well, this raises the question: how do we determine how much of the Drupal core we've actually covered with our tests? We're looking on a way to figure this out.

There are many coverage-testing tools, but nearly all of them are based on the same principle: they look at your code while you run your tests, count the physical number of lines that were hit and not hit, and return that result. A tool we've been looking at called provides a means of doing this in PHP. Relying on XDebug for the heavy lifting, Spike PHPCoverage is a relatively lightweight utility which generates a nice html report of our current test coverage. A demo report generated by an early attempt at determining code coverage can be found here. (Note: this data is not accurate, but gives an idea of what the test coverage reports will look like.)

We have gotten the test coverage tests to work, but due to the massive amounts of time it takes to run all tests— plus the fact that adding test coverage makes it about twice as slow— and you can see that this is going to take a while not only to run all tests, but to record which lines of code are called and then generate a huge series of html files on our test coverage. But it should be awesome once we're finished, as not only will we know what are status is in regard to overall SimpleTest testing, but also where, exactly, our weaknesses are.

Presentation on Simpletest, and what you can do to help

Our testing sprint was going on parallel to Drupal Camp Paris, and we decided to give a short session on SimpleTest. However, come time for our session, we were nowhere near ready to present on our new unit tests. So we decided that I would do a repeat of my dojo lesson on writing functional SimpleTests for the . Not only did this go amazingly well, but there were a lot of questions— people seemed genuinely interested, and some even volunteered to help us out tomorrow by actually writing some of the tests we were making easier to write. This is completely and insanely AWESOME. And the best part is, you can help too, even if you're not in Paris! We have a list of tests that need to be written, and now we need people like you to write them! This is your opportunity to push back the dreadful day of the Drupal 7 code freeze. Let's get to it!