WordPress:Automated Testing
Automated testing
Automated Testing is the act of running all or some test cases in an environment, where each individual test is run one after the other and combining the results. There are different ways to create automated testing suites and even then automated does not have to apply to a system where it is run without you manually starting it. Automated only means that it will run all of the test cases for you without you having to run each one by yourself.
The Automattic WordPress Tests is the formal and authoritative automated testing suite for WordPress. The only way to download is by checking out with Subversion and there isn't a bug tracker devoted to just it.
There is also another tool, based on the Automattic WordPress Tests, called WPTests, which allows for looking at the results based on its server configuration. The WPTests Respository also can only be downloaded by Subversion.
The test case type that the Automattic WordPress Tests uses are not units, therefore mocks do not need to be made. This allows for testing the entire scope of a test in the WordPress execution, much like it would work on a live web site. It should be noted that you should not run the Automattic WordPress Tests on a web site you don't want to be deleted. The Automattic WordPress Tests will remove all databases before and after it runs the tests, have a database devoted to just it.
This section is composed of four parts:
- Working with the Automattic WordPress Tests
- Working with the WPTests Repository and Site
- Writing Test Cases
- Submitting the Test Cases
Automattic WordPress Tests
Downloading
You will need to download WordPress:Subversion before you can continue, if you don't have it already. You should also have an empty MySQL database available. Create a new one if you have to, but you don't want anything in there.
Check out the latest testing suite from http://svn.automattic.com/wordpress-tests/
Configuration
- In the root folder of the checkout copy wp-config-sample.php to wp-config.php.
- Edit it to include your database name/user/password.
You might find it easier to just go to that WordPress installation and run the install, so that it creates the wp-config for you. After, you can copy that to the directory below and run the tests. The Automattic WordPress Tests will be looking for wp-config.php in the same directory as wp-test.php. This will also prevent an error in which wp-cron.php will fail, because it can't find wp-config.php in that directory.
Running The Tests
The wp-test.php will initiate the automated test suite run. There are some options for which to control what tests are run and other options which trigger how the automated test suite works. Without any arguments, all tests will be run with default options.
The average time for running all of the tests can range between 2 minutes and 7 minutes. This is with the current 399 test cases. When the amount increases, so will the time it will fully take to execute all of the tests.
Note for Windows Users: The wp-test.php makes use of getopt() function, which does not work on Windows before PHP 5.3. You will have to wait until PHP 5.3 comes out before you can use the command line on Windows or add some hacking to get it to work for you.
Run the tests by typing: php wp-test.php in the command line.
Arguments
- -v : You can run WordPress Mu instead of WordPress, by typing php wptests.php -v mu
- -t : Choose which test you want to run, this is the class name, which is the test suite with the test cases.
- -r : The directory of WordPress which will be used to test. You may create custom patches on other locations and this allows you to not have to checkout multiple Automattic WordPress Tests repositories for which to apply those patches for testing.
- -s : If this option exists, then known WordPress bugs will be skipped.
- -f : If this option exists, then it will force skipping known WordPress bugs.
The first three require a value to be after it, the last two only need to be typed. All arguments are optional.
Examples
For example, if you wanted to run the Plugin API Actions test suite, then you would type the following on the command line:
php wp-test.php -t WPTestActions
If you wanted to skip known bugs, you would type the following:
php wp-test.php -s
And if you wanted to force it:
php wp-test.php -s -f
WPTests Tool for Automattic WordPress Tests
The WPTests web site is setup to automatically run two minutes past the hour, every hour. Therefore it provides a history starting from the 17th of March 2008 onward. This holds some interest to see where tests failures rose and fell. Given that it does not track the WordPress Subversion revisions, this doesn't hold that much strength other than to show which tests failed and what errors there were.
There are two parts to the WPTests Site. The Main page which shows the last available test run and how long ago it was. If it is longer than an hour and a half, then you know that something is wrong (given that if it was the web site, the entire web site would be down).
The second part is the historical test runs, which you can look through and some of the later test runs allow you to check the results of the run.
The repository is located at [1] and requires Subversion.
Other requirements for running the WPTests tool on your system, is WAMP, Apache, PHP, and MySQL. If you install the latest version of WAMP, then you'll have to enable PDO_SQLite module, because it is disabled by default and the tool needs it.
Writing Tests
Test cases live in the wp-testcase subdirectory. All files in that directory will be included by default.
Extend the WPTestCase class to ensure your test is run. You can see the code in current tests for examples or read the PHPUnit Pocket Guide.
Submitting the Test Cases
The test cases should be submitted to Alex Shiels of Automattic. You can contact him at his blog or web site at Threshold State. You may also leave the test case at the WordPress Trac, if it is based on a ticket and inform Alex of the test case or test suite.
Alex is fairly good at getting the test cases in within three to seven days.