Sunday, 9 February 2014

Can you run phpunit tests on entire folders?

I'll warn you up front. This is a post about code and programming. If you've already fallen asleep it won't get any better :-)

My day job is about code and I see the purpose of it all as finding faster ways to do something than before.

Today we have quality control duties. We upgraded to the latest version of Moodle (2.6) and so we're checking all our plugins work. This means running a bunch of boring tests. Some automatic, some manual. Most not as automatic as I would like.

So of course this is a chance to learn something.

Moodle uses PHPunit and you run their unit tests through the command line. The advantage is that you can then use the power of the command line to automate things or at least that's the theory.

In this case we use Git Bash, not the windows command line itself. This gives us unix like functionality and a chance to learn some unix. Useful for a guy like myself who got trapped in the Microsoft castle early on in my career. I've been struggling to get out ever since.

I have the task of checking 12 question types, 3 reports and 2 other features. My first thought? How do I automate thee.

Individual tests

I started running each individual test with commands like

vendor/bin/phpunit qtype_ddimageortext_walkthrough_test question/type/ddimageortext/tests/walkthrough_test.php

learn more at phpunit: the command line test runner
Then I realised I needed to do this for each test file of each feature. That's a lot of files

Individual Folders

I asked around and came up with
find question/type/ddimageortext/tests/*_test.php -type f -print | xargs -n1 vendor/bin/phpunit

This taught me some Unix. I didn't realise that in Unix you can query the file system much like you query a database through sql. Fascinating.

What this sentence does is first find files for a given query then use xargs to execute the phpunit tests with the required parameters


The find command is everything before the |

find question/type/ddimageortext/tests/*_test.php -type f -print

You break it down as follows

  • find : Find items given relevant parameters
  • question/type/ddimageortext/tests/*_test.php:  all files in the given path matching the *_test.php expression
  • -type f : match files. For folders use d
  • -print : Print results to the command line. Otherwise you don't see anything.


The xargs command is everything after the |

xargs -n1 vendor/bin/phpunit

You break it down as follows

  • xargs : create a series of command line statement items given a utility and argument
  • argument -n1 take the first item passed from find as an argument to follow the phpunit statement
  • utility vendor/bin/phpunit phpunit is a utility xargs can call
That broke my head a little at first but now I have described it I finally feel I actually understand it :-)

Which test has been run

This was working really well. I could see all the tests in one folder had passed though I realised I didn't know which test file was being called each time. There was nothing to tell me the file name or relevant classes. Not critical but I could see the potential to search all question types and thus multiple folders. At that scale I need to easily identify which files have problems.

enter phpunit --debug
The --debug switch is the only option I could find to print out anything related to the test file being processed. It prints too much info, each individual test that is run but it's better than nothing. So I went with it.

That built the command up to
find question/type/ddmarker/tests/*_test.php -type f -print | xargs -n1 vendor/bin/phpunit --debug

All folders in question/type

Then of course I saw the potential of just running all the tests in the question/type folder because I'm going to test most of them anyway and it's quicker than checking each individual folder.

So I try:

find question/type/*tests/*_test.php -type f -print | xargs -n1 vendor/bin/phpunit --debug

and find two problems.

  1. There is too much information flashing by. I need this stored to a file
  2. The run aborts when it reaches the stack folder. Ignoring the tests after the stack folder

Save to file

This one is straightforward just add

> unit_tests.txt

to send the results to a file in the location your shell is currently at.

Exclude a folder

I tried various options to get round the stack errors including:
  • phpunit::filter patterns to avoid tests that errored and stopped the run.
  • find only specific folders
The first didn't work. The second just seemed more complicated than my final solution.

So I ended up telling the find command to ignore the stack folder. It took a little doing but once I got my head around how find works it became straightforward. 

I just needed to tell find to exclude anything with a path of stack. I did this using -not -path "*stack*"

Ending up with:
find question/type/*/tests/*_test.php -type f -not -path "*stack*" -print | xargs -n1 vendor/bin/phpunit --debug > unit_tests.txt

I just ran the command and it worked perfectly. I now have a file listing the results of all the unit tests in the question/types folder. I found two warnings of incomplete tests but nothing else. Result!!!

These are some of the references I used from the search find all directories except

Of course now it seems simple. The point is that I learnt a lot about unix. I now understand two unix utilities and that the point of unix is that it is
a bunch of utilities that you can link together in various ways like lego to achieve any end you want. 
This is quite literally a small step for my coding but a huge leap for my understanding of operating systems and why unix is the father of all we have today. I now understand why everyone else reveres it so much.


This would all seem like a lot of work for something you do rarely. That's what everyone tells me when I share this. You see I log all this and keep it for later. Last time I built a nice template for testing which I just brought out again this time and copied as a basis to begin with. I have the same work to do plus a few more features to test.

Now I have added much more to the template so I can get the job done to the same standard next time but faster. I've always found this iterative approach a great way to learn and also improve so I'm sticking with it. The only difference is that I'm now sharing it more openly. 

No comments: