Browser based testing with Watir - integrating into cruise control

[Update: Check out This Blog for an excellent way to get junit xml style output from ruby Test::Unit]

WATIR (www.watir.org) is a great functional testing tool that drives IE with Ruby. (It has plans to support Firefox and Safari too).

Since it's driven through the DOM of IE, it can drive the browser just as a normal user would, and do assertions on things like AJAX functionality. This also results in far less fragility than with a record/playback solution.

It's also driven through Test::Unit - Ruby's xUnit testing framework. This is good because you can integrate it into Cruise Control.

Justa couple notes on how to get this going:
1) Don't use the ServiceWrapper - just have a CI specific account login automatically, and have a startup script load cruise control. There are access control issues when running IE from the service account (especially if no one is currently logged into the local machine).

2) Route the output of your xUnit tests to a junit xml style file. Following something like http://meb.home.cern.ch/meb/xPyUnit.htm is a good way. Our solution was to create a different TestRunner, and override the finished() method. It's kind of hacky, so not worth publicizing :)

3) Return a non-zero error code from ruby if the tests fail, and set failonerror="true" in your ant script that calls this process. I do this (at the bottom of my main run-all script) like:

result = Test::Unit::AutoRunner.run(true)
exit 0 if result == nil
exit result

This invokes the same logic as an empty .rb that depends on test/unit - calling autorunner and searching the object space looking for tests to run.

Then you can do a merge on the results of the functional tests as if it was just a normal junit test.