In this post we’ll describe our continuous integration setup which runs all our test suites on a daily basis. The recent improvements in the Siesta automation package allows us to run big test suites. Since then, we have added support for Ext JS 4.1.3 to the stack, along with tests against the nightly builds of Ext JS. This means our products are being tested with Ext JS 4.1.1, 4.1.2, 4.1.3 + Ext nightlies, in IE, Chrome and Firefox – every night! On top of that, we’re also running certain date calculation tests in 30 different time zones to detect any daylight savings issues.
One of our main requirements for running nightly tests was to run them in real browsers. After all, this is where our products will be used. So, we decided to start with IE 9, Chrome and Firefox – which means we need Windows OS and Selenium WebDriver.
Another option would be to use the PhantomJS launcher, but we found that the WebKit version in PhantomJS is slightly different from the implementations in Chrome and Safari. The difference is very small, but sometimes it is actually enough to make a test to pass in Chrome/Safari and fail in PhantomJS. We still use PhantomJS during development since it’s a very handy tool for running tests from within your IDE.
To run our nightlies, we setup a dedicated server with Ubuntu 10.04. It has VirtualBox 4.2 installed, running a Windows 7 guest OS. Our server has 4 GB of memory, 2GB is dedicated for our virtual machines.
VirtualBox provides remote desktop access to the virtual machine, so one can connect using any RDP client. See this chapter in the VirtualBox manual for details. Using this connection you can configure the machine and install additional browsers, OS updates etc. Tip: To speed up the screen refresh rate, disable the wallpapers feature in Windows.
Our host server and guest Windows OS share a folder containing the source checkouts of our products. The host server is running Apache, available from guest using a fixed IP address which we’ve aliased to host “local” in “system32driversetchosts”. With this setup, we can run our tests easily by simply opening a URL in the guest OS: http://local/ExtScheduler2.x/tests/index.html
In our host system, we have a cron job executed at 1 AM every day. This cron job first fetches the latest sources from Git, then builds the products. Once our products have been built, it proceeds to launch the test suites:
vboxmanage guestcontrol nightlies execute --image "c:\WindowsSystem32\cmd.exe" --username bryntum --wait-exit "/C l:\siesta\bin\webdriver http://local/ExtScheduler2.x/tests/index-no-ui.html$EXTROOT --browser=chrome > $SAVETO\scheduler-chrome.txt"
The test reports produced from each browser are then combined and emailed to us using the “sendemail” command. Below you can see what we see in our mailbox every morning:
We are planning to extend our VM farm with Windows XP + IE8 and MacOS. Additionally, we are investigating how we can run the tests on actual mobile devices. Even further down the road, we will likely create a cloud-testing service. This will allow Siesta users to avoid all the manual setup described above and just focus on writing tests which can then be launched remotely accessing our cloud service, something like:
bin/remote http://project/harness.html --os=windowsxp --browser=ie8