Improving the Siesta automation

As you may already know, we use Siesta to test our own JavaScript components (Ext Scheduler and Ext Gantt) as well as Siesta itself. The test suites for our Ext JS components now consist of about 100 files for each product, meaning a couple of thousand assertions in total. More over we want our components to be compatible with at least the few most recent versions of Ext JS (4.1.0 and 4.1.1 at present time). This means we need to run our tests against 2 versions of Ext JS – doubling the size of the test suite to ~200 files for each of our products.

 Memory leak issues

While such a big test suite gives us great confidence in the quality of our products, until recently we were struggling to run the automated test suite because some browsers leak memory. We found that the browser memory usage continued to grow while running the test suite and finally the browser either froze or crashed. Other times tests would start to randomly fail, producing varying results. In fact, we had this problem even in good browsers, like Chrome and Firefox. Some versions of those browsers were leaking memory and some not – it seems this regression is periodically re-introduced. Conclusion: It seems we just cannot rely on the browser garbage collection mechanism and expect everything to work perfectly.

 A new approach

After reaching this conclusion, we did some experimenting and decided that the best approach would be to run the tests suite in smaller parts, re-starting the browser after each part is done. This approach keeps the memory usage low and gives consistent results across browsers. This ‘paging’ mechanism was implemented for both the PhantomJS and Selenium launchers. In addition we added a slight, configurable pause between each test, allowing slower/older browsers to “take a breath” before proceeding to next test.

Another recent and quite important improvement in the Siesta automation package is the introduction of the new Selenium launcher – based on WebDriver. WebDriver is sometimes called Selenium 2.0 – it is a new part of the Selenium project and we found that it’s much more stable than Selenium RC. The WebDriver launcher provides the same command line interface as the Selenium RC launcher, so migration to it should be smooth. The old Selenium RC launcher is now deprecated and scheduled for removal in one of the nearest releases.

Summing up…

So, now we happily run our big test suite before each release making sure you won’t have to report any bug twice and instead can spend your time developing your application, not debugging our components.

The next step for us is to complete setting up our Continuous Integration process by running nightly builds and tests on all browsers out there to detect any regressions early. We also plan on running our test suites against the bleeding edge builds of Ext JS (which the Sencha dev team kindly agreed to provide for us). This will mean smoother upgrades to newer versions of Ext JS and we also hope this will contribute to keeping future releases of Ext JS even more stable.

The improved automation support is available in the Siesta 1.1.1 released yesterday. It works great for us – please let us know how it works for you!

Enjoy the Siesta 🙂

Leave a Reply

1 Comment threads
0 Thread replies
Most reacted comment
Hottest comment thread
0 Comment authors
Running nightly tests with Siesta | Bryntum – Blog Recent comment authors
newest oldest most voted
Notify of

[…] describe our continuous integration setup which runs all our test suites on a daily basis. The recent improvements in the Siesta automation package allows us to run big test suites. Since then, we have added […]