Memory leak issues
While such a big test suite gives us great confidence in the quality of our products, until recently we were struggling to run the automated test suite because some browsers leak memory. We found that the browser memory usage continued to grow while running the test suite and finally the browser either froze or crashed. Other times tests would start to randomly fail, producing varying results. In fact, we had this problem even in good browsers, like Chrome and Firefox. Some versions of those browsers were leaking memory and some not – it seems this regression is periodically re-introduced. Conclusion: It seems we just cannot rely on the browser garbage collection mechanism and expect everything to work perfectly.
A new approach
After reaching this conclusion, we did some experimenting and decided that the best approach would be to run the tests suite in smaller parts, re-starting the browser after each part is done. This approach keeps the memory usage low and gives consistent results across browsers. This ‘paging’ mechanism was implemented for both the PhantomJS and Selenium launchers. In addition we added a slight, configurable pause between each test, allowing slower/older browsers to “take a breath” before proceeding to next test.
Another recent and quite important improvement in the Siesta automation package is the introduction of the new Selenium launcher – based on WebDriver. WebDriver is sometimes called Selenium 2.0 – it is a new part of the Selenium project and we found that it’s much more stable than Selenium RC. The WebDriver launcher provides the same command line interface as the Selenium RC launcher, so migration to it should be smooth. The old Selenium RC launcher is now deprecated and scheduled for removal in one of the nearest releases.
So, now we happily run our big test suite before each release making sure you won’t have to report any bug twice and instead can spend your time developing your application, not debugging our components.
The next step for us is to complete setting up our Continuous Integration process by running nightly builds and tests on all browsers out there to detect any regressions early. We also plan on running our test suites against the bleeding edge builds of Ext JS (which the Sencha dev team kindly agreed to provide for us). This will mean smoother upgrades to newer versions of Ext JS and we also hope this will contribute to keeping future releases of Ext JS even more stable.
The improved automation support is available in the Siesta 1.1.1 released yesterday. It works great for us – please let us know how it works for you!
Enjoy the Siesta 🙂