Siesta 4.0 Beta Is Now Available

We’re thrilled to announce the beta release of our new major Siesta version. In this release we have focused primarily on performance and stability. To achieve significant improvements in these areas we’ve redesigned how tests are launched in automation mode. With the new launchers, we can now easily run multiple tests simultaneously. We also added a number of other cool features, so let’s take a peek at what’s in the Siesta 4.0 package.

Parallelization

The major new feature introduced in 4.0 is the support for running the test suite in several parallel “workers” (or “threads”). The number of workers can be specified using the new command line argument “–max-workers”. Pairing this feature with cloud services such as Sauce or BrowserStack provides an instant performance boost. Let’s see what impact it will have on the execution time for the Siesta internal test suite:

1 worker – 15min 40sec

10 workers – 2min 20sec

Naturally the execution time is reduced by the number of workers available. So in this case, for 10 workers we are able to run our tests 7 times faster (the speed boost will be close to 10 for bigger suites). If you remember in Siesta 3 we introduced the “disabled sandboxing” feature, which also could give you up to 10 times performance boost. So compared to Siesta 2, for the best case scenario (non-DOM tests with sandboxing disabled) you can run your tests 100 times faster! 🙂

Running your tests suite in several parallel workers has certain implications. Notably the order of test execution is no longer strictly sequential as it was before. Some test positioned lower in your `Harness.start()` list, may start execution earlier than tests above it. This is simply because you can not run your tests in parallel and in the same time sequentially. You should not rely on your tests running sequentially anymore. Always aim to have your tests be fully isolated and not depending on execution of other tests.

The new architecture for the launcher is also much more stable and fault tolerant. If for any reason the test suite encounters some error such as WebDriver throwing an exception or temporarily losing the internet connection, the launcher will restart which doesn’t affect the overall test suite execution.

Ext JS 6 support

Siesta’s internal Ext JS layer has been updated to support the latest Ext JS 6 version, so if your application is built using Ext JS 6 – we recommend using Siesta 4 to test it.

Screenshot comparison

Another cool and frequently requested feature is the ability to compare two screenshots. The comparison can be done with either a previously taken image or with a corresponding screenshot from another set of screenshots. A diff image is then generated which highlights the areas where the two screenshots are different. A certain threshold (which is configurable) of dissimilarity is allowed.

Let’s say you have a simple application containing a list that should be centered:

Now let’s pretend in your next release, some conflicting CSS rule is introduced which displaces the list a bit to the left. This UI bug would be impossible to detect using regular unit or application tests.

When running your test in automation mode with the special --screenshot-compare-with-previous argument, Siesta will perform a comparison and assert that the screenshot looks like it did before.

>>webdriver localhost/myapp --screenshot-compare-with-previous
Launching test suite, OS: MacOS, browser: Chrome 45.0.2454.85
fail 2 - Screenshots are different
New screenshot : "/Documents/foo.png"
Previous screenshot : "/Documents/foo-prev.png"
Similarity: 0.638677
[FAIL] 4.testing-3rd-party-libraries/polymer/polymer.t.js


The generated diff image highlighting the differences can be seen below:

Screenshots of single elements

In previous versions of Siesta we supported taking screenshots of the entire browser window. In v4.0, we can also capture individual DOM elements using any of our standard ActionTarget selectors such as CSS, Component Query, Ext JS Component instances etc. This feature has actually allowed us to improve our documentation a bit, since previously a few of the screenshots in the docs were outdated. The images in the Siesta 4.0 documentation are now auto-generated at each release using a special small test suite to ensure they are always up-to-date.

Meet the new launcher in the team – SlimerJS.

We are also introducing a new automation launcher in this release – SlimerJS.

One of the driving reasons for this is the poor support of the PhantomJS project, which seems no longer actively maintained (the last release with several major known issues was made Jan 23rd 2015). Another reason is that SlimerJS has much better architecture than PhantomJS – it uses the real rendering engine of the Firefox browser. So you can trust the results from this launcher as if they would be received from the “real” Firefox browser which is a quite important thing.

Recorder improvements

The event recorder is one of Siesta’s most popular features and it has received some nice improvements too. It now generates a complete test case using a tree structure with t.it statements. This helps you avoid recording a huge flat list of actions which makes the UI test hard to read and understand. Instead, select some rows in the recorder grid, then right click and logically group your UI interactions and add a descriptive name to each test section.

And after creating the test section, the generated complete test code will look like this – ready to run immediately.

describe("New recording...", function(t) {
    t.it("Logging in", function(t) {
        t.chain(
            { click : "textfield[name=name] => .x-form-text" },

            { action : "type", text : "mike[TAB]password" },

            { click : "button[text=Login] => .x-btn-button" }
        );
    });
});

When you have recorded your test, Siesta now also offers you to re-play not only the entire test but also individual actions (see icons in the screenshot above). This is a great help as you debug and polish your recorded tests.

Test files auto-discovery

Some users provided feedback that it’s bothersome to include the test file name to the “Harness.start()” list every time you create a test. To address this, we have created a simple command line tool that auto-discovers the test files in the provided directory list and generates a JSON file with test descriptors. You can then modify this file if needed and add some config options to groups or for individual tests. These changes will be preserved on the next run of the tool. In such a setup it may also be more convenient to specify configuration options inside the test itself:

StartTest({ waitForTimeout : 90000 }, function (t) {

})

Popup window support

If you work with legacy applications using popup window, you will be happy to know that Siesta now supports interacting with popups opened by the target test page. You can click and type on any elements inside the popup window after switching to it with the new `switchTo` method.

Conclusion

This release is a major evolution of Siesta and we’re determined to continue trying to make your testing life easier. As always, except for any documented API changes, we aim for 100% backward compatibility in our releases. We encourage you to give it a try and report any changes or issues compared to the previous version in our forum. Happy testing! 🙂