Siesta Turbo Mode – Doubling Your UI Test Suite Execution Speed

Interested in speeding up your UI tests dramatically? Then this post is for you. In the latest Siesta release we introduced a new turboMode setting which, for us at Bryntum, has cut our UI test execution time in half. This means a lot when you have many and big test suites running often across multiple browsers. Read on to learn more about what happens behind the scenes when the new setting is used.

Mouse movement is expensive

Simulating mouse movement is an expensive process, since multiple events need to be fired for every pixel along a mouse path. Ever since the first release of Siesta, it always prioritised being exact over being fast to avoid having to write too detailed UI test instructions. The downside to being exact is that tests run slow and consume lots of CPU as every pixel is visited. Using turboMode, Siesta will only simulate mouse events for the start and end point of a path, essentially making mouse movement instant. In a big UI test which acts like a user, this will have a big impact.

Reducing internal timeouts

Using the new turboMode, Siesta reduces its own internal timeouts between each action in a chain. In non-turbo modes Siesta waits 100ms between each action by default, which may not seem like a lot. This delay was added back in the old IE days to deal with delays used internally by various UI frameworks. By having this small delay, tests could pass even if it had a “micro” race condition in it. But when you have a huge UI test suite, the small waits can add up to minutes of test execution time. Without the inter-step waits, you are responsible for applying “a brake” wherever needed to prevent test steps from running too fast. Simple example below illustrating difference between turbo and regular modes:

Test code


t.chain(
    { dblclick : 'button:textEquals(Show editor)' },
    { click : '.popup-window input[type=text]' },
    { type : '10:00 AM[ENTER]' }
);

In non-turbo mode what actually is happening;


t.chain(
    { dblclick : 'button:textEquals(Show editor)' },
    { waitFor : 100 },
    { click : '.popup-window input[type=text]' },
    { waitFor : 100 },
    { type : '10:00 AM[ENTER]' }
);

In turbo mode you should consider chain statements to execute synchronously.

If the first step triggers some form of asynchronous flow (e.g a CSS transition), it’s advisable to wait for such async flows before continuing your UI tests to guarantee stable tests. It’s also imperative that you use CSS selectors or similar (as opposed to DOM element instances) as your targets which Siesta can then wait for before executing the action.

So what does it mean in numbers?

Please see the below TeamCity screenshot, where you can clearly see the duration dropping by 10 minutes (~50%).

Any caveats?

Does it sound too good to be true? While this new mode is awesome and fast, it does require you to know your application code very well. Any async behavior needs to be fully understood and waitFor statements should be added where required to avoid race conditions. After enabling turboMode You will likely notice tests failing. This means you have race conditions in your application or test code that relies on the internal 100ms delay between Siesta actions. It’s a good idea to clean those up and make sure your test code is free of race conditions that cause test flakiness and demoralize your team.

Additionally, tests running in turboMode are hard (and not fun) to debug. This is why we have added new buttons to quickly toggle the execution speed. The recommended approach is to debug using Slow or Fast modes (which trigger mouse events for all points of a path) and switch to Turbo once test is green to assure it’s still stable also when running in super fast mode.

See the different speeds in action in this video of a monkey test with 500 random actions:

Summing up

We had to go through a few rounds of test stabilization internally and now our tests are much more robust and run super fast. We have also adopted a new coding guideline that prevents use of setTimeout/setInterval/requestAnimationFrame without associating such calls to an entity which has a life cycle (so they can be automatically cleaned up in case of destruction). This means we now have 0 free hanging setTimeout calls and thereby dramatically less chance of weird race conditions.

If test speed matters to you as it does to us we recommend you download the latest Siesta version and try the turboMode setting. Please let us know your feedback in the comments below, it matters a lot to us.

Happy testing!

For full details please see the change log.

Download Free Trial

1
Leave a Reply

avatar
1 Comment threads
0 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
1 Comment authors
v Recent comment authors
  Subscribe  
newest oldest most voted
Notify of
v
Guest
v

interesting update