Run testsuite several times in a row

Post by **AndersE** » Wed Jan 04, 2017 10:07 am

Hi,

I have a testcase that is failing something like 1/10 times.
Is there a way to run a testcase or testsuite X-amount of times in siesta?

BR,
Anders

Post by **nickolay** » Wed Jan 04, 2017 11:29 am

Hello,

That test probably has a race condition - you should investigate the place where it fails. If you will post the test, I can take a quick look.

Otherwise check the "--rerun-failed" command line option (re-runs the failed tests if there were not too many)

Post by **AndersE** » Wed Jan 04, 2017 2:04 pm

Hi,

I will try the --rerun flag, I couldn't find any documentation, could you post a link where I can read more about the different flags?

Think I managed to solve the issue with a couple of waitFor and extra clicks.

But regarding my original question, is it possible to rerun a testsuite several times somehow? This would help to make sure that the TCs are stable before integrating them into CI.

Thanks!
//Anders

Post by **nickolay** » Wed Jan 04, 2017 3:02 pm

Glad it works. It is not possible to re-run in the browser UI. The command line "--rerun-failed" option does that - re-runs the failed tests if number failed tests is less than 10% of total number of tests. It re-runs only 1 additional time. See the "bin/webdriver --help" for documentation on all options.

Post by **AndersE** » Wed Jan 04, 2017 4:06 pm

It does not work when running it in the CI though.
Then it fails with

fail 7 - The step in `t.chain()` call did not complete within required timeframe, chain can not proceed
        Step number: 7 (1-based)
        At line: 204

and the script looks like:

t.chain(
				
				{ waitForCQVisible: "StartView" , desc : "Wait for StartView" }, // => :textEquals(Min arbetstid)
			
				{ click: "button[text=Start] => .x-btn-inner-mainbtn-small" , desc : "Click Start menu" },
					
				{ click: "menuitem[text=Översikt planering] => .x-menu-item-text" , desc : "Click Översikt planering"},
				
				{ click : "button[text=Rapporter] => .x-btn-wrap" , desc : "Click Rapporter" },

				{ click : "menuitem[text=Ej attesterade tider] => .x-menu-item-text" , desc : "Click Ej attesterade tider"}, 
			
				{ click : "PDEjAttestRapport[title=Ej attesterade tider] button[text=Visa] => .x-btn-inner-default-small" , desc : "Click Ej attesterade tider > Visa"},
				
				{ click : "#ok => .x-btn-inner-default-small" , desc : "Click OK, BT13762"},

				{ click : "button[text=Stäng] => .x-btn-inner-default-small" , desc : "Click Stäng" },

and so on...
I also have one additional TC that fails in the same way...

Its a very simple test just making sure all items are reachable on a website. What is the difference between running the TCs in Siesta vs CI? Can I change something to make it work?

It has been running for about 100 times without failing now on my machine, i.e. not CI.

Post by **nickolay** » Wed Jan 04, 2017 4:31 pm

Hard to say w/o a reproducible test case. One thing to consider is that in automation mode browser start fresh, w/o cookies, so you might need to login.

Post by **AndersE** » Wed Jan 04, 2017 4:45 pm

I dont know how I would be able to provide you with a TC since its so specific to our software.

The scripts do contain an automatic login function, and that is done before the t.chain I posted above.
There must be some difference in execution of the TCs/SW depending on launching from siesta or cmd...

What can I do to get more info on what's going on, is there any additional logs to look through or can they be activated somehow? Any suggestions are more then welcome!

Post by **nickolay** » Wed Jan 04, 2017 6:42 pm

Are you running your automation scripts locally on your machine or in the cloud, like with BrowserStack or SauceLabs? The race conditions tend to appear when running in the cloud, because there the resource loading time is usually much longer than locally.

There's no difference in execution, except the clean browser state. Try to clear the cookies or run the same test in incognito window?

To log from the test in automation you can use "t.diag()" method + "--verbose" switch.

Post by **AndersE** » Thu Jan 05, 2017 9:51 am

We are using teamcity, so I make the TCs, testrun them on my local machine and then check them in to teamcity.

We use the following:
Command executable: Siesta/Bin/webdriver.bat
Command parameters: https://srv-t01/Avstigning/MvTest --teamcity --no-color

Post by **nickolay** » Thu Jan 05, 2017 10:52 am

Ok, I see nothing special. Still not able to pass the test in automation? Tried running in incognito page?

Support Forum

Run testsuite several times in a row