running parallel tests in browserstack

Get help with testing, discuss unit testing strategies etc.
User avatar
jayaprakasha
Posts: 9
Joined: Thu Apr 27, 2017 1:38 pm

running parallel tests in browserstack

Post by jayaprakasha » Tue Jul 11, 2017 2:35 pm

i understand we should use --max-workers to run tests in parallel in browser stack. Normally i goes like this..

webdriver http://localhost --filter="Workflow>" --rerun-failed --browserstack=user,key --cap browserName=chrome --cap platform=windows

But how to mention the browser capabilities if i need to execute 3 instances in parallel?

User avatar
nickolay
Core Developer
Core Developer
Posts: 3357
Joined: Mon May 16, 2011 10:48 am

Re: running parallel tests in browserstack

Post by nickolay » Tue Jul 11, 2017 3:07 pm

You don't mention that in capabilities, only in "normal" command line options:

Code: Select all

webdriver http://localhost --filter="Workflow>" --max-workers 3 --rerun-failed --browserstack=user,key --cap browserName=chrome --cap platform=windows
We offer training in both Ext JS and our products, read more here.
Read the API documentation

User avatar
jayaprakasha
Posts: 9
Joined: Thu Apr 27, 2017 1:38 pm

Re: running parallel tests in browserstack

Post by jayaprakasha » Tue Jul 11, 2017 3:12 pm

But when i run the above command, in browser-stack i cannot see parallel tests running. It shows that parallel tests running 1/5.

User avatar
nickolay
Core Developer
Core Developer
Posts: 3357
Joined: Mon May 16, 2011 10:48 am

Re: running parallel tests in browserstack

Post by nickolay » Tue Jul 11, 2017 5:41 pm

This may mean your test suite is quite small. It divides the suite in separate "chunks", 20 tests per "chunk" and runs those chunks in parallel. If you have <20 tests in suite there will be only 1 chunk. The size of the chunk can be specified with the "--chunk-size" option.
We offer training in both Ext JS and our products, read more here.
Read the API documentation

User avatar
jayaprakasha
Posts: 9
Joined: Thu Apr 27, 2017 1:38 pm

Re: running parallel tests in browserstack

Post by jayaprakasha » Thu Jul 13, 2017 8:30 am

Great! i can see the tests in parallel in browser stack as i gave the "--chunk-size 5" option. But, the test cases pass only in one thread/instance and the other thread always fails .

For example,
Test cases executed = 11
maxworkers =2
chunk size = 5
threads run in browser stack= 2
i can see 5 browser stack instances have run totally and 2 in parallel in which one thread always fails. Is it because one fails another one starts to try again and it tries for 5 attempts?(See attachment-only the 5th instance is running some of the test cases, all others gets stopped)

My command prompt throws this error for the failed threads,

<Exception from launcher>
While running big script: return (function (args) {
var me = Siesta.my.activeHarness

return me ? me[ 'launchAutomatedTests' ].apply(me, args)
Exception: TypeError: Cannot set property 'automationElementId' of undefined

TypeError: Cannot set property 'automationElementId' of undefined
at Object.<anonymous> (http://elsevier.uat.ordersetsmanager.com/application/
resources/siesta/siesta-all.js:46986:45)
at Object.map (http://elsevier.uat.ordersetsmanager.co ... n/resource
s/siesta/siesta-all.js:67:28)
at Object.launchAutomatedTests (http://elsevier.uat.ordersetsmanager.com/app
lication/resources/siesta/siesta-all.js:46983:43)
at Object.override [as launchAutomatedTests] (http://elsevier.uat.ordersetsm
anager.com/application/resources/siesta/siesta-all.js:817:32)
at __EXECUTE_SCRIPT_RESULT__.automationElementId (eval at execute (http://el
sevier.uat.ordersetsmanager.com/application/resources/siesta/siesta-all.js:46363
:17), <anonymous>:4:58)
at eval (eval at execute (http://elsevier.uat.ordersetsmanager.com/applicati
on/resources/siesta/siesta-all.js:46363:17), <anonymous>:5:15)
at eval (eval at execute (http://elsevier.uat.ordersetsmanager.com/applicati
on/resources/siesta/siesta-all.js:46363:17), <anonymous>:5:234)
at eval (<anonymous>)
at Object.execute (http://elsevier.uat.ordersetsmanager.co ... ation/reso
urces/siesta/siesta-all.js:46363:17)
at Object.override [as execute] (http://elsevier.uat.ordersetsmanager.com/ap
plication/resources/siesta/siesta-all.js:817:32)
</Exception from launcher>
Attachments
0001.JPG
0001.JPG (370.72 KiB) Viewed 3033 times

User avatar
nickolay
Core Developer
Core Developer
Posts: 3357
Joined: Mon May 16, 2011 10:48 am

Re: running parallel tests in browserstack

Post by nickolay » Sat Jul 15, 2017 11:21 am

Hm.. Hard to say. Yes, it will re-start the test, if it has failed because of some selenium error. Any chance you can provide a reproducible test case?

Perhaps this is something related to the test code itself. Can you try lets say keeping only 2 test in the suite and running it using --chunk-size=1 --max-workers=2
We offer training in both Ext JS and our products, read more here.
Read the API documentation

User avatar
jayaprakasha
Posts: 9
Joined: Thu Apr 27, 2017 1:38 pm

Re: running parallel tests in browserstack

Post by jayaprakasha » Mon Jul 17, 2017 3:21 pm

This is as i said before, one test case runs in one thread successfully. The other one doesn't run as the thread gets closed, a new thread opens and again fails. Finally that test case is failed with the below mentioned exception as shown in the cmd in the previous post.

User avatar
nickolay
Core Developer
Core Developer
Posts: 3357
Joined: Mon May 16, 2011 10:48 am

Re: running parallel tests in browserstack

Post by nickolay » Mon Jul 17, 2017 3:25 pm

Sorry, its not enough information to be able to understand/reproduce the problem.. In our setup (test suites of Bryntum products) running several chunks in parallel in BrowserStack works fine.

Perhaps the tests in other chunks somehow depends on the tests in the 1st chunk. This can be for example if 1st test logs into the application, the authentication cookie is received, saved in the browser, and then the following tests do not perform the log in, relying on being launched _after_ the 1st tests.

When running several tests in parallel the order of tests execution is not defined, so every test should be standalone (check that log in is required, allocate exclusive set of data in DB if necessary etc).
We offer training in both Ext JS and our products, read more here.
Read the API documentation

User avatar
jayaprakasha
Posts: 9
Joined: Thu Apr 27, 2017 1:38 pm

Re: running parallel tests in browserstack

Post by jayaprakasha » Mon Jul 17, 2017 4:05 pm

Thanks much..That was right! My test cases depends on the same login function. So i tried to execute two test cases that do no depend on each other(no login at all), but still i am facing the same exception and the thread gets closed.
My concern is that the exception is on siesta-all.js and not on my scripts, please take a look...

While running big script: return (function (args) {
var me = Siesta.my.activeHarness

return me ? me[ 'launchAutomatedTests' ].apply(me, args)
Exception: TypeError: Cannot set property 'automationElementId' of undefined

TypeError: Cannot set property 'automationElementId' of undefined
at Object.<anonymous> (http://elsevier.remote/application/resources/siesta/s
iesta-all.js:46986:45)
at Object.map (http://elsevier.remote/application/reso ... /siesta-al
l.js:67:28)
at Object.launchAutomatedTests (http://elsevier.remote/application/resources
/siesta/siesta-all.js:46983:43)
at Object.override [as launchAutomatedTests] (http://elsevier.remote/applica
tion/resources/siesta/siesta-all.js:817:32)
at __EXECUTE_SCRIPT_RESULT__.automationElementId (eval at execute (http://el
sevier.remote/application/resources/siesta/siesta-all.js:46363:17), <anonymous>:
4:58)
at eval (eval at execute (http://elsevier.remote/application/resources/siest
a/siesta-all.js:46363:17), <anonymous>:5:15)
at eval (eval at execute (http://elsevier.remote/application/resources/siest
a/siesta-all.js:46363:17), <anonymous>:5:213)
at eval (<anonymous>)
at Object.execute (http://elsevier.remote/application/reso ... esta/siest
a-all.js:46363:17)
at Object.override [as execute] (http://elsevier.remote/application/resource
s/siesta/siesta-all.js:817:32)
</Exception from launcher>

User avatar
nickolay
Core Developer
Core Developer
Posts: 3357
Joined: Mon May 16, 2011 10:48 am

Re: running parallel tests in browserstack

Post by nickolay » Mon Jul 17, 2017 5:12 pm

Hm.. What is your Siesta version? Try with the latest version? Everything definitely just works in our setup.
We offer training in both Ext JS and our products, read more here.
Read the API documentation

Post Reply