Get help with testing, discuss unit testing strategies etc.


Post by KaguChan »

Hello Forum,

currently we're in our trial and trying to setup a working pipeline for siesta test.

We started using web driver to setup our test pipeline (CI). It didn't work, so we tried out puppeteer. The effort to setup was insane, but in the end it seemed to work.

Then our pipelines started failing randomly (without adding tests or chaning anything).

Today i started investigating the issue - first we assumed it may just be the tests - we're still new to this and took beginner issues into consideration.

The issue is now the following:
Using puppeteer, about 43% of test fail (16 out of 37 test runs failed). Puppeteer needs at avarage 45-50 seconds per test run (when it succeeds)
Using webdriver, 3% failed (1 out of 30 tests). Webdriver needs at avarage 34 seconds per run.

I assume, partially that is just because extjs and its long loading time etc etc (using development build on a apache setup to use nyc).

Mostly puppeteer fails with the message "Inactivity timeout 180000ms has been exceeded"

As well we found out, that puppeteer reports 82 tests in our setup, webdriver reports 79. The web UI reports 82 tests as well.
We have a few complex tests which uses mouse movement, clicks etc. Once we excluded them, the test amount was the same with both setups (37 tests), so i guess there is something more to investigate with webdriver.
The result, however, didnt change - we had there as well some issues with randomly failing tests (raw "unit" tests, just creating instance of xy and test some methods which does not render something or do other fancy stuff)

We have the same issues using Windows 10
puppeteer: OS: Windows, agent: Chrome 76.0.3803.0
webdriver: OS: Windows, agent: Chrome 75.0.3770.142

and our automatic ci using linux
puppeteer: OS: Linux, agent: Chrome 76.0.3803.0


Command:
cross-env-shell siesta/bin/puppeteer https://tools/siesta-ci.html --headless --nyc.include=**/app/**/*.js --nyc.reporter=html --nyc.reporter=text-summary --report-format JUnit --report-file ../reports/junit/report-frontend.xml
(We're using the same for webdriver, just replaced puppeteer then)

We're using sencha cmd via ext-gen (node). Some errors just state "Unrecognized class name / alias: Ext.data.Store", properties beeing gone (or references) "Cannot read property 'disabled' of undefined" or the already stated timeout issue.


What can we do to get rid of this errors?
Is there any guide with a working example using webdriver in an docker environment for the CI process?

Regards
Kai

Logs (i cut out some since the thread has a character limit):
webdriver logs

>
> cross-env-shell siesta/bin/webdriver https://tools/siesta-ci.html --headless --nyc.include=**/app/**/*.js --nyc.reporter=html --nyc.reporter=text-summary --report-format JUnit --report-file ../reports/junit/report-frontend.xml

[31m[WARN] [39mUnrecognized coverage report format: text-summary
Launching test suite, OS: Windows, agent: Chrome 75.0.3770.142
[PASS]  tests/integration/authorized/mobile/mobile.spec.js
[PASS]  tests/integration/authorized/mobile/behavior.spec.js
[PASS]  tests/integration/unauthorized/login.spec.js
[PASS]  tests/integration/unauthorized/forgotPassword.spec.js
[PASS]  tests/unit/model/Brief.spec.js
[PASS]  tests/unit/view/components/base/Grid.spec.js
[PASS]  tests/unit/view/main/MainViewModel.spec.js
[PASS]  tests/unit/view/user/UserController.spec.js
79 passed, 0 failed assertions took 33.615s to complete

=============================== Coverage summary ===============================
Statements   : 22% ( 438/1991 )
Branches     : 18.54% ( 127/685 )
Functions    : 18.02% ( 82/455 )
Lines        : 22.31% ( 434/1945 )
================================================================================

>
> cross-env-shell siesta/bin/webdriver https://tools/siesta-ci.html --headless --nyc.include=**/app/**/*.js --nyc.reporter=html --nyc.reporter=text-summary --report-format JUnit --report-file ../reports/junit/report-frontend.xml

[31m[WARN] [39mUnrecognized coverage report format: text-summary
Launching test suite, OS: Windows, agent: Chrome 75.0.3770.142
[PASS]  tests/integration/authorized/mobile/mobile.spec.js
[PASS]  tests/integration/authorized/mobile/behavior.spec.js
[PASS]  tests/integration/unauthorized/login.spec.js
[PASS]  tests/integration/unauthorized/forgotPassword.spec.js
[PASS]  tests/unit/model/Brief.spec.js
[PASS]  tests/unit/view/components/base/Grid.spec.js
[PASS]  tests/unit/view/main/MainViewModel.spec.js
[PASS]  tests/unit/view/user/UserController.spec.js
79 passed, 0 failed assertions took 34.762s to complete

=============================== Coverage summary ===============================
Statements   : 22% ( 438/1991 )
Branches     : 18.54% ( 127/685 )
Functions    : 18.02% ( 82/455 )
Lines        : 22.31% ( 434/1945 )
================================================================================

>
> cross-env-shell siesta/bin/webdriver https://tools/siesta-ci.html --headless --nyc.include=**/app/**/*.js --nyc.reporter=html --nyc.reporter=text-summary --report-format JUnit --report-file ../reports/junit/report-frontend.xml

[31m[WARN] [39mUnrecognized coverage report format: text-summary
Launching test suite, OS: Windows, agent: Chrome 75.0.3770.142
[PASS]  tests/integration/authorized/mobile/mobile.spec.js
[PASS]  tests/integration/authorized/mobile/behavior.spec.js
[PASS]  tests/integration/unauthorized/login.spec.js
[PASS]  tests/integration/unauthorized/forgotPassword.spec.js
[PASS]  tests/unit/model/Brief.spec.js
[PASS]  tests/unit/view/components/base/Grid.spec.js
[PASS]  tests/unit/view/main/MainViewModel.spec.js
[PASS]  tests/unit/view/user/UserController.spec.js
79 passed, 0 failed assertions took 33.08s to complete

=============================== Coverage summary ===============================
Statements   : 22% ( 438/1991 )
Branches     : 18.54% ( 127/685 )
Functions    : 18.02% ( 82/455 )
Lines        : 22.31% ( 434/1945 )
================================================================================

>
> cross-env-shell siesta/bin/webdriver https://tools/siesta-ci.html --headless --nyc.include=**/app/**/*.js --nyc.reporter=html --nyc.reporter=text-summary --report-format JUnit --report-file ../reports/junit/report-frontend.xml

[31m[WARN] [39mUnrecognized coverage report format: text-summary
Launching test suite, OS: Windows, agent: Chrome 75.0.3770.142
[PASS]  tests/integration/authorized/mobile/mobile.spec.js
[PASS]  tests/integration/authorized/mobile/behavior.spec.js
[PASS]  tests/integration/unauthorized/login.spec.js
[PASS]  tests/integration/unauthorized/forgotPassword.spec.js
[PASS]  tests/unit/model/Brief.spec.js
[PASS]  tests/unit/view/components/base/Grid.spec.js
[PASS]  tests/unit/view/main/MainViewModel.spec.js
[PASS]  tests/unit/view/user/UserController.spec.js
79 passed, 0 failed assertions took 33.945s to complete

=============================== Coverage summary ===============================
Statements   : 22% ( 438/1991 )
Branches     : 18.54% ( 127/685 )
Functions    : 18.02% ( 82/455 )
Lines        : 22.31% ( 434/1945 )
================================================================================

>
> cross-env-shell siesta/bin/webdriver https://tools/siesta-ci.html --headless --nyc.include=**/app/**/*.js --nyc.reporter=html --nyc.reporter=text-summary --report-format JUnit --report-file ../reports/junit/report-frontend.xml

[31m[WARN] [39mUnrecognized coverage report format: text-summary
Launching test suite, OS: Windows, agent: Chrome 75.0.3770.142
[PASS]  tests/integration/authorized/mobile/mobile.spec.js
[PASS]  tests/integration/authorized/mobile/behavior.spec.js
[PASS]  tests/integration/unauthorized/login.spec.js
[PASS]  tests/integration/unauthorized/forgotPassword.spec.js
[PASS]  tests/unit/model/Brief.spec.js
[PASS]  tests/unit/view/components/base/Grid.spec.js
[PASS]  tests/unit/view/main/MainViewModel.spec.js
[PASS]  tests/unit/view/user/UserController.spec.js
79 passed, 0 failed assertions took 33.96s to complete

=============================== Coverage summary ===============================
Statements   : 22% ( 438/1991 )
Branches     : 18.54% ( 127/685 )
Functions    : 18.02% ( 82/455 )
Lines        : 22.31% ( 434/1945 )
================================================================================

>
> cross-env-shell siesta/bin/webdriver https://tools/siesta-ci.html --headless --nyc.include=**/app/**/*.js --nyc.reporter=html --nyc.reporter=text-summary --report-format JUnit --report-file ../reports/junit/report-frontend.xml

[31m[WARN] [39mUnrecognized coverage report format: text-summary
Launching test suite, OS: Windows, agent: Chrome 75.0.3770.142
[PASS]  tests/integration/authorized/mobile/mobile.spec.js
[PASS]  tests/integration/authorized/mobile/behavior.spec.js
[PASS]  tests/integration/unauthorized/login.spec.js
[PASS]  tests/integration/unauthorized/forgotPassword.spec.js
[PASS]  tests/unit/model/Brief.spec.js
[PASS]  tests/unit/view/components/base/Grid.spec.js
[PASS]  tests/unit/view/main/MainViewModel.spec.js
[PASS]  tests/unit/view/user/UserController.spec.js
79 passed, 0 failed assertions took 34.503s to complete

=============================== Coverage summary ===============================
Statements   : 22% ( 438/1991 )
Branches     : 18.54% ( 127/685 )
Functions    : 18.02% ( 82/455 )
Lines        : 22.31% ( 434/1945 )
================================================================================

>
> cross-env-shell siesta/bin/webdriver https://tools/siesta-ci.html --max-workers=10 --headless --nyc.include=**/app/**/*.js --nyc.reporter=html --nyc.reporter=text-summary --report-format JUnit --report-file ../reports/junit/report-frontend.xml

[31m[WARN] [39mUnrecognized coverage report format: text-summary
Launching test suite, OS: Windows, agent: Chrome 75.0.3770.142
[PASS]  tests/integration/authorized/mobile/mobile.spec.js
[PASS]  tests/integration/authorized/mobile/behavior.spec.js
[PASS]  tests/integration/unauthorized/login.spec.js
[PASS]  tests/integration/unauthorized/forgotPassword.spec.js
[PASS]  tests/unit/model/Brief.spec.js
[PASS]  tests/unit/view/components/base/Grid.spec.js
[PASS]  tests/unit/view/main/MainViewModel.spec.js
[PASS]  tests/unit/view/user/UserController.spec.js
79 passed, 0 failed assertions took 33.753s to complete

=============================== Coverage summary ===============================
Statements   : 22% ( 438/1991 )
Branches     : 18.54% ( 127/685 )
Functions    : 18.02% ( 82/455 )
Lines        : 22.31% ( 434/1945 )
================================================================================

>
> cross-env-shell siesta/bin/webdriver https://tools/siesta-ci.html --max-workers=10 --headless --nyc.include=**/app/**/*.js --nyc.reporter=html --nyc.reporter=text-summary --report-format JUnit --report-file ../reports/junit/report-frontend.xml

[31m[WARN] [39mUnrecognized coverage report format: text-summary
Launching test suite, OS: Windows, agent: Chrome 75.0.3770.142
[PASS]  tests/integration/authorized/mobile/mobile.spec.js
[PASS]  tests/integration/authorized/mobile/behavior.spec.js
[PASS]  tests/integration/unauthorized/login.spec.js
[PASS]  tests/integration/unauthorized/forgotPassword.spec.js
[PASS]  tests/unit/model/Brief.spec.js
[PASS]  tests/unit/view/components/base/Grid.spec.js
[PASS]  tests/unit/view/main/MainViewModel.spec.js
[PASS]  tests/unit/view/user/UserController.spec.js
79 passed, 0 failed assertions took 33.226s to complete

=============================== Coverage summary ===============================
Statements   : 22% ( 438/1991 )
Branches     : 18.54% ( 127/685 )
Functions    : 18.02% ( 82/455 )
Lines        : 22.31% ( 434/1945 )
================================================================================

>
> cross-env-shell siesta/bin/webdriver https://tools/siesta-ci.html --max-workers=10 --headless --nyc.include=**/app/**/*.js --nyc.reporter=html --nyc.reporter=text-summary --report-format JUnit --report-file ../reports/junit/report-frontend.xml

[31m[WARN] [39mUnrecognized coverage report format: text-summary
Launching test suite, OS: Windows, agent: Chrome 75.0.3770.142
[PASS]  tests/integration/authorized/mobile/mobile.spec.js
[PASS]  tests/integration/authorized/mobile/behavior.spec.js
[PASS]  tests/integration/unauthorized/login.spec.js
[PASS]  tests/integration/unauthorized/forgotPassword.spec.js
[PASS]  tests/unit/model/Brief.spec.js
[PASS]  tests/unit/view/components/base/Grid.spec.js
[PASS]  tests/unit/view/main/MainViewModel.spec.js
[PASS]  tests/unit/view/user/UserController.spec.js
79 passed, 0 failed assertions took 33.062s to complete

=============================== Coverage summary ===============================
Statements   : 22% ( 438/1991 )
Branches     : 18.54% ( 127/685 )
Functions    : 18.02% ( 82/455 )
Lines        : 22.31% ( 434/1945 )
================================================================================

>
> cross-env-shell siesta/bin/webdriver https://tools/siesta-ci.html --max-workers=10 --headless --nyc.include=**/app/**/*.js --nyc.reporter=html --nyc.reporter=text-summary --report-format JUnit --report-file ../reports/junit/report-frontend.xml

[31m[WARN] [39mUnrecognized coverage report format: text-summary
Launching test suite, OS: Windows, agent: Chrome 75.0.3770.142
[PASS]  tests/integration/authorized/mobile/mobile.spec.js
[PASS]  tests/integration/authorized/mobile/behavior.spec.js
[PASS]  tests/integration/unauthorized/login.spec.js
[PASS]  tests/integration/unauthorized/forgotPassword.spec.js
[PASS]  tests/unit/model/Brief.spec.js
should automatically configure multiColumnSort
    fail 1 - Subtest `should automatically configure multiColumnSort` threw an exception
    Error: [Ext.create] Unrecognized class name / alias: Ext.data.Store
        at line 1634, character 23, of https://tools/node_modules/@sencha/ext-core/src/class/ClassManager.js?_dc=1564997545040
        at line 8, character 29, of https://tools/tests/unit/view/components/base/Grid.spec.js
        at line 369, character 413, of https://tools/siesta/siesta-all.js
        at line 1, character 347, of <anonymous>
        at line 369, character 344, of https://tools/siesta/siesta-all.js
        at line 65, character 357, of https://tools/siesta/siesta-all.js
        at line 63, character 414, of https://tools/siesta/siesta-all.js
        at line 52, character 45, of https://tools/siesta/siesta-all.js
        at line 917, character 325, of https://tools/siesta/siesta-all.js
        at line 65, character 357, of https://tools/siesta/siesta-all.js
should automatically configure the `resetState` plugin
    fail 1 - Chain step threw an exception
    Error: [Ext.create] Unrecognized class name / alias: Tools.view.components.base.Grid
        at line 1634, character 23, of https://tools/node_modules/@sencha/ext-core/src/class/ClassManager.js?_dc=1564997545040
        at line 85, character 26, of https://tools/tests/unit/view/components/base/Grid.spec.js
        at line 307, character 368, of https://tools/siesta/siesta-all.js
        at line 1, character 347, of <anonymous>
        at line 307, character 352, of https://tools/siesta/siesta-all.js
        at line 218, character 53, of https://tools/siesta/siesta-all.js
        at line 65, character 357, of https://tools/siesta/siesta-all.js
        at line 217, character 35, of https://tools/siesta/siesta-all.js
        at line 65, character 357, of https://tools/siesta/siesta-all.js
        at line 216, character 132, of https://tools/siesta/siesta-all.js
containsPlugin
    fail 1 - Subtest `containsPlugin` threw an exception
    Error: [Ext.create] Unrecognized class name / alias: Tools.view.components.base.Grid
        at line 1634, character 23, of https://tools/node_modules/@sencha/ext-core/src/class/ClassManager.js?_dc=1564997545040
        at line 135, character 22, of https://tools/tests/unit/view/components/base/Grid.spec.js
        at line 369, character 413, of https://tools/siesta/siesta-all.js
        at line 1, character 347, of <anonymous>
        at line 369, character 344, of https://tools/siesta/siesta-all.js
        at line 65, character 357, of https://tools/siesta/siesta-all.js
        at line 63, character 414, of https://tools/siesta/siesta-all.js
        at line 52, character 45, of https://tools/siesta/siesta-all.js
        at line 917, character 325, of https://tools/siesta/siesta-all.js
        at line 65, character 357, of https://tools/siesta/siesta-all.js
[FAIL]  tests/unit/view/components/base/Grid.spec.js

[PASS]  tests/unit/view/main/MainViewModel.spec.js
[PASS]  tests/unit/view/user/UserController.spec.js
69 passed, 3 failed assertions took 32.293s to complete

=============================== Coverage summary ===============================
Statements   : 21.85% ( 435/1991 )
Branches     : 17.81% ( 122/685 )
Functions    : 18.02% ( 82/455 )
Lines        : 22.16% ( 431/1945 )
================================================================================
puppeteer logs

>
> cross-env-shell siesta/bin/puppeteer https://tools/siesta-ci.html --headless --nyc.include=**/app/**/*.js --nyc.reporter=html --nyc.reporter=text-summary --report-format JUnit --report-file ../reports/junit/report-frontend.xml

[31m[WARN] [39mUnrecognized coverage report format: text-summary
Launching test suite, OS: Windows, agent: Chrome 76.0.3803.0
[[32mPASS[39m]  tests/integration/authorized/mobile/mobile.spec.js
[[32mPASS[39m]  tests/unit/view/main/MainViewModel.spec.js
[[32mPASS[39m]  tests/integration/unauthorized/forgotPassword.spec.js
[[32mPASS[39m]  tests/unit/view/user/UserController.spec.js
[[32mPASS[39m]  tests/integration/authorized/mobile/behavior.spec.js
[[32mPASS[39m]  tests/unit/model/Brief.spec.js
[[32mPASS[39m]  tests/unit/view/components/base/Grid.spec.js
[1m[31mfail 1 - Chain step threw an exception
TypeError: Cannot read property 'disabled' of undefined
    at line 28, character 24, of https://tools/tests/integration/unauthorized/login.spec.js
    at line 307, character 368, of https://tools/siesta/siesta-all.js
    at line 1, character 347, of <anonymous>
    at line 307, character 352, of https://tools/siesta/siesta-all.js
    at line 218, character 53, of https://tools/siesta/siesta-all.js
    at line 65, character 357, of https://tools/siesta/siesta-all.js
    at line 217, character 35, of https://tools/siesta/siesta-all.js
    at line 65, character 357, of https://tools/siesta/siesta-all.js
    at line 218, character 30, of https://tools/siesta/siesta-all.js
    at line 307, character 3, of https://tools/siesta/siesta-all.js[39m[22m
[[31mFAIL[39m]  tests/integration/unauthorized/login.spec.js

67 passed, 1 failed assertions took 48.777s to complete

=============================== Coverage summary ===============================
Statements   : 20.84% ( 415/1991 )
Branches     : 17.37% ( 119/685 )
Functions    : 16.48% ( 75/455 )
Lines        : 21.13% ( 411/1945 )
================================================================================

>
> cross-env-shell siesta/bin/puppeteer https://tools/siesta-ci.html --headless --nyc.include=**/app/**/*.js --nyc.reporter=html --nyc.reporter=text-summary --report-format JUnit --report-file ../reports/junit/report-frontend.xml

[31m[WARN] [39mUnrecognized coverage report format: text-summary
Launching test suite, OS: Windows, agent: Chrome 76.0.3803.0
[[32mPASS[39m]  tests/integration/authorized/mobile/mobile.spec.js
[[32mPASS[39m]  tests/integration/unauthorized/forgotPassword.spec.js
[[32mPASS[39m]  tests/integration/authorized/mobile/behavior.spec.js
[[32mPASS[39m]  tests/unit/model/Brief.spec.js
[[32mPASS[39m]  tests/unit/view/components/base/Grid.spec.js
[[32mPASS[39m]  tests/integration/unauthorized/login.spec.js
[31m[WARN] [39mInactivity timeout 180000ms has been exceeded
[[32mPASS[39m]  tests/unit/view/main/MainViewModel.spec.js
[[32mPASS[39m]  tests/unit/view/user/UserController.spec.js
82 passed, 0 failed assertions took 3m 18s to complete

=============================== Coverage summary ===============================
Statements   : 22% ( 438/1991 )
Branches     : 18.54% ( 127/685 )
Functions    : 18.02% ( 82/455 )
Lines        : 22.31% ( 434/1945 )
================================================================================

>
> cross-env-shell siesta/bin/puppeteer https://tools/siesta-ci.html --headless --nyc.include=**/app/**/*.js --nyc.reporter=html --nyc.reporter=text-summary --report-format JUnit --report-file ../reports/junit/report-frontend.xml

[31m[WARN] [39mUnrecognized coverage report format: text-summary
Launching test suite, OS: Windows, agent: Chrome 76.0.3803.0
[[32mPASS[39m]  tests/integration/authorized/mobile/mobile.spec.js
[[32mPASS[39m]  tests/unit/view/main/MainViewModel.spec.js
[1misLoggedIn[22m
    [1m[31mfail 1 - Subtest `isLoggedIn` threw an exception
    ReferenceError: User is not defined
        at line 9, character 10, of https://tools/tests/unit/view/user/UserController.spec.js
        at line 369, character 413, of https://tools/siesta/siesta-all.js
        at line 1, character 347, of <anonymous>
        at line 369, character 344, of https://tools/siesta/siesta-all.js
        at line 65, character 357, of https://tools/siesta/siesta-all.js
        at line 63, character 414, of https://tools/siesta/siesta-all.js
        at line 52, character 45, of https://tools/siesta/siesta-all.js
        at line 917, character 325, of https://tools/siesta/siesta-all.js
        at line 65, character 357, of https://tools/siesta/siesta-all.js
        at line 914, character 366, of https://tools/siesta/siesta-all.js[39m[22m
[1mlogout[22m
    [1m[31mfail 1 - Subtest `logout` threw an exception
    ReferenceError: User is not defined
        at line 27, character 5, of https://tools/tests/unit/view/user/UserController.spec.js
        at line 369, character 413, of https://tools/siesta/siesta-all.js
        at line 1, character 347, of <anonymous>
        at line 369, character 344, of https://tools/siesta/siesta-all.js
        at line 65, character 357, of https://tools/siesta/siesta-all.js
        at line 63, character 414, of https://tools/siesta/siesta-all.js
        at line 52, character 45, of https://tools/siesta/siesta-all.js
        at line 917, character 325, of https://tools/siesta/siesta-all.js
        at line 65, character 357, of https://tools/siesta/siesta-all.js
        at line 914, character 366, of https://tools/siesta/siesta-all.js[39m[22m
[1misPasswordExpired[22m
    [1m[31mfail 1 - Subtest `isPasswordExpired` threw an exception
    ReferenceError: User is not defined
        at line 42, character 22, of https://tools/tests/unit/view/user/UserController.spec.js
        at line 369, character 413, of https://tools/siesta/siesta-all.js
        at line 1, character 347, of <anonymous>
        at line 369, character 344, of https://tools/siesta/siesta-all.js
        at line 65, character 357, of https://tools/siesta/siesta-all.js
        at line 63, character 414, of https://tools/siesta/siesta-all.js
        at line 52, character 45, of https://tools/siesta/siesta-all.js
        at line 917, character 325, of https://tools/siesta/siesta-all.js
        at line 65, character 357, of https://tools/siesta/siesta-all.js
        at line 914, character 366, of https://tools/siesta/siesta-all.js[39m[22m
[1misPasswordExpiredByDate[22m
    [1m[31mfail 1 - Subtest `isPasswordExpiredByDate` threw an exception
    ReferenceError: User is not defined
        at line 60, character 22, of https://tools/tests/unit/view/user/UserController.spec.js
        at line 369, character 413, of https://tools/siesta/siesta-all.js
        at line 1, character 347, of <anonymous>
        at line 369, character 344, of https://tools/siesta/siesta-all.js
        at line 65, character 357, of https://tools/siesta/siesta-all.js
        at line 63, character 414, of https://tools/siesta/siesta-all.js
        at line 52, character 45, of https://tools/siesta/siesta-all.js
        at line 917, character 325, of https://tools/siesta/siesta-all.js
        at line 65, character 357, of https://tools/siesta/siesta-all.js
        at line 914, character 366, of https://tools/siesta/siesta-all.js[39m[22m
[1mgetUser[22m
    [1m[31mfail 1 - Subtest `getUser` threw an exception
    ReferenceError: User is not defined
        at line 84, character 20, of https://tools/tests/unit/view/user/UserController.spec.js
        at line 369, character 413, of https://tools/siesta/siesta-all.js
        at line 1, character 347, of <anonymous>
        at line 369, character 344, of https://tools/siesta/siesta-all.js
        at line 65, character 357, of https://tools/siesta/siesta-all.js
        at line 63, character 414, of https://tools/siesta/siesta-all.js
        at line 52, character 45, of https://tools/siesta/siesta-all.js
        at line 917, character 325, of https://tools/siesta/siesta-all.js
        at line 65, character 357, of https://tools/siesta/siesta-all.js
        at line 914, character 366, of https://tools/siesta/siesta-all.js[39m[22m
[1mrequest[22m
    [1m[31mfail 1 - Subtest `request` threw an exception
    ReferenceError: User is not defined
        at line 100, character 5, of https://tools/tests/unit/view/user/UserController.spec.js
        at line 369, character 413, of https://tools/siesta/siesta-all.js
        at line 1, character 347, of <anonymous>
        at line 369, character 344, of https://tools/siesta/siesta-all.js
        at line 65, character 357, of https://tools/siesta/siesta-all.js
        at line 63, character 414, of https://tools/siesta/siesta-all.js
        at line 52, character 45, of https://tools/siesta/siesta-all.js
        at line 917, character 325, of https://tools/siesta/siesta-all.js
        at line 65, character 357, of https://tools/siesta/siesta-all.js
        at line 914, character 366, of https://tools/siesta/siesta-all.js[39m[22m
[[31mFAIL[39m]  tests/unit/view/user/UserController.spec.js

[[32mPASS[39m]  tests/integration/unauthorized/forgotPassword.spec.js
[[32mPASS[39m]  tests/integration/authorized/mobile/behavior.spec.js
[[32mPASS[39m]  tests/unit/model/Brief.spec.js
[[32mPASS[39m]  tests/unit/view/components/base/Grid.spec.js
[[32mPASS[39m]  tests/integration/unauthorized/login.spec.js
68 passed, 6 failed assertions took 45.377s to complete

=============================== Coverage summary ===============================
Statements   : 21.5% ( 428/1991 )
Branches     : 18.54% ( 127/685 )
Functions    : 17.14% ( 78/455 )
Lines        : 21.8% ( 424/1945 )
================================================================================

>
> cross-env-shell siesta/bin/puppeteer https://tools/siesta-ci.html --headless --nyc.include=**/app/**/*.js --nyc.reporter=html --nyc.reporter=text-summary --report-format JUnit --report-file ../reports/junit/report-frontend.xml

[31m[WARN] [39mUnrecognized coverage report format: text-summary
Launching test suite, OS: Windows, agent: Chrome 76.0.3803.0
[[32mPASS[39m]  tests/integration/authorized/mobile/mobile.spec.js
[[32mPASS[39m]  tests/unit/view/main/MainViewModel.spec.js
[[32mPASS[39m]  tests/integration/unauthorized/forgotPassword.spec.js
[[32mPASS[39m]  tests/unit/view/user/UserController.spec.js
[[32mPASS[39m]  tests/integration/authorized/mobile/behavior.spec.js
[[32mPASS[39m]  tests/unit/model/Brief.spec.js
[[32mPASS[39m]  tests/unit/view/components/base/Grid.spec.js
[[32mPASS[39m]  tests/integration/unauthorized/login.spec.js
82 passed, 0 failed assertions took 50.984s to complete

=============================== Coverage summary ===============================
Statements   : 22% ( 438/1991 )
Branches     : 18.54% ( 127/685 )
Functions    : 18.02% ( 82/455 )
Lines        : 22.31% ( 434/1945 )
================================================================================

>
> cross-env-shell siesta/bin/puppeteer https://tools/siesta-ci.html --headless --nyc.include=**/app/**/*.js --nyc.reporter=html --nyc.reporter=text-summary --report-format JUnit --report-file ../reports/junit/report-frontend.xml

[31m[WARN] [39mUnrecognized coverage report format: text-summary
Launching test suite, OS: Windows, agent: Chrome 76.0.3803.0
[[32mPASS[39m]  tests/integration/authorized/mobile/mobile.spec.js
[[32mPASS[39m]  tests/unit/view/main/MainViewModel.spec.js
[[32mPASS[39m]  tests/integration/unauthorized/forgotPassword.spec.js
[[32mPASS[39m]  tests/unit/view/user/UserController.spec.js
[[32mPASS[39m]  tests/integration/authorized/mobile/behavior.spec.js
[[32mPASS[39m]  tests/unit/model/Brief.spec.js
[[32mPASS[39m]  tests/unit/view/components/base/Grid.spec.js
[[32mPASS[39m]  tests/integration/unauthorized/login.spec.js
82 passed, 0 failed assertions took 51.123s to complete

=============================== Coverage summary ===============================
Statements   : 22% ( 438/1991 )
Branches     : 18.54% ( 127/685 )
Functions    : 18.02% ( 82/455 )
Lines        : 22.31% ( 434/1945 )
================================================================================

>
> cross-env-shell siesta/bin/puppeteer https://tools/siesta-ci.html --headless --nyc.include=**/app/**/*.js --nyc.reporter=html --nyc.reporter=text-summary --report-format JUnit --report-file ../reports/junit/report-frontend.xml

[31m[WARN] [39mUnrecognized coverage report format: text-summary
Launching test suite, OS: Windows, agent: Chrome 76.0.3803.0
[[32mPASS[39m]  tests/integration/authorized/mobile/mobile.spec.js
[[32mPASS[39m]  tests/integration/authorized/mobile/behavior.spec.js
[[32mPASS[39m]  tests/integration/unauthorized/login.spec.js
[31m[WARN] [39mInactivity timeout 180000ms has been exceeded
[31m[WARN] [39mInactivity timeout 180000ms has been exceeded
[[32mPASS[39m]  tests/unit/view/main/MainViewModel.spec.js
[[32mPASS[39m]  tests/integration/unauthorized/forgotPassword.spec.js
[[32mPASS[39m]  tests/unit/view/user/UserController.spec.js
[[32mPASS[39m]  tests/unit/model/Brief.spec.js
[[32mPASS[39m]  tests/unit/view/components/base/Grid.spec.js
82 passed, 0 failed assertions took 3m 34s to complete

=============================== Coverage summary ===============================
Statements   : 22% ( 438/1991 )
Branches     : 18.54% ( 127/685 )
Functions    : 18.02% ( 82/455 )
Lines        : 22.31% ( 434/1945 )
================================================================================

>
> cross-env-shell siesta/bin/puppeteer https://tools/siesta-ci.html --headless --nyc.include=**/app/**/*.js --nyc.reporter=html --nyc.reporter=text-summary --report-format JUnit --report-file ../reports/junit/report-frontend.xml

[31m[WARN] [39mUnrecognized coverage report format: text-summary
Launching test suite, OS: Windows, agent: Chrome 76.0.3803.0
[[32mPASS[39m]  tests/unit/view/main/MainViewModel.spec.js
[[32mPASS[39m]  tests/integration/authorized/mobile/mobile.spec.js
[[32mPASS[39m]  tests/unit/view/user/UserController.spec.js
[[32mPASS[39m]  tests/integration/authorized/mobile/behavior.spec.js
[[32mPASS[39m]  tests/integration/unauthorized/login.spec.js
[31m[WARN] [39mInactivity timeout 180000ms has been exceeded
[[32mPASS[39m]  tests/integration/unauthorized/forgotPassword.spec.js
[[32mPASS[39m]  tests/unit/model/Brief.spec.js
[[32mPASS[39m]  tests/unit/view/components/base/Grid.spec.js
82 passed, 0 failed assertions took 3m 27s to complete

=============================== Coverage summary ===============================
Statements   : 22% ( 438/1991 )
Branches     : 18.54% ( 127/685 )
Functions    : 18.02% ( 82/455 )
Lines        : 22.31% ( 434/1945 )
================================================================================

>
> cross-env-shell siesta/bin/puppeteer https://tools/siesta-ci.html --headless --nyc.include=**/app/**/*.js --nyc.reporter=html --nyc.reporter=text-summary --report-format JUnit --report-file ../reports/junit/report-frontend.xml

[31m[WARN] [39mUnrecognized coverage report format: text-summary
Launching test suite, OS: Windows, agent: Chrome 76.0.3803.0
[[32mPASS[39m]  tests/integration/authorized/mobile/mobile.spec.js
[[32mPASS[39m]  tests/unit/view/main/MainViewModel.spec.js
[[32mPASS[39m]  tests/integration/unauthorized/forgotPassword.spec.js
[[32mPASS[39m]  tests/unit/view/user/UserController.spec.js
[[32mPASS[39m]  tests/integration/authorized/mobile/behavior.spec.js
[[32mPASS[39m]  tests/unit/model/Brief.spec.js
[[32mPASS[39m]  tests/unit/view/components/base/Grid.spec.js
[[32mPASS[39m]  tests/integration/unauthorized/login.spec.js
82 passed, 0 failed assertions took 51.523s to complete

=============================== Coverage summary ===============================
Statements   : 22% ( 438/1991 )
Branches     : 18.54% ( 127/685 )
Functions    : 18.02% ( 82/455 )
Lines        : 22.31% ( 434/1945 )
================================================================================

Post by KaguChan »

After more testing i found out that:

- webdriver fails as well using Chrome 76
- puppeteer with version 77 (beta?) fails as well
- puppeteer with chrome version 75 runs through (as the local webdriver tests using the same version)

This can be accomplished by browsing to puppeteer in node_modules (inside siesta), opening package json and setting the "chrome_revision" to "650583"

This is a (very) dirty fix, but it seems to work.
Tomorrow ill try to include it in the pipeline and report here if it succeed.

Post by mats »

Very hard to say but most likely the issue is in the test, or due to an unstable version of Puppeteer. We recommend you to update puppeteer version to latest stable, edit package.json accordingly, then run "npm i" in the Siesta folder. Does this resolve it?

If not, can you please provide us with a test case we can execute?

Post by KaguChan »

Hello,

I just tried modifying the puppeteer version / chrome version downloaded.
I noticed the amount of errors going down using the 75 version (down to like 20%), 77 beta tried as well.
My computer just updated to Chrome 76 (as in puppeteer) two days ago. With webdriver still no errors.

On the CI server I did not yet try changing the version, but noticed that (using version 75 and 76) webdriver had 100% failed tests, maybe having a wrong setup - I don't know. Debugging pipelines is hard as hell and does not contribute to a good day ;)

The last two days we tried out different frameworks to find a reliable test setup, but mostly they lacked usability or other stuff (mostly code coverage) - props for siesta here.

On code base we tried different things today.

In some of our tests, we use the "setTimeout" function to wait for specific events.
Looking at the following test:
describe((s) => {
  s.it('formulas => versionNumber', (t) => {
    const viewModel = new ITVM.view.main.MainViewModel();

    t.chain(
      (next) => {
        viewModel.set('appVersion', 1);
        next();
      },
      { waitFor: 10 },
      (next) => {
        t.is(
          viewModel.get('versionNumber'),
          'v1',
          'versionNumber resolves to v1 when provided with simple number',
        );
        next();
      },
      (next) => {
        viewModel.set('appVersion', 'v2');
        next();
      },
      { waitFor: 10 },
      (next) => {
        t.is(
          viewModel.get('versionNumber'),
          'v2',
          'versionNumber resolves to v2 when provided with malformed string containing `v`',
        );
        next();
      },
      (next) => {
        viewModel.set('appVersion', 'vvvv3');
        next();
      },
      { waitFor: 10 },
      (next) => {
        t.is(
          viewModel.get('versionNumber'),
          'v3',
          'versionNumber resolves to v3 when provided with malformed string containing multiple `v`',
        );
        next();
      },
    );
  });
});
We have different persons doing the deployments via a configuration script. Code will always contain mistakes and humans do them as well. So we had a version number displayed like "vv0.1.9" - we switched to a formular to avoid this kind of mistake.

In this test, we use the "waitFor" - despite having 3 assertions, the output tells us having 6 tests - caused by the "waitFor".
To avoid the "wrong" number of tests, we switched in our integration tests - where we do more than just three assertions - to "setTimeout" (even if "t.diag" for example increases the test amount as well - is this intended anyways? If so, it would be cool to know why)

(The timeout in the provided test btw is to give the view model time to recalculate the formular, since it won't do it directly. I remember there being a function to let it happen directly, but we only need it in tests, so we just use timeouts.)

During our tests, we found out that using the "waitFor" or "waitForMilliseconds" methods improve the stability of the tests. I think it's worth investigating at this point, since it's weird seeing puppeteer losing track of the actual process or anything resolving in the timeouts over three minutes.

Another test, completely without timeouts, would be the following:
describe((s) => {
  s.afterEach(() => {
    hooks.sinon.restore();
  });

  s.it('isLoggedIn', (t) => {
    hooks.sinon.stub(localStorage, 'getItem').returns('x');

    t.is(User.isLoggedIn(), true, 'User is identified as logged in when token is set');

    localStorage.getItem.restore();
    hooks.sinon.stub(localStorage, 'getItem').returns(null);

    t.is(User.isLoggedIn(), false, 'User is identified as logged out when no token is set');
  });

  s.it('login', (t) => {
    t.is(1, 1, '1');
  });

  s.it('logout', (t) => {
    hooks
      .disableReload()
      .sinon.stub(localStorage, 'removeItem')
      .returns(true);

    User.logout();

    t.is(localStorage.removeItem.calledWith('token'), true, 'User token was reset');
    t.is(window.reload.called, true, 'Page reloaded after logout');
  });
Here we have a "hook" object, which just contains helper methods to stub some code via sinon.js - We do a lot of this stuff to avoid any application logic to be executed which for example would actually cause requests or something (or just to spy and check if some things are done correctly)


The last thing I tried was running only a few tests and log the results.
For now, we have 4 unit test files and 4 files doing actual rendering.

We executed each of them alone, then some in combination and so on.
The last two hours I started a total of 250 tests using puppeteer via cli.

Running tests by themselves caused absolutely no problems.
Running all tests together or groups of tests let puppeteer fail several times.
> cross-env-shell siesta/bin/puppeteer https://tools/siesta-ci.html --headless --nyc.include=**/app/**/*.js --nyc.reporter=html --nyc.reporter=text-summary --report-format JUnit --report-file ../reports/junit/report-frontend.xml

[31m[WARN] [39mUnrecognized coverage report format: text-summary
Launching test suite, OS: Windows, agent: Chrome 76.0.3803.0
[[32mPASS[39m]  tests/integration/authorized/mobile/mobile.spec.js
[[32mPASS[39m]  tests/unit/view/main/MainViewModel.spec.js
[[32mPASS[39m]  tests/integration/unauthorized/forgotPassword.spec.js
[[32mPASS[39m]  tests/unit/view/user/UserController.spec.js
[[32mPASS[39m]  tests/integration/authorized/mobile/behavior.spec.js
[[32mPASS[39m]  tests/unit/model/ams/Brief.spec.js
[[32mPASS[39m]  tests/unit/view/components/base/Grid.spec.js
[[32mPASS[39m]  tests/integration/unauthorized/login.spec.js
75 passed, 0 failed assertions took 51.31s to complete

=============================== Coverage summary ===============================
Statements   : 21.53% ( 425/1974 )
Branches     : 17.19% ( 116/675 )
Functions    : 17.45% ( 78/447 )
Lines        : 21.88% ( 422/1929 )
================================================================================
> cross-env-shell siesta/bin/puppeteer https://tools/siesta-ci.html --headless --nyc.include=**/app/**/*.js --nyc.reporter=html --nyc.reporter=text-summary --report-format JUnit --report-file ../reports/junit/report-frontend.xml

[31m[WARN] [39mUnrecognized coverage report format: text-summary
Launching test suite, OS: Windows, agent: Chrome 76.0.3803.0
[[32mPASS[39m]  tests/integration/authorized/mobile/mobile.spec.js
[[32mPASS[39m]  tests/unit/view/main/MainViewModel.spec.js
[[32mPASS[39m]  tests/integration/unauthorized/forgotPassword.spec.js
[[32mPASS[39m]  tests/unit/view/user/UserController.spec.js
[[32mPASS[39m]  tests/integration/authorized/mobile/behavior.spec.js
[[32mPASS[39m]  tests/unit/model/ams/Brief.spec.js
[1m[31mfail 1 - Chain step threw an exception
TypeError: Cannot read property 'disabled' of undefined
    at line 28, character 24, of https://tools/tests/integration/unauthorized/login.spec.js
    at line 307, character 368, of https://tools/siesta/siesta-all.js
    at line 1, character 347, of <anonymous>
    at line 307, character 352, of https://tools/siesta/siesta-all.js
    at line 218, character 53, of https://tools/siesta/siesta-all.js
    at line 65, character 357, of https://tools/siesta/siesta-all.js
    at line 217, character 35, of https://tools/siesta/siesta-all.js
    at line 65, character 357, of https://tools/siesta/siesta-all.js
    at line 218, character 30, of https://tools/siesta/siesta-all.js
    at line 307, character 3, of https://tools/siesta/siesta-all.js[39m[22m
[[31mFAIL[39m]  tests/integration/unauthorized/login.spec.js

[[32mPASS[39m]  tests/unit/view/components/base/Grid.spec.js
60 passed, 1 failed assertions took 43.5s to complete

=============================== Coverage summary ===============================
Statements   : 20.36% ( 402/1974 )
Branches     : 16% ( 108/675 )
Functions    : 15.88% ( 71/447 )
Lines        : 20.68% ( 399/1929 )
================================================================================
> cross-env-shell siesta/bin/puppeteer https://tools/siesta-ci.html --headless --nyc.include=**/app/**/*.js --nyc.reporter=html --nyc.reporter=text-summary --report-format JUnit --report-file ../reports/junit/report-frontend.xml

[31m[WARN] [39mUnrecognized coverage report format: text-summary
Launching test suite, OS: Windows, agent: Chrome 76.0.3803.0
[[32mPASS[39m]  tests/integration/authorized/mobile/mobile.spec.js
[[32mPASS[39m]  tests/unit/view/main/MainViewModel.spec.js
[31mfail 2 - Token should invisible at start
Failed assertion `is` at line 33 of tests/integration/unauthorized/forgotPassword.spec.js
Got  : true
Need : false[39m
[[31mFAIL[39m]  tests/integration/unauthorized/forgotPassword.spec.js

[[32mPASS[39m]  tests/integration/authorized/mobile/behavior.spec.js
[[32mPASS[39m]  tests/unit/view/user/UserController.spec.js
[[32mPASS[39m]  tests/unit/model/ams/Brief.spec.js
Here one more example of a test - this time a test doing actual stuff on the browser - this test often causes 3 minute timeouts...
One "classic" case is receiving null from the t.cq method (maybe even more timeout stuff in beforehand to let extjs have again more time?)
describe('LogIn', (t) => {
  let loginButton;
  let userName;
  let password;
  let wnd;

  t.chain(
    (next) => {
      hooks
        .loggedOut()
        .disableReload()
        .spy(localStorage, 'setItem')
        .spy(User, 'login')
        .app(next);
    },
    (next) => {
      [loginButton] = t.cq('button[text=Anmelden]');
      [userName] = t.cq('[name=username]');
      [password] = t.cq('[name=password]');

      next();
    },
    (next) => {
      t.diag('Check prerequisites');
      next();
    },
    (next) => {
      t.is(loginButton.disabled, true, 'login button is disabled at start');
      next();
    },
    (next) => {
      const [view] = t.cq('userLogin');
      const form = view.getForm();

      hooks.spy(form, 'isValid');
      hooks.spy(view, 'setLoading');

      t.click(userName, () => {
        t.type(userName, '[ENTER]', () => {
          t.is(form.isValid.called, true, 'Form is validated on enter');
          t.is(view.setLoading.called, false, 'View is not set to loading on invalid enter');

          next();
        });
      });
    },
    (next) => {
      userName.setValue('foo');
      password.setValue('bar');

      setTimeout(() => {
        t.is(loginButton.disabled, false, 'login button is enabled when form is filled');
        next();
      }, 10);
    },
    (next) => {
      t.diag('Check login click / failed login');
      next();
    },
  items: [
    {
      name: 'username',
      fieldLabel: 'Nutzername',
    },
    {
      inputType: 'password',
      name: 'password',
      fieldLabel: 'Passwort',
    },
  ],

  bbar: [
    {
      xtype: 'panel',
      html: '<a href="#user/reset">Passwort vergessen</a>',
      width: 120,
    },
    '->',
    {
      text: 'Anmelden',
      formBind: true,
      listeners: {
        click: 'onLoginClick',
      },
    },
  ],
The setup:
/* global Siesta */
const project = new Siesta.Project.Browser.ExtJS();

project.configure({
  title: 'ITVM Test Suite',
  preload: ['tests/sinon.js', 'tests/functions.js'],
});

project.plan(
  {
    group: '[Integration][Authorized][Mobile]',
    waitForAppReady: true,
    pageUrl: 'index.html?testing&phone',
    preload: 'inherit',
    items: [
      {
        title: 'Mobile Patches',
        url: 'tests/integration/authorized/mobile/mobile.spec.js',
      },
      {
        title: 'Mobile Behavior',
        url: 'tests/integration/authorized/mobile/behavior.spec.js',
      },
    ],
  },
  {
    group: '[Integration][Unauthorized]',
    waitForAppReady: true,
    pageUrl: 'index.html?testing',
    preload: 'inherit',
    items: [
      {
        title: 'Login',
        url: 'tests/integration/unauthorized/login.spec.js',
      },
      {
        title: 'Forgot Password',
        url: 'tests/integration/unauthorized/forgotPassword.spec.js',
      },
    ],
  },
  {
    group: '[UnitTests]',
    pageUrl: 'index.html?testing',
    preload: 'inherit',
    items: [
      {
        title: 'Brief Model',
        url: 'tests/unit/model/ams/Brief.spec.js',
      },
      {
        title: 'Base Grid',
        url: 'tests/unit/view/components/base/Grid.spec.js',
      },
      {
        title: 'Main ViewModel',
        url: 'tests/unit/view/main/MainViewModel.spec.js',
      },
      {
        title: 'UserController',
        url: 'tests/unit/view/user/UserController.spec.js',
      },
    ],
  },
);

project.start();

Right now i just ran "npm i --save puppeteer@1.19.0" in siesta folder and let my computer do 100 test runs with logs during the next few hours, will check the result tomorrow.

Post by Maxim Gorkovsky »

Hello.
Puppeteer/chrome instability aside, these tests could be improved. I recall when we started to run our tests on 3rd party platforms we suffered from tests instability, tests were failing randomly. It turned out that most common source of problem was that we expected components to be in a certain state, but they weren't. This is a reason why we have a rule: wait for condition, not the time. This is especially important when tested process is asynchronous: involves setTimeout, requestAnimationFrame (e.g. Ext.asap), animation etc

I see some of your tests could benefit from that rule too. For instance:
describe((s) => {
    s.it('formulas => versionNumber', (t) => {
        const viewModel = new ITVM.view.main.MainViewModel();

        t.chain(
            (next) => {
                viewModel.set('appVersion', 1);
                next();
            },
            {
                waitFor: () => viewModel.get('versionNumber') == 'v1',
                desc: 'versionNumber resolves to v1 when provided with simple number'
            },
            (next) => {
                viewModel.set('appVersion', 'v2');
                next();
            },
            { 
                waitFor: () => viewModel.get('versionNumber') == 'v2',
                desc: 'versionNumber resolves to v2 when provided with malformed string containing `v`'
            },
            (next) => {
                viewModel.set('appVersion', 'vvvv3');
                next();
            },
            {
                waitFor: () => viewModel.get('versionNumber') == 'v3',
                desc: 'versionNumber resolves to v3 when provided with malformed string containing multiple `v`'
            }
        );
    });
});
And this is how another test could look:
describe('LogIn', (t) => {
    let loginButton;
    let userName;
    let password;
    let wnd;

    t.chain(
        (next) => {
            hooks
                .loggedOut()
                .disableReload()
                .spy(localStorage, 'setItem')
                .spy(User, 'login')
                .app(next);
        },
        // Given unknown behavior of the .app() method, let's wait for condition
        // presumably button and fields would appear in the view at the same time
        { waitForCQVisible : 'button[text=Anmelden]' },
        // when button appeared - resolve other fields too
        (next) => {
            [loginButton] = t.cq('button[text=Anmelden]');
            [userName] = t.cq('[name=username]');
            [password] = t.cq('[name=password]');

            t.diag('Check prerequisites');
            
            t.is(loginButton.disabled, true, 'login button is disabled at start');
            
            const [view] = t.cq('userLogin');
            const form = view.getForm();

            hooks.spy(form, 'isValid');
            hooks.spy(view, 'setLoading');

            // If this causes problems, try replacing it with chain of steps
            // { click : '[name=username]' }
            // { type : '[ENTER]' }
            // next => { t.is(...); next(); }
            t.click(userName, () => {
                t.type(userName, '[ENTER]', () => {
                    t.is(form.isValid.called, true, 'Form is validated on enter');
                    t.is(view.setLoading.called, false, 'View is not set to loading on invalid enter');

                    next();
                });
            });
        },
        (next) => {
            userName.setValue('foo');
            password.setValue('bar');

            next();
        },
        // timeout replaced with condition
        {
            waitFor : () => !loginButton.disabled,
            desc: 'login button is enabled when form is filled'
        },
        (next) => {
            t.diag('Check login click / failed login');
            next();
        },
        
As you see, I tried to follow two rules:
1. When you expect component to be in the certain state - wait for component to be in that state
2. Do your best to avoid { waitFor : ms }, it is a race condition. That may work locally, but will almost certainly fail on a 3rd party testing platform. Wait for events, component conditions or even DOM conditions, if possible

Please let us know if this helps.

Post by KaguChan »

Hi,

thanks for the tipps.

It may be a good idea to mention the possible problems with timeouts in the documentation.

For the time beeing, we did three things:
- We disabled our integration tests, since there pretty complex and need to be rewritten, according your guide.
- We fixed every other test to avoid mentioned edge cases
- We updated the project configuration:
/* global Siesta */
const project = new Siesta.Project.Browser.ExtJS();
const loaderPath = {
  ITVM: 'app/src',
};

const group = (name, items, cfg = {}) => ({
  group: name,
  pageUrl: 'index.html?testing',
  preload: 'inherit',
  waitForAppReady: true,
  loaderPath,
  items,
  ...cfg,
});

project.configure({
  title: 'ITVM Test Suite',
  preload: ['tests/sinon.js', 'tests/functions.js'],
  suppressPassedWaitForAssertion: true,
  showTestDurationColumn: true,
  failOnExclusiveSpecsWhenAutomated: true,
  pauseBetweenTests: 50,
  loaderPath,
  requires: ['ITVM.view.main.MainViewModel', 'ITVM.view.components.base.Grid'],
});

project.plan(
  group('Models', [
    {
      title: 'Brief',
      url: 'tests/spec/models/Brief.spec.js',
    },
  ]),
  group('Base Components', [
    {
      title: 'Base Grid',
      url: 'tests/spec/components/Grid.spec.js',
    },
  ]),
  group('Formulas', [
    {
      title: 'Main ViewModel',
      url: 'tests/spec/formulas/MainViewModel.spec.js',
    },
  ]),
  group('Application Logic', [
    {
      title: 'UserController',
      url: 'tests/spec/misc/user/UserController.spec.js',
    },
  ]),
);

project.start();
With this, it seems to be more stable than before. Sometimes we're still experiencing the issue "class xy is not defined" - For this we tried the "require".

The integration tests will be tackled next to see if it helps.

Post Reply