Get help with testing, discuss unit testing strategies etc.


Post by klodoma »

Hi,

I am running an evaluation of Siesta Trial, version 5.2.2 and I am facing several issues with the code coverage.
I would appreciate help to understand and get it working.

First, I am executing this on 2 environments: Ubuntu 18 (running in VirtualBox) and Windows 10 system.

First of all, I am starting a webserver in the trial folder:
php -S localhost:80
The tests seems to be running, but I am trying to understand how the code coverage works, but that seems problematic.

I execute the following code and get different results on linux/windows:
bin/webdriver localhost/examples/sencha-extjs/ --headless --nyc.reporter=text --include code_coverage  --nyc.include='**/*.js'

On Linux:
Screenshot_354.png
Screenshot_354.png (147.51 KiB) Viewed 7335 times
Screenshot_355.png
Screenshot_355.png (252.59 KiB) Viewed 7335 times
On Windows:
Screenshot_356.png
Screenshot_356.png (52.47 KiB) Viewed 7335 times
Now on linux there is some code coverage present, on windows not, but I still don't get it how it works.

Following the 2 documentations:
https://www.bryntum.com/blog/siesta-5-1-0-released/
https://www.bryntum.com/docs/siesta/#!/guide/code_coverage

I tried several examples, but none works.

For example:
bin/webdriver localhost/examples/sencha-extjs/ --nyc.include='**/5.code_coverage' --nyc.reporter=text --include code_coverage
doesn't deliver any code coverage
Launching test suite, OS: Linux, agent: Chrome 74.0.3729.169
[PASS]  5.code_coverage/010_range.t.js
[PASS]  5.code_coverage/020_event.t.js
[PASS]  5.code_coverage/030_loader.t.js
[PASS]  5.code_coverage/040_pageUrl.t.js
7 passed, 0 failed assertions took 6.812s to complete
----------|----------|----------|----------|----------|-------------------|
File      |  % Stmts | % Branch |  % Funcs |  % Lines | Uncovered Line #s |
----------|----------|----------|----------|----------|-------------------|
All files |        0 |        0 |        0 |        0 |                   |
----------|----------|----------|----------|----------|-------------------|
Can you point me out in the right direction?



A second question I have is: how do you execute this code? You must have a webserver running or you map "examples" in your /etc/hosts file?
> bin/webdriver examples/sencha-extjs --nyc.include='**/5.code_coverage' --nyc.reporter=html --include code_coverage
Thanks in advance!

Post by nickolay »

This is strange, here's what I have (on Linux):
screenshot4.jpg
screenshot4.jpg (162.71 KiB) Viewed 7331 times
(Btw, it runs much faster if --nyc.include is fine-tuned to exclude the "ext-all" from instrumentation, like `**/5.code_coverage/**/*.js`)

In your case it seems only the "ext-all" is included in the report, which makes me think this something with the internal proxy, that Siesta uses for instrumentation.

I'm running local Apache instance and thats probably a bit different from using php built-in server.

Can you please run this command and post the output?

`bin/webdriver localhost/examples/sencha-extjs --include code_coverage --nyc.reporter=text --nyc.include='**/5.code_coverage/**/*.js' --headless --debug`

I'll investigate this deeper on Monday.

Post by klodoma »

Thanks for the feedback.
bin/webdriver localhost/examples/sencha-extjs --include code_coverage --nyc.reporter=text --nyc.include='**/5.code_coverage/**/*.js' --headless --debug
[DEBUG] Dispatcher start
[DEBUG] Starting instrumentation proxy
[DEBUG] HTTP Instrumentation proxy has started on :::42891
[DEBUG] HTTPS Instrumentation proxy has started on :::40349
[DEBUG] Runner setup: a Siesta.Launcher.Runner.WebDriverNodeJS
[DEBUG] Dispatcher setup starting
[DEBUG] Trying to create a WebDriver instance for browser: chrome
[DEBUG] WebDriver instantiated successfully
[DEBUG] PageId: 1, method : getConfigInfo starting, args size: 30
[DEBUG] PageId: 1, method : getConfigInfo, took 0.021s, result size : 2347
[DEBUG] Dispatcher setup completed, launching the suite
Launching test suite, OS: Linux, agent: Chrome 74.0.3729.169
[DEBUG] Launch runner, pagesCount: 1, max : 1, pages left: 1
[DEBUG] PageId: 1, method : launchAutomatedTests starting, args size: 428
[DEBUG] PageId: 1, method : launchAutomatedTests, took 0.057s, result size : 4
[DEBUG] Page polling has started
[DEBUG] PROXYING request for https://cdn.sencha.com/ext/gpl/4.2.1/resources/css/ext-all.css will instrument: false
[DEBUG] PROXYING request for https://cdn.sencha.com/ext/gpl/4.2.1/ext-all-debug.js will instrument: false
[DEBUG] PROXYING request for https://cdn.sencha.com/ext/gpl/4.2.1/resources/ext-theme-classic/ext-theme-classic-all.css will instrument: false
[DEBUG] PageId: 1, method : getAutomationState starting, args size: 2
[DEBUG] PageId: 1, method : getAutomationState, took 0.306s, result size : 2626
[PASS]  5.code_coverage/010_range.t.js
[PASS]  5.code_coverage/020_event.t.js
[DEBUG] PageId: 1, method : getAutomationState starting, args size: 2
[DEBUG] PageId: 1, method : getAutomationState, took 0.02s, result size : 2362
[PASS]  5.code_coverage/030_loader.t.js
[PASS]  5.code_coverage/040_pageUrl.t.js
[DEBUG] Recevied results for all tests in the suite, proceeding to finalization
7 passed, 0 failed assertions took 3.885s to complete
[DEBUG] Generating reports (if any)
----------|----------|----------|----------|----------|-------------------|
File      |  % Stmts | % Branch |  % Funcs |  % Lines | Uncovered Line #s |
----------|----------|----------|----------|----------|-------------------|
All files |        0 |        0 |        0 |        0 |                   |
----------|----------|----------|----------|----------|-------------------|
[DEBUG] Reports processed (if any)
[DEBUG] Page close started: 1
I ran this with Apache, I thought maybe it makes a difference, but the result is the same. Same result on windows.
Let me know if you need more info. I'll wait for your feedback.

Post by nickolay »

Just a wild guess, if you are blocked by this (I'm still fixing the firefox automation in trial) - perhaps the problem is, that Chrome has a setting "bypass proxy for localhost" enabled? In my setup I have different host name 'lh', aliased in /etc/hosts, thats why it works for me and requests goes through the proxy.

You can try to alias the host using different host name, or wait some time.

Post by klodoma »

I am not blocked by this. As I wrote I am running an evaluation.
My current target is to understand how the code coverage works and for that I would need to see the results. But it's not blocking anything unless you tell me we'll find a solution in 2 months :)

I'll give it a shot with another host name and let you know.

Post by klodoma »

Ok, the localhost proxy did something.

I don't seem to get the --nyc.include working properly, it's not matching anything...
With --nyc.all I get some coverage report (haven't checked it yet in detail)
Screenshot_361.png
Screenshot_361.png (244.46 KiB) Viewed 7308 times


See the 2 results below, log files are attached.
d:\Temp\siesta-5.2.2-trial>bin\webdriver 10.20.20.10:3000/examples/sencha-extjs/ --include code_coverage --nyc.reporter=text --nyc.all --headless --debug > log.all.txt

d:\Temp\siesta-5.2.2-trial>bin\webdriver 10.20.20.10:3000/examples/sencha-extjs/ --include code_coverage --nyc.reporter=text --nyc.include='**/*code_coverage*/*.js' --headless --debug > log.filtered.txt
Attachments
log.filtered.txt
(5.99 KiB) Downloaded 270 times
log.all.txt
(8.54 KiB) Downloaded 234 times

Post by nickolay »

Try with `--nyc.include='**/*code_coverage*/**/*.js'` ?

Post by klodoma »

nickolay wrote: Mon May 27, 2019 2:02 pm Try with `--nyc.include='**/*code_coverage*/**/*.js'` ?
Same. 0 files.

Post by nickolay »

This is weird, here's what I have
screenshot5.jpg
screenshot5.jpg (325.8 KiB) Viewed 7305 times
Blaming the port in the url, we currently strip the protocol only. Will check.

Post by klodoma »

Tried it also without port:
bin\webdriver local/examples/sencha-extjs/index-no-ui.html --headless --debug --include code_coverage --nyc.reporter=text --nyc.include='**/*code_coverage*/*.js'
The proxy notice is there:
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/010_range.t.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/020_event.t.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/030_loader.t.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/040_pageUrl.t.js will instrument: false
[DEBUG] PROXYING request for https://cdn.sencha.com/ext/gpl/4.2.1/resources/css/ext-all.css will instrument: false
[DEBUG] PROXYING request for https://cdn.sencha.com/ext/gpl/4.2.1/ext-all-debug.js will instrument: false
[DEBUG] PROXYING request for https://cdn.sencha.com/ext/gpl/4.2.1/resources/ext-theme-classic/ext-theme-classic-all.css will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/several_classes_in_one_file.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/some_folder/preload_file.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/some_folder/folder2/preload_file.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/010_range.t.js will instrument: false
[DEBUG] PROXYING request for https://local/resources/css/font-awesome/fonts/fontawesome-webfont.woff2?v=4.4.0 will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/several_classes_in_one_file.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/some_folder/preload_file.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/some_folder/folder2/preload_file.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/020_event.t.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/several_classes_in_one_file.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/030_loader.t.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/lib/My/Store/EventStore.js?_dc=1558961938247 will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/lib/My/Store/Base.js?_dc=1558961938261 will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/lib/My/Util/SomeClass.js?_dc=1558961938278 will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/pageUrl.html will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/040_pageUrl.t.js will instrument: false
[DEBUG] PROXYING request for https://local/examples/sencha-extjs/5.code_coverage/lib/My/Store/ResourceStore.js?_dc=1558961938429 will instrument: false
it seems that the filtering --nyc.include='**/*code_coverage*/*.js' is not forwared properly or smth like that

Post Reply