[Bug]: UI seems to lock on Windows making tests time out

[pbrzosko]

Version

1.42.1

Steps to reproduce

1. Clone my repo at https://github.com/pbrzosko/playwright-lock-issue
2. Run npm install
3. Run npm run pw on a Windows machine
4. Execute tests in the UI

Expected behavior

All tests should run similar amount of time and not time out. Here is a screenshot of them running on OSX:

Actual behavior

On Windows, it seems like tests are locking each other on some resource. Thus they start to time out and run very slowly:

Additional context

When run with one worker, it seems to work fine.

Environment

System:
    OS: Windows 10 10.0.19045
    CPU: (24) x64 12th Gen Intel(R) Core(TM) i7-12850HX
    Memory: 6.77 GB / 31.69 GB
  Binaries:
    Node: 18.12.0 - C:\Program Files\nodejs\node.EXE
    npm: 8.19.2 - C:\Program Files\nodejs\npm.CMD
  IDEs:
    VSCode: 1.87.2 - C:\Users\rgzc\AppData\Local\Programs\Microsoft VS Code\bin\code.CMD
  Languages:
    Bash: 5.0.17 - C:\Windows\System32\bash.EXE
  npmPackages:
    @playwright/experimental-ct-vue: ^1.42.1 => 1.42.1

[pbrzosko]

We have checked tracing impact, according to this comment: #30629 (comment), but I am not sure how that changes anything? It just confirms, that writing trace make tests run slower, but it does not explain timeouts for those tests in UI mode:

[yury-s]

Do you have any new findings since the previous bug was filed? I tried running the suite on windows and was not able to reproduce the issue, here is what I get:

The numbers are a higher but not as drastically as in your case:

You can try reducing the number of workers in the ui mode. I don't see what action we can take here based on the provided information.

[pbrzosko]

No I don't have anything new. But what I have are tests which run around 1s on OSX regardless whether it is one worker, multiple workers, console or UI mode.
Then I have a windows machine where tests run around 1s with single worker in console or UI mode, and then are 5 times slower with multiple workers in the console and 10 times or more slower in the UI. And the problem seem to occur only if we introduce parallel mode.
This basically means for us that we can not run tests in parallel for moste of the devs. So this slows down our development and that is an issue for us.
Should I concvince my boss that the whole team needs to switch to OSX because otherwise they need to wait 10 times more for tests to finish?
You have executed the repo on Windows. Please now check on Linux or OSX and see whether there is a difference.

[pbrzosko]

The processor that we run it on (i7-12850HX) has 16 cores and 24 threads. I have a feeling that this 80% worker setting in playwright is actually either resulting in 16 or more workers.
Basically the more workers we have the worse is the issue. So this sounds like a lock problem, right?

If it wasn't then it wouldn't matter If I would use 5 workers or 15 workers - the time increase would be the same because of trace or something else that UI does. Buy this is not the case.

Maybe try to run it on a similar processor.

Not sure though why do I need to try to figure out ways to find the issue?

[yury-s]

Performance difference on Mac for me is similar to Windows (https://github.com/microsoft/playwright/assets/9798949/94dd6273-55a9-4a3e-9877-1d2ec82f3bc8, https://github.com/microsoft/playwright/assets/9798949/9adafec5-6673-4f44-8b9f-5a394e0f0a88) Removing workers: '80%' setting from the config and using the default value makes the tests pass faster actually.

But what I have are tests which run around 1s on OSX regardless whether it is one worker, multiple workers, console or UI mode.

That does not match what I see, on macOS they are also at least 2x slower in parallel mode.

Then I have a windows machine where tests run around 1s with single worker in console or UI mode, and then are 5 times slower with multiple workers in the console and 10 times or more slower in the UI. And the problem seem to occur only if we introduce parallel mode.

Neither me nor @dgozman have been able to reproduce 10x difference, the tests run ~5x slower on Windows in UI mode. This is probably due to difference in the hardware (disk?) and system configuration.

The UI mode is expected to be slower than normal execution. The more parallel workers you run, the more contention that may cause. 10x slow down is too high, but it appears to be specific to your environment which we cannot reproduce locally.

[pbrzosko]

We have tried to narrow down the issue. The only thing we see in the UI that it is the browser setup that takes majority of time - like 9.5s when the test times out at 10. We tried various tools like "Process Monitor" on Windows to see what is happening, but there is not much there. We only see process creating files with trace for the UI.
Do you maybe have some hints how can we narrow down the problem?
We also did PWTEST_PROFILE_DIR and loaded result files in Chrome Dev Tools - majority of time spent is in idle state of the worker. So seems like it is waiting for something, but I would like to know for what? Do you have some hints?

[dgozman]

@pbrzosko I am not really sure how to help you. Perhaps launching a browser triggers some security checks on your Windows?

We did some general improvements to tracing performance that will be released in v1.45 release. I'll look into windows-specific issues more closely for the next v1.46 release as well.