Ask only for the settings a test uses, and print skip reasons

[jwrosewell]

What this changes

Three problems reported by the language repositories that run
dotnet test --filter TestCategory=Contract against their web examples.

1. The suite demanded settings it did not use

TestInitialiser read CLOUD_ROOT_URL in [AssemblyInitialize], so an unset
variable aborted the whole run whatever the tests being run needed. The
Contract tests then read PAID_RESOURCE_KEY as well, in
BrowserCache/SessionStorageCacheTests.cs and in the TestInitialize methods
of the three classes under ClientSideOverrides. An example that is already
running was pointed at its data and given its key by whoever started it, and
ExternalExampleApp.StartAsync ignores both values, so an on-premise example
had nothing to give and the callers were setting placeholders to get a run
started.

Now nothing is read for the run as a whole. The cloud URL is read where it is
used, and the options for starting an example are built in one place,
ExampleApps.BuildOptions:

• an example this suite launches needs both values, and a missing one fails
  that test naming the variable,
• an example that is already running (EXAMPLE_URL) needs neither and is
  passed whatever happens to be set,
• CloudInternal, which talks to the cloud directly, still needs them and
  still fails naming the variable.

Examples/ContractSettingsTests.cs covers those rules by case. They read
through a lookup the test supplies, so they do not change the environment of
the run around them.

README.md said "A missing variable only fails the test that reads it", which
was not true. It now says which category needs which variable, and that a run
against a running example needs no cloud URL and no key.

2. Skip reasons never reached the console

A test that cannot run says why and reports itself as skipped, but the console
logger dotnet test uses prints the name of a skipped test and nothing else.
A CI log therefore showed a row of skips with no way to tell a missing example
from a stale page, and the reason was only visible by re-running with
--logger "console;verbosity=detailed".

Writing the reason from inside the test does not work, because output written
in a test goes to the test host and never reaches the console. Two other
routes were tried and failed as well, being the console logger verbosity set
in a run settings file, which dotnet test ignores because it adds its own
console logger, and writing to the raw standard output handle, which goes
nowhere for the same reason as the test output.

A logger runs in the process that writes the console output, so this adds a
small one in TestLogger. It is a separate assembly because the test platform
only looks for a logger in one whose name ends with ".TestLogger", and it is
enabled by test.runsettings, which the test project points at through
RunSettingsFilePath. A plain dotnet test picks it up with no extra
arguments, so no caller has to change how it calls the suite.

3. The comment about the rust example was out of date

ClientSideOverrides/ExampleRenderTests.cs said java and rust render no
device id server-side. The rust example renders one, in the same table as the
device type (examples/device-detection-examples/src/web_support/mod.rs in
the rust repository lists ("Device Id", "deviceid") among the properties its
server-side table renders). Only java is named now.

No rust-specific skip depended on the comment. The check reads the device id
where it is rendered and skips the assertion where the page has no such row,
which applies to every language, and it is unchanged. It was left conditional
rather than demanded for everything except java, because the same tests run
against on-premise examples too and those pages were not all checked.

Also

• The python example is launched through the interpreter inside its virtual
  environment, which is Scripts/python.exe on Windows and bin/python
  elsewhere. The descriptor assumed bin/python, which does not exist on
  Windows. Creating the environment now falls back from python3 to python,
  as a Windows install may have no working python3.
• The repository had no CI of its own, so a change here could only be tried by
  a language repository checking out main. The "Build and test" workflow
  builds the suite and runs the tests that need no browser, no cloud and no
  keys, on every push and pull request. It sets no environment variables,
  which is the point of the first change.

Evidence

The Selenium tests themselves were not run, as the machine used has no browser
and must not start one. Everything below is from real runs of the tests that
need no browser.

Before, on main, with no environment variables set:

dotnet test -c Release --no-build --filter "TestCategory!=Contract&TestCategory!=CloudInternal"
  Failed StartAsync_BuildFails_ThrowsWithBuildOutput
   Assembly Initialization method ...TestInitialiser.TestInitialise threw exception.
   System.InvalidOperationException: Required environment variable 'CLOUD_ROOT_URL'
   is not set.. Aborting test execution.
Failed!  - Failed: 11, Passed: 0, Skipped: 0, Total: 11

After, on this branch, same command and still no variables set:

Passed!  - Failed: 0, Passed: 24, Skipped: 0, Total: 24

A Contract test against an example that is already running, with no cloud URL
and no key. On main it aborts on the missing variable. On this branch it gets
all the way to the browser step and fails only because there is no grid on the
machine it was run on, which is what proves the settings are no longer in the
way:

EXAMPLE_URL=http://localhost:8080/ SELENIUM_URL=http://localhost:65000/wd/hub
dotnet test -c Release --no-build --filter "FullyQualifiedName~ExampleRenderTests"

main:   Assembly Initialization method ...TestInitialise threw exception.
        Required environment variable 'CLOUD_ROOT_URL' is not set.. Aborting test execution.

branch: Test method ...Example_RendersRealDetectionResult threw exception:
        OpenQA.Selenium.WebDriverException: ... sending an HTTP request to the remote
        WebDriver server for URL http://localhost:65000/wd/hub/session ... refused

A CloudInternal test with nothing set still fails, clearly and on its own:

dotnet test -c Release --no-build --filter "FullyQualifiedName~CoepTests"
   Initialization method ...COEP.CoepTests.Init threw exception.
   System.InvalidOperationException: Required environment variable 'CLOUD_ROOT_URL' is not set..
Failed!  - Failed: 1, Passed: 0, Skipped: 0, Total: 1

An example this suite launches still needs both values and says which one is
missing:

EXAMPLE_LANG=dotnet dotnet test -c Release --no-build --filter "FullyQualifiedName~ExampleRenderTests"
   Initialization method ...ExampleRenderTests.Init threw exception.
   System.InvalidOperationException: Required environment variable 'CLOUD_ROOT_URL' is not set..

Skip reasons, from a plain dotnet test with no logger arguments:

EXAMPLE_LANG=bogus dotnet test -c Release --no-build --filter "TestCategory=Contract"
  Skipped Example_RendersRealDetectionResult, because: Initialization method
  ...ExampleRenderTests.Init threw exception. No example descriptor registered for
  EXAMPLE_LANG='bogus'. Known: dotnet, java, node, python, php, rust..
  Skipped Example_RendersRealDetectionResult [1 ms]
  ...
Skipped! - Failed: 0, Passed: 0, Skipped: 9, Total: 9

On main the same case, run with placeholder values set as the CI scripts do
so that the run starts at all, prints the nine names and no reasons at all:

  Skipped Example_RendersRealDetectionResult [1 ms]
  Skipped SessionStorageCache_Chrome (True)
  ...
Skipped! - Failed: 0, Passed: 0, Skipped: 9, Total: 9

The "Build and test" workflow added here runs on this pull request, so the
checks on it are the same evidence in real CI rather than on one machine.

What a reviewer should decide

• Whether the language CI scripts should now stop setting placeholder values
  for CLOUD_ROOT_URL and PAID_RESOURCE_KEY when they run an on-premise
  example. Nothing outside this repository was changed here, and a placeholder
  still works, it is simply no longer needed.
• Whether the device id should be demanded for every language except java,
  rather than checked where present. That needs someone to confirm that every
  on-premise example page renders one.

Related

51Degrees/common-ci#242 makes a nightly step fail when the repository script
it ran failed. Until that is in, a failing run of this suite leaves the job
green, so the skip reasons added here are printed into a log nobody is sent
to look at. The two do not depend on each other and can be merged in either
order.

[jwrosewell]

The "Build and test" workflow added in this branch has run on this pull request and passed, with no environment variables set at all: "Build succeeded" and "Passed! - Failed: 0, Passed: 24, Skipped: 0, Total: 24" (run 35162545567, ubuntu-latest, .NET 10). That is the first change proved in real CI rather than on one machine, because on main the same command with nothing set aborts the whole run on the missing CLOUD_ROOT_URL.