FIX: Run the browser tests on ARM64 Linux instead of reporting passes that never happened

[jwrosewell]

What was wrong

On an ARM64 Linux runner the browser suite reported nine passes that never
happened. The job was green and the log said
Failed! - Failed: 9, Passed: 0, Total: 9 - SeleniumApiTests.dll, with every
one of the nine dying in about two milliseconds on:

OpenQA.Selenium.WebDriverException: Error running Selenium Manager process:
.../runtimes/linux/native/selenium-manager --browser "chrome" ...
---> System.ComponentModel.Win32Exception: An error occurred trying to start
process '.../runtimes/linux/native/selenium-manager' ... Exec format error

Selenium Manager is the helper program the Selenium package ships to fetch a
matching driver. Up to Selenium.WebDriver 4.48 the only Linux build of it was
for x64, and it sat under runtimes/linux/native, a folder whose name carries
no architecture. On ARM64 Linux that x64 build was picked, could not start, and
no browser was ever launched.

What this runner actually provides

Read from the runner image manifests rather than guessed at.

Ubuntu2204-Arm64-Readme.md, Ubuntu2404-Arm64-Readme.md and
Ubuntu2604-Arm64-Readme.md all carry the same short list:

### Browsers and Drivers
- Selenium server 4.48.0
- Mozilla Firefox 155.0
- Geckodriver 0.37.1

#### Environment variables
| CHROMEWEBDRIVER   |                                     |
| EDGEWEBDRIVER     |                                     |
| GECKOWEBDRIVER    | /usr/local/share/gecko_driver       |
| SELENIUM_JAR_PATH | /usr/share/java/selenium-server.jar |

The x64 manifests add Google Chrome, ChromeDriver, Chromium, Microsoft Edge and
Microsoft Edge WebDriver, and fill in CHROMEWEBDRIVER and EDGEWEBDRIVER. So
on ARM64 there is no Chrome, no Chromium, no ChromeDriver, no Edge and no Edge
driver, and the two blank rows mean the variables are unset rather than set to
nothing. The packer templates say why: the ARM64 build runs only
install-firefox.sh and install-selenium.sh, whilst the x64 build also runs
install-google-chrome.sh and install-microsoft-edge.sh, and the Chrome
script is hard-wired to amd64 and linux64 throughout. Adding Chrome to those
images is tracked at actions/runner-images#14567, still open.

.NET is not a constraint. The ARM64 images carry the same SDK list as the x64
ones, 8.0, 9.0 and 10.0.

Upstream, checked by fetching the artefacts:

• Chrome for Testing publishes linux-arm64 for Chrome, ChromeDriver and
  chrome-headless-shell on every channel, and
  chromedriver-linux-arm64.zip answers 200.
• Mozilla publishes geckodriver-v0.37.1-linux-aarch64.tar.gz and a
  linux-aarch64 Firefox.
• Microsoft publishes nothing for ARM64 Linux. Its Edge apt index at
  binary-arm64 is zero bytes where binary-amd64 is about a megabyte, and
  edgedriver_linux-arm64.zip answers 404. Selenium Manager's own binary
  carries the string "Linux arm64 is not supported yet by Microsoft Edge".
• Selenium.WebDriver 4.49.0, published 9 September 2026, splits the Linux
  folder by architecture and adds an ARM64 build. Listing the two packages
  side by side: 4.48.0 has runtimes/linux/native/selenium-manager alone, an
  x86-64 ELF, and 4.49.0 has runtimes/linux-x64/native/selenium-manager plus
  runtimes/linux-arm64/native/selenium-manager, an AArch64 ELF.
  SeleniumManager.cs in 4.49 picks the folder from
  RuntimeInformation.ProcessArchitecture.

What changed

• Selenium.WebDriver moves to 4.49.0, which is the version that fixes the
  cause. The reason is written next to the package reference so nobody
  downgrades it by accident.
• Helpers/BrowserDrivers.cs creates every driver the suite uses, from one
  place. Twelve copies of the same remote-or-local block across ten files now
  call it. It prefers a driver the machine already provides, named by
  CHROMEWEBDRIVER, GECKOWEBDRIVER or EDGEWEBDRIVER or found on the path,
  because a machine that ships one is faster and steadier than downloading one
  for every run, and because that is what makes Firefox work on ARM64 with no
  setup at all. Handing Selenium a driver service that already knows its path
  stops it calling Selenium Manager, which is the behaviour this relies on.
  Only when the machine provides nothing is Selenium Manager left to fetch one.
• The failure says what is wrong. When no driver can be started the message
  names the browser, the variable that was read, the file names looked for on
  the path and the runtime identifier of the machine, and then quotes what
  Selenium said. Edge on ARM64 Linux says plainly that Microsoft publishes
  neither the browser nor the driver for it, rather than failing on something
  that cannot be fixed here.
• Selenium/DriverSmokeTests.cs, category Browser, starts each browser
  against a page served from the test process and checks the browser really ran
  the script on it. No cloud, no resource key and no example application, so
  the test can go on any runner and answer one question, which is whether a
  browser starts there. They fail rather than skip when none can, because a run
  that quietly skipped them would say nothing, and saying nothing is the fault
  they were written for.
• Helpers/BrowserDriversTests.cs covers the rules for finding a driver,
  including the case an ARM64 runner actually presents, which is
  CHROMEWEBDRIVER set to nothing.
• The suite's own CI gains a second job that runs the Browser tests on
  ubuntu-latest, ubuntu-22.04-arm and ubuntu-24.04-arm, and prints what
  each runner provides before it does, so a later change of image is readable
  from the log.

Which repositories skip ARM today

Two skip the browser suite on ARM64 Linux, each with a note about the missing
Selenium Manager build, and each can drop the skip once this is merged and the
Selenium version has reached them:

• device-detection-dotnet, ci/run-integration-tests.ps1, the
  $skipSeleniumOnArm guard and the warning it prints.
• device-detection-node, ci/run-integration-tests.ps1, the early exit on
  Linux ARM64.

One has an ARM64 Linux job and no skip, which is where the fault was seen:

• device-detection-java, ci/run-integration-tests.ps1. It was running the
  suite on ARM and getting nine failures that never reached the job result,
  because dotnet test returning non-zero does not stop a PowerShell script
  that has not asked it to. Making the suite honest is only half of it, as a
  caller that drops the exit code reports a green job either way, so that half
  is device-detection-java#592.

Two run the suite but carry no ARM64 image in their matrix at all, so nothing
there needs removing, although both drop the exit code the same way and would
hide a failure if an ARM job were added:

• device-detection-php, ci/run-integration-tests.ps1
• device-detection-python, ci/run-integration-tests.ps1

rust has no ARM64 Linux job either.

How this was proved

The machine this was written on cannot run a browser, so it was proved in CI.

This repository's own CI, run
35198239429,
all four jobs green. Each browser job prints what its runner provides before it
starts anything. On ubuntu-24.04-arm:

arch: aarch64
CHROMEWEBDRIVER: <unset>
GECKOWEBDRIVER: /usr/local/share/gecko_driver
EDGEWEBDRIVER: <unset>
google-chrome: <not on the path>
chromium: <not on the path>
chromedriver: <not on the path>
firefox: /usr/bin/firefox
geckodriver: /usr/bin/geckodriver
microsoft-edge: <not on the path>
msedgedriver: <not on the path>

and then:

Skipped Edge_RunsThePage, because: Headless Edge on the Ubuntu runners hangs on
the first navigation, and Microsoft publishes no Edge at all for ARM64 Linux.
Chrome and Firefox cover both architectures.
Passed! - Failed: 0, Passed: 2, Skipped: 1, Total: 3, Duration: 23 s

Firefox runs on the driver the image provides. Chrome runs although the image
has none, because Selenium Manager fetches the Chrome for Testing linux-arm64
build. ubuntu-22.04-arm gives the same result in 25 seconds, and
ubuntu-latest in 15 seconds.

The Edge smoke test was written to run, and was ignored after its first run
proved headless Edge on the x64 Ubuntu runner accepts the session and then hangs
on the first navigation for the whole sixty second timeout. That is the fault
the other Edge tests in this suite were already ignored for, and it is not the
ARM64 one, so the test is kept with its reason where a reader will find it.

End to end through a language repository, which is where the fault was
reported. device-detection-java run
35198427368,
dispatched with dryrun=true on the branch of
device-detection-java#592,
with the suite pinned to this branch for that run only. Its Ubuntu_ARM_Java_17
job says:

Running Selenium tests...
Cloning the Selenium contract tests from 'fix/arm-linux-browser-drivers'
Passed! - Failed: 0, Passed: 9, Skipped: 0, Total: 9, Duration: 40 s

The same nine tests that died in about 155 milliseconds without starting a
browser now drive a real browser for forty seconds and pass.

Locally, dotnet build -c Release and the tests that need no browser pass, 30
of them.

Why this is still a draft, 17 September 2026

CI is green on this head, in
https://github.com/51Degrees/selenium-api-tests/actions/runs/35198526469,
where build-and-test and the three Browsers on ... jobs all pass.

It stays a draft only because its base is
fix/contract-settings-and-skip-reasons, the branch behind
#10, which has not
merged. Taking this out of draft would let the overnight merge push
these commits into that branch rather than into main, which would
rewrite what pull request 10 is asking for and throw away the run that
proved it.

Once pull request 10 has merged, retarget this one at main first and
then take it out of draft. Retargeting before the merge would be wrong
as well, because the diff would then carry pull request 10's commits
too. Do not let pull request 10 merge with its branch deleted while this
one is still pointing at it, because a stacked pull request closes when
its base branch goes and it cannot be reopened.