Skip to content

TEST: 51Did browser acceptance tests, driving the PMP demo - #8

Merged
oleksandrlazarenko-pi merged 8 commits into
mainfrom
feature/create-last-browser-tests
Sep 22, 2026
Merged

oleksandrlazarenko-pi merged 8 commits into
mainfrom
feature/create-last-browser-tests

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Why

The browser tests belong in this shared suite so that the same tests can
check every implementation. They drive the demo where the preference
platform lives, with everything the tests need placed in that demo, so that
when the demo is copied into the other languages the same tests show every
copy behaves the same way.

So the fifteen 51Did browser acceptance tests move here from the cloud
service, where they were drafted, and drive the demo Examples/Cloud/pmp-web
in
device-detection-dotnet-examples
(device-detection-dotnet-examples#952)
instead of pages the tests built themselves. The demo was moved to
Examples/Cloud/pmp-web, with the project file pmp-web.csproj, from its earlier folder on 15 September 2026.

What is here

Six commits, the fifth and sixth under the review follow up below. 34558ec is the tests exactly as they were when drafted, with
only the namespace moved and a nullable context added, so a reviewer can
compare them line for line. 6f07e30 makes them drive the demo, e1a9d70
corrects one test's description of how PMP finds the client script,
which is by its tag, and 69499f0 reads the resource key from the names
51Degrees agreed for keys and launches the demo from its new path.

  • Browser51Did/, category Browser51Did. What the tests prove, the route
    each loads and the variables are in
    Browser51Did/README.md.
  • Examples/ExampleApps.cs gains ExampleApps.Demos and TryCreateDemo.
    The dotnet demo is launched from Examples/Cloud/pmp-web the way the GettingStarted-Web
    example is, with exactly 51DEGREES_RESOURCE_KEY,
    51DEGREES_CLOUD_ENDPOINT and ASPNETCORE_URLS. A demo is chosen with
    DEMO_URL or DEMO_LANG, apart from the Contract tests' EXAMPLE_URL and
    EXAMPLE_LANG, because the two are different apps. DEMO_MODE chooses
    cloud or pipeline pages, and the client script's request path comes
    from that mode.
  • The key and endpoint names follow the 51Degrees naming scheme agreed on
    8 September 2026. The suite reads 51DEGREES_RESOURCE_KEY first and
    _51DEGREES_RESOURCE_KEY_51DID where that is unset, which is the name CI
    sets for a resource key carrying the 51Did product, and always hands the
    key to the demo as 51DEGREES_RESOURCE_KEY. The endpoint is
    51DEGREES_CLOUD_ENDPOINT alone. DEMO_URL, DEMO_LANG and DEMO_MODE
    choose the demo rather than carry a key, so the scheme does not name them.
  • Browser51DidTestAttribute takes the resource key out of every test
    result, being messages, stack traces and output, because this repository
    and its CI logs are public and a failure message routinely quotes a script
    source naming the key. A test not marked with it fails.
  • Every assertion from the cloud copy is kept, and the consent test also
    checks the stub delivers exactly TcString.Personalized().
  • The common path's ordering assertion counts rounds inside the page, at the
    press and at the first frame the share card is visible. Polling from the
    test failed 3 times in 7 before the move and once in 10 over HTTPS after
    it, and a timeline recorded in the page showed the card visible 198 ms
    before the answered round finished in the failing run, so the page had not
    waited and the poll had looked late.

Proof

Against a locally run cloud on http://localhost:5050 carrying the client
script from javascript-templates eb02894 through
FiftyOne.Pipeline.JavaScriptBuilder 4.5.105-createlast.1, with the dotnet
demo launched by the suite:

Total tests: 15
     Passed: 14
    Skipped: 1

The skip is IsGdpr_ReadFromTheClientScript_SetsGdprApplies, which cannot
pass until IsGdpr is released in the cloud and says so. The guard reported
Client script guard passed for the dotnet demo's /cloud/ pages, with the
iteration limit message and __51d_pmp found in the served script. The
unredacted output of that run held the resource key no times.

The common path over HTTPS, run alone ten times against the same demo with
the in-page measure, passed ten times.

After 69499f0, dotnet build -c Release gave 0 warnings and 0 errors, and
the category run with neither resource key name set gave 15 skipped, each
reason naming 51DEGREES_RESOURCE_KEY and _51DEGREES_RESOURCE_KEY_51DID.
The run of 14 passed and 1 skipped above was on e1a9d70 with the demo at
its old path, and has not been repeated since the move.

Outstanding

  • /pipeline/ mode now runs against the demo's /pipeline/ pages. There the
    suite gave 15 tests, 8 passed, 6 failed and 1 skipped. Five failures are
    the share card, which PMP now shows only where third party
    cookies are confirmed, and one is a defect in this suite's
    NoClientScriptTag_PmpAddsItAndTheAnswerStillCreates in that mode,
    described in the demo pull request.
  • The consumer that runs this category stopped on its first run at the
    check for the new client script, as it must until the cloud release
    carries that script.
  • The tests that create a 51Did for a standard or personalized answer need
    a resource key whose products include CloudV5FODiD. The consumer that runs
    this category builds a throwaway record from the licence in
    _51DEGREES_LICENSE_KEY_51DID. The harness asks the service first, and
    skips with the service's own reason when the key cannot create.
  • The consumer that runs this suite pins both this repository and the demo
    repository, and the two pins have to move together, to bf56d0e or later
    here and to the demo commit that carries Examples/Cloud/pmp-web,
    because this suite now launches the demo from the new path.

Depends on

It merges together with the demo pull request,
device-detection-dotnet-examples#952,
after the cloud release that carries the new client script.

Review follow up, 15 September 2026

  • 13a6692 DOC: Say what the pipeline mode proves now the demo serves its
    pages. Browser51Did/README.md said every run uses /cloud/ until a demo
    serves the pipeline pages, and that the tests about the PMP adding a
    script prove the cloud's script in either mode. Both were stale once the
    demo gained /pipeline/ and 69499f0 wired the mode in. The section now
    says the dotnet demo serves /pipeline/, that those pages show the user
    prompt block only with the pipeline packages carrying the new template,
    and that NoClientScriptTag_PmpAddsItAndTheAnswerStillCreates fails
    in that mode because Visitor.ClientRequests looks on the pipeline's
    path while the PMP adds the cloud's script. Step 2 of adding a demo in
    another language names /pipeline/ too. Verified by reading Demo.cs
    (DemoMode.JsonEndpointPath), Visitor.cs lines 165 to 170 and
    PmpAddsClientScriptTests.cs lines 89 and 94. dotnet build SeleniumApiTests.csproj at 69499f0 gave 0 warnings and 0 errors, and
    a README change does not alter that. The commit count under "What is
    here" was four and is corrected to five in this edit.

Review follow up, 16 September 2026

  • bf56d0e FIX: Say PMP throughout, and drop a helper naming an address
    that is going away. Two things. First the naming rule, which is that every
    name uses pmp and every sentence says PMP rather than "the platform".
    PlatformAcceptanceTests and PlatformAddsClientScriptTests are renamed
    with their files, along with Visitor.WaitForPmp, PmpLoaded, PmpState,
    PmpPreference, OpenPmp, Harness.NoPmpMessage, the five test methods
    and the CommonPath parameter. Three things keep the older spelling on
    purpose and the commit message says why: Routes.NoPlatform and
    Routes.PlatformOnly, whose values are the URL paths the demo serves, so
    renaming the pair needs the demo's own page names to move in the same
    release; the client script's own console wording, which NoPmpMessage
    matches as a substring; and the consent platform tests, where a consent
    management platform is a different product from PMP.
  • Second, Harness.PlatformLoaderUrl() named {CloudUrl}/api/v4/pmp and
    nothing called it. That address answers 404 once the loader takes the
    resource key in its path, as api/v4/pmp/<resource key>.js, so the helper
    is removed rather than corrected. An address the suite does not ask for is
    better absent than wrong.
  • Four comments pointed at another repository's internal paths and change
    numbers, which a reader here cannot follow, and one of them printed into
    the test output. Each now says what the thing is instead. The same goes for
    the references to an internal list of demonstrations.
  • dotnet build SeleniumApiTests.csproj -c Release gave 0 warnings and
    0 errors after the rename. The change is names and comments only, so the
    run recorded under Proof still stands, and the browser run has not been
    repeated since.

… they were

The fifteen browser acceptance tests for the 51Did user prompt work were
written into the cloud repository at
Did/Tests/Selenium.Context.Integration/Browser51Did on the branch
feature/create-last-f. They prove client script behaviour that every
implementation shares, so they belong in this suite, where every
implementation's example can be pointed at them.

This commit is the files as they were, with only what they need to build
here: the namespace moved under FiftyOne.Pipeline.Cloud.SeleniumTests, a
nullable context per file because this project does not enable one, and
the published FiftyOne.Did parser the tests read an identifier with. Every
class keeps TestCategory("Browser51Did"). Configuration, the suite's own
furniture and the implementation under test follow in the next commits,
so this one can be compared line for line with the cloud copy.
The 51Did browser acceptance tests must drive the demo where the
Preference Management Platform lives, with every page and every piece of
state the tests need placed in that demo, so that each language mirrors
the demo and these same tests prove every one of them. The dotnet demo is
Examples/Cloud/PreferenceManagement-Web in device-detection-dotnet-examples.

- Pages.cs and PageServer.cs are gone. Each test loads a demo route under
  /cloud/, being common, common-script-first, change, two/one, two/two,
  consent, no-platform, platform-only or named-object, as
  site-a.localtest or site-b.localtest on the demo's port. The routes are
  named once in Demo.cs.
- Examples/ExampleApps.cs gains ExampleApps.Demos, with a dotnet entry
  following the GettingStarted-Web one, and TryCreateDemo. A demo is chosen
  apart from the Contract tests' example, with DEMO_URL for one already
  running and DEMO_LANG for one to launch, dotnet unless named, because the
  two are different apps. The demo is started with exactly
  51DEGREES_RESOURCE_KEY and 51DEGREES_CLOUD_ENDPOINT, the input variables
  every language's demo reads, and ASPNETCORE_URLS for its port.
  TestsCommon/TestConfig.cs reads those two names, and CLOUD_ROOT_URL keeps
  its meaning for the rest of the suite.
- The path the client script posts to comes from the demo's mode, being the
  cloud's /api/v4/json on /cloud/ pages and the descriptor's
  JsonEndpointPath on /pipeline/ pages, chosen with DEMO_MODE.
- Every assertion is kept. The consent test gains one, that the string
  reaching the cloud is exactly TcString.Personalized(), because every
  language's stub consent platform has to deliver that same string.
- Browser51DidTestAttribute replaces [TestMethod] on all fifteen tests and
  takes the resource key out of every result before the runner sees it,
  being the failure message, the stack trace and all output. The base
  class fails any test not marked with it, because this suite is public and
  a failure message routinely quotes a script source naming the key.
- The tests that create a 51Did for a standard or personalized answer need
  a resource key whose products include CloudV5FODiD. The harness asks the
  service first and skips with the service's own reason where the key
  cannot create.
- The shared choice skips only where the cloud is plain HTTP and not
  localhost, because both browsers kept its Secure SameSite=None cookie
  from http://localhost:5050 when measured.

The share card ordering is counted inside the page. The common path
asserts that the share card appears before any further client round
finishes after the visitor answers. It used to read the finished count
from the test before the press and then poll the page every 100 ms, and
that failed 3 times in 7 runs before the move and once in 10 over HTTPS
against the demo. A timeline recorded inside the page caught that failure:

  0ms recording started | 1ms request 3 seen already finished |
  1ms request 3 finished | 291ms pressed standard |
  302ms request 4 seen carrying an answer |
  302ms share card in the dialog | 302ms share card visible |
  500ms request 4 finished

with the assertion reporting Expected:<1>. Actual:<2>. The card was up
198 ms before the answered round finished, so nothing waited, and the
test's first poll that saw the card came after 500 ms. The two counts are
now taken inside the page, at the moment of the press and at the first
animation frame in which the card is visible, and the assertion is still
that they are equal. It still fails where the platform waits, because a
round finishing before the card becomes visible is counted first within a
frame, and the failure message carries the whole timeline.

Against a locally run cloud, with the demo launched by the suite, the
category gave 14 passed and 1 skipped, the skip being the IsGdpr test that
cannot pass until that property is released.
… from pmp-web

The 51Did browser tests read 51DEGREES_RESOURCE_KEY first and, where that is unset, _51DEGREES_RESOURCE_KEY_51DID, the name continuous integration sets for a resource key carrying the 51Did product. The demo is still given the key as 51DEGREES_RESOURCE_KEY. A missing key now names both variables in the skip reason, and both READMEs list the second name.

The dotnet demo moved from Examples/Cloud/PreferenceManagement-Web to Examples/Cloud/pmp-web, so the launcher and the README point there.
The section said every run uses /cloud/ until a demo serves the pipeline pages, and that the tests about the PMP adding a script prove the cloud's script in either mode. The dotnet demo serves /pipeline/ now, and in that mode NoClientScriptTag_PlatformAddsItAndTheAnswerStillCreates reads Visitor.ClientRequests, which looks on the pipeline's path while the PMP adds the cloud's script, so the section now says that. Step 2 of adding a demo names /pipeline/ too.
…going away

The suite called the Preference Management Platform "the platform" in
prose and "Platform" in identifiers, against the rule that every name
uses pmp and every sentence says PMP. Renamed the two test classes and
their files, the Visitor and Harness members, the five test methods and
the CommonPath parameter, and swept the prose.

Left as they were, on purpose:

1. Routes.NoPlatform and Routes.PlatformOnly, because their values are
   the URL paths the demo serves and the constant should keep naming the
   URL it produces. Renaming the pair needs the demo's own page names to
   move in the same release.
2. The client script's console wording quoted in Harness, because
   NoPmpMessage is matched against that output as a substring and
   rewording it would break the match.
3. The consent platform tests, where a consent management platform is a
   different product from PMP.

Harness.PlatformLoaderUrl() named {CloudUrl}/api/v4/pmp and nothing
called it. That address answers 404 once the loader takes the resource
key in its path, so the helper is removed rather than corrected. An
address the suite does not ask for is better absent than wrong.

Four comments pointed at another repository's internal paths and change
numbers, which a reader here cannot follow, and one of them printed into
the test output. Each now says what the thing is instead. The same goes
for the references to an internal list of demonstrations.

Build: dotnet build SeleniumApiTests.csproj -c Release, 0 warnings, 0
errors.
@jwrosewell jwrosewell changed the title TEST: 51Did browser acceptance tests, driving the preference management demo TEST: 51Did browser acceptance tests, driving the PMP demo Sep 15, 2026
The redaction attribute edited the TestResult MSTest had already built.
Assigning TestFailureException on a result that already has one does not
replace it, MSTest aggregates, and it has already copied the first
message into an internal member this assembly cannot set. A failing test
therefore reported "One or more errors occurred. (<the key>) (<the
redaction>)" - the resource key and the redaction of it side by side, in
a public repository's public build log.

The attribute now returns a new TestResult, which has no first exception
to aggregate with and no such member. DisplayName is redacted too, so
the rule is that nothing on a reported result carries the key, with no
exceptions to remember.

Harness.Around spliced 160 raw bytes of the cloud-rendered client script
into the guard's evidence. That evidence is written from ClassInitialize
and so never passed through the attribute at all. It now redacts inside
itself rather than at its one call site, because a caller cannot tell
from the returned text whether it needs it.

Harness.Resource is read once when the class is first touched, so no
test can set it. Each redacting method gained an internal overload
taking the key as an argument, with the public form delegating to it.

Browser51Did/RedactionTests.cs covers both, six tests needing no cloud,
demo or browser. Three of the six fail against the pre-fix code.
@oleksandrlazarenko-pi
oleksandrlazarenko-pi marked this pull request as ready for review September 16, 2026 14:48
Brings in #10 (contract settings, skip reasons) and #11 (ARM64 Linux
browser drivers). Three conflicts, all where both sides added:

- TestsCommon/TestConfig.cs: main made every variable read through the
  lookup a TestConfig is given. The demo settings are added on top, and
  RequireFirst now reads through Optional too, so a configuration built
  for a test no longer ignores its lookup for the demo's resource key.
  Browser51Did/DemoSettingsTests.cs covers it.
- Examples/ExampleApps.cs: main's BuildOptions and the demo registry
  are independent, and both are kept.
- README.md: both new category rows are kept.
@oleksandrlazarenko-pi
oleksandrlazarenko-pi merged commit 03582ea into main Sep 22, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: cleanup Tidying that changes no behaviour

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants